add delta sync and fail fetch process

scripts/benchmark/README.md

@@ -0,0 +1,70 @@
+# RPKI Benchmarks (Stage2, selected_der_v2)
+
+This directory contains a reproducible, one-click benchmark to measure **decode + profile validate**
+performance for all supported object types and compare **OURS** against the **Routinator baseline**
+(`rpki` crate `=0.19.1` with `repository` feature).
+
+## What it measures
+
+Dataset:
+
+- Fixtures: `rpki/tests/benchmark/selected_der_v2/`
+- Objects: `cer`, `crl`, `manifest` (`.mft`), `roa`, `aspa` (`.asa`)
+- Samples: 10 quantiles per type (`min/p01/p10/p25/p50/p75/p90/p95/p99/max`) → 50 files total
+
+Metrics:
+
+- **decode+validate**: `decode_der` (parse + profile validate) for each object file
+- **landing** (OURS only): `PackFile::from_bytes_compute_sha256` + CBOR encode + `RocksDB put_raw`
+- **compare**: ratio `ours_ns/op ÷ rout_ns/op` for decode+validate
+
+## Default benchmark settings
+
+Both OURS and Routinator baseline use the same run settings:
+
+- warmup: `10` iterations
+- rounds: `3`
+- adaptive loop target: `min_round_ms=200` (with an internal max of `1_000_000` iters)
+- strict DER: `true` (baseline)
+- cert inspect: `false` (baseline)
+
+You can override the settings via environment variables in the runner script:
+
+- `BENCH_WARMUP_ITERS` (default `10`)
+- `BENCH_ROUNDS` (default `3`)
+- `BENCH_MIN_ROUND_MS` (default `200`)
+
+## One-click run (OURS + Routinator compare)
+
+From the `rpki/` crate directory:
+
+```bash
+./scripts/benchmark/run_stage2_selected_der_v2_release.sh
+```
+
+Outputs are written under:
+
+- `rpki/target/bench/`
+  - OURS decode+validate: `stage2_selected_der_v2_decode_release_<TS>.{md,csv}`
+  - OURS landing: `stage2_selected_der_v2_landing_release_<TS>.{md,csv}`
+  - Routinator: `stage2_selected_der_v2_routinator_decode_release_<TS>.{md,csv}`
+  - Compare: `stage2_selected_der_v2_compare_ours_vs_routinator_decode_release_<TS>.{md,csv}`
+  - Summary: `stage2_selected_der_v2_compare_summary_<TS>.md`
+
+### Why decode and landing are separated
+
+The underlying benchmark can run in `BENCH_MODE=both`, but the **landing** part writes to RocksDB
+and may trigger background work (e.g., compactions) that can **skew subsequent decode timings**.
+For a fair OURS-vs-Routinator comparison, the runner script:
+
+- runs `BENCH_MODE=decode_validate` for comparison, and
+- runs `BENCH_MODE=landing` separately for landing-only numbers.
+
+## Notes
+
+- The Routinator baseline benchmark is implemented in-repo under:
+  - `rpki/benchmark/routinator_object_bench/`
+  - It pins `rpki = "=0.19.1"` in its `Cargo.toml`.
+- This benchmark is implemented as an `#[ignore]` integration test:
+  - `rpki/tests/bench_stage2_decode_profile_selected_der_v2.rs`
+  - The runner script invokes it with `cargo test --release ... -- --ignored --nocapture`.

scripts/benchmark/run_stage2_selected_der_v2_release.sh

@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Stage2 (selected_der_v2) decode+profile validate benchmark.
+# Runs:
+#  1) OURS decode+validate benchmark and writes MD/CSV.
+#  2) OURS landing benchmark and writes MD/CSV.
+#  3) Routinator baseline decode benchmark (rpki crate =0.19.1).
+#  4) Produces a joined compare CSV/MD and a short geomean summary.
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$ROOT_DIR"
+
+OUT_DIR="$ROOT_DIR/target/bench"
+mkdir -p "$OUT_DIR"
+
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+
+WARMUP_ITERS="${BENCH_WARMUP_ITERS:-10}"
+ROUNDS="${BENCH_ROUNDS:-3}"
+MIN_ROUND_MS="${BENCH_MIN_ROUND_MS:-200}"
+
+OURS_MD="$OUT_DIR/stage2_selected_der_v2_decode_release_${TS}.md"
+OURS_CSV="$OUT_DIR/stage2_selected_der_v2_decode_release_${TS}.csv"
+
+OURS_LANDING_MD="$OUT_DIR/stage2_selected_der_v2_landing_release_${TS}.md"
+OURS_LANDING_CSV="$OUT_DIR/stage2_selected_der_v2_landing_release_${TS}.csv"
+
+ROUT_MD="$OUT_DIR/stage2_selected_der_v2_routinator_decode_release_${TS}.md"
+ROUT_CSV="$OUT_DIR/stage2_selected_der_v2_routinator_decode_release_${TS}.csv"
+
+COMPARE_MD="$OUT_DIR/stage2_selected_der_v2_compare_ours_vs_routinator_decode_release_${TS}.md"
+COMPARE_CSV="$OUT_DIR/stage2_selected_der_v2_compare_ours_vs_routinator_decode_release_${TS}.csv"
+
+SUMMARY_MD="$OUT_DIR/stage2_selected_der_v2_compare_summary_${TS}.md"
+
+echo "[1/4] OURS: decode+validate benchmark (release)..." >&2
+BENCH_MODE="decode_validate" \
+BENCH_WARMUP_ITERS="$WARMUP_ITERS" \
+BENCH_ROUNDS="$ROUNDS" \
+BENCH_MIN_ROUND_MS="$MIN_ROUND_MS" \
+BENCH_OUT_MD="$OURS_MD" \
+BENCH_OUT_CSV="$OURS_CSV" \
+  cargo test --release --test bench_stage2_decode_profile_selected_der_v2 -- --ignored --nocapture >/dev/null
+
+echo "[2/4] OURS: landing benchmark (release)..." >&2
+BENCH_MODE="landing" \
+BENCH_WARMUP_ITERS="$WARMUP_ITERS" \
+BENCH_ROUNDS="$ROUNDS" \
+BENCH_MIN_ROUND_MS="$MIN_ROUND_MS" \
+BENCH_OUT_MD_LANDING="$OURS_LANDING_MD" \
+BENCH_OUT_CSV_LANDING="$OURS_LANDING_CSV" \
+  cargo test --release --test bench_stage2_decode_profile_selected_der_v2 -- --ignored --nocapture >/dev/null
+
+echo "[3/4] Routinator baseline + compare join..." >&2
+OURS_CSV="$OURS_CSV" \
+ROUT_CSV="$ROUT_CSV" \
+ROUT_MD="$ROUT_MD" \
+COMPARE_CSV="$COMPARE_CSV" \
+COMPARE_MD="$COMPARE_MD" \
+WARMUP_ITERS="$WARMUP_ITERS" \
+ROUNDS="$ROUNDS" \
+MIN_ROUND_MS="$MIN_ROUND_MS" \
+  scripts/stage2_perf_compare_m4.sh >/dev/null
+
+echo "[4/4] Summary (geomean ratios)..." >&2
+python3 - "$COMPARE_CSV" "$SUMMARY_MD" <<'PY'
+import csv
+import math
+import sys
+from pathlib import Path
+from datetime import datetime, timezone
+
+in_csv = Path(sys.argv[1])
+out_md = Path(sys.argv[2])
+
+rows = list(csv.DictReader(in_csv.open(newline="")))
+ratios = {}
+for r in rows:
+    ratios.setdefault(r["type"], []).append(float(r["ratio_ours_over_rout"]))
+
+def geomean(vals):
+    return math.exp(sum(math.log(v) for v in vals) / len(vals))
+
+def p50(vals):
+    v = sorted(vals)
+    n = len(v)
+    if n % 2 == 1:
+        return v[n // 2]
+    return (v[n // 2 - 1] + v[n // 2]) / 2.0
+
+all_vals = [float(r["ratio_ours_over_rout"]) for r in rows]
+types = ["all"] + sorted(ratios.keys())
+
+now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+lines = []
+lines.append("# Stage2 selected_der_v2 compare summary (release)\n\n")
+lines.append(f"- recorded_at_utc: `{now}`\n")
+lines.append(f"- inputs_csv: `{in_csv}`\n\n")
+lines.append("| type | n | min | p50 | geomean | max | >1 count |\n")
+lines.append("|---|---:|---:|---:|---:|---:|---:|\n")
+
+for t in types:
+    vals = all_vals if t == "all" else ratios[t]
+    vals_sorted = sorted(vals)
+    lines.append(
+        f"| {t} | {len(vals_sorted)} | {vals_sorted[0]:.4f} | {p50(vals_sorted):.4f} | "
+        f"{geomean(vals_sorted):.4f} | {vals_sorted[-1]:.4f} | {sum(1 for v in vals_sorted if v>1.0)} |\n"
+    )
+
+out_md.write_text("".join(lines), encoding="utf-8")
+print(out_md)
+PY
+
+echo "Done." >&2
+echo "- OURS decode MD:   $OURS_MD" >&2
+echo "- OURS decode CSV:  $OURS_CSV" >&2
+echo "- OURS landing MD:  $OURS_LANDING_MD" >&2
+echo "- OURS landing CSV: $OURS_LANDING_CSV" >&2
+echo "- Routinator:       $ROUT_MD" >&2
+echo "- Compare MD:       $COMPARE_MD" >&2
+echo "- Compare CSV:      $COMPARE_CSV" >&2
+echo "- Summary MD:       $SUMMARY_MD" >&2