20260316迭代增加delta replay以及multi-rir

replay 对比，五个RIR 输出vrp与routinator一致
2026-03-16 22:54:48 +08:00
parent 73d8ebb5c1
commit 557a69cbd2
24 changed files with 4678 additions and 55 deletions
--- a/scripts/payload_replay/README.md
+++ b/scripts/payload_replay/README.md
@@ -2,6 +2,69 @@

 本目录提供基于本地 payload archive 的手工 replay 入口。

+## `multi_rir_case_info.py`
+
+用于从 multi-RIR bundle 中解析指定 `rir` 的输入路径、对照 CSV、fixture、以及 Routinator replay timing 基线。
+
+示例：
+
+```bash
+python3 scripts/payload_replay/multi_rir_case_info.py \
+  --bundle-root ../../rpki/target/live/20260316-112341-multi-final3 \
+  --rir afrinic
+```
+
+也支持输出 shell 环境变量：
+
+```bash
+python3 scripts/payload_replay/multi_rir_case_info.py \
+  --bundle-root ../../rpki/target/live/20260316-112341-multi-final3 \
+  --rir afrinic \
+  --format env
+```
+
+## `run_multi_rir_replay_case.sh`
+
+统一的 multi-RIR 入口。给定 `rir` 和模式后，它会自动选择该 RIR 的：
+
+- snapshot/base replay 输入
+- delta replay 输入
+- 对照 CSV
+- TAL / TA fixture
+- trust anchor 名称
+
+用法：
+
+```bash
+./scripts/payload_replay/run_multi_rir_replay_case.sh <rir> [describe|snapshot|delta|both]
+```
+
+示例：
+
+```bash
+./scripts/payload_replay/run_multi_rir_replay_case.sh afrinic describe
+./scripts/payload_replay/run_multi_rir_replay_case.sh lacnic snapshot
+./scripts/payload_replay/run_multi_rir_replay_case.sh arin delta
+./scripts/payload_replay/run_multi_rir_replay_case.sh ripe both
+```
+
+
+脚本会自动：
+
+- 从 multi-RIR bundle 中选择指定 RIR 的 snapshot/base 与 delta 输入
+- 读取该 RIR 的 Routinator `base-replay` / `delta-replay` timing 基线
+- 使用该 RIR `timings/base-replay.json` 与 `timings/delta-replay.json` 的 `startedAt` 作为 replay `--validation-time`
+- 在 `target/live/multi_rir_replay_runs/<rir>/` 下生成：
+  - snapshot replay 产物
+  - delta replay 产物
+  - per-RIR 合并 case report（含 correctness + timing compare）
+
+默认 bundle 根目录为：
+
+- `../../rpki/target/live/20260316-112341-multi-final3`
+
+也可以通过 `BUNDLE_ROOT` 覆盖。
+
 ## `run_apnic_replay.sh`

 默认使用：
@@ -98,3 +161,74 @@ python3 scripts/payload_replay/report_to_routinator_csv.py \
  - compare summary
  - `only_in_ours.csv`
  - `only_in_record.csv`
+
+## `run_apnic_delta_replay.sh`
+
+使用 APNIC delta demo 数据集运行 base + delta replay：
+
+```bash
+./scripts/payload_replay/run_apnic_delta_replay.sh
+```
+
+默认输入：
+
+- `target/live/apnic_delta_demo/20260315-170223-autoplay/base-payload-archive`
+- `target/live/apnic_delta_demo/20260315-170223-autoplay/base-locks.json`
+- `target/live/apnic_delta_demo/20260315-170223-autoplay/payload-delta-archive`
+- `target/live/apnic_delta_demo/20260315-170223-autoplay/locks-delta.json`
+- `tests/fixtures/tal/apnic-rfc7730-https.tal`
+- `tests/fixtures/ta/apnic-ta.cer`
+
+输出目录默认：`target/live/payload_delta_replay_runs/`
+
+## `run_apnic_delta_replay.sh` compare outputs
+
+脚本现在在 delta replay 结束后还会额外生成：
+
+- `vrps.csv`
+- compare summary Markdown
+- `only_in_ours.csv`
+- `only_in_record.csv`
+
+默认 compare 输入是：
+
+- `target/live/apnic_delta_demo/20260315-170223-autoplay/record-delta.csv`
+
+也可以通过环境变量覆盖：
+
+- `TRUST_ANCHOR`
+- `ROUTINATOR_RECORD_CSV`
+- `VRPS_CSV`
+- `COMPARE_SUMMARY_MD`
+- `ONLY_IN_OURS_CSV`
+- `ONLY_IN_RECORD_CSV`
+
+## `write_multi_rir_case_report.py`
+
+把某个 RIR 的 snapshot replay 与 delta replay 的 `meta.json`、compare summary 以及 Routinator timing 基线合并成一个 per-RIR Markdown/JSON 报告。
+
+该脚本通常由 `run_multi_rir_replay_case.sh <rir> both` 自动调用。
+
+## `run_multi_rir_replay_suite.sh`
+
+顺序执行 5 个 RIR（或环境变量 `RIRS` 指定的子集）的 `both` 模式，并最终生成 multi-RIR 汇总报告。
+
+```bash
+./scripts/payload_replay/run_multi_rir_replay_suite.sh
+```
+
+可覆盖环境变量：
+
+- `BUNDLE_ROOT`
+- `SUITE_OUT_DIR`
+- `RIRS`
+
+最终输出：
+
+- `<suite_out_dir>/<rir>/<rir>_case_report.md`
+- `<suite_out_dir>/multi_rir_summary.md`
+- `<suite_out_dir>/multi_rir_summary.json`
+
+## `write_multi_rir_summary.py`
+
+汇总 5 个 RIR 的 per-RIR case report，生成 correctness + timing 总表与几何平均比值。
--- a/scripts/payload_replay/multi_rir_case_info.py
+++ b/scripts/payload_replay/multi_rir_case_info.py
@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import shlex
+import sys
+from pathlib import Path
+
+
+RIR_CONFIG = {
+    "afrinic": {
+        "tal": "tests/fixtures/tal/afrinic.tal",
+        "ta": "tests/fixtures/ta/afrinic-ta.cer",
+        "trust_anchor": "afrinic",
+    },
+    "apnic": {
+        "tal": "tests/fixtures/tal/apnic-rfc7730-https.tal",
+        "ta": "tests/fixtures/ta/apnic-ta.cer",
+        "trust_anchor": "apnic",
+    },
+    "arin": {
+        "tal": "tests/fixtures/tal/arin.tal",
+        "ta": "tests/fixtures/ta/arin-ta.cer",
+        "trust_anchor": "arin",
+    },
+    "lacnic": {
+        "tal": "tests/fixtures/tal/lacnic.tal",
+        "ta": "tests/fixtures/ta/lacnic-ta.cer",
+        "trust_anchor": "lacnic",
+    },
+    "ripe": {
+        "tal": "tests/fixtures/tal/ripe-ncc.tal",
+        "ta": "tests/fixtures/ta/ripe-ncc-ta.cer",
+        "trust_anchor": "ripe",
+    },
+}
+
+
+def default_repo_root() -> Path:
+    return Path(__file__).resolve().parents[2]
+
+
+def default_bundle_root(repo_root: Path) -> Path:
+    return (repo_root / "../../rpki/target/live/20260316-112341-multi-final3").resolve()
+
+
+def require_path(path: Path, kind: str) -> Path:
+    if kind == "dir" and not path.is_dir():
+        raise SystemExit(f"missing directory: {path}")
+    if kind == "file" and not path.is_file():
+        raise SystemExit(f"missing file: {path}")
+    return path
+
+
+def load_timing_summary(bundle_root: Path) -> dict:
+    timing_path = require_path(bundle_root / "timing-summary.json", "file")
+    return json.loads(timing_path.read_text(encoding="utf-8"))
+
+
+def build_case(bundle_root: Path, repo_root: Path, rir: str) -> dict:
+    if rir not in RIR_CONFIG:
+        raise SystemExit(
+            f"unsupported rir: {rir}; expected one of: {', '.join(sorted(RIR_CONFIG))}"
+        )
+
+    rir_root = require_path(bundle_root / rir, "dir")
+    cfg = RIR_CONFIG[rir]
+    timing_summary = load_timing_summary(bundle_root)
+    if rir not in timing_summary:
+        raise SystemExit(f"timing-summary.json missing entry for rir: {rir}")
+    timing_entry = timing_summary[rir]
+    durations = timing_entry.get("durations") or {}
+
+    base_timing = require_path(rir_root / "timings" / "base-replay.json", "file")
+    delta_timing = require_path(rir_root / "timings" / "delta-replay.json", "file")
+    base_timing_obj = json.loads(base_timing.read_text(encoding="utf-8"))
+    delta_timing_obj = json.loads(delta_timing.read_text(encoding="utf-8"))
+
+    case = {
+        "bundle_root": str(bundle_root),
+        "repo_root": str(repo_root),
+        "rir": rir,
+        "trust_anchor": cfg["trust_anchor"],
+        "rir_root": str(rir_root),
+        "base_archive": str(require_path(rir_root / "base-payload-archive", "dir")),
+        "base_locks": str(require_path(rir_root / "base-locks.json", "file")),
+        "base_vrps_csv": str(require_path(rir_root / "base-vrps.csv", "file")),
+        "delta_archive": str(require_path(rir_root / "payload-delta-archive", "dir")),
+        "delta_locks": str(require_path(rir_root / "locks-delta.json", "file")),
+        "delta_record_csv": str(require_path(rir_root / "record-delta.csv", "file")),
+        "replay_delta_csv": str(require_path(rir_root / "replay-delta.csv", "file")),
+        "verification_json": str(require_path(rir_root / "verification.json", "file")),
+        "readme": str(require_path(rir_root / "README.md", "file")),
+        "timings_dir": str(require_path(rir_root / "timings", "dir")),
+        "base_timing_json": str(base_timing),
+        "delta_timing_json": str(delta_timing),
+        "tal_path": str(require_path(repo_root / cfg["tal"], "file")),
+        "ta_path": str(require_path(repo_root / cfg["ta"], "file")),
+        "validation_times": {
+            "snapshot": base_timing_obj["startedAt"],
+            "delta": delta_timing_obj["startedAt"],
+        },
+        "routinator_timings": {
+            "base_replay_seconds": float(durations["base-replay"]),
+            "delta_replay_seconds": float(durations["delta-replay"]),
+        },
+    }
+    return case
+
+
+def emit_env(case: dict) -> str:
+    ordered = {
+        "BUNDLE_ROOT": case["bundle_root"],
+        "RIR": case["rir"],
+        "TRUST_ANCHOR": case["trust_anchor"],
+        "RIR_ROOT": case["rir_root"],
+        "TAL_PATH": case["tal_path"],
+        "TA_PATH": case["ta_path"],
+        "PAYLOAD_REPLAY_ARCHIVE": case["base_archive"],
+        "PAYLOAD_REPLAY_LOCKS": case["base_locks"],
+        "ROUTINATOR_BASE_RECORD_CSV": case["base_vrps_csv"],
+        "PAYLOAD_BASE_ARCHIVE": case["base_archive"],
+        "PAYLOAD_BASE_LOCKS": case["base_locks"],
+        "PAYLOAD_DELTA_ARCHIVE": case["delta_archive"],
+        "PAYLOAD_DELTA_LOCKS": case["delta_locks"],
+        "ROUTINATOR_DELTA_RECORD_CSV": case["delta_record_csv"],
+        "SNAPSHOT_VALIDATION_TIME": case["validation_times"]["snapshot"],
+        "DELTA_VALIDATION_TIME": case["validation_times"]["delta"],
+        "ROUTINATOR_BASE_REPLAY_SECONDS": str(case["routinator_timings"]["base_replay_seconds"]),
+        "ROUTINATOR_DELTA_REPLAY_SECONDS": str(case["routinator_timings"]["delta_replay_seconds"]),
+    }
+    return "\n".join(
+        f"export {key}={shlex.quote(value)}" for key, value in ordered.items()
+    )
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Resolve one RIR case inside a multi-RIR replay bundle")
+    parser.add_argument("--bundle-root", type=Path, default=None)
+    parser.add_argument("--repo-root", type=Path, default=None)
+    parser.add_argument("--rir", required=True, choices=sorted(RIR_CONFIG))
+    parser.add_argument("--format", choices=["json", "env"], default="json")
+    args = parser.parse_args()
+
+    repo_root = (args.repo_root or default_repo_root()).resolve()
+    bundle_root = (args.bundle_root or default_bundle_root(repo_root)).resolve()
+    case = build_case(bundle_root, repo_root, args.rir)
+
+    if args.format == "env":
+        print(emit_env(case))
+    else:
+        print(json.dumps(case, ensure_ascii=False, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
--- a/scripts/payload_replay/run_apnic_delta_replay.sh
+++ b/scripts/payload_replay/run_apnic_delta_replay.sh
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$ROOT_DIR"
+
+DELTA_ROOT="${DELTA_ROOT:-$ROOT_DIR/target/live/apnic_delta_demo/20260315-170223-autoplay}"
+TAL_PATH="${TAL_PATH:-$ROOT_DIR/tests/fixtures/tal/apnic-rfc7730-https.tal}"
+TA_PATH="${TA_PATH:-$ROOT_DIR/tests/fixtures/ta/apnic-ta.cer}"
+PAYLOAD_BASE_ARCHIVE="${PAYLOAD_BASE_ARCHIVE:-$DELTA_ROOT/base-payload-archive}"
+PAYLOAD_BASE_LOCKS="${PAYLOAD_BASE_LOCKS:-$DELTA_ROOT/base-locks.json}"
+PAYLOAD_DELTA_ARCHIVE="${PAYLOAD_DELTA_ARCHIVE:-$DELTA_ROOT/payload-delta-archive}"
+PAYLOAD_DELTA_LOCKS="${PAYLOAD_DELTA_LOCKS:-$DELTA_ROOT/locks-delta.json}"
+VALIDATION_TIME="${VALIDATION_TIME:-2026-03-15T10:00:00Z}"
+PAYLOAD_BASE_VALIDATION_TIME="${PAYLOAD_BASE_VALIDATION_TIME:-}"
+TRUST_ANCHOR="${TRUST_ANCHOR:-apnic}"
+ROUTINATOR_RECORD_CSV="${ROUTINATOR_RECORD_CSV:-$DELTA_ROOT/record-delta.csv}"
+MAX_DEPTH="${MAX_DEPTH:-}"
+MAX_INSTANCES="${MAX_INSTANCES:-}"
+OUT_DIR="${OUT_DIR:-$ROOT_DIR/target/live/payload_delta_replay_runs}"
+mkdir -p "$OUT_DIR"
+
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+RUN_NAME="${RUN_NAME:-apnic_delta_replay_${TS}}"
+DB_DIR="${DB_DIR:-$OUT_DIR/${RUN_NAME}_db}"
+REPORT_JSON="${REPORT_JSON:-$OUT_DIR/${RUN_NAME}_report.json}"
+RUN_LOG="${RUN_LOG:-$OUT_DIR/${RUN_NAME}_run.log}"
+META_JSON="${META_JSON:-$OUT_DIR/${RUN_NAME}_meta.json}"
+SUMMARY_MD="${SUMMARY_MD:-$OUT_DIR/${RUN_NAME}_summary.md}"
+VRPS_CSV="${VRPS_CSV:-$OUT_DIR/${RUN_NAME}_vrps.csv}"
+COMPARE_SUMMARY_MD="${COMPARE_SUMMARY_MD:-$OUT_DIR/${RUN_NAME}_compare_summary.md}"
+ONLY_IN_OURS_CSV="${ONLY_IN_OURS_CSV:-$OUT_DIR/${RUN_NAME}_only_in_ours.csv}"
+ONLY_IN_RECORD_CSV="${ONLY_IN_RECORD_CSV:-$OUT_DIR/${RUN_NAME}_only_in_record.csv}"
+
+cmd=(cargo run --release --bin rpki --
+  --db "$DB_DIR"
+  --tal-path "$TAL_PATH"
+  --ta-path "$TA_PATH"
+  --payload-base-archive "$PAYLOAD_BASE_ARCHIVE"
+  --payload-base-locks "$PAYLOAD_BASE_LOCKS"
+  --payload-delta-archive "$PAYLOAD_DELTA_ARCHIVE"
+  --payload-delta-locks "$PAYLOAD_DELTA_LOCKS"
+  --validation-time "$VALIDATION_TIME"
+  --report-json "$REPORT_JSON")
+
+if [[ -n "$MAX_DEPTH" ]]; then
+  cmd+=(--max-depth "$MAX_DEPTH")
+fi
+if [[ -n "$MAX_INSTANCES" ]]; then
+  cmd+=(--max-instances "$MAX_INSTANCES")
+fi
+
+run_start_s="$(date +%s)"
+(
+  echo "# command:"
+  printf '%q ' "${cmd[@]}"
+  echo
+  echo
+  "${cmd[@]}"
+) 2>&1 | tee "$RUN_LOG" >/dev/null
+run_end_s="$(date +%s)"
+run_duration_s="$((run_end_s - run_start_s))"
+
+PAYLOAD_BASE_ARCHIVE="$PAYLOAD_BASE_ARCHIVE" \
+PAYLOAD_BASE_LOCKS="$PAYLOAD_BASE_LOCKS" \
+PAYLOAD_DELTA_ARCHIVE="$PAYLOAD_DELTA_ARCHIVE" \
+PAYLOAD_DELTA_LOCKS="$PAYLOAD_DELTA_LOCKS" \
+PAYLOAD_BASE_VALIDATION_TIME="$PAYLOAD_BASE_VALIDATION_TIME" \
+DB_DIR="$DB_DIR" \
+REPORT_JSON="$REPORT_JSON" \
+RUN_LOG="$RUN_LOG" \
+VALIDATION_TIME="$VALIDATION_TIME" \
+RUN_DURATION_S="$run_duration_s" \
+python3 - "$REPORT_JSON" "$META_JSON" "$SUMMARY_MD" <<'PY'
+import json
+import os
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+report_path = Path(sys.argv[1])
+meta_path = Path(sys.argv[2])
+summary_path = Path(sys.argv[3])
+rep = json.loads(report_path.read_text(encoding='utf-8'))
+now = datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
+meta = {
+  'recorded_at_utc': now,
+  'payload_base_archive': os.environ['PAYLOAD_BASE_ARCHIVE'],
+  'payload_base_locks': os.environ['PAYLOAD_BASE_LOCKS'],
+  'payload_delta_archive': os.environ['PAYLOAD_DELTA_ARCHIVE'],
+  'payload_delta_locks': os.environ['PAYLOAD_DELTA_LOCKS'],
+  'db_dir': os.environ['DB_DIR'],
+  'report_json': os.environ['REPORT_JSON'],
+  'run_log': os.environ['RUN_LOG'],
+  'validation_time_arg': os.environ['VALIDATION_TIME'],
+  'base_validation_time_arg': os.environ.get('PAYLOAD_BASE_VALIDATION_TIME') or os.environ['VALIDATION_TIME'],
+  'durations_secs': {'rpki_run': int(os.environ['RUN_DURATION_S'])},
+  'counts': {
+    'publication_points_processed': rep['tree']['instances_processed'],
+    'publication_points_failed': rep['tree']['instances_failed'],
+    'vrps': len(rep['vrps']),
+    'aspas': len(rep['aspas']),
+    'audit_publication_points': len(rep['publication_points']),
+  },
+}
+meta_path.write_text(json.dumps(meta, ensure_ascii=False, indent=2)+'\n', encoding='utf-8')
+summary = []
+summary.append('# Payload Delta Replay Summary\n\n')
+for key in ['payload_base_archive','payload_base_locks','payload_delta_archive','payload_delta_locks','db_dir','report_json','base_validation_time_arg','validation_time_arg']:
+    summary.append(f'- {key}: `{meta[key]}`\n')
+summary.append('\n## Results\n\n| metric | value |\n|---|---:|\n')
+for k,v in meta['counts'].items():
+    summary.append(f'| {k} | {v} |\n')
+summary.append('\n## Durations\n\n| step | seconds |\n|---|---:|\n')
+for k,v in meta['durations_secs'].items():
+    summary.append(f'| {k} | {v} |\n')
+summary_path.write_text(''.join(summary), encoding='utf-8')
+print(summary_path)
+PY
+
+python3 scripts/payload_replay/report_to_routinator_csv.py \
+  --report "$REPORT_JSON" \
+  --out "$VRPS_CSV" \
+  --trust-anchor "$TRUST_ANCHOR" >/dev/null
+
+if [[ -f "$ROUTINATOR_RECORD_CSV" ]]; then
+  ./scripts/payload_replay/compare_with_routinator_record.sh \
+    "$VRPS_CSV" \
+    "$ROUTINATOR_RECORD_CSV" \
+    "$COMPARE_SUMMARY_MD" \
+    "$ONLY_IN_OURS_CSV" \
+    "$ONLY_IN_RECORD_CSV" >/dev/null
+fi
+
+echo "== payload delta replay run complete ==" >&2
+echo "- db: $DB_DIR" >&2
+echo "- report: $REPORT_JSON" >&2
+echo "- run log: $RUN_LOG" >&2
+echo "- meta json: $META_JSON" >&2
+echo "- summary md: $SUMMARY_MD" >&2
+echo "- vrps csv: $VRPS_CSV" >&2
+if [[ -f "$COMPARE_SUMMARY_MD" ]]; then
+  echo "- compare summary: $COMPARE_SUMMARY_MD" >&2
+  echo "- only in ours: $ONLY_IN_OURS_CSV" >&2
+  echo "- only in record: $ONLY_IN_RECORD_CSV" >&2
+fi
--- a/scripts/payload_replay/run_multi_rir_replay_case.sh
+++ b/scripts/payload_replay/run_multi_rir_replay_case.sh
@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$ROOT_DIR"
+
+if [[ $# -lt 1 || $# -gt 2 ]]; then
+  echo "usage: $0 <rir> [describe|snapshot|delta|both]" >&2
+  exit 2
+fi
+
+RIR="$1"
+MODE="${2:-both}"
+BUNDLE_ROOT="${BUNDLE_ROOT:-$ROOT_DIR/../../rpki/target/live/20260316-112341-multi-final3}"
+CASE_INFO_SCRIPT="$ROOT_DIR/scripts/payload_replay/multi_rir_case_info.py"
+CASE_REPORT_SCRIPT="$ROOT_DIR/scripts/payload_replay/write_multi_rir_case_report.py"
+MULTI_RIR_OUT_DIR="${MULTI_RIR_OUT_DIR:-$ROOT_DIR/target/live/multi_rir_replay_runs/$RIR}"
+mkdir -p "$MULTI_RIR_OUT_DIR"
+
+eval "$(python3 "$CASE_INFO_SCRIPT" --bundle-root "$BUNDLE_ROOT" --rir "$RIR" --format env)"
+
+SNAPSHOT_DB_DIR="${SNAPSHOT_DB_DIR:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_replay_db}"
+SNAPSHOT_REPORT_MD="${SNAPSHOT_REPORT_MD:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_compare_summary.md}"
+SNAPSHOT_META_JSON="${SNAPSHOT_META_JSON:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_meta.json}"
+SNAPSHOT_RUN_LOG="${SNAPSHOT_RUN_LOG:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_run.log}"
+SNAPSHOT_REPORT_JSON="${SNAPSHOT_REPORT_JSON:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_report.json}"
+SNAPSHOT_VRPS_CSV="${SNAPSHOT_VRPS_CSV:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_vrps.csv}"
+SNAPSHOT_ONLY_OURS="${SNAPSHOT_ONLY_OURS:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_only_in_ours.csv}"
+SNAPSHOT_ONLY_RECORD="${SNAPSHOT_ONLY_RECORD:-$MULTI_RIR_OUT_DIR/${RIR}_snapshot_only_in_record.csv}"
+
+DELTA_DB_DIR="${DELTA_DB_DIR:-$MULTI_RIR_OUT_DIR/${RIR}_delta_replay_db}"
+DELTA_REPORT_MD="${DELTA_REPORT_MD:-$MULTI_RIR_OUT_DIR/${RIR}_delta_compare_summary.md}"
+DELTA_META_JSON="${DELTA_META_JSON:-$MULTI_RIR_OUT_DIR/${RIR}_delta_meta.json}"
+DELTA_RUN_LOG="${DELTA_RUN_LOG:-$MULTI_RIR_OUT_DIR/${RIR}_delta_run.log}"
+DELTA_REPORT_JSON="${DELTA_REPORT_JSON:-$MULTI_RIR_OUT_DIR/${RIR}_delta_report.json}"
+DELTA_VRPS_CSV="${DELTA_VRPS_CSV:-$MULTI_RIR_OUT_DIR/${RIR}_delta_vrps.csv}"
+DELTA_ONLY_OURS="${DELTA_ONLY_OURS:-$MULTI_RIR_OUT_DIR/${RIR}_delta_only_in_ours.csv}"
+DELTA_ONLY_RECORD="${DELTA_ONLY_RECORD:-$MULTI_RIR_OUT_DIR/${RIR}_delta_only_in_record.csv}"
+
+CASE_REPORT_JSON="${CASE_REPORT_JSON:-$MULTI_RIR_OUT_DIR/${RIR}_case_report.json}"
+CASE_REPORT_MD="${CASE_REPORT_MD:-$MULTI_RIR_OUT_DIR/${RIR}_case_report.md}"
+
+case "$MODE" in
+  describe)
+    python3 "$CASE_INFO_SCRIPT" --bundle-root "$BUNDLE_ROOT" --rir "$RIR"
+    ;;
+  snapshot)
+    rm -rf "$SNAPSHOT_DB_DIR"
+    ROUTINATOR_RECORD_CSV="$ROUTINATOR_BASE_RECORD_CSV" \
+      VALIDATION_TIME="$SNAPSHOT_VALIDATION_TIME" \
+      OUT_DIR="$MULTI_RIR_OUT_DIR" \
+      DB_DIR="$SNAPSHOT_DB_DIR" \
+      RUN_NAME="${RUN_NAME:-${RIR}_snapshot_replay}" \
+      META_JSON="$SNAPSHOT_META_JSON" \
+      RUN_LOG="$SNAPSHOT_RUN_LOG" \
+      REPORT_JSON="$SNAPSHOT_REPORT_JSON" \
+      VRPS_CSV="$SNAPSHOT_VRPS_CSV" \
+      COMPARE_SUMMARY_MD="$SNAPSHOT_REPORT_MD" \
+      ONLY_IN_OURS_CSV="$SNAPSHOT_ONLY_OURS" \
+      ONLY_IN_RECORD_CSV="$SNAPSHOT_ONLY_RECORD" \
+      ./scripts/payload_replay/run_apnic_replay.sh
+    ;;
+  delta)
+    rm -rf "$DELTA_DB_DIR"
+    ROUTINATOR_RECORD_CSV="$ROUTINATOR_DELTA_RECORD_CSV" \
+      VALIDATION_TIME="$DELTA_VALIDATION_TIME" \
+      PAYLOAD_BASE_VALIDATION_TIME="$SNAPSHOT_VALIDATION_TIME" \
+      OUT_DIR="$MULTI_RIR_OUT_DIR" \
+      DB_DIR="$DELTA_DB_DIR" \
+      RUN_NAME="${RUN_NAME:-${RIR}_delta_replay}" \
+      DELTA_ROOT="$RIR_ROOT" \
+      META_JSON="$DELTA_META_JSON" \
+      RUN_LOG="$DELTA_RUN_LOG" \
+      REPORT_JSON="$DELTA_REPORT_JSON" \
+      VRPS_CSV="$DELTA_VRPS_CSV" \
+      COMPARE_SUMMARY_MD="$DELTA_REPORT_MD" \
+      ONLY_IN_OURS_CSV="$DELTA_ONLY_OURS" \
+      ONLY_IN_RECORD_CSV="$DELTA_ONLY_RECORD" \
+      ./scripts/payload_replay/run_apnic_delta_replay.sh
+    ;;
+  both)
+    rm -rf "$SNAPSHOT_DB_DIR" "$DELTA_DB_DIR"
+    ROUTINATOR_RECORD_CSV="$ROUTINATOR_BASE_RECORD_CSV" \
+      VALIDATION_TIME="$SNAPSHOT_VALIDATION_TIME" \
+      OUT_DIR="$MULTI_RIR_OUT_DIR" \
+      DB_DIR="$SNAPSHOT_DB_DIR" \
+      RUN_NAME="${RUN_NAME_SNAPSHOT:-${RIR}_snapshot_replay}" \
+      META_JSON="$SNAPSHOT_META_JSON" \
+      RUN_LOG="$SNAPSHOT_RUN_LOG" \
+      REPORT_JSON="$SNAPSHOT_REPORT_JSON" \
+      VRPS_CSV="$SNAPSHOT_VRPS_CSV" \
+      COMPARE_SUMMARY_MD="$SNAPSHOT_REPORT_MD" \
+      ONLY_IN_OURS_CSV="$SNAPSHOT_ONLY_OURS" \
+      ONLY_IN_RECORD_CSV="$SNAPSHOT_ONLY_RECORD" \
+      ./scripts/payload_replay/run_apnic_replay.sh
+    ROUTINATOR_RECORD_CSV="$ROUTINATOR_DELTA_RECORD_CSV" \
+      VALIDATION_TIME="$DELTA_VALIDATION_TIME" \
+      PAYLOAD_BASE_VALIDATION_TIME="$SNAPSHOT_VALIDATION_TIME" \
+      OUT_DIR="$MULTI_RIR_OUT_DIR" \
+      DB_DIR="$DELTA_DB_DIR" \
+      RUN_NAME="${RUN_NAME_DELTA:-${RIR}_delta_replay}" \
+      DELTA_ROOT="$RIR_ROOT" \
+      META_JSON="$DELTA_META_JSON" \
+      RUN_LOG="$DELTA_RUN_LOG" \
+      REPORT_JSON="$DELTA_REPORT_JSON" \
+      VRPS_CSV="$DELTA_VRPS_CSV" \
+      COMPARE_SUMMARY_MD="$DELTA_REPORT_MD" \
+      ONLY_IN_OURS_CSV="$DELTA_ONLY_OURS" \
+      ONLY_IN_RECORD_CSV="$DELTA_ONLY_RECORD" \
+      ./scripts/payload_replay/run_apnic_delta_replay.sh
+    python3 "$CASE_REPORT_SCRIPT" \
+      --rir "$RIR" \
+      --snapshot-meta "$SNAPSHOT_META_JSON" \
+      --snapshot-compare "$SNAPSHOT_REPORT_MD" \
+      --delta-meta "$DELTA_META_JSON" \
+      --delta-compare "$DELTA_REPORT_MD" \
+      --routinator-base-seconds "$ROUTINATOR_BASE_REPLAY_SECONDS" \
+      --routinator-delta-seconds "$ROUTINATOR_DELTA_REPLAY_SECONDS" \
+      --out-md "$CASE_REPORT_MD" \
+      --out-json "$CASE_REPORT_JSON" >/dev/null
+    echo "- case report: $CASE_REPORT_MD" >&2
+    echo "- case report json: $CASE_REPORT_JSON" >&2
+    ;;
+  *)
+    echo "unsupported mode: $MODE; expected describe|snapshot|delta|both" >&2
+    exit 2
+    ;;
+esac
--- a/scripts/payload_replay/run_multi_rir_replay_suite.sh
+++ b/scripts/payload_replay/run_multi_rir_replay_suite.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$ROOT_DIR"
+
+BUNDLE_ROOT="${BUNDLE_ROOT:-$ROOT_DIR/../../rpki/target/live/20260316-112341-multi-final3}"
+SUITE_OUT_DIR="${SUITE_OUT_DIR:-$ROOT_DIR/target/live/multi_rir_replay_runs}"
+RIRS="${RIRS:-afrinic apnic arin lacnic ripe}"
+CASE_SCRIPT="$ROOT_DIR/scripts/payload_replay/run_multi_rir_replay_case.sh"
+SUMMARY_SCRIPT="$ROOT_DIR/scripts/payload_replay/write_multi_rir_summary.py"
+
+mkdir -p "$SUITE_OUT_DIR"
+
+for rir in $RIRS; do
+  MULTI_RIR_OUT_DIR="$SUITE_OUT_DIR/$rir" \
+    BUNDLE_ROOT="$BUNDLE_ROOT" \
+    "$CASE_SCRIPT" "$rir" both
+  echo "completed $rir" >&2
+  echo >&2
+ done
+
+python3 "$SUMMARY_SCRIPT" \
+  --case-root "$SUITE_OUT_DIR" \
+  --out-md "$SUITE_OUT_DIR/multi_rir_summary.md" \
+  --out-json "$SUITE_OUT_DIR/multi_rir_summary.json" \
+  --rirs $RIRS >/dev/null
+
+echo "== multi-RIR replay suite complete ==" >&2
+echo "- suite_out_dir: $SUITE_OUT_DIR" >&2
+echo "- summary_md: $SUITE_OUT_DIR/multi_rir_summary.md" >&2
+echo "- summary_json: $SUITE_OUT_DIR/multi_rir_summary.json" >&2
--- a/scripts/payload_replay/write_multi_rir_case_report.py
+++ b/scripts/payload_replay/write_multi_rir_case_report.py
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Generate one multi-RIR replay case report")
+    p.add_argument("--rir", required=True)
+    p.add_argument("--snapshot-meta", required=True)
+    p.add_argument("--snapshot-compare", required=True)
+    p.add_argument("--delta-meta", required=True)
+    p.add_argument("--delta-compare", required=True)
+    p.add_argument("--routinator-base-seconds", required=True, type=float)
+    p.add_argument("--routinator-delta-seconds", required=True, type=float)
+    p.add_argument("--out-md", required=True)
+    p.add_argument("--out-json", required=True)
+    return p.parse_args()
+
+
+def read_json(path: str) -> dict:
+    return json.loads(Path(path).read_text(encoding="utf-8"))
+
+
+def parse_compare_md(path: str) -> dict:
+    lines = Path(path).read_text(encoding="utf-8").splitlines()
+    out = {}
+    for line in lines:
+        if not line.startswith("| "):
+            continue
+        parts = [p.strip() for p in line.strip("|").split("|")]
+        if len(parts) != 2:
+            continue
+        key, value = parts
+        if key in {"metric", "---"}:
+            continue
+        try:
+            out[key] = int(value)
+        except ValueError:
+            pass
+    return out
+
+
+def ratio(ours: float, baseline: float) -> float | None:
+    if baseline <= 0:
+        return None
+    return ours / baseline
+
+
+def build_report(args: argparse.Namespace) -> dict:
+    snapshot_meta = read_json(args.snapshot_meta)
+    delta_meta = read_json(args.delta_meta)
+    snapshot_compare = parse_compare_md(args.snapshot_compare)
+    delta_compare = parse_compare_md(args.delta_compare)
+
+    snapshot_ours = float(snapshot_meta["durations_secs"]["rpki_run"])
+    delta_ours = float(delta_meta["durations_secs"]["rpki_run"])
+
+    report = {
+        "rir": args.rir,
+        "snapshot": {
+            "meta_json": str(Path(args.snapshot_meta).resolve()),
+            "compare_md": str(Path(args.snapshot_compare).resolve()),
+            "ours_seconds": snapshot_ours,
+            "routinator_seconds": args.routinator_base_seconds,
+            "ratio": ratio(snapshot_ours, args.routinator_base_seconds),
+            "compare": snapshot_compare,
+            "match": snapshot_compare.get("only_in_ours", -1) == 0
+            and snapshot_compare.get("only_in_record", -1) == 0,
+            "counts": snapshot_meta.get("counts", {}),
+        },
+        "delta": {
+            "meta_json": str(Path(args.delta_meta).resolve()),
+            "compare_md": str(Path(args.delta_compare).resolve()),
+            "ours_seconds": delta_ours,
+            "routinator_seconds": args.routinator_delta_seconds,
+            "ratio": ratio(delta_ours, args.routinator_delta_seconds),
+            "compare": delta_compare,
+            "match": delta_compare.get("only_in_ours", -1) == 0
+            and delta_compare.get("only_in_record", -1) == 0,
+            "counts": delta_meta.get("counts", {}),
+        },
+    }
+    return report
+
+
+def write_md(path: Path, report: dict) -> None:
+    snapshot = report["snapshot"]
+    delta = report["delta"]
+    lines = []
+    lines.append(f"# {report['rir'].upper()} Replay Report\n\n")
+    lines.append("## Summary\n\n")
+    lines.append("| mode | match | ours_s | routinator_s | ratio | only_in_ours | only_in_record |\n")
+    lines.append("|---|---|---:|---:|---:|---:|---:|\n")
+    lines.append(
+        f"| snapshot | {str(snapshot['match']).lower()} | {snapshot['ours_seconds']:.3f} | {snapshot['routinator_seconds']:.3f} | {snapshot['ratio']:.3f} | {snapshot['compare'].get('only_in_ours', 0)} | {snapshot['compare'].get('only_in_record', 0)} |\n"
+    )
+    lines.append(
+        f"| delta | {str(delta['match']).lower()} | {delta['ours_seconds']:.3f} | {delta['routinator_seconds']:.3f} | {delta['ratio']:.3f} | {delta['compare'].get('only_in_ours', 0)} | {delta['compare'].get('only_in_record', 0)} |\n"
+    )
+    lines.append("\n## Snapshot Inputs\n\n")
+    lines.append(f"- meta_json: `{snapshot['meta_json']}`\n")
+    lines.append(f"- compare_md: `{snapshot['compare_md']}`\n")
+    lines.append("\n## Delta Inputs\n\n")
+    lines.append(f"- meta_json: `{delta['meta_json']}`\n")
+    lines.append(f"- compare_md: `{delta['compare_md']}`\n")
+    lines.append("\n## Counts\n\n")
+    lines.append("### Snapshot\n\n")
+    for k, v in sorted(snapshot.get("counts", {}).items()):
+        lines.append(f"- {k}: `{v}`\n")
+    lines.append("\n### Delta\n\n")
+    for k, v in sorted(delta.get("counts", {}).items()):
+        lines.append(f"- {k}: `{v}`\n")
+    path.write_text("".join(lines), encoding="utf-8")
+
+
+def main() -> int:
+    args = parse_args()
+    report = build_report(args)
+    out_json = Path(args.out_json)
+    out_md = Path(args.out_md)
+    out_json.parent.mkdir(parents=True, exist_ok=True)
+    out_md.parent.mkdir(parents=True, exist_ok=True)
+    out_json.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+    write_md(out_md, report)
+    print(out_md)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
--- a/scripts/payload_replay/write_multi_rir_summary.py
+++ b/scripts/payload_replay/write_multi_rir_summary.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import math
+from pathlib import Path
+
+DEFAULT_RIRS = ["afrinic", "apnic", "arin", "lacnic", "ripe"]
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Aggregate per-RIR replay case reports")
+    p.add_argument("--case-root", required=True, help="directory containing <rir>/<rir>_case_report.json")
+    p.add_argument("--out-md", required=True)
+    p.add_argument("--out-json", required=True)
+    p.add_argument("--rirs", nargs="*", default=None, help="RIRs to include (default: all 5)")
+    return p.parse_args()
+
+
+def read_case(case_root: Path, rir: str) -> dict:
+    path = case_root / rir / f"{rir}_case_report.json"
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def geomean(values: list[float]) -> float:
+    vals = [v for v in values if v > 0]
+    if not vals:
+        return 0.0
+    return math.exp(sum(math.log(v) for v in vals) / len(vals))
+
+
+def build_summary(cases: list[dict]) -> dict:
+    snapshot_ratios = [c["snapshot"]["ratio"] for c in cases]
+    delta_ratios = [c["delta"]["ratio"] for c in cases]
+    return {
+        "cases": cases,
+        "summary": {
+            "snapshot_all_match": all(c["snapshot"]["match"] for c in cases),
+            "delta_all_match": all(c["delta"]["match"] for c in cases),
+            "snapshot_ratio_geomean": geomean(snapshot_ratios),
+            "delta_ratio_geomean": geomean(delta_ratios),
+            "all_ratio_geomean": geomean(snapshot_ratios + delta_ratios),
+        },
+    }
+
+
+def write_md(path: Path, data: dict) -> None:
+    lines = []
+    lines.append("# Multi-RIR Replay Summary\n\n")
+    lines.append("## Correctness + Timing\n\n")
+    lines.append("| RIR | snapshot_match | snapshot_ours_s | snapshot_routinator_s | snapshot_ratio | delta_match | delta_ours_s | delta_routinator_s | delta_ratio |\n")
+    lines.append("|---|---|---:|---:|---:|---|---:|---:|---:|\n")
+    for case in data["cases"]:
+        lines.append(
+            f"| {case['rir']} | {str(case['snapshot']['match']).lower()} | {case['snapshot']['ours_seconds']:.3f} | {case['snapshot']['routinator_seconds']:.3f} | {case['snapshot']['ratio']:.3f} | {str(case['delta']['match']).lower()} | {case['delta']['ours_seconds']:.3f} | {case['delta']['routinator_seconds']:.3f} | {case['delta']['ratio']:.3f} |\n"
+        )
+    s = data["summary"]
+    lines.append("\n## Aggregate Metrics\n\n")
+    lines.append("| metric | value |\n")
+    lines.append("|---|---:|\n")
+    lines.append(f"| snapshot_all_match | {str(s['snapshot_all_match']).lower()} |\n")
+    lines.append(f"| delta_all_match | {str(s['delta_all_match']).lower()} |\n")
+    lines.append(f"| snapshot_ratio_geomean | {s['snapshot_ratio_geomean']:.3f} |\n")
+    lines.append(f"| delta_ratio_geomean | {s['delta_ratio_geomean']:.3f} |\n")
+    lines.append(f"| all_ratio_geomean | {s['all_ratio_geomean']:.3f} |\n")
+    path.write_text("".join(lines), encoding="utf-8")
+
+
+def main() -> int:
+    args = parse_args()
+    case_root = Path(args.case_root)
+    rirs = args.rirs or DEFAULT_RIRS
+    cases = [read_case(case_root, rir) for rir in rirs]
+    data = build_summary(cases)
+    out_md = Path(args.out_md)
+    out_json = Path(args.out_json)
+    out_md.parent.mkdir(parents=True, exist_ok=True)
+    out_json.parent.mkdir(parents=True, exist_ok=True)
+    out_json.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+    write_md(out_md, data)
+    print(out_md)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())