20260716 增加verification-only缓存正确性复核

scripts/verification/README.md

@@ -0,0 +1,50 @@
+# verification-only 使用说明
+
+`verification-only` 用于复核启用验证缓存的最新成功 normal run。它冻结当前仓库视图，关闭 PP、ROA、child certificate cache、prefetch、VCIR 持久化和网络访问，在源 CIR 的同一验证时间重新执行完整验证，并比较 CCR 的 MFT、VRP、VAP、TA、router key 五类 state digest。
+
+## 前提
+
+- 源 run 必须是当前 state DB 对应的最新成功 normal run。
+- 源 run 必须包含 `input.cir`、`result.ccr`、`run-summary.json` 和 `validation-contract.json`。
+- state root 必须包含 `db/work-db`、`db/repo-bytes.db` 和 `meta/current-state-binding.json`。
+- 源 normal run 必须启用至少一种验证缓存；仅启用 transport prefetch 不算验证缓存。
+- verification-only 与 normal soak 共用 `state/locks/state-execution.lock`，不能绕过 wrapper 并发执行。
+
+## 执行
+
+在 portable package 根目录执行：
+
+```bash
+./scripts/verification/run_verification_only.sh \
+  --source-run-dir /data/ours-rp/runs/run_0123 \
+  --source-state-root /data/ours-rp/state \
+  --out-dir /data/ours-rp/verification-runs/run_0123
+```
+
+验证时间自动从源 `input.cir` 读取，不接受命令行覆盖。成功后 scratch work-db 默认删除；排障时可加 `--keep-scratch`。
+
+## 输出
+
+- `input.cir`、`result.ccr`、`report.json`
+- `vrps.csv`、`vaps.csv`
+- `stage-timing.json`、`validation-events.jsonl`
+- `validation-contract.json`、`verification-meta.json`
+- `compare/ccr-state-compare.json`、`compare/ccr-state-compare.md`
+- `stdout.log`、`stderr.log`、`process-time.txt`、`wrapper-timing.json`
+
+通过条件是 compare JSON 中 `stateDigestMatch=true`。`producedAt` 不参与比较。
+
+## 隔离保证
+
+- 不分配 normal run id，不更新 lifecycle、periodic snapshot 计数和 retention。
+- 源 work-db 只用于创建 checkpoint，后续写入只发生在独立 scratch work-db。
+- 源 repo-bytes.db 以只读模式打开。
+- 输出目录必须不存在，防止覆盖旧验证证据。
+
+## 常见失败
+
+- `current state binding`：所选 run 已不是当前 state 对应的最新成功 run，需选择 binding 指向的 run。
+- `differs from current state binding`：源 CIR、CCR 或 contract 已变化，不能继续复核。
+- `blob is missing`：repository view 引用的原始 bytes 不完整，验证不会启动。
+- `did not enable a validation cache`：源 run 不是缓存正确性对比的有效基准。
+- `CCR state digest mismatch`：查看 `compare/` 下 JSON/Markdown，按不匹配状态集合继续排查。

scripts/verification/run_verification_only.sh

@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'USAGE'
+Usage:
+  run_verification_only.sh \
+    --source-run-dir <run_dir> \
+    --source-state-root <state_dir> \
+    --out-dir <verification_dir> \
+    [--rpki-bin <path>] [--keep-scratch]
+
+validation time is always read from the source CIR. The wrapper serializes access
+with normal soak runs and never writes into the normal run sequence.
+USAGE
+}
+
+SOURCE_RUN_DIR=""
+SOURCE_STATE_ROOT=""
+OUT_DIR=""
+RPKI_BIN=""
+KEEP_SCRATCH=0
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --source-run-dir) SOURCE_RUN_DIR="${2:?missing value}"; shift 2 ;;
+    --source-state-root) SOURCE_STATE_ROOT="${2:?missing value}"; shift 2 ;;
+    --out-dir) OUT_DIR="${2:?missing value}"; shift 2 ;;
+    --rpki-bin) RPKI_BIN="${2:?missing value}"; shift 2 ;;
+    --keep-scratch) KEEP_SCRATCH=1; shift ;;
+    -h|--help) usage; exit 0 ;;
+    *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
+  esac
+done
+
+[[ -n "$SOURCE_RUN_DIR" ]] || { echo "--source-run-dir is required" >&2; exit 2; }
+[[ -n "$SOURCE_STATE_ROOT" ]] || { echo "--source-state-root is required" >&2; exit 2; }
+[[ -n "$OUT_DIR" ]] || { echo "--out-dir is required" >&2; exit 2; }
+if [[ -z "$RPKI_BIN" ]]; then
+  RPKI_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/target/release/rpki"
+fi
+[[ -x "$RPKI_BIN" ]] || { echo "rpki binary is not executable: $RPKI_BIN" >&2; exit 2; }
+command -v flock >/dev/null || { echo "flock is required" >&2; exit 2; }
+command -v python3 >/dev/null || { echo "python3 is required" >&2; exit 2; }
+
+now_ms() {
+  python3 -c 'import time; print(time.time_ns() // 1_000_000)'
+}
+
+mkdir -p "$SOURCE_STATE_ROOT/locks" "$(dirname "$OUT_DIR")"
+[[ ! -e "$OUT_DIR" ]] || { echo "verification output already exists: $OUT_DIR" >&2; exit 2; }
+LOCK_PATH="$SOURCE_STATE_ROOT/locks/state-execution.lock"
+STDOUT_TMP="${OUT_DIR}.stdout.tmp"
+STDERR_TMP="${OUT_DIR}.stderr.tmp"
+PROCESS_TIME_TMP="${OUT_DIR}.process-time.tmp"
+WAIT_STARTED_MS="$(now_ms)"
+exec {LOCK_FD}>"$LOCK_PATH"
+flock "$LOCK_FD"
+LOCK_ACQUIRED_MS="$(now_ms)"
+
+ARGS=(
+  --verification-only
+  --verification-source-run "$SOURCE_RUN_DIR"
+  --verification-source-state "$SOURCE_STATE_ROOT"
+  --verification-out "$OUT_DIR"
+)
+if [[ "$KEEP_SCRATCH" -eq 1 ]]; then
+  ARGS+=(--keep-verification-scratch)
+fi
+
+set +e
+if [[ -x /usr/bin/time ]]; then
+  RPKI_STATE_EXECUTION_LOCK_HELD=1 /usr/bin/time -v -o "$PROCESS_TIME_TMP" -- \
+    "$RPKI_BIN" "${ARGS[@]}" >"$STDOUT_TMP" 2>"$STDERR_TMP"
+else
+  RPKI_STATE_EXECUTION_LOCK_HELD=1 "$RPKI_BIN" "${ARGS[@]}" >"$STDOUT_TMP" 2>"$STDERR_TMP"
+fi
+EXIT_CODE=$?
+set -e
+LOCK_RELEASED_MS="$(now_ms)"
+flock -u "$LOCK_FD"
+exec {LOCK_FD}>&-
+
+mkdir -p "$OUT_DIR"
+mv "$STDOUT_TMP" "$OUT_DIR/stdout.log"
+mv "$STDERR_TMP" "$OUT_DIR/stderr.log"
+if [[ -f "$PROCESS_TIME_TMP" ]]; then
+  mv "$PROCESS_TIME_TMP" "$OUT_DIR/process-time.txt"
+fi
+python3 - "$OUT_DIR/wrapper-timing.json" "$WAIT_STARTED_MS" "$LOCK_ACQUIRED_MS" "$LOCK_RELEASED_MS" "$EXIT_CODE" <<'PY'
+import json
+import sys
+from pathlib import Path
+
+path = Path(sys.argv[1])
+wait_started_ms, lock_acquired_ms, lock_released_ms, exit_code = map(int, sys.argv[2:])
+path.write_text(json.dumps({
+    "lockWaitMs": lock_acquired_ms - wait_started_ms,
+    "lockHeldMs": lock_released_ms - lock_acquired_ms,
+    "exitCode": exit_code,
+}, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+PY
+exit "$EXIT_CODE"