50 lines
2.5 KiB
Markdown
50 lines
2.5 KiB
Markdown
# Compatibility harness (M5)
|
|
|
|
`verify_run_abi.sh` validates the required files and semantic fields for a
|
|
successful normal run. It is intentionally independent of absolute paths and
|
|
wall-clock timestamps, so it can be used for native and Docker run directories:
|
|
|
|
```bash
|
|
tests/compat/verify_run_abi.sh /path/to/runs/run_0001
|
|
```
|
|
|
|
`compare_runs.py` compares an original-runtime run with a component run. It
|
|
checks the normal-run status/count contract, report and validation contract,
|
|
and canonical CIR/CCR/VRP/VAP payloads. It normalises only the exceptions in
|
|
`baseline-manifest.toml` (including runtime telemetry that is measured
|
|
separately). Validation time is treated as a fixed input, while only lifecycle
|
|
timestamps and CCR `producedAt` are normalised; all other semantic payload
|
|
differences are failures:
|
|
|
|
```bash
|
|
tests/compat/compare_runs.py \
|
|
/path/to/original/runs/run_0001 \
|
|
/path/to/panda/runs/run_0001
|
|
```
|
|
|
|
The M5 harness will run the same frozen input against the original `rpki`
|
|
runtime, native `panda-rpki-validator`, and the Docker runtime. It will compare
|
|
the normal-run output ABI in `docs/output-abi.md`, then record stage timing,
|
|
CPU, RSS, state size and artifact size for five serial repetitions per profile.
|
|
|
|
M4 provides the ABI verifier and a real snapshot/warm-delta staging run. The
|
|
canonical comparator is now available for M5; five-run performance comparison
|
|
and the complete profile matrix remain M5 deliverables.
|
|
`verification-only` is excluded.
|
|
|
|
For live-RIR evidence, run both images with the same `RIRS` and fixed
|
|
`--validation-time`, then compare `input.cir` and the decoded `result.ccr` in
|
|
separate steps. `compare_runs.py` deliberately remains strict about report,
|
|
validation-event and CSV order, so a live RRDP/rsync source race can produce a
|
|
file-level mismatch even when the CCR state digest matches. Use
|
|
`ccr_state_compare` plus `triage_ccr_cir_pair` to distinguish source/input
|
|
drift from validator behavior; the 2026-09-01 APNIC/all5 evidence is recorded in
|
|
`specs/develop/20260901/m5_live_rir_image_artifact_comparison_milestone_report.md`.
|
|
The follow-up remote-231 serial all5 run (old image first, then new image;
|
|
one snapshot plus three deltas per side with cache/prefetch/parallel flags) is
|
|
recorded in
|
|
`specs/develop/20260901_2/m5_live_all5_cache_prefetch_serial_4run_milestone_report.md`.
|
|
Its strict file mismatches remain failures until the same frozen input is
|
|
replayed on both sides; live CCR/CIR differences must first be triaged with
|
|
the state comparator rather than treated as code parity.
|