rpki/scripts/benchmark

RPKI Benchmarks (Stage2, selected_der_v2)

This directory contains a reproducible, one-click benchmark to measure decode + profile validate performance for all supported object types and compare OURS against the Routinator baseline (rpki crate =0.19.1 with repository feature).

What it measures

Dataset:

• Fixtures: rpki/tests/benchmark/selected_der_v2/
• Objects: cer, crl, manifest (.mft), roa, aspa (.asa)
• Samples: 10 quantiles per type (min/p01/p10/p25/p50/p75/p90/p95/p99/max) → 50 files total

Metrics:

• decode+validate: decode_der (parse + profile validate) for each object file
• landing (OURS only): PackFile::from_bytes_compute_sha256 + CBOR encode + RocksDB put_raw
• compare: ratio ours_ns/op ÷ rout_ns/op for decode+validate

Default benchmark settings

Both OURS and Routinator baseline use the same run settings:

• warmup: 10 iterations
• rounds: 3
• adaptive loop target: min_round_ms=200 (with an internal max of 1_000_000 iters)
• strict DER: true (baseline)
• cert inspect: false (baseline)

You can override the settings via environment variables in the runner script:

• BENCH_WARMUP_ITERS (default 10)
• BENCH_ROUNDS (default 3)
• BENCH_MIN_ROUND_MS (default 200)

One-click run (OURS + Routinator compare)

From the rpki/ crate directory:

./scripts/benchmark/run_stage2_selected_der_v2_release.sh

Outputs are written under:

• rpki/target/bench/
  • OURS decode+validate: stage2_selected_der_v2_decode_release_<TS>.{md,csv}
  • OURS landing: stage2_selected_der_v2_landing_release_<TS>.{md,csv}
  • Routinator: stage2_selected_der_v2_routinator_decode_release_<TS>.{md,csv}
  • Compare: stage2_selected_der_v2_compare_ours_vs_routinator_decode_release_<TS>.{md,csv}
  • Summary: stage2_selected_der_v2_compare_summary_<TS>.md

Why decode and landing are separated

The underlying benchmark can run in BENCH_MODE=both, but the landing part writes to RocksDB and may trigger background work (e.g., compactions) that can skew subsequent decode timings. For a fair OURS-vs-Routinator comparison, the runner script:

• runs BENCH_MODE=decode_validate for comparison, and
• runs BENCH_MODE=landing separately for landing-only numbers.

Notes

• The Routinator baseline benchmark is implemented in-repo under:
  • rpki/benchmark/routinator_object_bench/
  • It pins rpki = "=0.19.1" in its Cargo.toml.
• This benchmark is implemented as an #[ignore] integration test:
  • rpki/tests/bench_stage2_decode_profile_selected_der_v2.rs
  • The runner script invokes it with cargo test --release ... -- --ignored --nocapture.