71 lines
2.6 KiB
Markdown
71 lines
2.6 KiB
Markdown
# RPKI Benchmarks (Stage2, selected_der_v2)
|
|
|
|
This directory contains a reproducible, one-click benchmark to measure **decode + profile validate**
|
|
performance for all supported object types and compare **OURS** against the **Routinator baseline**
|
|
(`rpki` crate `=0.19.1` with `repository` feature).
|
|
|
|
## What it measures
|
|
|
|
Dataset:
|
|
|
|
- Fixtures: `rpki/tests/benchmark/selected_der_v2/`
|
|
- Objects: `cer`, `crl`, `manifest` (`.mft`), `roa`, `aspa` (`.asa`)
|
|
- Samples: 10 quantiles per type (`min/p01/p10/p25/p50/p75/p90/p95/p99/max`) → 50 files total
|
|
|
|
Metrics:
|
|
|
|
- **decode+validate**: `decode_der` (parse + profile validate) for each object file
|
|
- **landing** (OURS only): `PackFile::from_bytes_compute_sha256` + CBOR encode + `RocksDB put_raw`
|
|
- **compare**: ratio `ours_ns/op ÷ rout_ns/op` for decode+validate
|
|
|
|
## Default benchmark settings
|
|
|
|
Both OURS and Routinator baseline use the same run settings:
|
|
|
|
- warmup: `10` iterations
|
|
- rounds: `3`
|
|
- adaptive loop target: `min_round_ms=200` (with an internal max of `1_000_000` iters)
|
|
- strict DER: `true` (baseline)
|
|
- cert inspect: `false` (baseline)
|
|
|
|
You can override the settings via environment variables in the runner script:
|
|
|
|
- `BENCH_WARMUP_ITERS` (default `10`)
|
|
- `BENCH_ROUNDS` (default `3`)
|
|
- `BENCH_MIN_ROUND_MS` (default `200`)
|
|
|
|
## One-click run (OURS + Routinator compare)
|
|
|
|
From the `rpki/` crate directory:
|
|
|
|
```bash
|
|
./scripts/benchmark/run_stage2_selected_der_v2_release.sh
|
|
```
|
|
|
|
Outputs are written under:
|
|
|
|
- `rpki/target/bench/`
|
|
- OURS decode+validate: `stage2_selected_der_v2_decode_release_<TS>.{md,csv}`
|
|
- OURS landing: `stage2_selected_der_v2_landing_release_<TS>.{md,csv}`
|
|
- Routinator: `stage2_selected_der_v2_routinator_decode_release_<TS>.{md,csv}`
|
|
- Compare: `stage2_selected_der_v2_compare_ours_vs_routinator_decode_release_<TS>.{md,csv}`
|
|
- Summary: `stage2_selected_der_v2_compare_summary_<TS>.md`
|
|
|
|
### Why decode and landing are separated
|
|
|
|
The underlying benchmark can run in `BENCH_MODE=both`, but the **landing** part writes to RocksDB
|
|
and may trigger background work (e.g., compactions) that can **skew subsequent decode timings**.
|
|
For a fair OURS-vs-Routinator comparison, the runner script:
|
|
|
|
- runs `BENCH_MODE=decode_validate` for comparison, and
|
|
- runs `BENCH_MODE=landing` separately for landing-only numbers.
|
|
|
|
## Notes
|
|
|
|
- The Routinator baseline benchmark is implemented in-repo under:
|
|
- `rpki/benchmark/routinator_object_bench/`
|
|
- It pins `rpki = "=0.19.1"` in its `Cargo.toml`.
|
|
- This benchmark is implemented as an `#[ignore]` integration test:
|
|
- `rpki/tests/bench_stage2_decode_profile_selected_der_v2.rs`
|
|
- The runner script invokes it with `cargo test --release ... -- --ignored --nocapture`.
|