# RPKI Benchmarks (Stage2, selected_der_v2)

This directory contains a reproducible, one-click benchmark to measure **decode + profile validate**
performance for all supported object types and compare **OURS** against the **Routinator baseline**
(`rpki` crate `=0.19.1` with `repository` feature).

## What it measures

Dataset:

- Fixtures: `rpki/tests/benchmark/selected_der_v2/`
- Objects: `cer`, `crl`, `manifest` (`.mft`), `roa`, `aspa` (`.asa`)
- Samples: 10 quantiles per type (`min/p01/p10/p25/p50/p75/p90/p95/p99/max`) → 50 files total

Metrics:

- **decode+validate**: `decode_der` (parse + profile validate) for each object file
- **landing** (OURS only): `PackFile::from_bytes_compute_sha256` + CBOR encode + `RocksDB put_raw`
- **compare**: ratio `ours_ns/op ÷ rout_ns/op` for decode+validate

## Default benchmark settings

Both OURS and Routinator baseline use the same run settings:

- warmup: `10` iterations
- rounds: `3`
- adaptive loop target: `min_round_ms=200` (with an internal max of `1_000_000` iters)
- strict DER: `true` (baseline)
- cert inspect: `false` (baseline)

You can override the settings via environment variables in the runner script:

- `BENCH_WARMUP_ITERS` (default `10`)
- `BENCH_ROUNDS` (default `3`)
- `BENCH_MIN_ROUND_MS` (default `200`)

## One-click run (OURS + Routinator compare)

From the `rpki/` crate directory:

```bash
./scripts/benchmark/run_stage2_selected_der_v2_release.sh
```

Outputs are written under:

- `rpki/target/bench/`
  - OURS decode+validate: `stage2_selected_der_v2_decode_release_<TS>.{md,csv}`
  - OURS landing: `stage2_selected_der_v2_landing_release_<TS>.{md,csv}`
  - Routinator: `stage2_selected_der_v2_routinator_decode_release_<TS>.{md,csv}`
  - Compare: `stage2_selected_der_v2_compare_ours_vs_routinator_decode_release_<TS>.{md,csv}`
  - Summary: `stage2_selected_der_v2_compare_summary_<TS>.md`

### Why decode and landing are separated

The underlying benchmark can run in `BENCH_MODE=both`, but the **landing** part writes to RocksDB
and may trigger background work (e.g., compactions) that can **skew subsequent decode timings**.
For a fair OURS-vs-Routinator comparison, the runner script:

- runs `BENCH_MODE=decode_validate` for comparison, and
- runs `BENCH_MODE=landing` separately for landing-only numbers.

## Notes

- The Routinator baseline benchmark is implemented in-repo under:
  - `rpki/benchmark/routinator_object_bench/`
  - It pins `rpki = "=0.19.1"` in its `Cargo.toml`.
- This benchmark is implemented as an `#[ignore]` integration test:
  - `rpki/tests/bench_stage2_decode_profile_selected_der_v2.rs`
  - The runner script invokes it with `cargo test --release ... -- --ignored --nocapture`.