freeze initial panda-rpki staging
This commit is contained in:
commit
15c8de82ba
20
.dockerignore
Normal file
20
.dockerignore
Normal file
@ -0,0 +1,20 @@
|
||||
.git
|
||||
.git/**
|
||||
.gitignore
|
||||
target
|
||||
artifacts
|
||||
state
|
||||
runs
|
||||
logs
|
||||
tmp
|
||||
docker-out
|
||||
provenance/generated
|
||||
crates/panda-rpki-validator/tests
|
||||
tests
|
||||
specs
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
*.local
|
||||
*.tmp
|
||||
*.swp
|
||||
27
.github/workflows/ci.yml
vendored
Normal file
27
.github/workflows/ci.yml
vendored
Normal file
@ -0,0 +1,27 @@
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
rust:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
toolchain: 1.92.0
|
||||
components: rustfmt, clippy
|
||||
- run: cargo fmt --all --check
|
||||
- run: cargo build --locked -p panda-rpki-validator
|
||||
- run: cargo test --locked -p panda-rpki-validator
|
||||
- run: cargo clippy --locked -p panda-rpki-validator --all-targets -- -D warnings
|
||||
|
||||
docker-runtime:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: docker buildx build --platform linux/amd64 --load -t panda-rpki-validator:ci -f docker/validator-runtime.Dockerfile .
|
||||
- run: ./scripts/docker/verify_image.sh --image panda-rpki-validator:ci
|
||||
- run: docker run --rm panda-rpki-validator:ci run --help
|
||||
16
.gitignore
vendored
Normal file
16
.gitignore
vendored
Normal file
@ -0,0 +1,16 @@
|
||||
/target/
|
||||
/artifacts/
|
||||
/state/
|
||||
/runs/
|
||||
/logs/
|
||||
/tmp/
|
||||
/docker-out/
|
||||
/provenance/generated/
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
*.local
|
||||
*.tmp
|
||||
*.swp
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
10
CONTRIBUTING.md
Normal file
10
CONTRIBUTING.md
Normal file
@ -0,0 +1,10 @@
|
||||
# Contributing to panda-rpki
|
||||
|
||||
This repository is private staging for the `panda-rpki-validator` extraction.
|
||||
External contributions are not enabled while ownership, license and contribution
|
||||
governance are pending.
|
||||
|
||||
During M3–M6, changes must preserve `docs/output-abi.md`, update the relevant
|
||||
allowlist/provenance entry, and pass the locked Rust build and Docker smoke
|
||||
checks. Do not add live RIR data, private state, credentials, internal hosts,
|
||||
or copied history.
|
||||
2633
Cargo.lock
generated
Normal file
2633
Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
4
Cargo.toml
Normal file
4
Cargo.toml
Normal file
@ -0,0 +1,4 @@
|
||||
[workspace]
|
||||
members = ["crates/panda-rpki-validator"]
|
||||
default-members = ["crates/panda-rpki-validator"]
|
||||
resolver = "3"
|
||||
100
README.md
Normal file
100
README.md
Normal file
@ -0,0 +1,100 @@
|
||||
# panda-rpki
|
||||
|
||||
`panda-rpki` is the staging workspace for the independently distributable RPKI
|
||||
synchronization and validation component. The first component is
|
||||
`panda-rpki-validator` (Cargo package and CLI); its Rust import path is
|
||||
`panda_rpki_validator`.
|
||||
|
||||
This directory is currently a **private staging repository**. It has no copied
|
||||
history from the existing private `rpki` repository. The normal synchronization
|
||||
and validation path has now been extracted, together with its daemon and
|
||||
Docker lifecycle wrapper. The first M5 canonical snapshot/delta baseline and
|
||||
amd64/arm64 staging image checks are complete; remaining profile/performance
|
||||
gates still block any public release. Do not use this staging build as a
|
||||
production validator.
|
||||
|
||||
## Current milestone
|
||||
|
||||
M4/M5 contain the first functional extraction from source commit
|
||||
`74cbebbd3334ac0063761c1a97a88ee000cc2a57`: normal RRDP/rsync synchronization,
|
||||
RPKI validation, RocksDB state, CIR/CCR/report/CSV outputs, the run daemon, and
|
||||
the snapshot/delta lifecycle wrapper. The `verification-only` mode is
|
||||
intentionally deferred. The project license is intentionally **TBD**; no
|
||||
external contributions or public release are accepted until ownership and
|
||||
licensing are approved.
|
||||
|
||||
## Build and test the staging component
|
||||
|
||||
```bash
|
||||
cargo build --locked -p panda-rpki-validator
|
||||
cargo test --locked -p panda-rpki-validator
|
||||
cargo fmt --all --check
|
||||
cargo clippy --locked -p panda-rpki-validator --all-targets -- -D warnings
|
||||
./scripts/docker/build_image.sh --arch amd64 --allow-dirty --no-save
|
||||
```
|
||||
|
||||
The extracted test suite currently passes `735` tests with `1` ignored test in
|
||||
the normal profile. The APNIC offline snapshot/delta profile has also passed
|
||||
the M5 canonical comparator and a five-run pinned release smoke baseline; the
|
||||
full cache/fallback/failure/profile matrix is still pending.
|
||||
|
||||
For a deterministic single-RIR run against the checked-in test repository,
|
||||
build the binaries first and invoke the lifecycle wrapper with
|
||||
`RPKI_EXTRA_ARGS=--disable-rrdp --rsync-local-dir ...`; the wrapper writes the
|
||||
normal run ABI under `RUN_ROOT/runs/run_0001/`.
|
||||
|
||||
## Docker smoke
|
||||
|
||||
```bash
|
||||
./scripts/docker/build_image.sh --arch amd64 --allow-dirty
|
||||
./scripts/docker/verify_image.sh \
|
||||
--image panda-rpki-validator:0.1.0-dirty-amd64
|
||||
```
|
||||
|
||||
The runtime image contains both `panda-rpki-validator` and
|
||||
`panda-rpki-validator-daemon`, the complete normal-run wrapper, and the
|
||||
redistributable TAL/TA fixtures. It uses one persistent data root mounted at
|
||||
`/var/lib/panda-rpki-validator`; `run`, `run-validator`, and `daemon` are
|
||||
entrypoint subcommands. The wrapper preserves the output contract documented
|
||||
in
|
||||
[`docs/output-abi.md`](docs/output-abi.md).
|
||||
|
||||
For native or container A/B output checks, run the ABI verifier and canonical
|
||||
comparator against two retained run directories:
|
||||
|
||||
```bash
|
||||
tests/compat/verify_run_abi.sh /path/to/run_0001
|
||||
tests/compat/compare_runs.py \
|
||||
/path/to/original/runs/run_0001 \
|
||||
/path/to/panda/runs/run_0001
|
||||
```
|
||||
|
||||
The comparator has matched the original runtime for the APNIC offline
|
||||
snapshot/delta baseline. Five serial release samples on that small single-RIR
|
||||
profile stay within the current wall-time gate under a pinned CPU. A live APNIC
|
||||
run also produced an identical decoded CCR state (MFT/VRP/VAP/TA/RK); only the
|
||||
time-bearing `producedAt` byte differed in the raw DER. A concurrent all-RIR
|
||||
(`all5`) run is retained as network/fallback evidence, but its source was not an
|
||||
atomic snapshot (the original image timed out on the ARIN RRDP notification),
|
||||
so its different CCR state is not a code-parity verdict. See the detailed
|
||||
[`M5 live RIR comparison report`](../specs/develop/20260901/m5_live_rir_image_artifact_comparison_milestone_report.md).
|
||||
The follow-up remote-231 serial all5 run (old image first, then this image;
|
||||
one snapshot plus three warm deltas per image with cache/prefetch/parallel
|
||||
flags) also completed 4/4 runs on each side and retained full artifacts. Its
|
||||
live CCR/CIR and timing differences are documented separately and are not a
|
||||
frozen-input parity result: [`remote-231 all5 cache/prefetch report`](../specs/develop/20260901_2/m5_live_all5_cache_prefetch_serial_4run_milestone_report.md).
|
||||
The new image also completed a separate remote-231 all5 long sequence of one
|
||||
snapshot plus ten deltas; the timing and cache counters are retained as live
|
||||
health evidence, not as a frozen-input parity gate: [`one snapshot + ten delta
|
||||
timing report`](../specs/develop/20260901_2/m5_new_image_all5_snapshot_10delta_timing_report.md).
|
||||
Cache, RRDP/fallback, constraints, replay and failure-path profiles remain in
|
||||
M5.
|
||||
|
||||
## Repository status
|
||||
|
||||
- `verification-only`: deferred to a separate backlog item.
|
||||
- License and copyright owner: TBD; do not add a speculative `LICENSE` file.
|
||||
- Staging Git remote: `https://git.nasp.fit/yuyr/panda-rpki.git` (private
|
||||
staging target selected); public visibility, registry prefix, signing identity,
|
||||
and release tags remain unselected until release governance is complete.
|
||||
- Source provenance baseline: `provenance/source-baseline.toml`.
|
||||
12
SECURITY.md
Normal file
12
SECURITY.md
Normal file
@ -0,0 +1,12 @@
|
||||
# Security policy
|
||||
|
||||
This project is not public yet and does not have a public vulnerability intake
|
||||
address. Do not publish vulnerability details in an issue or commit. Report
|
||||
urgent findings to the project owner through the private project channel and
|
||||
include the affected commit, image tag/digest, reproduction steps, and whether
|
||||
state or output data is exposed.
|
||||
|
||||
The public contact, response SLA, supported release branches, and disclosure
|
||||
process will be added after ownership and license approval. Container images
|
||||
must never contain credentials, production state, private registry settings,
|
||||
or internal hostnames.
|
||||
49
crates/panda-rpki-validator/Cargo.toml
Normal file
49
crates/panda-rpki-validator/Cargo.toml
Normal file
@ -0,0 +1,49 @@
|
||||
[package]
|
||||
name = "panda-rpki-validator"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
rust-version = "1.92"
|
||||
description = "Panda RPKI synchronization and validation runtime"
|
||||
publish = false
|
||||
|
||||
[lib]
|
||||
name = "panda_rpki_validator"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "panda-rpki-validator"
|
||||
path = "src/main.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "panda-rpki-validator-daemon"
|
||||
path = "src/bin/panda-rpki-validator-daemon.rs"
|
||||
|
||||
[features]
|
||||
default = ["full"]
|
||||
full = []
|
||||
profile = ["dep:pprof", "dep:flate2"]
|
||||
|
||||
[dependencies]
|
||||
asn1-rs = "0.7.1"
|
||||
der-parser = { version = "10.0.0", features = ["serialize"] }
|
||||
hex = "0.4.3"
|
||||
base64 = "0.22.1"
|
||||
sha2 = "0.10.8"
|
||||
thiserror = "2.0.18"
|
||||
time = "0.3.45"
|
||||
ring = "0.17.14"
|
||||
x509-parser = { version = "0.18.0", features = ["verify"] }
|
||||
url = "2.5.8"
|
||||
serde = { version = "1.0.218", features = ["derive"] }
|
||||
serde_json = { version = "1.0.140", features = ["raw_value"] }
|
||||
toml = "0.8.20"
|
||||
rocksdb = { version = "0.22.0", default-features = false, features = ["lz4"] }
|
||||
serde_cbor = "0.11.2"
|
||||
memmap2 = "0.9.10"
|
||||
roxmltree = "0.20.0"
|
||||
quick-xml = "0.37.2"
|
||||
uuid = { version = "1.7.0", features = ["v4"] }
|
||||
reqwest = { version = "0.12.12", default-features = false, features = ["blocking", "rustls-tls", "gzip", "brotli", "deflate"] }
|
||||
pprof = { version = "0.14.1", optional = true, features = ["flamegraph", "prost-codec"] }
|
||||
flate2 = { version = "1.0.35", optional = true }
|
||||
tempfile = "3.16.0"
|
||||
1
crates/panda-rpki-validator/src/analysis/mod.rs
Normal file
1
crates/panda-rpki-validator/src/analysis/mod.rs
Normal file
@ -0,0 +1 @@
|
||||
pub mod timing;
|
||||
353
crates/panda-rpki-validator/src/analysis/timing.rs
Normal file
353
crates/panda-rpki-validator/src/analysis/timing.rs
Normal file
@ -0,0 +1,353 @@
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct TimingHandle {
|
||||
inner: Arc<Mutex<TimingCollector>>,
|
||||
}
|
||||
|
||||
impl TimingHandle {
|
||||
pub fn new(meta: TimingMeta) -> Self {
|
||||
Self {
|
||||
inner: Arc::new(Mutex::new(TimingCollector::new(meta))),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn span_phase(&self, phase: &'static str) -> TimingSpanGuard<'_> {
|
||||
TimingSpanGuard {
|
||||
handle: self.clone(),
|
||||
kind: TimingSpanKind::Phase(phase),
|
||||
start: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn span_rrdp_repo<'a>(&self, repo_uri: &'a str) -> TimingSpanGuard<'a> {
|
||||
TimingSpanGuard {
|
||||
handle: self.clone(),
|
||||
kind: TimingSpanKind::RrdpRepo(repo_uri),
|
||||
start: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn span_rrdp_repo_step<'a>(
|
||||
&self,
|
||||
repo_uri: &'a str,
|
||||
step: &'static str,
|
||||
) -> TimingSpanGuard<'a> {
|
||||
TimingSpanGuard {
|
||||
handle: self.clone(),
|
||||
kind: TimingSpanKind::RrdpRepoStep { repo_uri, step },
|
||||
start: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn span_publication_point<'a>(&self, manifest_rsync_uri: &'a str) -> TimingSpanGuard<'a> {
|
||||
TimingSpanGuard {
|
||||
handle: self.clone(),
|
||||
kind: TimingSpanKind::PublicationPoint(manifest_rsync_uri),
|
||||
start: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn set_meta(&self, update: TimingMetaUpdate<'_>) {
|
||||
let mut g = self.inner.lock().expect("timing lock");
|
||||
if let Some(v) = update.tal_url {
|
||||
g.meta.tal_url = Some(v.to_string());
|
||||
}
|
||||
if let Some(v) = update.db_path {
|
||||
g.meta.db_path = Some(v.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
pub fn record_count(&self, key: &'static str, inc: u64) {
|
||||
let mut g = self.inner.lock().expect("timing lock");
|
||||
g.counts
|
||||
.entry(key)
|
||||
.and_modify(|v| *v = v.saturating_add(inc))
|
||||
.or_insert(inc);
|
||||
}
|
||||
|
||||
pub fn counts_snapshot(&self) -> HashMap<String, u64> {
|
||||
let g = self.inner.lock().expect("timing lock");
|
||||
g.counts
|
||||
.iter()
|
||||
.map(|(key, value)| ((*key).to_string(), *value))
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn report_snapshot(&self, top_n: usize) -> TimingReportV1 {
|
||||
let g = self.inner.lock().expect("timing lock");
|
||||
g.to_report(top_n)
|
||||
}
|
||||
|
||||
/// Record a phase duration directly in nanoseconds.
|
||||
///
|
||||
/// This is useful when aggregating sub-phase timings locally (to reduce lock contention)
|
||||
/// and then emitting a single record per publication point.
|
||||
pub fn record_phase_nanos(&self, phase: &'static str, nanos: u64) {
|
||||
let mut g = self.inner.lock().expect("timing lock");
|
||||
g.phases.record(phase, nanos);
|
||||
}
|
||||
|
||||
pub fn record_publication_point_nanos(&self, manifest_rsync_uri: &str, nanos: u64) {
|
||||
let mut g = self.inner.lock().expect("timing lock");
|
||||
g.publication_points.record(manifest_rsync_uri, nanos);
|
||||
}
|
||||
|
||||
pub fn record_publication_point_step_nanos(
|
||||
&self,
|
||||
manifest_rsync_uri: &str,
|
||||
step: &'static str,
|
||||
nanos: u64,
|
||||
) {
|
||||
let mut g = self.inner.lock().expect("timing lock");
|
||||
g.publication_point_steps
|
||||
.record(&format!("{manifest_rsync_uri}::{step}"), nanos);
|
||||
}
|
||||
|
||||
pub fn write_json(&self, path: &Path, top_n: usize) -> Result<(), String> {
|
||||
let report = {
|
||||
let g = self.inner.lock().expect("timing lock");
|
||||
g.to_report(top_n)
|
||||
};
|
||||
|
||||
let f = std::fs::File::create(path)
|
||||
.map_err(|e| format!("create timing json failed: {}: {e}", path.display()))?;
|
||||
serde_json::to_writer_pretty(f, &report)
|
||||
.map_err(|e| format!("write timing json failed: {e}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn record_duration(&self, kind: TimingSpanKind<'_>, duration: Duration) {
|
||||
let nanos_u64 = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
|
||||
let mut g = self.inner.lock().expect("timing lock");
|
||||
match kind {
|
||||
TimingSpanKind::Phase(name) => g.phases.record(name, nanos_u64),
|
||||
TimingSpanKind::RrdpRepo(uri) => g.rrdp_repos.record(uri, nanos_u64),
|
||||
TimingSpanKind::RrdpRepoStep { repo_uri, step } => g
|
||||
.rrdp_repo_steps
|
||||
.record(&format!("{repo_uri}::{step}"), nanos_u64),
|
||||
TimingSpanKind::PublicationPoint(uri) => g.publication_points.record(uri, nanos_u64),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TimingMeta {
|
||||
pub recorded_at_utc_rfc3339: String,
|
||||
pub validation_time_utc_rfc3339: String,
|
||||
pub tal_url: Option<String>,
|
||||
pub db_path: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct TimingMetaUpdate<'a> {
|
||||
pub tal_url: Option<&'a str>,
|
||||
pub db_path: Option<&'a str>,
|
||||
}
|
||||
|
||||
pub struct TimingSpanGuard<'a> {
|
||||
handle: TimingHandle,
|
||||
kind: TimingSpanKind<'a>,
|
||||
start: Instant,
|
||||
}
|
||||
|
||||
impl Drop for TimingSpanGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
self.handle
|
||||
.record_duration(self.kind.clone(), self.start.elapsed());
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
enum TimingSpanKind<'a> {
|
||||
Phase(&'static str),
|
||||
RrdpRepo(&'a str),
|
||||
RrdpRepoStep {
|
||||
repo_uri: &'a str,
|
||||
step: &'static str,
|
||||
},
|
||||
PublicationPoint(&'a str),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct DurationStats {
|
||||
pub count: u64,
|
||||
pub total_nanos: u64,
|
||||
}
|
||||
|
||||
impl DurationStats {
|
||||
fn record(&mut self, nanos: u64) {
|
||||
self.count = self.count.saturating_add(1);
|
||||
self.total_nanos = self.total_nanos.saturating_add(nanos);
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default)]
|
||||
struct DurationStatsMap {
|
||||
map: HashMap<String, DurationStats>,
|
||||
}
|
||||
|
||||
impl DurationStatsMap {
|
||||
fn record(&mut self, key: &str, nanos: u64) {
|
||||
self.map.entry(key.to_string()).or_default().record(nanos);
|
||||
}
|
||||
|
||||
fn top(&self, n: usize) -> Vec<TopDurationEntry> {
|
||||
let mut v = self
|
||||
.map
|
||||
.iter()
|
||||
.map(|(k, s)| TopDurationEntry {
|
||||
key: k.clone(),
|
||||
count: s.count,
|
||||
total_nanos: s.total_nanos,
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
v.sort_by(|a, b| b.total_nanos.cmp(&a.total_nanos));
|
||||
v.truncate(n);
|
||||
v
|
||||
}
|
||||
}
|
||||
|
||||
struct TimingCollector {
|
||||
meta: TimingMeta,
|
||||
counts: HashMap<&'static str, u64>,
|
||||
phases: DurationStatsMap,
|
||||
rrdp_repos: DurationStatsMap,
|
||||
rrdp_repo_steps: DurationStatsMap,
|
||||
publication_points: DurationStatsMap,
|
||||
publication_point_steps: DurationStatsMap,
|
||||
}
|
||||
|
||||
impl TimingCollector {
|
||||
fn new(meta: TimingMeta) -> Self {
|
||||
Self {
|
||||
meta,
|
||||
counts: HashMap::new(),
|
||||
phases: DurationStatsMap::default(),
|
||||
rrdp_repos: DurationStatsMap::default(),
|
||||
rrdp_repo_steps: DurationStatsMap::default(),
|
||||
publication_points: DurationStatsMap::default(),
|
||||
publication_point_steps: DurationStatsMap::default(),
|
||||
}
|
||||
}
|
||||
|
||||
fn to_report(&self, top_n: usize) -> TimingReportV1 {
|
||||
TimingReportV1 {
|
||||
format_version: 1,
|
||||
meta: self.meta.clone(),
|
||||
counts: self
|
||||
.counts
|
||||
.iter()
|
||||
.map(|(k, v)| ((*k).to_string(), *v))
|
||||
.collect(),
|
||||
phases: self
|
||||
.phases
|
||||
.map
|
||||
.iter()
|
||||
.map(|(k, s)| (k.clone(), s.clone()))
|
||||
.collect(),
|
||||
top_rrdp_repos: self.rrdp_repos.top(top_n),
|
||||
top_rrdp_repo_steps: self.rrdp_repo_steps.top(top_n),
|
||||
top_publication_points: self.publication_points.top(top_n),
|
||||
top_publication_point_steps: self.publication_point_steps.top(top_n),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct TimingReportV1 {
|
||||
pub format_version: u64,
|
||||
pub meta: TimingMeta,
|
||||
pub counts: HashMap<String, u64>,
|
||||
pub phases: HashMap<String, DurationStats>,
|
||||
pub top_rrdp_repos: Vec<TopDurationEntry>,
|
||||
pub top_rrdp_repo_steps: Vec<TopDurationEntry>,
|
||||
pub top_publication_points: Vec<TopDurationEntry>,
|
||||
pub top_publication_point_steps: Vec<TopDurationEntry>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TopDurationEntry {
|
||||
pub key: String,
|
||||
pub count: u64,
|
||||
pub total_nanos: u64,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn timing_handle_writes_json_with_phases_and_tops() {
|
||||
let meta = TimingMeta {
|
||||
recorded_at_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
|
||||
validation_time_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
|
||||
tal_url: Some("https://example.test/x.tal".to_string()),
|
||||
db_path: Some("db".to_string()),
|
||||
};
|
||||
let h = TimingHandle::new(meta);
|
||||
|
||||
{
|
||||
let _p = h.span_phase("tal_bootstrap");
|
||||
}
|
||||
{
|
||||
let _r = h.span_rrdp_repo("https://rrdp.example.test/notification.xml");
|
||||
}
|
||||
{
|
||||
let _s = h.span_rrdp_repo_step(
|
||||
"https://rrdp.example.test/notification.xml",
|
||||
"fetch_notification",
|
||||
);
|
||||
}
|
||||
{
|
||||
let _pp = h.span_publication_point("rsync://example.test/repo/manifest.mft");
|
||||
}
|
||||
h.record_count("vrps", 42);
|
||||
h.record_publication_point_nanos("rsync://example.test/repo/manifest.mft", 1_000_000);
|
||||
h.record_publication_point_step_nanos(
|
||||
"rsync://example.test/repo/manifest.mft",
|
||||
"fresh_snapshot_prepare",
|
||||
1_000_000,
|
||||
);
|
||||
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join("timing.json");
|
||||
h.write_json(&path, 10).expect("write_json");
|
||||
|
||||
let rep: TimingReportV1 =
|
||||
serde_json::from_slice(&std::fs::read(&path).expect("read timing.json"))
|
||||
.expect("parse timing.json");
|
||||
assert_eq!(rep.format_version, 1);
|
||||
assert!(rep.phases.contains_key("tal_bootstrap"));
|
||||
assert_eq!(rep.counts.get("vrps").copied(), Some(42));
|
||||
assert!(
|
||||
rep.top_rrdp_repos
|
||||
.iter()
|
||||
.any(|e| e.key.contains("rrdp.example.test")),
|
||||
"expected repo in top list"
|
||||
);
|
||||
assert!(
|
||||
rep.top_rrdp_repo_steps
|
||||
.iter()
|
||||
.any(|e| e.key.contains("fetch_notification")),
|
||||
"expected repo step in top list"
|
||||
);
|
||||
assert!(
|
||||
rep.top_publication_points
|
||||
.iter()
|
||||
.any(|e| e.key.contains("manifest.mft")),
|
||||
"expected PP in top list"
|
||||
);
|
||||
assert!(
|
||||
rep.top_publication_point_steps
|
||||
.iter()
|
||||
.any(|e| e.key.contains("fresh_snapshot_prepare")),
|
||||
"expected PP step in top list"
|
||||
);
|
||||
}
|
||||
}
|
||||
342
crates/panda-rpki-validator/src/audit.rs
Normal file
342
crates/panda-rpki-validator/src/audit.rs
Normal file
@ -0,0 +1,342 @@
|
||||
use serde::Serialize;
|
||||
use sha2::Digest;
|
||||
|
||||
use crate::policy::Policy;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AuditObjectKind {
|
||||
Manifest,
|
||||
Crl,
|
||||
Certificate,
|
||||
RouterCertificate,
|
||||
Roa,
|
||||
Aspa,
|
||||
Other,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AuditObjectResult {
|
||||
Ok,
|
||||
Skipped,
|
||||
Error,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct ObjectAuditEntry {
|
||||
pub rsync_uri: String,
|
||||
pub sha256_hex: String,
|
||||
pub kind: AuditObjectKind,
|
||||
pub result: AuditObjectResult,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub detail: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditWarning {
|
||||
pub message: String,
|
||||
pub category: String,
|
||||
pub rfc_refs: Vec<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub context: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct QueryAuditManifest {
|
||||
pub schema_version: u32,
|
||||
pub status: String,
|
||||
pub events_path: String,
|
||||
pub events_count: u64,
|
||||
pub events_sha256: String,
|
||||
pub writer_version: u32,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ValidationEventCounts {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub objects: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub warnings: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub vrps: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub aspas: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ValidationEvent {
|
||||
pub schema_version: u32,
|
||||
pub seq: u64,
|
||||
pub event_type: String,
|
||||
pub validation_time: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub pp_node_id: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub pp_manifest_uri: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub pp_rsync_base_uri: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_sync_phase: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_terminal_state: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub object_uri: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub sha256: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub object_type: Option<AuditObjectKind>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub result: Option<AuditObjectResult>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub reason: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub counts: Option<ValidationEventCounts>,
|
||||
}
|
||||
|
||||
impl From<&crate::report::Warning> for AuditWarning {
|
||||
fn from(w: &crate::report::Warning) -> Self {
|
||||
Self {
|
||||
message: w.message.clone(),
|
||||
category: w.category.as_str().to_string(),
|
||||
rfc_refs: w.rfc_refs.iter().map(|r| r.0.to_string()).collect(),
|
||||
context: w.context.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct PublicationPointAudit {
|
||||
/// Monotonic node ID assigned by the traversal engine.
|
||||
///
|
||||
/// Present when running via the Stage2 tree engine; may be absent in ad-hoc runs.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub node_id: Option<u64>,
|
||||
/// Parent node ID in the traversal tree.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub parent_node_id: Option<u64>,
|
||||
/// Provenance metadata for non-root nodes (how this CA instance was discovered).
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub discovered_from: Option<DiscoveredFrom>,
|
||||
|
||||
pub rsync_base_uri: String,
|
||||
pub manifest_rsync_uri: String,
|
||||
pub publication_point_rsync_uri: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub rrdp_notification_uri: Option<String>,
|
||||
|
||||
pub source: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_sync_source: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_sync_phase: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_sync_duration_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub repo_sync_error: Option<String>,
|
||||
pub repo_terminal_state: String,
|
||||
pub this_update_rfc3339_utc: String,
|
||||
pub next_update_rfc3339_utc: String,
|
||||
pub verified_at_rfc3339_utc: String,
|
||||
|
||||
pub warnings: Vec<AuditWarning>,
|
||||
pub objects: Vec<ObjectAuditEntry>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct DiscoveredFrom {
|
||||
pub parent_manifest_rsync_uri: String,
|
||||
pub child_ca_certificate_rsync_uri: String,
|
||||
pub child_ca_certificate_sha256_hex: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct TreeSummary {
|
||||
pub instances_processed: usize,
|
||||
pub instances_failed: usize,
|
||||
pub warnings: Vec<AuditWarning>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditRunMeta {
|
||||
pub validation_time_rfc3339_utc: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AuditDownloadKind {
|
||||
RrdpNotification,
|
||||
RrdpSnapshot,
|
||||
RrdpDelta,
|
||||
Rsync,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditDownloadObjectsStat {
|
||||
pub objects_count: u64,
|
||||
pub objects_bytes_total: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditDownloadEvent {
|
||||
pub kind: AuditDownloadKind,
|
||||
pub uri: String,
|
||||
pub started_at_rfc3339_utc: String,
|
||||
pub finished_at_rfc3339_utc: String,
|
||||
pub duration_ms: u64,
|
||||
pub success: bool,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub error: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub bytes: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub objects: Option<AuditDownloadObjectsStat>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditDownloadKindStats {
|
||||
pub ok_total: u64,
|
||||
pub fail_total: u64,
|
||||
pub duration_ms_total: u64,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub bytes_total: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub objects_count_total: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub objects_bytes_total: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditDownloadStats {
|
||||
pub events_total: u64,
|
||||
/// Statistics keyed by serialized `AuditDownloadKind` string (e.g. "rrdp_snapshot").
|
||||
pub by_kind: std::collections::BTreeMap<String, AuditDownloadKindStats>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn publication_point_audit_serializes_object_audit_only() {
|
||||
let audit = PublicationPointAudit {
|
||||
objects: vec![ObjectAuditEntry {
|
||||
rsync_uri: "rsync://example.test/repo/fresh.roa".to_string(),
|
||||
sha256_hex: "11".repeat(32),
|
||||
kind: AuditObjectKind::Roa,
|
||||
result: AuditObjectResult::Ok,
|
||||
detail: None,
|
||||
}],
|
||||
..PublicationPointAudit::default()
|
||||
};
|
||||
|
||||
let value = serde_json::to_value(&audit).expect("serialize audit");
|
||||
assert!(value.get("objects").is_some());
|
||||
assert!(value.get("cir_fresh_objects").is_none());
|
||||
assert!(value.get("cir_cached_objects").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn warning_audit_keeps_the_warning_category() {
|
||||
let warning = crate::report::Warning::new("BER-compatible CMS accepted")
|
||||
.with_category(crate::report::WarningCategory::BerCompatibleCmsEncoding);
|
||||
|
||||
let audit_warning = AuditWarning::from(&warning);
|
||||
assert_eq!(audit_warning.category, "ber_compatible_cms_encoding");
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditRepoSyncStateStat {
|
||||
pub count: u64,
|
||||
pub duration_ms_total: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditRepoSyncStats {
|
||||
pub publication_points_total: u64,
|
||||
pub by_phase: std::collections::BTreeMap<String, AuditRepoSyncStateStat>,
|
||||
pub by_terminal_state: std::collections::BTreeMap<String, AuditRepoSyncStateStat>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditReportV1 {
|
||||
pub format_version: u32,
|
||||
pub meta: AuditRunMeta,
|
||||
pub policy: Policy,
|
||||
pub tree: TreeSummary,
|
||||
pub publication_points: Vec<PublicationPointAudit>,
|
||||
|
||||
pub vrps: Vec<VrpOutput>,
|
||||
pub aspas: Vec<AspaOutput>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct AuditReportV2 {
|
||||
pub format_version: u32,
|
||||
pub meta: AuditRunMeta,
|
||||
pub policy: Policy,
|
||||
pub tree: TreeSummary,
|
||||
pub publication_points: Vec<PublicationPointAudit>,
|
||||
|
||||
pub vrps: Vec<VrpOutput>,
|
||||
pub aspas: Vec<AspaOutput>,
|
||||
|
||||
pub downloads: Vec<AuditDownloadEvent>,
|
||||
pub download_stats: AuditDownloadStats,
|
||||
pub repo_sync_stats: AuditRepoSyncStats,
|
||||
#[serde(rename = "queryAudit", skip_serializing_if = "Option::is_none")]
|
||||
pub query_audit: Option<QueryAuditManifest>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct VrpOutput {
|
||||
pub asn: u32,
|
||||
pub prefix: String,
|
||||
pub max_length: u16,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct AspaOutput {
|
||||
pub customer_as_id: u32,
|
||||
pub provider_as_ids: Vec<u32>,
|
||||
}
|
||||
|
||||
pub fn sha256_hex_from_32(bytes: &[u8; 32]) -> String {
|
||||
hex::encode(bytes)
|
||||
}
|
||||
|
||||
pub fn sha256_hex(bytes: &[u8]) -> String {
|
||||
let digest = sha2::Sha256::digest(bytes);
|
||||
hex::encode(digest)
|
||||
}
|
||||
|
||||
pub fn format_roa_ip_prefix(p: &crate::data_model::roa::IpPrefix) -> String {
|
||||
let addr = p.addr_bytes();
|
||||
match p.afi {
|
||||
crate::data_model::roa::RoaAfi::Ipv4 => {
|
||||
format!(
|
||||
"{}.{}.{}.{}{}",
|
||||
addr[0],
|
||||
addr[1],
|
||||
addr[2],
|
||||
addr[3],
|
||||
format!("/{}", p.prefix_len)
|
||||
)
|
||||
}
|
||||
crate::data_model::roa::RoaAfi::Ipv6 => {
|
||||
let mut parts = Vec::with_capacity(8);
|
||||
for i in 0..8 {
|
||||
let hi = addr[i * 2] as u16;
|
||||
let lo = addr[i * 2 + 1] as u16;
|
||||
parts.push(format!("{:x}", (hi << 8) | lo));
|
||||
}
|
||||
format!("{}{}", parts.join(":"), format!("/{}", p.prefix_len))
|
||||
}
|
||||
}
|
||||
}
|
||||
170
crates/panda-rpki-validator/src/audit_downloads.rs
Normal file
170
crates/panda-rpki-validator/src/audit_downloads.rs
Normal file
@ -0,0 +1,170 @@
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use crate::audit::{
|
||||
AuditDownloadEvent, AuditDownloadKind, AuditDownloadKindStats, AuditDownloadObjectsStat,
|
||||
AuditDownloadStats,
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct DownloadLogHandle {
|
||||
inner: Arc<Mutex<Vec<AuditDownloadEvent>>>,
|
||||
}
|
||||
|
||||
impl DownloadLogHandle {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
pub fn record_event(&self, event: AuditDownloadEvent) {
|
||||
self.inner.lock().expect("download log lock").push(event);
|
||||
}
|
||||
|
||||
pub fn snapshot_events(&self) -> Vec<AuditDownloadEvent> {
|
||||
self.inner.lock().expect("download log lock").clone()
|
||||
}
|
||||
|
||||
pub fn stats_from_events(events: &[AuditDownloadEvent]) -> AuditDownloadStats {
|
||||
let mut out = AuditDownloadStats {
|
||||
events_total: events.len() as u64,
|
||||
by_kind: BTreeMap::new(),
|
||||
};
|
||||
for e in events {
|
||||
let kind_key = match e.kind {
|
||||
AuditDownloadKind::RrdpNotification => "rrdp_notification",
|
||||
AuditDownloadKind::RrdpSnapshot => "rrdp_snapshot",
|
||||
AuditDownloadKind::RrdpDelta => "rrdp_delta",
|
||||
AuditDownloadKind::Rsync => "rsync",
|
||||
}
|
||||
.to_string();
|
||||
|
||||
let st = out
|
||||
.by_kind
|
||||
.entry(kind_key)
|
||||
.or_insert_with(|| AuditDownloadKindStats {
|
||||
ok_total: 0,
|
||||
fail_total: 0,
|
||||
duration_ms_total: 0,
|
||||
bytes_total: None,
|
||||
objects_count_total: None,
|
||||
objects_bytes_total: None,
|
||||
});
|
||||
if e.success {
|
||||
st.ok_total = st.ok_total.saturating_add(1);
|
||||
} else {
|
||||
st.fail_total = st.fail_total.saturating_add(1);
|
||||
}
|
||||
st.duration_ms_total = st.duration_ms_total.saturating_add(e.duration_ms);
|
||||
if let Some(b) = e.bytes {
|
||||
st.bytes_total = Some(st.bytes_total.unwrap_or(0).saturating_add(b));
|
||||
}
|
||||
if let Some(objects) = &e.objects {
|
||||
st.objects_count_total = Some(
|
||||
st.objects_count_total
|
||||
.unwrap_or(0)
|
||||
.saturating_add(objects.objects_count),
|
||||
);
|
||||
st.objects_bytes_total = Some(
|
||||
st.objects_bytes_total
|
||||
.unwrap_or(0)
|
||||
.saturating_add(objects.objects_bytes_total),
|
||||
);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
pub fn stats(&self) -> AuditDownloadStats {
|
||||
let events = self.snapshot_events();
|
||||
Self::stats_from_events(&events)
|
||||
}
|
||||
|
||||
pub fn span_download<'a>(
|
||||
&'a self,
|
||||
kind: AuditDownloadKind,
|
||||
uri: &'a str,
|
||||
) -> DownloadSpanGuard<'a> {
|
||||
DownloadSpanGuard {
|
||||
handle: self,
|
||||
kind,
|
||||
uri,
|
||||
start_instant: Instant::now(),
|
||||
started_at: time::OffsetDateTime::now_utc(),
|
||||
bytes: None,
|
||||
objects: None,
|
||||
error: None,
|
||||
success: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct DownloadSpanGuard<'a> {
|
||||
handle: &'a DownloadLogHandle,
|
||||
kind: AuditDownloadKind,
|
||||
uri: &'a str,
|
||||
start_instant: Instant,
|
||||
started_at: time::OffsetDateTime,
|
||||
bytes: Option<u64>,
|
||||
objects: Option<AuditDownloadObjectsStat>,
|
||||
error: Option<String>,
|
||||
success: Option<bool>,
|
||||
}
|
||||
|
||||
impl DownloadSpanGuard<'_> {
|
||||
pub fn set_bytes(&mut self, bytes: u64) {
|
||||
self.bytes = Some(bytes);
|
||||
}
|
||||
|
||||
pub fn set_objects(&mut self, objects_count: u64, objects_bytes_total: u64) {
|
||||
self.objects = Some(AuditDownloadObjectsStat {
|
||||
objects_count,
|
||||
objects_bytes_total,
|
||||
});
|
||||
}
|
||||
|
||||
pub fn set_ok(&mut self) {
|
||||
self.success = Some(true);
|
||||
}
|
||||
|
||||
pub fn set_err(&mut self, msg: impl Into<String>) {
|
||||
self.success = Some(false);
|
||||
self.error = Some(msg.into());
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for DownloadSpanGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
let finished_at = time::OffsetDateTime::now_utc();
|
||||
let dur = self.start_instant.elapsed();
|
||||
let duration_ms = duration_to_ms(dur);
|
||||
let started_at_rfc3339_utc = self
|
||||
.started_at
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&Rfc3339)
|
||||
.unwrap_or_else(|_| "<format-error>".to_string());
|
||||
let finished_at_rfc3339_utc = finished_at
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&Rfc3339)
|
||||
.unwrap_or_else(|_| "<format-error>".to_string());
|
||||
let success = self.success.unwrap_or(false);
|
||||
let event = AuditDownloadEvent {
|
||||
kind: self.kind.clone(),
|
||||
uri: self.uri.to_string(),
|
||||
started_at_rfc3339_utc,
|
||||
finished_at_rfc3339_utc,
|
||||
duration_ms,
|
||||
success,
|
||||
error: if success { None } else { self.error.clone() },
|
||||
bytes: self.bytes,
|
||||
objects: self.objects.clone(),
|
||||
};
|
||||
self.handle.record_event(event);
|
||||
}
|
||||
}
|
||||
|
||||
fn duration_to_ms(d: Duration) -> u64 {
|
||||
let ms = d.as_millis();
|
||||
ms.min(u128::from(u64::MAX)) as u64
|
||||
}
|
||||
@ -0,0 +1,6 @@
|
||||
fn main() {
|
||||
let code = panda_rpki_validator::daemon::main_entry();
|
||||
if code != 0 {
|
||||
std::process::exit(code);
|
||||
}
|
||||
}
|
||||
839
crates/panda-rpki-validator/src/blob_store.rs
Normal file
839
crates/panda-rpki-validator/src/blob_store.rs
Normal file
@ -0,0 +1,839 @@
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
|
||||
use rocksdb::{DB, Options, WriteBatch};
|
||||
|
||||
use crate::storage::{
|
||||
RawByHashEntry, RocksDbMemoryDbSnapshot, RocksStore, StorageError, StorageResult,
|
||||
memory_db_snapshot_for_column_families,
|
||||
};
|
||||
|
||||
const RAW_BY_HASH_KEY_PREFIX: &str = "rawbyhash:";
|
||||
const RAW_BLOB_KEY_PREFIX: &str = "rawblob:";
|
||||
const REPO_BYTES_KEY_PREFIX: &str = "sha256:";
|
||||
|
||||
fn raw_by_hash_key(sha256_hex: &str) -> String {
|
||||
format!("{RAW_BY_HASH_KEY_PREFIX}{sha256_hex}")
|
||||
}
|
||||
|
||||
fn raw_blob_key(sha256_hex: &str) -> String {
|
||||
format!("{RAW_BLOB_KEY_PREFIX}{sha256_hex}")
|
||||
}
|
||||
|
||||
fn repo_bytes_key(sha256_hex: &str) -> String {
|
||||
format!("{REPO_BYTES_KEY_PREFIX}{sha256_hex}")
|
||||
}
|
||||
|
||||
fn validate_blob_sha256_hex(sha256_hex: &str) -> StorageResult<()> {
|
||||
if sha256_hex.len() != 64 || !sha256_hex.as_bytes().iter().all(u8::is_ascii_hexdigit) {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "raw_blob",
|
||||
detail: format!("invalid sha256 hex: {sha256_hex}"),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_blob_bytes(bytes: &[u8]) -> StorageResult<()> {
|
||||
if bytes.is_empty() {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "raw_blob",
|
||||
detail: "bytes must not be empty".to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub trait RawObjectStore {
|
||||
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>>;
|
||||
|
||||
fn get_raw_entries_batch(
|
||||
&self,
|
||||
sha256_hexes: &[String],
|
||||
) -> StorageResult<Vec<Option<RawByHashEntry>>>;
|
||||
|
||||
fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
|
||||
self.get_raw_entry(sha256_hex)
|
||||
.map(|entry| entry.map(|entry| entry.bytes))
|
||||
}
|
||||
|
||||
fn get_blob_bytes_batch(&self, sha256_hexes: &[String]) -> StorageResult<Vec<Option<Vec<u8>>>> {
|
||||
self.get_raw_entries_batch(sha256_hexes).map(|entries| {
|
||||
entries
|
||||
.into_iter()
|
||||
.map(|entry| entry.map(|entry| entry.bytes))
|
||||
.collect()
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct ExternalRawStoreDb {
|
||||
path: PathBuf,
|
||||
db: Arc<DB>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct ExternalRepoBytesDb {
|
||||
path: PathBuf,
|
||||
db: Arc<DB>,
|
||||
read_only: bool,
|
||||
secondary: bool,
|
||||
}
|
||||
|
||||
impl ExternalRawStoreDb {
|
||||
pub fn open(path: impl Into<PathBuf>) -> StorageResult<Self> {
|
||||
let path = path.into();
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
}
|
||||
let mut opts = Options::default();
|
||||
opts.create_if_missing(true);
|
||||
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
|
||||
opts.set_max_open_files(512);
|
||||
let db = DB::open(&opts, &path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(Self {
|
||||
path,
|
||||
db: Arc::new(db),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn put_raw_entry(&self, entry: &RawByHashEntry) -> StorageResult<()> {
|
||||
entry.validate_internal()?;
|
||||
let key = raw_by_hash_key(&entry.sha256_hex);
|
||||
let blob_key = raw_blob_key(&entry.sha256_hex);
|
||||
let value = serde_cbor::to_vec(entry).map_err(|e| StorageError::Codec {
|
||||
entity: "raw_by_hash",
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
let blob_value = entry.bytes.clone();
|
||||
self.db
|
||||
.write({
|
||||
let mut batch = WriteBatch::default();
|
||||
batch.put(key.as_bytes(), value);
|
||||
batch.put(blob_key.as_bytes(), blob_value);
|
||||
batch
|
||||
})
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn put_raw_entries_batch(&self, entries: &[RawByHashEntry]) -> StorageResult<()> {
|
||||
if entries.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let mut batch = WriteBatch::default();
|
||||
for entry in entries {
|
||||
entry.validate_internal()?;
|
||||
let key = raw_by_hash_key(&entry.sha256_hex);
|
||||
let blob_key = raw_blob_key(&entry.sha256_hex);
|
||||
let value = serde_cbor::to_vec(entry).map_err(|e| StorageError::Codec {
|
||||
entity: "raw_by_hash",
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
batch.put(key.as_bytes(), value);
|
||||
batch.put(blob_key.as_bytes(), entry.bytes.as_slice());
|
||||
}
|
||||
self.db
|
||||
.write(batch)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn put_blob_bytes_batch(&self, blobs: &[(String, Vec<u8>)]) -> StorageResult<()> {
|
||||
if blobs.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let mut batch = WriteBatch::default();
|
||||
for (sha256_hex, bytes) in blobs {
|
||||
validate_blob_sha256_hex(sha256_hex)?;
|
||||
validate_blob_bytes(bytes)?;
|
||||
let blob_key = raw_blob_key(sha256_hex);
|
||||
batch.put(blob_key.as_bytes(), bytes.as_slice());
|
||||
}
|
||||
self.db
|
||||
.write(batch)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn delete_raw_entry(&self, sha256_hex: &str) -> StorageResult<()> {
|
||||
let key = raw_by_hash_key(sha256_hex);
|
||||
let blob_key = raw_blob_key(sha256_hex);
|
||||
self.db
|
||||
.write({
|
||||
let mut batch = WriteBatch::default();
|
||||
batch.delete(key.as_bytes());
|
||||
batch.delete(blob_key.as_bytes());
|
||||
batch
|
||||
})
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))
|
||||
}
|
||||
|
||||
pub fn path(&self) -> &PathBuf {
|
||||
&self.path
|
||||
}
|
||||
|
||||
pub(crate) fn memory_snapshot(&self, label: impl Into<String>) -> RocksDbMemoryDbSnapshot {
|
||||
memory_db_snapshot_for_column_families(label, self.db.as_ref(), None)
|
||||
}
|
||||
}
|
||||
|
||||
impl ExternalRepoBytesDb {
|
||||
pub fn open(path: impl Into<PathBuf>) -> StorageResult<Self> {
|
||||
let path = path.into();
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
}
|
||||
let mut opts = Options::default();
|
||||
opts.create_if_missing(true);
|
||||
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
|
||||
opts.set_max_open_files(512);
|
||||
let db = DB::open(&opts, &path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(Self {
|
||||
path,
|
||||
db: Arc::new(db),
|
||||
read_only: false,
|
||||
secondary: false,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn open_read_only(path: impl Into<PathBuf>) -> StorageResult<Self> {
|
||||
let path = path.into();
|
||||
let mut opts = Options::default();
|
||||
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
|
||||
opts.set_max_open_files(512);
|
||||
let db = DB::open_for_read_only(&opts, &path, false)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(Self {
|
||||
path,
|
||||
db: Arc::new(db),
|
||||
read_only: true,
|
||||
secondary: false,
|
||||
})
|
||||
}
|
||||
|
||||
/// Open the repo-bytes DB as a RocksDB secondary instance.
|
||||
///
|
||||
/// Unlike `open_read_only` (a frozen point-in-time view), a secondary
|
||||
/// instance can follow the live primary via `try_catch_up_with_primary`,
|
||||
/// which is required when the soak keeps writing new object bytes while
|
||||
/// the query service is running.
|
||||
pub fn open_as_secondary(
|
||||
path: impl Into<PathBuf>,
|
||||
secondary_path: impl Into<PathBuf>,
|
||||
) -> StorageResult<Self> {
|
||||
let path = path.into();
|
||||
let secondary_path = secondary_path.into();
|
||||
if let Some(parent) = secondary_path.parent() {
|
||||
std::fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
}
|
||||
let mut opts = Options::default();
|
||||
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
|
||||
opts.set_max_open_files(512);
|
||||
let db = DB::open_as_secondary(&opts, &path, &secondary_path)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(Self {
|
||||
path,
|
||||
db: Arc::new(db),
|
||||
read_only: true,
|
||||
secondary: true,
|
||||
})
|
||||
}
|
||||
|
||||
/// Pull the secondary view up to the primary's current state. No-op for
|
||||
/// non-secondary handles.
|
||||
pub fn try_catch_up_with_primary(&self) -> StorageResult<()> {
|
||||
if self.secondary {
|
||||
self.db
|
||||
.try_catch_up_with_primary()
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn is_secondary(&self) -> bool {
|
||||
self.secondary
|
||||
}
|
||||
|
||||
pub fn put_blob_bytes_batch(&self, blobs: &[(String, Vec<u8>)]) -> StorageResult<()> {
|
||||
if blobs.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
if self.read_only {
|
||||
return Err(StorageError::RocksDb(format!(
|
||||
"repo-bytes DB is read-only: {}",
|
||||
self.path.display()
|
||||
)));
|
||||
}
|
||||
let mut batch = WriteBatch::default();
|
||||
for (sha256_hex, bytes) in blobs {
|
||||
validate_blob_sha256_hex(sha256_hex)?;
|
||||
validate_blob_bytes(bytes)?;
|
||||
let key = repo_bytes_key(sha256_hex);
|
||||
batch.put(key.as_bytes(), bytes.as_slice());
|
||||
}
|
||||
self.db
|
||||
.write(batch)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn is_read_only(&self) -> bool {
|
||||
self.read_only
|
||||
}
|
||||
|
||||
pub(crate) fn require_existing_blob_bytes_batch(
|
||||
&self,
|
||||
blobs: &[(String, Vec<u8>)],
|
||||
) -> StorageResult<()> {
|
||||
if blobs.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let hashes = blobs
|
||||
.iter()
|
||||
.map(|(sha256_hex, _)| sha256_hex.clone())
|
||||
.collect::<Vec<_>>();
|
||||
let existing = self.get_blob_bytes_batch(&hashes)?;
|
||||
for ((sha256_hex, expected_bytes), actual_bytes) in blobs.iter().zip(existing) {
|
||||
match actual_bytes {
|
||||
Some(actual_bytes) if actual_bytes == *expected_bytes => {}
|
||||
Some(_) => {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "read_only_repo_bytes",
|
||||
detail: format!("existing bytes differ for SHA-256 {sha256_hex}"),
|
||||
});
|
||||
}
|
||||
None => {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "read_only_repo_bytes",
|
||||
detail: format!("blob is missing for SHA-256 {sha256_hex}"),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
|
||||
validate_blob_sha256_hex(sha256_hex)?;
|
||||
let key = repo_bytes_key(sha256_hex);
|
||||
self.db
|
||||
.get(key.as_bytes())
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))
|
||||
}
|
||||
|
||||
pub fn get_blob_bytes_batch(
|
||||
&self,
|
||||
sha256_hexes: &[String],
|
||||
) -> StorageResult<Vec<Option<Vec<u8>>>> {
|
||||
if sha256_hexes.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let keys: Vec<String> = sha256_hexes
|
||||
.iter()
|
||||
.map(|hash| {
|
||||
validate_blob_sha256_hex(hash)?;
|
||||
Ok::<String, StorageError>(repo_bytes_key(hash))
|
||||
})
|
||||
.collect::<Result<_, _>>()?;
|
||||
self.db
|
||||
.multi_get(keys.iter().map(|key| key.as_bytes()))
|
||||
.into_iter()
|
||||
.map(|res| res.map_err(|e| StorageError::RocksDb(e.to_string())))
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn path(&self) -> &PathBuf {
|
||||
&self.path
|
||||
}
|
||||
|
||||
pub(crate) fn memory_snapshot(&self, label: impl Into<String>) -> RocksDbMemoryDbSnapshot {
|
||||
memory_db_snapshot_for_column_families(label, self.db.as_ref(), None)
|
||||
}
|
||||
}
|
||||
|
||||
impl RawObjectStore for RocksStore {
|
||||
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
|
||||
self.get_raw_by_hash_entry(sha256_hex)
|
||||
}
|
||||
|
||||
fn get_raw_entries_batch(
|
||||
&self,
|
||||
sha256_hexes: &[String],
|
||||
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
|
||||
self.get_raw_by_hash_entries_batch(sha256_hexes)
|
||||
}
|
||||
|
||||
fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
|
||||
RocksStore::get_blob_bytes(self, sha256_hex)
|
||||
}
|
||||
|
||||
fn get_blob_bytes_batch(&self, sha256_hexes: &[String]) -> StorageResult<Vec<Option<Vec<u8>>>> {
|
||||
RocksStore::get_blob_bytes_batch(self, sha256_hexes)
|
||||
}
|
||||
}
|
||||
|
||||
impl RawObjectStore for ExternalRawStoreDb {
|
||||
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
|
||||
let key = raw_by_hash_key(sha256_hex);
|
||||
let Some(bytes) = self
|
||||
.db
|
||||
.get(key.as_bytes())
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
else {
|
||||
return Ok(None);
|
||||
};
|
||||
let entry =
|
||||
serde_cbor::from_slice::<RawByHashEntry>(&bytes).map_err(|e| StorageError::Codec {
|
||||
entity: "raw_by_hash",
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
entry.validate_internal()?;
|
||||
Ok(Some(entry))
|
||||
}
|
||||
|
||||
fn get_raw_entries_batch(
|
||||
&self,
|
||||
sha256_hexes: &[String],
|
||||
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
|
||||
if sha256_hexes.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let keys: Vec<String> = sha256_hexes
|
||||
.iter()
|
||||
.map(|hash| raw_by_hash_key(hash))
|
||||
.collect();
|
||||
self.db
|
||||
.multi_get(keys.iter().map(|key| key.as_bytes()))
|
||||
.into_iter()
|
||||
.map(|res| {
|
||||
let maybe = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
match maybe {
|
||||
Some(bytes) => {
|
||||
let entry =
|
||||
serde_cbor::from_slice::<RawByHashEntry>(&bytes).map_err(|e| {
|
||||
StorageError::Codec {
|
||||
entity: "raw_by_hash",
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
entry.validate_internal()?;
|
||||
Ok(Some(entry))
|
||||
}
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
|
||||
let key = raw_blob_key(sha256_hex);
|
||||
self.db
|
||||
.get(key.as_bytes())
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))
|
||||
}
|
||||
|
||||
fn get_blob_bytes_batch(&self, sha256_hexes: &[String]) -> StorageResult<Vec<Option<Vec<u8>>>> {
|
||||
if sha256_hexes.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let keys: Vec<String> = sha256_hexes.iter().map(|hash| raw_blob_key(hash)).collect();
|
||||
self.db
|
||||
.multi_get(keys.iter().map(|key| key.as_bytes()))
|
||||
.into_iter()
|
||||
.map(|res| res.map_err(|e| StorageError::RocksDb(e.to_string())))
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ExternalRawStoreDb, ExternalRepoBytesDb, RawObjectStore};
|
||||
use crate::storage::{RawByHashEntry, RocksStore, StorageError, StorageResult};
|
||||
use std::collections::HashMap;
|
||||
|
||||
fn sha256_hex(bytes: &[u8]) -> String {
|
||||
use sha2::{Digest, Sha256};
|
||||
hex::encode(Sha256::digest(bytes))
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct MockRawStore {
|
||||
entries: HashMap<String, RawByHashEntry>,
|
||||
}
|
||||
|
||||
impl RawObjectStore for MockRawStore {
|
||||
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
|
||||
Ok(self.entries.get(sha256_hex).cloned())
|
||||
}
|
||||
|
||||
fn get_raw_entries_batch(
|
||||
&self,
|
||||
sha256_hexes: &[String],
|
||||
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
|
||||
Ok(sha256_hexes
|
||||
.iter()
|
||||
.map(|hash| self.entries.get(hash).cloned())
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rocks_store_raw_object_store_reads_single_and_batch_entries() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(td.path()).expect("open rocksdb");
|
||||
|
||||
let a = b"object-a".to_vec();
|
||||
let b = b"object-b".to_vec();
|
||||
let a_hash = sha256_hex(&a);
|
||||
let b_hash = sha256_hex(&b);
|
||||
|
||||
store
|
||||
.put_raw_by_hash_entry(&RawByHashEntry::from_bytes(a_hash.clone(), a.clone()))
|
||||
.expect("put a");
|
||||
store
|
||||
.put_raw_by_hash_entry(&RawByHashEntry::from_bytes(b_hash.clone(), b.clone()))
|
||||
.expect("put b");
|
||||
|
||||
let single = store
|
||||
.get_raw_entry(&a_hash)
|
||||
.expect("get single")
|
||||
.expect("present");
|
||||
assert_eq!(single.bytes, a);
|
||||
|
||||
let batch = store
|
||||
.get_raw_entries_batch(&[a_hash.clone(), "00".repeat(32), b_hash.clone()])
|
||||
.expect("get batch");
|
||||
assert_eq!(batch.len(), 3);
|
||||
assert_eq!(
|
||||
batch[0].as_ref().map(|entry| entry.bytes.as_slice()),
|
||||
Some(a.as_slice())
|
||||
);
|
||||
assert!(batch[1].is_none());
|
||||
assert_eq!(
|
||||
batch[2].as_ref().map(|entry| entry.bytes.as_slice()),
|
||||
Some(b.as_slice())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_raw_store_db_roundtrips_entries() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
|
||||
let mut entry = RawByHashEntry::from_bytes(sha256_hex(b"blob"), b"blob".to_vec());
|
||||
entry
|
||||
.origin_uris
|
||||
.push("rsync://example.test/repo/a.cer".to_string());
|
||||
entry.object_type = Some("cer".to_string());
|
||||
raw_store.put_raw_entry(&entry).expect("put raw entry");
|
||||
|
||||
let got = raw_store
|
||||
.get_raw_entry(&entry.sha256_hex)
|
||||
.expect("read raw entry")
|
||||
.expect("entry exists");
|
||||
assert_eq!(got, entry);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_raw_store_db_batch_writes_and_reads() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
|
||||
let a = RawByHashEntry::from_bytes(sha256_hex(b"a"), b"a".to_vec());
|
||||
let b = RawByHashEntry::from_bytes(sha256_hex(b"b"), b"b".to_vec());
|
||||
raw_store
|
||||
.put_raw_entries_batch(&[a.clone(), b.clone()])
|
||||
.expect("batch put");
|
||||
|
||||
let batch = raw_store
|
||||
.get_raw_entries_batch(&[a.sha256_hex.clone(), b.sha256_hex.clone()])
|
||||
.expect("batch get");
|
||||
assert_eq!(batch.len(), 2);
|
||||
assert_eq!(batch[0], Some(a));
|
||||
assert_eq!(batch[1], Some(b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn raw_object_store_default_blob_helpers_return_bytes_only() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store = ExternalRawStoreDb::open(td.path().join("nested/raw-store.db"))
|
||||
.expect("open raw store");
|
||||
|
||||
let mut entry = RawByHashEntry::from_bytes(sha256_hex(b"blob"), b"blob".to_vec());
|
||||
entry
|
||||
.origin_uris
|
||||
.push("rsync://example.test/repo/blob.roa".to_string());
|
||||
raw_store.put_raw_entry(&entry).expect("put raw entry");
|
||||
|
||||
let single = raw_store
|
||||
.get_blob_bytes(&entry.sha256_hex)
|
||||
.expect("get blob bytes")
|
||||
.expect("entry exists");
|
||||
assert_eq!(single, b"blob".to_vec());
|
||||
|
||||
let batch = raw_store
|
||||
.get_blob_bytes_batch(&[entry.sha256_hex.clone(), "00".repeat(32)])
|
||||
.expect("get blob bytes batch");
|
||||
assert_eq!(batch, vec![Some(b"blob".to_vec()), None]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn raw_object_store_default_blob_helpers_work_for_custom_store() {
|
||||
let mut store = MockRawStore::default();
|
||||
let a = RawByHashEntry::from_bytes(sha256_hex(b"a"), b"a".to_vec());
|
||||
let b = RawByHashEntry::from_bytes(sha256_hex(b"b"), b"b".to_vec());
|
||||
store.entries.insert(a.sha256_hex.clone(), a.clone());
|
||||
store.entries.insert(b.sha256_hex.clone(), b.clone());
|
||||
|
||||
let single = store
|
||||
.get_blob_bytes(&a.sha256_hex)
|
||||
.expect("single blob bytes")
|
||||
.expect("present");
|
||||
assert_eq!(single, b"a".to_vec());
|
||||
|
||||
let batch = store
|
||||
.get_blob_bytes_batch(&[a.sha256_hex.clone(), "00".repeat(32), b.sha256_hex.clone()])
|
||||
.expect("batch blob bytes");
|
||||
assert_eq!(batch, vec![Some(b"a".to_vec()), None, Some(b"b".to_vec())]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rocks_store_blob_helpers_use_external_raw_store_fast_path() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open_with_external_raw_store(
|
||||
&td.path().join("db"),
|
||||
&td.path().join("raw-store.db"),
|
||||
)
|
||||
.expect("open store with external raw store");
|
||||
|
||||
let entry = RawByHashEntry::from_bytes(sha256_hex(b"blob-fast"), b"blob-fast".to_vec());
|
||||
store.put_raw_by_hash_entry(&entry).expect("put");
|
||||
|
||||
let single = store
|
||||
.get_blob_bytes(&entry.sha256_hex)
|
||||
.expect("single blob bytes")
|
||||
.expect("present");
|
||||
assert_eq!(single, b"blob-fast".to_vec());
|
||||
|
||||
let batch = store
|
||||
.get_blob_bytes_batch(&[entry.sha256_hex.clone(), "00".repeat(32)])
|
||||
.expect("batch blob bytes");
|
||||
assert_eq!(batch, vec![Some(b"blob-fast".to_vec()), None]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_raw_store_db_delete_removes_entry() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
|
||||
let entry = RawByHashEntry::from_bytes(sha256_hex(b"gone"), b"gone".to_vec());
|
||||
raw_store.put_raw_entry(&entry).expect("put");
|
||||
assert!(
|
||||
raw_store
|
||||
.get_raw_entry(&entry.sha256_hex)
|
||||
.unwrap()
|
||||
.is_some()
|
||||
);
|
||||
|
||||
raw_store
|
||||
.delete_raw_entry(&entry.sha256_hex)
|
||||
.expect("delete entry");
|
||||
assert!(
|
||||
raw_store
|
||||
.get_raw_entry(&entry.sha256_hex)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn put_blob_bytes_batch_round_trips_without_raw_entry() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
|
||||
let a = (sha256_hex(b"blob-a"), b"blob-a".to_vec());
|
||||
let b = (sha256_hex(b"blob-b"), b"blob-b".to_vec());
|
||||
raw_store
|
||||
.put_blob_bytes_batch(&[a.clone(), b.clone()])
|
||||
.expect("put blobs");
|
||||
|
||||
assert_eq!(
|
||||
raw_store.get_blob_bytes(&a.0).expect("get blob a"),
|
||||
Some(a.1.clone())
|
||||
);
|
||||
assert_eq!(
|
||||
raw_store.get_blob_bytes(&b.0).expect("get blob b"),
|
||||
Some(b.1.clone())
|
||||
);
|
||||
assert!(raw_store.get_raw_entry(&a.0).expect("get raw a").is_none());
|
||||
assert!(raw_store.get_raw_entry(&b.0).expect("get raw b").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn put_blob_bytes_batch_rejects_invalid_inputs() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
|
||||
let err = raw_store
|
||||
.put_blob_bytes_batch(&[("zz".repeat(32), b"blob".to_vec())])
|
||||
.expect_err("invalid hash should fail");
|
||||
assert!(matches!(err, StorageError::InvalidData { .. }));
|
||||
|
||||
let err = raw_store
|
||||
.put_blob_bytes_batch(&[(sha256_hex(b"blob"), Vec::new())])
|
||||
.expect_err("empty bytes should fail");
|
||||
assert!(matches!(err, StorageError::InvalidData { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_raw_store_db_rejects_invalid_entry_on_put() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
|
||||
let bad = RawByHashEntry {
|
||||
sha256_hex: "11".repeat(32),
|
||||
bytes: b"blob".to_vec(),
|
||||
origin_uris: Vec::new(),
|
||||
object_type: None,
|
||||
encoding: None,
|
||||
};
|
||||
let err = raw_store
|
||||
.put_raw_entry(&bad)
|
||||
.expect_err("invalid hash should fail");
|
||||
assert!(matches!(err, StorageError::InvalidData { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_raw_store_db_reports_codec_error_for_corrupt_value() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
raw_store
|
||||
.db
|
||||
.put(b"rawbyhash:deadbeef", b"not-cbor")
|
||||
.expect("inject corrupt bytes");
|
||||
|
||||
let err = raw_store
|
||||
.get_raw_entry("deadbeef")
|
||||
.expect_err("corrupt value should fail");
|
||||
assert!(matches!(
|
||||
err,
|
||||
StorageError::Codec {
|
||||
entity: "raw_by_hash",
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_raw_store_db_batch_returns_empty_for_empty_request() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
|
||||
let entries = raw_store
|
||||
.get_raw_entries_batch(&[])
|
||||
.expect("empty batch succeeds");
|
||||
assert!(entries.is_empty());
|
||||
raw_store
|
||||
.put_raw_entries_batch(&[])
|
||||
.expect("empty put succeeds");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_repo_bytes_db_roundtrips_blob_bytes_without_raw_entry() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let repo_bytes =
|
||||
ExternalRepoBytesDb::open(td.path().join("repo-bytes.db")).expect("open repo bytes");
|
||||
let bytes = b"repo-bytes-object".to_vec();
|
||||
let hash = sha256_hex(&bytes);
|
||||
|
||||
repo_bytes
|
||||
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
|
||||
.expect("put repo bytes");
|
||||
|
||||
assert_eq!(
|
||||
repo_bytes.get_blob_bytes(&hash).expect("get repo bytes"),
|
||||
Some(bytes.clone())
|
||||
);
|
||||
assert_eq!(
|
||||
repo_bytes
|
||||
.get_blob_bytes_batch(&[hash, "00".repeat(32)])
|
||||
.expect("get repo bytes batch"),
|
||||
vec![Some(bytes), None]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_repo_bytes_db_rejects_invalid_inputs() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let repo_bytes =
|
||||
ExternalRepoBytesDb::open(td.path().join("repo-bytes.db")).expect("open repo bytes");
|
||||
|
||||
assert!(
|
||||
repo_bytes
|
||||
.put_blob_bytes_batch(&[("not-a-valid-hash".to_string(), b"blob".to_vec())])
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
repo_bytes
|
||||
.put_blob_bytes_batch(&[(sha256_hex(b"blob"), Vec::new())])
|
||||
.is_err()
|
||||
);
|
||||
assert!(repo_bytes.get_blob_bytes("not-a-valid-hash").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn external_repo_bytes_db_secondary_catches_up_with_live_primary() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let primary_path = td.path().join("repo-bytes.db");
|
||||
let secondary_path = td.path().join("repo-bytes.secondary");
|
||||
|
||||
let primary = ExternalRepoBytesDb::open(&primary_path).expect("open primary");
|
||||
let bytes_a = b"repo-bytes-a".to_vec();
|
||||
let hash_a = sha256_hex(&bytes_a);
|
||||
primary
|
||||
.put_blob_bytes_batch(&[(hash_a.clone(), bytes_a.clone())])
|
||||
.expect("put a");
|
||||
|
||||
let secondary = ExternalRepoBytesDb::open_as_secondary(&primary_path, &secondary_path)
|
||||
.expect("open secondary");
|
||||
assert!(secondary.is_secondary());
|
||||
|
||||
// A secondary open does not necessarily see pre-existing data until it
|
||||
// catches up with the primary's manifest.
|
||||
secondary
|
||||
.try_catch_up_with_primary()
|
||||
.expect("initial catch up");
|
||||
assert_eq!(
|
||||
secondary
|
||||
.get_blob_bytes(&hash_a)
|
||||
.expect("get a via secondary"),
|
||||
Some(bytes_a)
|
||||
);
|
||||
|
||||
// Bytes written by the primary *after* the secondary open become
|
||||
// visible after another catch-up (this is the live-soak scenario).
|
||||
let bytes_b = b"repo-bytes-b".to_vec();
|
||||
let hash_b = sha256_hex(&bytes_b);
|
||||
primary
|
||||
.put_blob_bytes_batch(&[(hash_b.clone(), bytes_b.clone())])
|
||||
.expect("put b after secondary open");
|
||||
secondary
|
||||
.try_catch_up_with_primary()
|
||||
.expect("second catch up");
|
||||
assert_eq!(
|
||||
secondary
|
||||
.get_blob_bytes(&hash_b)
|
||||
.expect("get b via secondary"),
|
||||
Some(bytes_b)
|
||||
);
|
||||
}
|
||||
}
|
||||
516
crates/panda-rpki-validator/src/ccr/accumulator.rs
Normal file
516
crates/panda-rpki-validator/src/ccr/accumulator.rs
Normal file
@ -0,0 +1,516 @@
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use crate::ccr::build::{
|
||||
build_aspa_payload_state, build_roa_payload_state, build_router_key_state_from_runtime,
|
||||
build_trust_anchor_state,
|
||||
};
|
||||
use crate::ccr::encode::encode_manifest_state_payload_der;
|
||||
use crate::ccr::hash::compute_state_hash;
|
||||
use crate::ccr::manifest_location::select_manifest_signed_object_location_from_der;
|
||||
use crate::ccr::model::{
|
||||
CcrDigestAlgorithm, ManifestInstance, ManifestState, RpkiCanonicalCacheRepresentation,
|
||||
};
|
||||
use crate::data_model::common::BigUnsigned;
|
||||
use crate::data_model::ta::TrustAnchor;
|
||||
use crate::storage::VcirCcrManifestProjection;
|
||||
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CcrManifestContribution {
|
||||
pub manifest_rsync_uri: String,
|
||||
pub hash: Vec<u8>,
|
||||
pub size: u64,
|
||||
pub aki: Vec<u8>,
|
||||
pub manifest_number_be: Vec<u8>,
|
||||
pub this_update: time::OffsetDateTime,
|
||||
pub locations_der: Vec<Vec<u8>>,
|
||||
pub subordinate_skis: Vec<Vec<u8>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct CcrAccumulatorMemoryStats {
|
||||
pub trust_anchor_count: u64,
|
||||
pub manifest_count: u64,
|
||||
pub estimated_heap_bytes: u64,
|
||||
pub string_bytes: u64,
|
||||
pub string_capacity_bytes: u64,
|
||||
pub vec_payload_bytes: u64,
|
||||
pub vec_capacity_bytes: u64,
|
||||
pub locations_der_count: u64,
|
||||
pub subordinate_ski_count: u64,
|
||||
pub btree_key_capacity_bytes: u64,
|
||||
pub btree_entry_shallow_bytes: u64,
|
||||
}
|
||||
|
||||
impl CcrManifestContribution {
|
||||
fn from_projection(projection: &VcirCcrManifestProjection) -> Result<Self, String> {
|
||||
let this_update = projection
|
||||
.manifest_this_update
|
||||
.parse()
|
||||
.map_err(|e| format!("parse projection manifest_this_update failed: {e}"))?;
|
||||
Ok(Self {
|
||||
manifest_rsync_uri: projection.manifest_rsync_uri.clone(),
|
||||
hash: projection.manifest_sha256.clone(),
|
||||
size: projection.manifest_size,
|
||||
aki: projection.manifest_ee_aki.clone(),
|
||||
manifest_number_be: projection.manifest_number_be.clone(),
|
||||
this_update,
|
||||
locations_der: vec![select_manifest_signed_object_location_from_der(
|
||||
&projection.manifest_rsync_uri,
|
||||
&projection.manifest_sia_locations_der,
|
||||
)?],
|
||||
subordinate_skis: projection.subordinate_skis.clone(),
|
||||
})
|
||||
}
|
||||
|
||||
fn to_manifest_instance(&self) -> ManifestInstance {
|
||||
ManifestInstance {
|
||||
hash: self.hash.clone(),
|
||||
size: self.size,
|
||||
aki: self.aki.clone(),
|
||||
manifest_number: BigUnsigned {
|
||||
bytes_be: self.manifest_number_be.clone(),
|
||||
},
|
||||
this_update: self.this_update,
|
||||
locations: self.locations_der.clone(),
|
||||
subordinates: self.subordinate_skis.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn add_memory_stats(&self, stats: &mut CcrAccumulatorMemoryStats) {
|
||||
stats.string_bytes += self.manifest_rsync_uri.len() as u64;
|
||||
stats.string_capacity_bytes += self.manifest_rsync_uri.capacity() as u64;
|
||||
stats.estimated_heap_bytes += self.manifest_rsync_uri.capacity() as u64;
|
||||
|
||||
add_vec_stats(&self.hash, stats);
|
||||
add_vec_stats(&self.aki, stats);
|
||||
add_vec_stats(&self.manifest_number_be, stats);
|
||||
add_vec_of_vec_stats(&self.locations_der, stats);
|
||||
add_vec_of_vec_stats(&self.subordinate_skis, stats);
|
||||
stats.locations_der_count += self.locations_der.len() as u64;
|
||||
stats.subordinate_ski_count += self.subordinate_skis.len() as u64;
|
||||
}
|
||||
}
|
||||
|
||||
fn add_vec_stats(value: &Vec<u8>, stats: &mut CcrAccumulatorMemoryStats) {
|
||||
stats.vec_payload_bytes += value.len() as u64;
|
||||
stats.vec_capacity_bytes += value.capacity() as u64;
|
||||
stats.estimated_heap_bytes += value.capacity() as u64;
|
||||
}
|
||||
|
||||
fn add_vec_of_vec_stats(values: &Vec<Vec<u8>>, stats: &mut CcrAccumulatorMemoryStats) {
|
||||
let outer_capacity = values.capacity() * std::mem::size_of::<Vec<u8>>();
|
||||
stats.vec_payload_bytes += (values.len() * std::mem::size_of::<Vec<u8>>()) as u64;
|
||||
stats.vec_capacity_bytes += outer_capacity as u64;
|
||||
stats.estimated_heap_bytes += outer_capacity as u64;
|
||||
for value in values {
|
||||
add_vec_stats(value, stats);
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CcrAccumulator {
|
||||
trust_anchors: Vec<TrustAnchor>,
|
||||
manifests_by_hash: BTreeMap<Vec<u8>, CcrManifestContribution>,
|
||||
most_recent_update: time::OffsetDateTime,
|
||||
}
|
||||
|
||||
impl CcrAccumulator {
|
||||
pub fn new(trust_anchors: Vec<TrustAnchor>) -> Self {
|
||||
Self {
|
||||
trust_anchors,
|
||||
manifests_by_hash: BTreeMap::new(),
|
||||
most_recent_update: time::OffsetDateTime::UNIX_EPOCH,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn append_manifest_projection(
|
||||
&mut self,
|
||||
projection: &VcirCcrManifestProjection,
|
||||
) -> Result<(), String> {
|
||||
let contribution = CcrManifestContribution::from_projection(projection)?;
|
||||
match self.manifests_by_hash.get(contribution.hash.as_slice()) {
|
||||
Some(existing) if existing != &contribution => {
|
||||
return Err(format!(
|
||||
"duplicate manifest hash with conflicting content for URI: {}",
|
||||
contribution.manifest_rsync_uri
|
||||
));
|
||||
}
|
||||
Some(_) => {}
|
||||
None => {
|
||||
self.manifests_by_hash
|
||||
.insert(contribution.hash.clone(), contribution.clone());
|
||||
}
|
||||
}
|
||||
if contribution.this_update > self.most_recent_update {
|
||||
self.most_recent_update = contribution.this_update;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn finish(
|
||||
&self,
|
||||
produced_at: time::OffsetDateTime,
|
||||
vrps: &[Vrp],
|
||||
aspas: &[AspaAttestation],
|
||||
router_keys: &[RouterKeyPayload],
|
||||
) -> Result<RpkiCanonicalCacheRepresentation, String> {
|
||||
let manifest_instances = self
|
||||
.manifests_by_hash
|
||||
.values()
|
||||
.map(CcrManifestContribution::to_manifest_instance)
|
||||
.collect::<Vec<_>>();
|
||||
let manifest_payload_der = encode_manifest_state_payload_der(&manifest_instances)
|
||||
.map_err(|e| format!("manifest state encoding failed: {e}"))?;
|
||||
let manifest_state = ManifestState {
|
||||
mis: manifest_instances,
|
||||
most_recent_update: self.most_recent_update,
|
||||
hash: compute_state_hash(&manifest_payload_der),
|
||||
};
|
||||
let vrp_state = build_roa_payload_state(vrps).map_err(|e| e.to_string())?;
|
||||
let aspa_state = build_aspa_payload_state(aspas).map_err(|e| e.to_string())?;
|
||||
let ta_state = build_trust_anchor_state(&self.trust_anchors).map_err(|e| e.to_string())?;
|
||||
let router_key_state =
|
||||
build_router_key_state_from_runtime(router_keys).map_err(|e| e.to_string())?;
|
||||
Ok(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at,
|
||||
mfts: Some(manifest_state),
|
||||
vrps: Some(vrp_state),
|
||||
vaps: Some(aspa_state),
|
||||
tas: Some(ta_state),
|
||||
rks: Some(router_key_state),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn manifest_count(&self) -> usize {
|
||||
self.manifests_by_hash.len()
|
||||
}
|
||||
|
||||
pub fn memory_stats(&self) -> CcrAccumulatorMemoryStats {
|
||||
let mut stats = CcrAccumulatorMemoryStats {
|
||||
trust_anchor_count: self.trust_anchors.len() as u64,
|
||||
manifest_count: self.manifests_by_hash.len() as u64,
|
||||
..CcrAccumulatorMemoryStats::default()
|
||||
};
|
||||
stats.estimated_heap_bytes +=
|
||||
(self.trust_anchors.capacity() * std::mem::size_of::<TrustAnchor>()) as u64;
|
||||
for trust_anchor in &self.trust_anchors {
|
||||
add_vec_stats(&trust_anchor.tal.raw, &mut stats);
|
||||
add_vec_of_string_stats(&trust_anchor.tal.comments, &mut stats);
|
||||
stats.vec_payload_bytes +=
|
||||
(trust_anchor.tal.ta_uris.len() * std::mem::size_of::<url::Url>()) as u64;
|
||||
stats.vec_capacity_bytes +=
|
||||
(trust_anchor.tal.ta_uris.capacity() * std::mem::size_of::<url::Url>()) as u64;
|
||||
stats.estimated_heap_bytes +=
|
||||
(trust_anchor.tal.ta_uris.capacity() * std::mem::size_of::<url::Url>()) as u64;
|
||||
for uri in &trust_anchor.tal.ta_uris {
|
||||
stats.string_bytes += uri.as_str().len() as u64;
|
||||
stats.string_capacity_bytes += uri.as_str().len() as u64;
|
||||
stats.estimated_heap_bytes += uri.as_str().len() as u64;
|
||||
}
|
||||
add_vec_stats(&trust_anchor.tal.subject_public_key_info_der, &mut stats);
|
||||
add_vec_stats(&trust_anchor.ta_certificate.raw_der, &mut stats);
|
||||
if let Some(uri) = &trust_anchor.resolved_ta_uri {
|
||||
stats.string_bytes += uri.as_str().len() as u64;
|
||||
stats.string_capacity_bytes += uri.as_str().len() as u64;
|
||||
stats.estimated_heap_bytes += uri.as_str().len() as u64;
|
||||
}
|
||||
}
|
||||
|
||||
stats.btree_entry_shallow_bytes = (self.manifests_by_hash.len()
|
||||
* (std::mem::size_of::<Vec<u8>>() + std::mem::size_of::<CcrManifestContribution>()))
|
||||
as u64;
|
||||
stats.estimated_heap_bytes += stats.btree_entry_shallow_bytes;
|
||||
for (key, contribution) in &self.manifests_by_hash {
|
||||
stats.btree_key_capacity_bytes += key.capacity() as u64;
|
||||
stats.estimated_heap_bytes += key.capacity() as u64;
|
||||
contribution.add_memory_stats(&mut stats);
|
||||
}
|
||||
stats
|
||||
}
|
||||
}
|
||||
|
||||
fn add_vec_of_string_stats(values: &Vec<String>, stats: &mut CcrAccumulatorMemoryStats) {
|
||||
let outer_capacity = values.capacity() * std::mem::size_of::<String>();
|
||||
stats.vec_payload_bytes += (values.len() * std::mem::size_of::<String>()) as u64;
|
||||
stats.vec_capacity_bytes += outer_capacity as u64;
|
||||
stats.estimated_heap_bytes += outer_capacity as u64;
|
||||
for value in values {
|
||||
stats.string_bytes += value.len() as u64;
|
||||
stats.string_capacity_bytes += value.capacity() as u64;
|
||||
stats.estimated_heap_bytes += value.capacity() as u64;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::export::build_ccr_from_run;
|
||||
use crate::ccr::verify::verify_content_info;
|
||||
use crate::data_model::manifest::ManifestObject;
|
||||
use crate::data_model::oid::{OID_AD_RPKI_NOTIFY, OID_AD_SIGNED_OBJECT};
|
||||
use crate::data_model::rc::{AccessDescription, SubjectInfoAccess};
|
||||
use crate::data_model::roa::{IpPrefix, RoaAfi};
|
||||
use crate::data_model::ta::TrustAnchor;
|
||||
use crate::data_model::tal::Tal;
|
||||
use crate::storage::{
|
||||
PackTime, RawByHashEntry, RocksStore, ValidatedCaInstanceResult, ValidatedManifestMeta,
|
||||
VcirArtifactKind, VcirArtifactRole, VcirArtifactValidationStatus, VcirAuditSummary,
|
||||
VcirCcrManifestProjection, VcirChildEntry, VcirInstanceGate, VcirRelatedArtifact,
|
||||
VcirSummary,
|
||||
};
|
||||
use sha2::Digest;
|
||||
|
||||
fn sample_trust_anchor() -> TrustAnchor {
|
||||
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let tal_bytes = std::fs::read(base.join("tests/fixtures/tal/apnic-rfc7730-https.tal"))
|
||||
.expect("read tal");
|
||||
let ta_der = std::fs::read(base.join("tests/fixtures/ta/apnic-ta.cer")).expect("read ta");
|
||||
let tal = Tal::decode_bytes(&tal_bytes).expect("decode tal");
|
||||
TrustAnchor::bind_der(tal, &ta_der, None).expect("bind ta")
|
||||
}
|
||||
|
||||
fn sample_vcir_and_manifest(
|
||||
store: &RocksStore,
|
||||
) -> (
|
||||
ValidatedCaInstanceResult,
|
||||
Vec<Vrp>,
|
||||
Vec<AspaAttestation>,
|
||||
Vec<RouterKeyPayload>,
|
||||
) {
|
||||
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let manifest_der = std::fs::read(
|
||||
base.join(
|
||||
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft",
|
||||
),
|
||||
)
|
||||
.expect("read manifest");
|
||||
let manifest = ManifestObject::decode_der(&manifest_der).expect("decode manifest");
|
||||
let manifest_uri = match manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.expect("manifest sia")
|
||||
{
|
||||
SubjectInfoAccess::Ee(ee_sia) => ee_sia
|
||||
.access_descriptions
|
||||
.iter()
|
||||
.find(|ad| {
|
||||
ad.access_method_oid == OID_AD_SIGNED_OBJECT
|
||||
&& ad.access_location.starts_with("rsync://")
|
||||
})
|
||||
.expect("manifest rsync signedObject")
|
||||
.access_location
|
||||
.clone(),
|
||||
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
|
||||
};
|
||||
let manifest_hash = hex::encode(sha2::Sha256::digest(&manifest_der));
|
||||
let mut raw = RawByHashEntry::from_bytes(manifest_hash.clone(), manifest_der.clone());
|
||||
raw.origin_uris.push(manifest_uri.clone());
|
||||
raw.object_type = Some("mft".to_string());
|
||||
raw.encoding = Some("der".to_string());
|
||||
store.put_raw_by_hash_entry(&raw).expect("put raw");
|
||||
|
||||
let projection = VcirCcrManifestProjection {
|
||||
manifest_rsync_uri: manifest_uri.clone(),
|
||||
manifest_sha256: sha2::Sha256::digest(&manifest_der).to_vec(),
|
||||
manifest_size: manifest_der.len() as u64,
|
||||
manifest_ee_aki: manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.authority_key_identifier
|
||||
.clone()
|
||||
.expect("manifest aki"),
|
||||
manifest_number_be: manifest.manifest.manifest_number.bytes_be.clone(),
|
||||
manifest_this_update: PackTime::from_utc_offset_datetime(manifest.manifest.this_update),
|
||||
manifest_sia_locations_der: match manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.expect("manifest sia")
|
||||
{
|
||||
SubjectInfoAccess::Ee(ee_sia) => vec![
|
||||
crate::ccr::manifest_location::select_manifest_signed_object_location(
|
||||
&manifest_uri,
|
||||
&ee_sia.access_descriptions,
|
||||
)
|
||||
.expect("select manifest signedObject"),
|
||||
],
|
||||
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
|
||||
},
|
||||
subordinate_skis: vec![vec![0x33; 20]],
|
||||
};
|
||||
|
||||
let vcir = ValidatedCaInstanceResult {
|
||||
manifest_rsync_uri: manifest_uri.clone(),
|
||||
parent_manifest_rsync_uri: None,
|
||||
tal_id: "apnic".to_string(),
|
||||
ca_subject_name: "CN=test".to_string(),
|
||||
ca_ski: "11".repeat(20),
|
||||
issuer_ski: "22".repeat(20),
|
||||
last_successful_validation_time: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.this_update,
|
||||
),
|
||||
current_manifest_rsync_uri: manifest_uri.clone(),
|
||||
current_crl_rsync_uri: format!("{manifest_uri}.crl"),
|
||||
validated_manifest_meta: ValidatedManifestMeta {
|
||||
validated_manifest_number: manifest.manifest.manifest_number.bytes_be.clone(),
|
||||
validated_manifest_this_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.this_update,
|
||||
),
|
||||
validated_manifest_next_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
},
|
||||
ccr_manifest_projection: projection.clone(),
|
||||
instance_gate: VcirInstanceGate {
|
||||
manifest_next_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
current_crl_next_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
self_ca_not_after: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
instance_effective_until: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
},
|
||||
child_entries: vec![VcirChildEntry {
|
||||
child_manifest_rsync_uri: "rsync://example.test/repo/child.mft".to_string(),
|
||||
child_cert_rsync_uri: "rsync://example.test/repo/child.cer".to_string(),
|
||||
child_cert_hash: "aa".repeat(32),
|
||||
child_ski: "33".repeat(20),
|
||||
child_rsync_base_uri: "rsync://example.test/repo/".to_string(),
|
||||
child_publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
|
||||
child_rrdp_notification_uri: None,
|
||||
child_effective_ip_resources: None,
|
||||
child_effective_as_resources: None,
|
||||
accepted_at_validation_time: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.this_update,
|
||||
),
|
||||
}],
|
||||
local_outputs: Vec::new(),
|
||||
related_artifacts: vec![VcirRelatedArtifact {
|
||||
artifact_role: VcirArtifactRole::Manifest,
|
||||
artifact_kind: VcirArtifactKind::Mft,
|
||||
uri: Some(manifest_uri.clone()),
|
||||
sha256: manifest_hash,
|
||||
object_type: Some("mft".to_string()),
|
||||
validation_status: VcirArtifactValidationStatus::Accepted,
|
||||
reject_reason: None,
|
||||
}],
|
||||
summary: VcirSummary {
|
||||
local_vrp_count: 0,
|
||||
local_aspa_count: 0,
|
||||
local_router_key_count: 0,
|
||||
child_count: 1,
|
||||
accepted_object_count: 1,
|
||||
rejected_object_count: 0,
|
||||
},
|
||||
audit_summary: VcirAuditSummary {
|
||||
failed_fetch_eligible: true,
|
||||
last_failed_fetch_reason: None,
|
||||
warning_count: 0,
|
||||
audit_flags: Vec::new(),
|
||||
},
|
||||
};
|
||||
|
||||
let vrps = vec![Vrp {
|
||||
asn: 64496,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv4,
|
||||
prefix_len: 8,
|
||||
addr: [10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
},
|
||||
max_length: 8,
|
||||
}];
|
||||
let aspas = vec![AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![64497],
|
||||
}];
|
||||
let router_keys = vec![RouterKeyPayload {
|
||||
as_id: 64496,
|
||||
ski: vec![0x11; 20],
|
||||
spki_der: vec![0x30, 0x00],
|
||||
source_object_uri: "rsync://example.test/repo/router.cer".to_string(),
|
||||
source_object_hash: hex::encode([0x11; 32]),
|
||||
source_ee_cert_hash: hex::encode([0x11; 32]),
|
||||
item_effective_until: PackTime::from_utc_offset_datetime(
|
||||
time::OffsetDateTime::now_utc() + time::Duration::hours(1),
|
||||
),
|
||||
}];
|
||||
(vcir, vrps, aspas, router_keys)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_finish_matches_builder_on_fresh_vcir_inputs() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(td.path()).expect("open rocksdb");
|
||||
let (vcir, vrps, aspas, router_keys) = sample_vcir_and_manifest(&store);
|
||||
store.put_vcir(&vcir).expect("put vcir");
|
||||
let trust_anchor = sample_trust_anchor();
|
||||
|
||||
let builder_ccr = build_ccr_from_run(
|
||||
&store,
|
||||
&[trust_anchor.clone()],
|
||||
&vrps,
|
||||
&aspas,
|
||||
&router_keys,
|
||||
time::OffsetDateTime::now_utc(),
|
||||
)
|
||||
.expect("build ccr from run");
|
||||
|
||||
let mut accumulator = CcrAccumulator::new(vec![trust_anchor]);
|
||||
accumulator
|
||||
.append_manifest_projection(&vcir.ccr_manifest_projection)
|
||||
.expect("append manifest projection");
|
||||
let accumulated_ccr = accumulator
|
||||
.finish(time::OffsetDateTime::now_utc(), &vrps, &aspas, &router_keys)
|
||||
.expect("finish accumulator");
|
||||
|
||||
assert_eq!(builder_ccr.mfts, accumulated_ccr.mfts);
|
||||
assert_eq!(builder_ccr.vrps, accumulated_ccr.vrps);
|
||||
assert_eq!(builder_ccr.vaps, accumulated_ccr.vaps);
|
||||
assert_eq!(builder_ccr.tas, accumulated_ccr.tas);
|
||||
assert_eq!(builder_ccr.rks, accumulated_ccr.rks);
|
||||
let ci = crate::ccr::model::CcrContentInfo::new(accumulated_ccr);
|
||||
verify_content_info(&ci).expect("verify accumulated ccr");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_sanitizes_historical_projection_with_rpki_notify() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(td.path()).expect("open rocksdb");
|
||||
let (mut vcir, _, _, _) = sample_vcir_and_manifest(&store);
|
||||
let notify_der =
|
||||
crate::ccr::manifest_location::encode_access_description_der(&AccessDescription {
|
||||
access_method_oid: OID_AD_RPKI_NOTIFY.to_string(),
|
||||
access_location: "https://rrdp.example.test/notification.xml".to_string(),
|
||||
})
|
||||
.expect("encode rpkiNotify");
|
||||
let expected_location = vcir.ccr_manifest_projection.manifest_sia_locations_der[0].clone();
|
||||
vcir.ccr_manifest_projection
|
||||
.manifest_sia_locations_der
|
||||
.push(notify_der);
|
||||
|
||||
let mut accumulator = CcrAccumulator::new(Vec::new());
|
||||
accumulator
|
||||
.append_manifest_projection(&vcir.ccr_manifest_projection)
|
||||
.expect("sanitize historical projection");
|
||||
let contribution = accumulator
|
||||
.manifests_by_hash
|
||||
.values()
|
||||
.next()
|
||||
.expect("manifest contribution");
|
||||
assert_eq!(contribution.locations_der, vec![expected_location]);
|
||||
}
|
||||
}
|
||||
975
crates/panda-rpki-validator/src/ccr/build.rs
Normal file
975
crates/panda-rpki-validator/src/ccr/build.rs
Normal file
@ -0,0 +1,975 @@
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
use serde::Serialize;
|
||||
use sha2::Digest;
|
||||
|
||||
use crate::ccr::encode::{
|
||||
encode_aspa_payload_state_payload_der, encode_manifest_state_payload_der,
|
||||
encode_roa_payload_state_payload_der, encode_router_key_state_payload_der,
|
||||
encode_trust_anchor_state_payload_der,
|
||||
};
|
||||
use crate::ccr::hash::compute_state_hash;
|
||||
use crate::ccr::manifest_location::select_manifest_signed_object_location;
|
||||
use crate::ccr::model::{
|
||||
AspaPayloadSet, AspaPayloadState, ManifestInstance, ManifestState, RoaPayloadSet,
|
||||
RoaPayloadState, RouterKey, RouterKeySet, RouterKeyState, TrustAnchorState,
|
||||
};
|
||||
use crate::data_model::manifest::ManifestObject;
|
||||
use crate::data_model::rc::SubjectInfoAccess;
|
||||
use crate::data_model::roa::RoaAfi;
|
||||
use crate::data_model::router_cert::BgpsecRouterCertificate;
|
||||
use crate::data_model::ta::TrustAnchor;
|
||||
use crate::storage::{RocksStore, ValidatedCaInstanceResult, VcirArtifactRole};
|
||||
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrBuildError {
|
||||
#[error("trust anchor set must not be empty")]
|
||||
EmptyTrustAnchors,
|
||||
|
||||
#[error("trust anchor certificate missing SubjectKeyIdentifier")]
|
||||
MissingTrustAnchorSki,
|
||||
|
||||
#[error("ROA payload state encoding failed: {0}")]
|
||||
RoaEncode(String),
|
||||
|
||||
#[error("ASPA payload state encoding failed: {0}")]
|
||||
AspaEncode(String),
|
||||
|
||||
#[error("TrustAnchor state encoding failed: {0}")]
|
||||
TrustAnchorEncode(String),
|
||||
|
||||
#[error("manifest artifact missing in VCIR: {0}")]
|
||||
MissingManifestArtifact(String),
|
||||
|
||||
#[error("manifest raw bytes missing in store for {manifest_rsync_uri}: {sha256_hex}")]
|
||||
MissingManifestRawBytes {
|
||||
manifest_rsync_uri: String,
|
||||
sha256_hex: String,
|
||||
},
|
||||
|
||||
#[error("manifest raw bytes load failed for {manifest_rsync_uri}: {detail}")]
|
||||
LoadManifestRawBytes {
|
||||
manifest_rsync_uri: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("manifest decode failed for {manifest_rsync_uri}: {detail}")]
|
||||
ManifestDecode {
|
||||
manifest_rsync_uri: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("manifest EE certificate missing AuthorityKeyIdentifier: {0}")]
|
||||
ManifestEeMissingAki(String),
|
||||
|
||||
#[error("manifest EE certificate missing Subject Information Access extension: {0}")]
|
||||
ManifestEeMissingSia(String),
|
||||
|
||||
#[error("manifest EE certificate SIA is not the EE form: {0}")]
|
||||
ManifestEeSiaWrongVariant(String),
|
||||
|
||||
#[error(
|
||||
"manifest EE certificate SIA location selection failed for {manifest_rsync_uri}: {detail}"
|
||||
)]
|
||||
ManifestSiaLocationSelection {
|
||||
manifest_rsync_uri: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("manifest child SKI is not valid lowercase hex or not 20 bytes: {0}")]
|
||||
InvalidChildSki(String),
|
||||
|
||||
#[error("manifest state encoding failed: {0}")]
|
||||
ManifestEncode(String),
|
||||
|
||||
#[error("duplicate manifest hash with conflicting content for URI: {0}")]
|
||||
DuplicateManifestHashConflict(String),
|
||||
|
||||
#[error("router key state encoding failed: {0}")]
|
||||
RouterKeyEncode(String),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct ManifestStateBuildBreakdown {
|
||||
pub vcir_count: usize,
|
||||
pub unique_manifest_count: usize,
|
||||
pub find_manifest_artifact_ms: u64,
|
||||
pub load_manifest_blob_ms: u64,
|
||||
pub decode_manifest_der_ms: u64,
|
||||
pub build_manifest_instance_ms: u64,
|
||||
pub dedup_manifest_instance_ms: u64,
|
||||
pub encode_manifest_state_ms: u64,
|
||||
pub total_ms: u64,
|
||||
}
|
||||
|
||||
pub fn build_roa_payload_state(vrps: &[Vrp]) -> Result<RoaPayloadState, CcrBuildError> {
|
||||
let mut grouped: BTreeMap<u32, BTreeSet<RoaPayloadKey>> = BTreeMap::new();
|
||||
for vrp in vrps {
|
||||
grouped
|
||||
.entry(vrp.asn)
|
||||
.or_default()
|
||||
.insert(RoaPayloadKey::from_vrp(vrp));
|
||||
}
|
||||
|
||||
let mut rps = Vec::with_capacity(grouped.len());
|
||||
for (asn, entries) in grouped {
|
||||
let mut families: BTreeMap<u16, Vec<RoaPayloadKey>> = BTreeMap::new();
|
||||
for entry in entries {
|
||||
families.entry(entry.afi).or_default().push(entry);
|
||||
}
|
||||
let mut ip_addr_blocks = Vec::with_capacity(families.len());
|
||||
for (afi, entries) in families {
|
||||
ip_addr_blocks.push(encode_roa_ip_address_family(afi, &entries));
|
||||
}
|
||||
rps.push(RoaPayloadSet {
|
||||
as_id: asn,
|
||||
ip_addr_blocks,
|
||||
});
|
||||
}
|
||||
|
||||
let payload_der = encode_roa_payload_state_payload_der(&rps)
|
||||
.map_err(|e| CcrBuildError::RoaEncode(e.to_string()))?;
|
||||
Ok(RoaPayloadState {
|
||||
rps,
|
||||
hash: compute_state_hash(&payload_der),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn build_aspa_payload_state(
|
||||
attestations: &[AspaAttestation],
|
||||
) -> Result<AspaPayloadState, CcrBuildError> {
|
||||
let mut grouped: BTreeMap<u32, BTreeSet<u32>> = BTreeMap::new();
|
||||
for attestation in attestations {
|
||||
grouped
|
||||
.entry(attestation.customer_as_id)
|
||||
.or_default()
|
||||
.extend(attestation.provider_as_ids.iter().copied());
|
||||
}
|
||||
|
||||
let aps: Vec<AspaPayloadSet> = grouped
|
||||
.into_iter()
|
||||
.map(|(customer_as_id, providers)| AspaPayloadSet {
|
||||
customer_as_id,
|
||||
providers: providers.into_iter().collect(),
|
||||
})
|
||||
.collect();
|
||||
|
||||
let payload_der = encode_aspa_payload_state_payload_der(&aps)
|
||||
.map_err(|e| CcrBuildError::AspaEncode(e.to_string()))?;
|
||||
Ok(AspaPayloadState {
|
||||
aps,
|
||||
hash: compute_state_hash(&payload_der),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn build_trust_anchor_state(
|
||||
trust_anchors: &[TrustAnchor],
|
||||
) -> Result<TrustAnchorState, CcrBuildError> {
|
||||
if trust_anchors.is_empty() {
|
||||
return Err(CcrBuildError::EmptyTrustAnchors);
|
||||
}
|
||||
let mut skis: BTreeSet<Vec<u8>> = BTreeSet::new();
|
||||
for ta in trust_anchors {
|
||||
let ski = ta
|
||||
.ta_certificate
|
||||
.rc_ca
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_key_identifier
|
||||
.clone()
|
||||
.ok_or(CcrBuildError::MissingTrustAnchorSki)?;
|
||||
skis.insert(ski);
|
||||
}
|
||||
let skis: Vec<Vec<u8>> = skis.into_iter().collect();
|
||||
let payload_der = encode_trust_anchor_state_payload_der(&skis)
|
||||
.map_err(|e| CcrBuildError::TrustAnchorEncode(e.to_string()))?;
|
||||
Ok(TrustAnchorState {
|
||||
skis,
|
||||
hash: compute_state_hash(&payload_der),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn build_manifest_state_from_vcirs(
|
||||
store: &RocksStore,
|
||||
vcirs: &[ValidatedCaInstanceResult],
|
||||
) -> Result<ManifestState, CcrBuildError> {
|
||||
build_manifest_state_from_vcirs_with_breakdown(store, vcirs).map(|(state, _)| state)
|
||||
}
|
||||
|
||||
pub fn build_manifest_state_from_vcirs_with_breakdown(
|
||||
store: &RocksStore,
|
||||
vcirs: &[ValidatedCaInstanceResult],
|
||||
) -> Result<(ManifestState, ManifestStateBuildBreakdown), CcrBuildError> {
|
||||
let total_started = std::time::Instant::now();
|
||||
let mut breakdown = ManifestStateBuildBreakdown {
|
||||
vcir_count: vcirs.len(),
|
||||
..ManifestStateBuildBreakdown::default()
|
||||
};
|
||||
let mut find_manifest_artifact_duration = std::time::Duration::ZERO;
|
||||
let mut load_manifest_blob_duration = std::time::Duration::ZERO;
|
||||
let mut decode_manifest_der_duration = std::time::Duration::ZERO;
|
||||
let mut build_manifest_instance_duration = std::time::Duration::ZERO;
|
||||
let mut dedup_manifest_instance_duration = std::time::Duration::ZERO;
|
||||
let mut mis_by_hash: BTreeMap<Vec<u8>, ManifestInstance> = BTreeMap::new();
|
||||
let mut most_recent_update = time::OffsetDateTime::UNIX_EPOCH;
|
||||
|
||||
for vcir in vcirs {
|
||||
let started = std::time::Instant::now();
|
||||
let manifest_artifact = vcir
|
||||
.related_artifacts
|
||||
.iter()
|
||||
.find(|artifact| {
|
||||
artifact.artifact_role == VcirArtifactRole::Manifest
|
||||
&& artifact.uri.as_deref() == Some(vcir.current_manifest_rsync_uri.as_str())
|
||||
})
|
||||
.ok_or_else(|| {
|
||||
CcrBuildError::MissingManifestArtifact(vcir.current_manifest_rsync_uri.clone())
|
||||
})?;
|
||||
find_manifest_artifact_duration += started.elapsed();
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let raw_bytes = store
|
||||
.get_blob_bytes(&manifest_artifact.sha256)
|
||||
.map_err(|e| CcrBuildError::LoadManifestRawBytes {
|
||||
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
|
||||
detail: e.to_string(),
|
||||
})?
|
||||
.ok_or_else(|| CcrBuildError::MissingManifestRawBytes {
|
||||
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
|
||||
sha256_hex: manifest_artifact.sha256.clone(),
|
||||
})?;
|
||||
load_manifest_blob_duration += started.elapsed();
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let manifest =
|
||||
ManifestObject::decode_der(&raw_bytes).map_err(|e| CcrBuildError::ManifestDecode {
|
||||
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
decode_manifest_der_duration += started.elapsed();
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let ee = &manifest.signed_object.signed_data.certificates[0].resource_cert;
|
||||
let aki = ee
|
||||
.tbs
|
||||
.extensions
|
||||
.authority_key_identifier
|
||||
.clone()
|
||||
.ok_or_else(|| {
|
||||
CcrBuildError::ManifestEeMissingAki(vcir.current_manifest_rsync_uri.clone())
|
||||
})?;
|
||||
let sia = ee
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.ok_or_else(|| {
|
||||
CcrBuildError::ManifestEeMissingSia(vcir.current_manifest_rsync_uri.clone())
|
||||
})?;
|
||||
let locations = match sia {
|
||||
SubjectInfoAccess::Ee(ee_sia) => vec![
|
||||
select_manifest_signed_object_location(
|
||||
&vcir.current_manifest_rsync_uri,
|
||||
&ee_sia.access_descriptions,
|
||||
)
|
||||
.map_err(|detail| CcrBuildError::ManifestSiaLocationSelection {
|
||||
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
|
||||
detail,
|
||||
})?,
|
||||
],
|
||||
SubjectInfoAccess::Ca(_) => {
|
||||
return Err(CcrBuildError::ManifestEeSiaWrongVariant(
|
||||
vcir.current_manifest_rsync_uri.clone(),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
let subordinates = collect_subordinate_ski_bytes(vcir)?;
|
||||
let this_update = vcir
|
||||
.validated_manifest_meta
|
||||
.validated_manifest_this_update
|
||||
.parse()
|
||||
.map_err(|e| CcrBuildError::ManifestDecode {
|
||||
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
|
||||
detail: format!("invalid validated_manifest_this_update: {e}"),
|
||||
})?;
|
||||
if this_update > most_recent_update {
|
||||
most_recent_update = this_update;
|
||||
}
|
||||
|
||||
let instance = ManifestInstance {
|
||||
hash: sha2::Sha256::digest(&raw_bytes).to_vec(),
|
||||
size: raw_bytes.len() as u64,
|
||||
aki,
|
||||
manifest_number: crate::data_model::common::BigUnsigned {
|
||||
bytes_be: vcir
|
||||
.validated_manifest_meta
|
||||
.validated_manifest_number
|
||||
.clone(),
|
||||
},
|
||||
this_update,
|
||||
locations,
|
||||
subordinates,
|
||||
};
|
||||
build_manifest_instance_duration += started.elapsed();
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
match mis_by_hash.get(instance.hash.as_slice()) {
|
||||
Some(existing) if existing != &instance => {
|
||||
return Err(CcrBuildError::DuplicateManifestHashConflict(
|
||||
vcir.current_manifest_rsync_uri.clone(),
|
||||
));
|
||||
}
|
||||
Some(_) => {}
|
||||
None => {
|
||||
mis_by_hash.insert(instance.hash.clone(), instance);
|
||||
}
|
||||
}
|
||||
dedup_manifest_instance_duration += started.elapsed();
|
||||
}
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let mis: Vec<ManifestInstance> = mis_by_hash.into_values().collect();
|
||||
let payload_der = encode_manifest_state_payload_der(&mis)
|
||||
.map_err(|e| CcrBuildError::ManifestEncode(e.to_string()))?;
|
||||
let state = ManifestState {
|
||||
mis,
|
||||
most_recent_update,
|
||||
hash: compute_state_hash(&payload_der),
|
||||
};
|
||||
breakdown.encode_manifest_state_ms = started.elapsed().as_millis() as u64;
|
||||
breakdown.find_manifest_artifact_ms = find_manifest_artifact_duration.as_millis() as u64;
|
||||
breakdown.load_manifest_blob_ms = load_manifest_blob_duration.as_millis() as u64;
|
||||
breakdown.decode_manifest_der_ms = decode_manifest_der_duration.as_millis() as u64;
|
||||
breakdown.build_manifest_instance_ms = build_manifest_instance_duration.as_millis() as u64;
|
||||
breakdown.dedup_manifest_instance_ms = dedup_manifest_instance_duration.as_millis() as u64;
|
||||
breakdown.unique_manifest_count = state.mis.len();
|
||||
breakdown.total_ms = total_started.elapsed().as_millis() as u64;
|
||||
Ok((state, breakdown))
|
||||
}
|
||||
|
||||
fn collect_subordinate_ski_bytes(
|
||||
vcir: &ValidatedCaInstanceResult,
|
||||
) -> Result<Vec<Vec<u8>>, CcrBuildError> {
|
||||
let mut skis = BTreeSet::new();
|
||||
for child in &vcir.child_entries {
|
||||
let bytes = hex::decode(&child.child_ski)
|
||||
.map_err(|_| CcrBuildError::InvalidChildSki(child.child_ski.clone()))?;
|
||||
if bytes.len() != 20 {
|
||||
return Err(CcrBuildError::InvalidChildSki(child.child_ski.clone()));
|
||||
}
|
||||
skis.insert(bytes);
|
||||
}
|
||||
Ok(skis.into_iter().collect())
|
||||
}
|
||||
|
||||
pub fn build_router_key_state(
|
||||
router_certs: &[BgpsecRouterCertificate],
|
||||
) -> Result<RouterKeyState, CcrBuildError> {
|
||||
let mut grouped: BTreeMap<u32, BTreeSet<RouterKey>> = BTreeMap::new();
|
||||
for cert in router_certs {
|
||||
let key = RouterKey {
|
||||
ski: cert.subject_key_identifier.clone(),
|
||||
spki_der: cert.spki_der.clone(),
|
||||
};
|
||||
for asn in &cert.asns {
|
||||
grouped.entry(*asn).or_default().insert(key.clone());
|
||||
}
|
||||
}
|
||||
|
||||
let rksets: Vec<RouterKeySet> = grouped
|
||||
.into_iter()
|
||||
.map(|(as_id, router_keys)| RouterKeySet {
|
||||
as_id,
|
||||
router_keys: router_keys.into_iter().collect(),
|
||||
})
|
||||
.collect();
|
||||
|
||||
let payload_der = encode_router_key_state_payload_der(&rksets)
|
||||
.map_err(|e| CcrBuildError::RouterKeyEncode(e.to_string()))?;
|
||||
Ok(RouterKeyState {
|
||||
rksets,
|
||||
hash: compute_state_hash(&payload_der),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn build_router_key_state_from_runtime(
|
||||
router_keys: &[RouterKeyPayload],
|
||||
) -> Result<RouterKeyState, CcrBuildError> {
|
||||
let mut grouped: BTreeMap<u32, BTreeSet<RouterKey>> = BTreeMap::new();
|
||||
for router_key in router_keys {
|
||||
grouped
|
||||
.entry(router_key.as_id)
|
||||
.or_default()
|
||||
.insert(RouterKey {
|
||||
ski: router_key.ski.clone(),
|
||||
spki_der: router_key.spki_der.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
let rksets: Vec<RouterKeySet> = grouped
|
||||
.into_iter()
|
||||
.map(|(as_id, router_keys)| RouterKeySet {
|
||||
as_id,
|
||||
router_keys: router_keys.into_iter().collect(),
|
||||
})
|
||||
.collect();
|
||||
|
||||
let payload_der = encode_router_key_state_payload_der(&rksets)
|
||||
.map_err(|e| CcrBuildError::RouterKeyEncode(e.to_string()))?;
|
||||
Ok(RouterKeyState {
|
||||
rksets,
|
||||
hash: compute_state_hash(&payload_der),
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
struct RoaPayloadKey {
|
||||
afi: u16,
|
||||
addr: Vec<u8>,
|
||||
prefix_len: u8,
|
||||
max_length: u8,
|
||||
}
|
||||
|
||||
impl RoaPayloadKey {
|
||||
fn from_vrp(vrp: &Vrp) -> Self {
|
||||
let afi = match vrp.prefix.afi {
|
||||
RoaAfi::Ipv4 => 1,
|
||||
RoaAfi::Ipv6 => 2,
|
||||
};
|
||||
Self {
|
||||
afi,
|
||||
addr: vrp.prefix.addr.to_vec(),
|
||||
prefix_len: vrp.prefix.prefix_len as u8,
|
||||
max_length: vrp.max_length as u8,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_roa_ip_address_family(afi: u16, entries: &[RoaPayloadKey]) -> Vec<u8> {
|
||||
let address_family = afi.to_be_bytes();
|
||||
let addresses = entries
|
||||
.iter()
|
||||
.map(encode_roa_ip_address)
|
||||
.collect::<Vec<_>>();
|
||||
encode_sequence(&[
|
||||
encode_octet_string(&address_family),
|
||||
encode_sequence(&addresses),
|
||||
])
|
||||
}
|
||||
|
||||
fn encode_roa_ip_address(entry: &RoaPayloadKey) -> Vec<u8> {
|
||||
let (unused_bits, content) = encode_prefix_bit_string(&entry.addr, entry.prefix_len);
|
||||
let mut fields = vec![encode_bit_string(unused_bits, &content)];
|
||||
if entry.max_length != entry.prefix_len {
|
||||
fields.push(encode_integer_u8(entry.max_length));
|
||||
}
|
||||
encode_sequence(&fields)
|
||||
}
|
||||
|
||||
fn encode_prefix_bit_string(addr: &[u8], prefix_len: u8) -> (u8, Vec<u8>) {
|
||||
if prefix_len == 0 {
|
||||
return (0, Vec::new());
|
||||
}
|
||||
let octets = ((prefix_len as usize) + 7) / 8;
|
||||
let mut content = addr[..octets].to_vec();
|
||||
let rem = prefix_len % 8;
|
||||
let unused = if rem == 0 { 0 } else { 8 - rem };
|
||||
if unused > 0 {
|
||||
let mask = 0xFFu8 << unused;
|
||||
let last = content.last_mut().expect("octets > 0");
|
||||
*last &= mask;
|
||||
}
|
||||
(unused, content)
|
||||
}
|
||||
|
||||
fn encode_integer_u8(v: u8) -> Vec<u8> {
|
||||
encode_integer_bytes(vec![v])
|
||||
}
|
||||
|
||||
fn encode_integer_bytes(mut bytes: Vec<u8>) -> Vec<u8> {
|
||||
if bytes.is_empty() {
|
||||
bytes.push(0);
|
||||
}
|
||||
if bytes[0] & 0x80 != 0 {
|
||||
bytes.insert(0, 0);
|
||||
}
|
||||
encode_tlv(0x02, bytes)
|
||||
}
|
||||
|
||||
fn encode_bit_string(unused_bits: u8, content: &[u8]) -> Vec<u8> {
|
||||
let mut value = Vec::with_capacity(content.len() + 1);
|
||||
value.push(unused_bits);
|
||||
value.extend_from_slice(content);
|
||||
encode_tlv(0x03, value)
|
||||
}
|
||||
|
||||
fn encode_octet_string(bytes: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0x04, bytes.to_vec())
|
||||
}
|
||||
|
||||
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
|
||||
let total_len: usize = elements.iter().map(Vec::len).sum();
|
||||
let mut buf = Vec::with_capacity(total_len);
|
||||
for element in elements {
|
||||
buf.extend_from_slice(element);
|
||||
}
|
||||
encode_tlv(0x30, buf)
|
||||
}
|
||||
|
||||
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(1 + 9 + value.len());
|
||||
out.push(tag);
|
||||
encode_length(value.len(), &mut out);
|
||||
out.extend_from_slice(&value);
|
||||
out
|
||||
}
|
||||
|
||||
fn encode_length(len: usize, out: &mut Vec<u8>) {
|
||||
if len < 0x80 {
|
||||
out.push(len as u8);
|
||||
return;
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
let mut value = len;
|
||||
while value > 0 {
|
||||
bytes.push((value & 0xFF) as u8);
|
||||
value >>= 8;
|
||||
}
|
||||
bytes.reverse();
|
||||
out.push(0x80 | (bytes.len() as u8));
|
||||
out.extend_from_slice(&bytes);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::decode::decode_content_info;
|
||||
use crate::ccr::encode::{
|
||||
encode_aspa_payload_state, encode_content_info, encode_trust_anchor_state,
|
||||
};
|
||||
use crate::ccr::model::{CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation};
|
||||
use crate::data_model::roa::{IpPrefix, RoaAfi};
|
||||
use crate::data_model::ta::TrustAnchor;
|
||||
use crate::data_model::tal::Tal;
|
||||
|
||||
fn sample_vrp_v4(asn: u32, a: [u8; 4], prefix_len: u16, max_length: u16) -> Vrp {
|
||||
Vrp {
|
||||
asn,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv4,
|
||||
prefix_len,
|
||||
addr: [a[0], a[1], a[2], a[3], 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
},
|
||||
max_length,
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_vrp_v6(asn: u32, a: [u8; 16], prefix_len: u16, max_length: u16) -> Vrp {
|
||||
Vrp {
|
||||
asn,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv6,
|
||||
prefix_len,
|
||||
addr: a,
|
||||
},
|
||||
max_length,
|
||||
}
|
||||
}
|
||||
|
||||
fn decode_family_block(block: &[u8]) -> (u16, Vec<(u8, Vec<u8>, Option<u8>)>) {
|
||||
let mut top = crate::data_model::common::DerReader::new(block);
|
||||
let mut seq = top.take_sequence().expect("family seq");
|
||||
assert!(top.is_empty());
|
||||
let afi_bytes = seq.take_octet_string().expect("afi octet string");
|
||||
let afi = u16::from_be_bytes([afi_bytes[0], afi_bytes[1]]);
|
||||
let mut addrs = seq.take_sequence().expect("addresses seq");
|
||||
let mut entries = Vec::new();
|
||||
while !addrs.is_empty() {
|
||||
let mut addr_seq = addrs.take_sequence().expect("roa ip addr seq");
|
||||
let (unused_bits, content) = addr_seq.take_bit_string().expect("bit string");
|
||||
let prefix_len = (content.len() * 8) as u8 - unused_bits;
|
||||
let max_len = if addr_seq.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(addr_seq.take_uint_u64().expect("maxlen") as u8)
|
||||
};
|
||||
entries.push((prefix_len, content.to_vec(), max_len));
|
||||
}
|
||||
(afi, entries)
|
||||
}
|
||||
|
||||
fn sample_trust_anchor(path_tal: &str, path_ta: &str) -> TrustAnchor {
|
||||
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let tal_bytes = std::fs::read(base.join(path_tal)).expect("read tal");
|
||||
let ta_der = std::fs::read(base.join(path_ta)).expect("read ta");
|
||||
let tal = Tal::decode_bytes(&tal_bytes).expect("decode tal");
|
||||
TrustAnchor::bind_der(tal, &ta_der, None).expect("bind ta")
|
||||
}
|
||||
|
||||
fn sample_router_cert(asns: Vec<u32>, ski_fill: u8, spki_fill: u8) -> BgpsecRouterCertificate {
|
||||
let ta = sample_trust_anchor(
|
||||
"tests/fixtures/tal/apnic-rfc7730-https.tal",
|
||||
"tests/fixtures/ta/apnic-ta.cer",
|
||||
);
|
||||
BgpsecRouterCertificate {
|
||||
raw_der: vec![0x01, 0x02],
|
||||
resource_cert: ta.ta_certificate.rc_ca,
|
||||
subject_key_identifier: vec![ski_fill; 20],
|
||||
spki_der: vec![0x30, 0x03, 0x03, 0x01, spki_fill],
|
||||
asns,
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_manifest_vcir(manifest_der: &[u8], child_ski_hex: &str) -> ValidatedCaInstanceResult {
|
||||
let manifest = ManifestObject::decode_der(manifest_der).expect("decode manifest fixture");
|
||||
let manifest_uri = match manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.expect("manifest sia")
|
||||
{
|
||||
SubjectInfoAccess::Ee(ee_sia) => ee_sia
|
||||
.access_descriptions
|
||||
.iter()
|
||||
.find(|ad| {
|
||||
ad.access_method_oid == crate::data_model::oid::OID_AD_SIGNED_OBJECT
|
||||
&& ad.access_location.starts_with("rsync://")
|
||||
})
|
||||
.expect("manifest rsync signedObject")
|
||||
.access_location
|
||||
.clone(),
|
||||
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
|
||||
};
|
||||
let hash = hex::encode(sha2::Sha256::digest(manifest_der));
|
||||
let now = manifest.manifest.this_update;
|
||||
let next = manifest.manifest.next_update;
|
||||
let projection = crate::storage::VcirCcrManifestProjection {
|
||||
manifest_rsync_uri: manifest_uri.clone(),
|
||||
manifest_sha256: sha2::Sha256::digest(manifest_der).to_vec(),
|
||||
manifest_size: manifest_der.len() as u64,
|
||||
manifest_ee_aki: manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.authority_key_identifier
|
||||
.clone()
|
||||
.expect("manifest aki"),
|
||||
manifest_number_be: manifest.manifest.manifest_number.bytes_be.clone(),
|
||||
manifest_this_update: crate::storage::PackTime::from_utc_offset_datetime(now),
|
||||
manifest_sia_locations_der: match manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.expect("manifest sia")
|
||||
{
|
||||
SubjectInfoAccess::Ee(ee_sia) => vec![
|
||||
select_manifest_signed_object_location(
|
||||
&manifest_uri,
|
||||
&ee_sia.access_descriptions,
|
||||
)
|
||||
.expect("select manifest signedObject"),
|
||||
],
|
||||
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
|
||||
},
|
||||
subordinate_skis: vec![hex::decode(child_ski_hex).expect("decode child ski")],
|
||||
};
|
||||
crate::storage::ValidatedCaInstanceResult {
|
||||
manifest_rsync_uri: manifest_uri.clone(),
|
||||
parent_manifest_rsync_uri: None,
|
||||
tal_id: "test-tal".to_string(),
|
||||
ca_subject_name: "CN=test".to_string(),
|
||||
ca_ski: "11".repeat(20),
|
||||
issuer_ski: "22".repeat(20),
|
||||
last_successful_validation_time: crate::storage::PackTime::from_utc_offset_datetime(
|
||||
now,
|
||||
),
|
||||
current_manifest_rsync_uri: manifest_uri.clone(),
|
||||
current_crl_rsync_uri: format!("{manifest_uri}.crl"),
|
||||
validated_manifest_meta: crate::storage::ValidatedManifestMeta {
|
||||
validated_manifest_number: manifest.manifest.manifest_number.bytes_be.clone(),
|
||||
validated_manifest_this_update: crate::storage::PackTime::from_utc_offset_datetime(
|
||||
now,
|
||||
),
|
||||
validated_manifest_next_update: crate::storage::PackTime::from_utc_offset_datetime(
|
||||
next,
|
||||
),
|
||||
},
|
||||
ccr_manifest_projection: projection,
|
||||
instance_gate: crate::storage::VcirInstanceGate {
|
||||
manifest_next_update: crate::storage::PackTime::from_utc_offset_datetime(next),
|
||||
current_crl_next_update: crate::storage::PackTime::from_utc_offset_datetime(next),
|
||||
self_ca_not_after: crate::storage::PackTime::from_utc_offset_datetime(next),
|
||||
instance_effective_until: crate::storage::PackTime::from_utc_offset_datetime(next),
|
||||
},
|
||||
child_entries: vec![crate::storage::VcirChildEntry {
|
||||
child_manifest_rsync_uri: format!("{manifest_uri}/child.mft"),
|
||||
child_cert_rsync_uri: format!("{manifest_uri}/child.cer"),
|
||||
child_cert_hash: "aa".repeat(32),
|
||||
child_ski: child_ski_hex.to_string(),
|
||||
child_rsync_base_uri: format!("{manifest_uri}/"),
|
||||
child_publication_point_rsync_uri: format!("{manifest_uri}/"),
|
||||
child_rrdp_notification_uri: None,
|
||||
child_effective_ip_resources: None,
|
||||
child_effective_as_resources: None,
|
||||
accepted_at_validation_time: crate::storage::PackTime::from_utc_offset_datetime(
|
||||
now,
|
||||
),
|
||||
}],
|
||||
local_outputs: Vec::new(),
|
||||
related_artifacts: vec![crate::storage::VcirRelatedArtifact {
|
||||
artifact_role: crate::storage::VcirArtifactRole::Manifest,
|
||||
artifact_kind: crate::storage::VcirArtifactKind::Mft,
|
||||
uri: Some(manifest_uri.clone()),
|
||||
sha256: hash,
|
||||
object_type: Some("mft".to_string()),
|
||||
validation_status: crate::storage::VcirArtifactValidationStatus::Accepted,
|
||||
reject_reason: None,
|
||||
}],
|
||||
summary: crate::storage::VcirSummary {
|
||||
local_vrp_count: 0,
|
||||
local_aspa_count: 0,
|
||||
local_router_key_count: 0,
|
||||
child_count: 1,
|
||||
accepted_object_count: 1,
|
||||
rejected_object_count: 0,
|
||||
},
|
||||
audit_summary: crate::storage::VcirAuditSummary {
|
||||
failed_fetch_eligible: true,
|
||||
last_failed_fetch_reason: None,
|
||||
warning_count: 0,
|
||||
audit_flags: Vec::new(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_manifest_state_from_vcirs_collects_current_manifests_and_hashes_payload() {
|
||||
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let manifest_a = std::fs::read(base.join("tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft")).expect("read manifest a");
|
||||
let manifest_b = std::fs::read(base.join(
|
||||
"tests/fixtures/repository/ca.rg.net/rpki/RGnet-OU/bW-_qXU9uNhGQz21NR2ansB8lr0.mft",
|
||||
))
|
||||
.expect("read manifest b");
|
||||
let vcir_a = sample_manifest_vcir(&manifest_a, &"33".repeat(20));
|
||||
let vcir_b = sample_manifest_vcir(&manifest_b, &"44".repeat(20));
|
||||
let store_dir = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(store_dir.path()).expect("open rocksdb");
|
||||
for (vcir, bytes) in [(&vcir_a, &manifest_a), (&vcir_b, &manifest_b)] {
|
||||
let artifact = &vcir.related_artifacts[0];
|
||||
let mut raw =
|
||||
crate::storage::RawByHashEntry::from_bytes(artifact.sha256.clone(), bytes.to_vec());
|
||||
raw.origin_uris
|
||||
.push(vcir.current_manifest_rsync_uri.clone());
|
||||
raw.object_type = Some("mft".to_string());
|
||||
raw.encoding = Some("der".to_string());
|
||||
store.put_raw_by_hash_entry(&raw).expect("put raw manifest");
|
||||
}
|
||||
|
||||
let state = build_manifest_state_from_vcirs(&store, &[vcir_a.clone(), vcir_b.clone()])
|
||||
.expect("build manifest state");
|
||||
assert_eq!(state.mis.len(), 2);
|
||||
assert!(state.mis[0].hash < state.mis[1].hash);
|
||||
let expected_subordinates = [
|
||||
hex::decode(vcir_a.child_entries[0].child_ski.clone()).unwrap(),
|
||||
hex::decode(vcir_b.child_entries[0].child_ski.clone()).unwrap(),
|
||||
];
|
||||
let actual_subordinates = state
|
||||
.mis
|
||||
.iter()
|
||||
.map(|mi| {
|
||||
assert_eq!(mi.subordinates.len(), 1);
|
||||
assert!(!mi.locations.is_empty());
|
||||
mi.subordinates[0].clone()
|
||||
})
|
||||
.collect::<std::collections::BTreeSet<_>>();
|
||||
let expected_subordinates = expected_subordinates
|
||||
.into_iter()
|
||||
.collect::<std::collections::BTreeSet<_>>();
|
||||
assert_eq!(actual_subordinates, expected_subordinates);
|
||||
let payload_der =
|
||||
encode_manifest_state_payload_der(&state.mis).expect("encode mis payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
let max_time = [
|
||||
vcir_a
|
||||
.validated_manifest_meta
|
||||
.validated_manifest_this_update
|
||||
.parse()
|
||||
.unwrap(),
|
||||
vcir_b
|
||||
.validated_manifest_meta
|
||||
.validated_manifest_this_update
|
||||
.parse()
|
||||
.unwrap(),
|
||||
]
|
||||
.into_iter()
|
||||
.max()
|
||||
.unwrap();
|
||||
assert_eq!(state.most_recent_update, max_time);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_manifest_state_from_vcirs_empty_uses_epoch() {
|
||||
let store_dir = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(store_dir.path()).expect("open rocksdb");
|
||||
let state = build_manifest_state_from_vcirs(&store, &[]).expect("empty manifest state");
|
||||
assert!(state.mis.is_empty());
|
||||
assert_eq!(state.most_recent_update, time::OffsetDateTime::UNIX_EPOCH);
|
||||
let payload_der =
|
||||
encode_manifest_state_payload_der(&state.mis).expect("encode mis payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_router_key_state_groups_dedupes_and_sorts() {
|
||||
let certs = vec![
|
||||
sample_router_cert(vec![64497], 0x21, 0xA1),
|
||||
sample_router_cert(vec![64496, 64497], 0x11, 0xA0),
|
||||
sample_router_cert(vec![64496], 0x11, 0xA0),
|
||||
sample_router_cert(vec![64496], 0x11, 0xA2),
|
||||
];
|
||||
let state = build_router_key_state(&certs).expect("build router key state");
|
||||
assert_eq!(state.rksets.len(), 2);
|
||||
assert_eq!(state.rksets[0].as_id, 64496);
|
||||
assert_eq!(state.rksets[1].as_id, 64497);
|
||||
assert_eq!(state.rksets[0].router_keys.len(), 2);
|
||||
assert_eq!(state.rksets[1].router_keys.len(), 2);
|
||||
assert!(state.rksets[0].router_keys[0].ski <= state.rksets[0].router_keys[1].ski);
|
||||
let payload_der =
|
||||
encode_router_key_state_payload_der(&state.rksets).expect("encode rk payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_router_key_state_empty_is_valid_and_hashes_empty_sequence() {
|
||||
let state = build_router_key_state(&[]).expect("empty router key state");
|
||||
assert!(state.rksets.is_empty());
|
||||
let payload_der =
|
||||
encode_router_key_state_payload_der(&state.rksets).expect("encode rk payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_roa_payload_state_groups_dedupes_and_sorts() {
|
||||
let vrps = vec![
|
||||
sample_vrp_v4(64497, [10, 1, 0, 0], 16, 16),
|
||||
sample_vrp_v4(64496, [10, 0, 0, 0], 8, 8),
|
||||
sample_vrp_v4(64496, [10, 0, 0, 0], 8, 8),
|
||||
sample_vrp_v4(64496, [10, 1, 0, 0], 16, 24),
|
||||
sample_vrp_v6(
|
||||
64496,
|
||||
[0x20, 0x01, 0x0d, 0xb8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
32,
|
||||
48,
|
||||
),
|
||||
];
|
||||
let state = build_roa_payload_state(&vrps).expect("build roa state");
|
||||
assert_eq!(state.rps.len(), 2);
|
||||
assert_eq!(state.rps[0].as_id, 64496);
|
||||
assert_eq!(state.rps[1].as_id, 64497);
|
||||
assert_eq!(state.rps[0].ip_addr_blocks.len(), 2);
|
||||
let (afi4, entries4) = decode_family_block(&state.rps[0].ip_addr_blocks[0]);
|
||||
let (afi6, entries6) = decode_family_block(&state.rps[0].ip_addr_blocks[1]);
|
||||
assert_eq!(afi4, 1);
|
||||
assert_eq!(afi6, 2);
|
||||
assert_eq!(entries4.len(), 2);
|
||||
assert_eq!(entries4[0], (8, vec![10], None));
|
||||
assert_eq!(entries4[1], (16, vec![10, 1], Some(24)));
|
||||
assert_eq!(entries6.len(), 1);
|
||||
assert_eq!(entries6[0], (32, vec![0x20, 0x01, 0x0d, 0xb8], Some(48)));
|
||||
let payload_der = encode_roa_payload_state_payload_der(&state.rps).expect("encode payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_roa_payload_state_empty_is_valid_and_hashes_empty_sequence() {
|
||||
let state = build_roa_payload_state(&[]).expect("empty roa state");
|
||||
assert!(state.rps.is_empty());
|
||||
let payload_der = encode_roa_payload_state_payload_der(&state.rps).expect("encode payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_aspa_payload_state_merges_and_sorts() {
|
||||
let aspas = vec![
|
||||
AspaAttestation {
|
||||
customer_as_id: 64497,
|
||||
provider_as_ids: vec![65002],
|
||||
},
|
||||
AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![65003, 65001],
|
||||
},
|
||||
AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![65002, 65001],
|
||||
},
|
||||
];
|
||||
let state = build_aspa_payload_state(&aspas).expect("build aspa state");
|
||||
assert_eq!(state.aps.len(), 2);
|
||||
assert_eq!(state.aps[0].customer_as_id, 64496);
|
||||
assert_eq!(state.aps[0].providers, vec![65001, 65002, 65003]);
|
||||
assert_eq!(state.aps[1].customer_as_id, 64497);
|
||||
let encoded = encode_aspa_payload_state(&state).expect("encode aspa state");
|
||||
let decoded = decode_content_info(
|
||||
&encode_content_info(&CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at: time::OffsetDateTime::now_utc(),
|
||||
mfts: None,
|
||||
vrps: None,
|
||||
vaps: Some(state.clone()),
|
||||
tas: None,
|
||||
rks: None,
|
||||
}))
|
||||
.expect("encode ccr"),
|
||||
)
|
||||
.expect("decode ccr");
|
||||
assert_eq!(decoded.content.vaps, Some(state));
|
||||
assert!(!encoded.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_trust_anchor_state_collects_sorted_unique_skis() {
|
||||
let apnic = sample_trust_anchor(
|
||||
"tests/fixtures/tal/apnic-rfc7730-https.tal",
|
||||
"tests/fixtures/ta/apnic-ta.cer",
|
||||
);
|
||||
let arin = sample_trust_anchor(
|
||||
"tests/fixtures/tal/arin.tal",
|
||||
"tests/fixtures/ta/arin-ta.cer",
|
||||
);
|
||||
let state = build_trust_anchor_state(&[apnic.clone(), arin.clone(), apnic])
|
||||
.expect("build ta state");
|
||||
assert_eq!(state.skis.len(), 2);
|
||||
assert!(state.skis[0] < state.skis[1]);
|
||||
let payload_der =
|
||||
encode_trust_anchor_state_payload_der(&state.skis).expect("encode ta payload");
|
||||
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
|
||||
let encoded = encode_trust_anchor_state(&state).expect("encode ta state");
|
||||
assert!(!encoded.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_trust_anchor_state_rejects_empty_and_missing_ski() {
|
||||
let err = build_trust_anchor_state(&[]).expect_err("empty TAs must fail");
|
||||
assert!(err.to_string().contains("trust anchor set"), "{err}");
|
||||
|
||||
let mut ta = sample_trust_anchor(
|
||||
"tests/fixtures/tal/apnic-rfc7730-https.tal",
|
||||
"tests/fixtures/ta/apnic-ta.cer",
|
||||
);
|
||||
ta.ta_certificate
|
||||
.rc_ca
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_key_identifier = None;
|
||||
let err = build_trust_anchor_state(&[ta]).expect_err("missing ski must fail");
|
||||
assert!(err.to_string().contains("SubjectKeyIdentifier"), "{err}");
|
||||
}
|
||||
}
|
||||
235
crates/panda-rpki-validator/src/ccr/compare_view.rs
Normal file
235
crates/panda-rpki-validator/src/ccr/compare_view.rs
Normal file
@ -0,0 +1,235 @@
|
||||
use std::collections::BTreeSet;
|
||||
use std::io::Write;
|
||||
use std::path::Path;
|
||||
|
||||
use crate::ccr::{CcrContentInfo, extract_vrp_rows};
|
||||
use crate::validation::objects::{AspaAttestation, Vrp};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub struct VrpCompareRow {
|
||||
pub asn: String,
|
||||
pub ip_prefix: String,
|
||||
pub max_length: String,
|
||||
pub trust_anchor: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub struct VapCompareRow {
|
||||
pub customer_asn: String,
|
||||
pub providers: String,
|
||||
pub trust_anchor: String,
|
||||
}
|
||||
|
||||
fn normalize_asn(asn: u32) -> String {
|
||||
format!("AS{asn}")
|
||||
}
|
||||
|
||||
pub fn canonical_vrp_prefix(prefix: &crate::data_model::roa::IpPrefix) -> String {
|
||||
let mut addr = prefix.addr_bytes().to_vec();
|
||||
let total_bits = match prefix.afi {
|
||||
crate::data_model::roa::RoaAfi::Ipv4 => 32usize,
|
||||
crate::data_model::roa::RoaAfi::Ipv6 => 128usize,
|
||||
};
|
||||
let keep = usize::from(prefix.prefix_len);
|
||||
for bit in keep..total_bits {
|
||||
let byte = bit / 8;
|
||||
let offset = 7 - (bit % 8);
|
||||
addr[byte] &= !(1u8 << offset);
|
||||
}
|
||||
match prefix.afi {
|
||||
crate::data_model::roa::RoaAfi::Ipv4 => {
|
||||
let ipv4 = std::net::Ipv4Addr::new(addr[0], addr[1], addr[2], addr[3]);
|
||||
format!("{ipv4}/{}", prefix.prefix_len)
|
||||
}
|
||||
crate::data_model::roa::RoaAfi::Ipv6 => {
|
||||
let mut bytes = [0u8; 16];
|
||||
bytes.copy_from_slice(&addr[..16]);
|
||||
let ipv6 = std::net::Ipv6Addr::from(bytes);
|
||||
format!("{ipv6}/{}", prefix.prefix_len)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn build_vrp_compare_rows(vrps: &[Vrp]) -> BTreeSet<VrpCompareRow> {
|
||||
vrps.iter()
|
||||
.map(|vrp| VrpCompareRow {
|
||||
asn: normalize_asn(vrp.asn),
|
||||
ip_prefix: canonical_vrp_prefix(&vrp.prefix),
|
||||
max_length: vrp.max_length.to_string(),
|
||||
trust_anchor: "unknown".to_string(),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn build_vap_compare_rows(aspas: &[AspaAttestation]) -> BTreeSet<VapCompareRow> {
|
||||
aspas
|
||||
.iter()
|
||||
.map(|aspa| {
|
||||
let mut providers = aspa.provider_as_ids.iter().copied().collect::<Vec<_>>();
|
||||
providers.sort_unstable();
|
||||
providers.dedup();
|
||||
VapCompareRow {
|
||||
customer_asn: normalize_asn(aspa.customer_as_id),
|
||||
providers: providers
|
||||
.into_iter()
|
||||
.map(normalize_asn)
|
||||
.collect::<Vec<_>>()
|
||||
.join(";"),
|
||||
trust_anchor: "unknown".to_string(),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn decode_ccr_compare_views(
|
||||
content_info: &CcrContentInfo,
|
||||
) -> Result<(BTreeSet<VrpCompareRow>, BTreeSet<VapCompareRow>), String> {
|
||||
let vrps = extract_vrp_rows(content_info)
|
||||
.map_err(|e| format!("extract vrp rows from ccr failed: {e}"))?
|
||||
.into_iter()
|
||||
.map(|(asn, prefix, max_length)| VrpCompareRow {
|
||||
asn: normalize_asn(asn),
|
||||
ip_prefix: prefix,
|
||||
max_length: max_length.to_string(),
|
||||
trust_anchor: "unknown".to_string(),
|
||||
})
|
||||
.collect::<BTreeSet<_>>();
|
||||
|
||||
let vaps = content_info
|
||||
.content
|
||||
.vaps
|
||||
.as_ref()
|
||||
.map(|state| {
|
||||
state
|
||||
.aps
|
||||
.iter()
|
||||
.map(|vap| VapCompareRow {
|
||||
customer_asn: normalize_asn(vap.customer_as_id),
|
||||
providers: vap
|
||||
.providers
|
||||
.iter()
|
||||
.copied()
|
||||
.map(normalize_asn)
|
||||
.collect::<Vec<_>>()
|
||||
.join(";"),
|
||||
trust_anchor: "unknown".to_string(),
|
||||
})
|
||||
.collect::<BTreeSet<_>>()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
|
||||
Ok((vrps, vaps))
|
||||
}
|
||||
|
||||
pub fn write_vrp_csv(path: &Path, rows: &BTreeSet<VrpCompareRow>) -> Result<(), String> {
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
|
||||
}
|
||||
let mut file = std::io::BufWriter::new(
|
||||
std::fs::File::create(path)
|
||||
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?,
|
||||
);
|
||||
writeln!(file, "ASN,IP Prefix,Max Length,Trust Anchor").map_err(|e| e.to_string())?;
|
||||
for row in rows {
|
||||
writeln!(
|
||||
file,
|
||||
"{},{},{},{}",
|
||||
row.asn, row.ip_prefix, row.max_length, row.trust_anchor
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn write_vap_csv(path: &Path, rows: &BTreeSet<VapCompareRow>) -> Result<(), String> {
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
|
||||
}
|
||||
let mut file = std::io::BufWriter::new(
|
||||
std::fs::File::create(path)
|
||||
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?,
|
||||
);
|
||||
writeln!(file, "Customer ASN,Providers,Trust Anchor").map_err(|e| e.to_string())?;
|
||||
for row in rows {
|
||||
writeln!(
|
||||
file,
|
||||
"{},{},{}",
|
||||
row.customer_asn, row.providers, row.trust_anchor
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::{
|
||||
CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation,
|
||||
build_aspa_payload_state, build_roa_payload_state,
|
||||
};
|
||||
use crate::data_model::roa::{IpPrefix, RoaAfi};
|
||||
|
||||
#[test]
|
||||
fn build_vap_compare_rows_sorts_and_dedups_providers() {
|
||||
let rows = build_vap_compare_rows(&[AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![64498, 64497, 64498],
|
||||
}]);
|
||||
let row = rows.iter().next().expect("one row");
|
||||
assert_eq!(row.customer_asn, "AS64496");
|
||||
assert_eq!(row.providers, "AS64497;AS64498");
|
||||
assert_eq!(row.trust_anchor, "unknown");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_ccr_compare_views_extracts_vrps_and_vaps() {
|
||||
let vrps = build_roa_payload_state(&[Vrp {
|
||||
asn: 64496,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv4,
|
||||
prefix_len: 24,
|
||||
addr: [192, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
},
|
||||
max_length: 24,
|
||||
}])
|
||||
.expect("build vrps");
|
||||
let vaps = build_aspa_payload_state(&[AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![64497],
|
||||
}])
|
||||
.expect("build vaps");
|
||||
let content = CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at: time::OffsetDateTime::now_utc(),
|
||||
mfts: None,
|
||||
vrps: Some(vrps),
|
||||
vaps: Some(vaps),
|
||||
tas: None,
|
||||
rks: None,
|
||||
});
|
||||
let (vrp_rows, vap_rows) =
|
||||
decode_ccr_compare_views(&content).expect("decode compare views");
|
||||
assert_eq!(vrp_rows.len(), 1);
|
||||
assert_eq!(vap_rows.len(), 1);
|
||||
assert_eq!(vap_rows.iter().next().unwrap().providers, "AS64497");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_vrp_compare_rows_canonicalizes_ipv6_prefix_text() {
|
||||
let rows = build_vrp_compare_rows(&[Vrp {
|
||||
asn: 64496,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv6,
|
||||
prefix_len: 32,
|
||||
addr: [0x20, 0x01, 0x0d, 0xb8, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
|
||||
},
|
||||
max_length: 48,
|
||||
}]);
|
||||
let row = rows.iter().next().expect("row");
|
||||
assert_eq!(row.ip_prefix, "2001:db8::/32");
|
||||
}
|
||||
}
|
||||
541
crates/panda-rpki-validator/src/ccr/decode.rs
Normal file
541
crates/panda-rpki-validator/src/ccr/decode.rs
Normal file
@ -0,0 +1,541 @@
|
||||
use crate::ccr::model::{
|
||||
AspaPayloadSet, AspaPayloadState, CCR_VERSION_V0, CcrContentInfo, CcrDigestAlgorithm,
|
||||
ManifestInstance, ManifestState, RoaPayloadSet, RoaPayloadState, RouterKey, RouterKeySet,
|
||||
RouterKeyState, RpkiCanonicalCacheRepresentation, TrustAnchorState,
|
||||
};
|
||||
use crate::data_model::common::{BigUnsigned, DerReader};
|
||||
use crate::data_model::oid::{OID_CT_RPKI_CCR, OID_CT_RPKI_CCR_RAW, OID_SHA256, OID_SHA256_RAW};
|
||||
use der_parser::der::parse_der_oid;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrDecodeError {
|
||||
#[error("DER parse error: {0}")]
|
||||
Parse(String),
|
||||
|
||||
#[error("unexpected contentType OID: expected {expected}, got {actual}")]
|
||||
UnexpectedContentType {
|
||||
expected: &'static str,
|
||||
actual: String,
|
||||
},
|
||||
|
||||
#[error("unexpected digest algorithm OID: expected {expected}, got {actual}")]
|
||||
UnexpectedDigestAlgorithm {
|
||||
expected: &'static str,
|
||||
actual: String,
|
||||
},
|
||||
|
||||
#[error("CCR model validation failed after decode: {0}")]
|
||||
Validate(String),
|
||||
}
|
||||
|
||||
pub fn decode_content_info(der: &[u8]) -> Result<CcrContentInfo, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ContentInfo".into(),
|
||||
));
|
||||
}
|
||||
let content_type_raw = seq.take_tag(0x06).map_err(CcrDecodeError::Parse)?;
|
||||
if content_type_raw != OID_CT_RPKI_CCR_RAW {
|
||||
return Err(CcrDecodeError::UnexpectedContentType {
|
||||
expected: OID_CT_RPKI_CCR,
|
||||
actual: oid_string(content_type_raw)?,
|
||||
});
|
||||
}
|
||||
let inner = seq.take_tag(0xA0).map_err(CcrDecodeError::Parse)?;
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing fields in ContentInfo".into(),
|
||||
));
|
||||
}
|
||||
let content = decode_ccr(inner)?;
|
||||
let ci = CcrContentInfo::new(content);
|
||||
ci.validate().map_err(CcrDecodeError::Validate)?;
|
||||
Ok(ci)
|
||||
}
|
||||
|
||||
pub fn decode_ccr(der: &[u8]) -> Result<RpkiCanonicalCacheRepresentation, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse("trailing bytes after CCR".into()));
|
||||
}
|
||||
|
||||
let version = if !seq.is_empty() && seq.peek_tag().map_err(CcrDecodeError::Parse)? == 0xA0 {
|
||||
let explicit = seq.take_tag(0xA0).map_err(CcrDecodeError::Parse)?;
|
||||
let mut inner = DerReader::new(explicit);
|
||||
let version = inner.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
|
||||
if !inner.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes inside CCR version EXPLICIT".into(),
|
||||
));
|
||||
}
|
||||
version
|
||||
} else {
|
||||
CCR_VERSION_V0
|
||||
};
|
||||
|
||||
let hash_alg = decode_digest_algorithm(seq.take_sequence().map_err(CcrDecodeError::Parse)?)?;
|
||||
let produced_at = parse_generalized_time(seq.take_tag(0x18).map_err(CcrDecodeError::Parse)?)?;
|
||||
|
||||
let mut mfts = None;
|
||||
let mut vrps = None;
|
||||
let mut vaps = None;
|
||||
let mut tas = None;
|
||||
let mut rks = None;
|
||||
while !seq.is_empty() {
|
||||
let tag = seq.peek_tag().map_err(CcrDecodeError::Parse)?;
|
||||
let (tag_read, value) = seq.take_any().map_err(CcrDecodeError::Parse)?;
|
||||
debug_assert_eq!(tag, tag_read);
|
||||
match tag {
|
||||
0xA1 => mfts = Some(decode_manifest_state(value)?),
|
||||
0xA2 => vrps = Some(decode_roa_payload_state(value)?),
|
||||
0xA3 => vaps = Some(decode_aspa_payload_state(value)?),
|
||||
0xA4 => tas = Some(decode_trust_anchor_state(value)?),
|
||||
0xA5 => rks = Some(decode_router_key_state(value)?),
|
||||
_ => {
|
||||
return Err(CcrDecodeError::Parse(format!(
|
||||
"unexpected CCR field tag 0x{tag:02X}"
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let ccr = RpkiCanonicalCacheRepresentation {
|
||||
version,
|
||||
hash_alg,
|
||||
produced_at,
|
||||
mfts,
|
||||
vrps,
|
||||
vaps,
|
||||
tas,
|
||||
rks,
|
||||
};
|
||||
ccr.validate().map_err(CcrDecodeError::Validate)?;
|
||||
Ok(ccr)
|
||||
}
|
||||
|
||||
fn decode_manifest_state(explicit_der: &[u8]) -> Result<ManifestState, CcrDecodeError> {
|
||||
let mut outer = DerReader::new(explicit_der);
|
||||
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !outer.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ManifestState".into(),
|
||||
));
|
||||
}
|
||||
let mis_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut mis_reader = DerReader::new(mis_der);
|
||||
let mut mis = Vec::new();
|
||||
while !mis_reader.is_empty() {
|
||||
let (_tag, full, _value) = mis_reader.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
mis.push(decode_manifest_instance(full)?);
|
||||
}
|
||||
let most_recent_update =
|
||||
parse_generalized_time(seq.take_tag(0x18).map_err(CcrDecodeError::Parse)?)?;
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing fields in ManifestState".into(),
|
||||
));
|
||||
}
|
||||
Ok(ManifestState {
|
||||
mis,
|
||||
most_recent_update,
|
||||
hash,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_manifest_instance(der: &[u8]) -> Result<ManifestInstance, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ManifestInstance".into(),
|
||||
));
|
||||
}
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let size = seq.take_uint_u64().map_err(CcrDecodeError::Parse)?;
|
||||
let aki = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let manifest_number = decode_big_unsigned(seq.take_tag(0x02).map_err(CcrDecodeError::Parse)?)?;
|
||||
let this_update = parse_generalized_time(seq.take_tag(0x18).map_err(CcrDecodeError::Parse)?)?;
|
||||
let locations_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut locations_reader = DerReader::new(locations_der);
|
||||
let mut locations = Vec::new();
|
||||
while !locations_reader.is_empty() {
|
||||
let (_tag, full, _value) = locations_reader
|
||||
.take_any_full()
|
||||
.map_err(CcrDecodeError::Parse)?;
|
||||
locations.push(full.to_vec());
|
||||
}
|
||||
let subordinates = if !seq.is_empty() {
|
||||
let subordinate_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(subordinate_der);
|
||||
let mut out = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
out.push(
|
||||
reader
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec(),
|
||||
);
|
||||
}
|
||||
out
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
Ok(ManifestInstance {
|
||||
hash,
|
||||
size,
|
||||
aki,
|
||||
manifest_number,
|
||||
this_update,
|
||||
locations,
|
||||
subordinates,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_roa_payload_state(explicit_der: &[u8]) -> Result<RoaPayloadState, CcrDecodeError> {
|
||||
let mut outer = DerReader::new(explicit_der);
|
||||
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !outer.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ROAPayloadState".into(),
|
||||
));
|
||||
}
|
||||
let payload_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(payload_der);
|
||||
let mut rps = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
rps.push(decode_roa_payload_set(full)?);
|
||||
}
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
Ok(RoaPayloadState { rps, hash })
|
||||
}
|
||||
|
||||
fn decode_roa_payload_set(der: &[u8]) -> Result<RoaPayloadSet, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ROAPayloadSet".into(),
|
||||
));
|
||||
}
|
||||
let as_id = seq.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
|
||||
let blocks_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(blocks_der);
|
||||
let mut ip_addr_blocks = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
ip_addr_blocks.push(full.to_vec());
|
||||
}
|
||||
Ok(RoaPayloadSet {
|
||||
as_id,
|
||||
ip_addr_blocks,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_aspa_payload_state(explicit_der: &[u8]) -> Result<AspaPayloadState, CcrDecodeError> {
|
||||
let mut outer = DerReader::new(explicit_der);
|
||||
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !outer.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ASPAPayloadState".into(),
|
||||
));
|
||||
}
|
||||
let payload_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(payload_der);
|
||||
let mut aps = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
aps.push(decode_aspa_payload_set(full)?);
|
||||
}
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
Ok(AspaPayloadState { aps, hash })
|
||||
}
|
||||
|
||||
fn decode_aspa_payload_set(der: &[u8]) -> Result<AspaPayloadSet, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after ASPAPayloadSet".into(),
|
||||
));
|
||||
}
|
||||
let customer_as_id = seq.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
|
||||
let providers_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(providers_der);
|
||||
let mut providers = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
providers.push(reader.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32);
|
||||
}
|
||||
Ok(AspaPayloadSet {
|
||||
customer_as_id,
|
||||
providers,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_trust_anchor_state(explicit_der: &[u8]) -> Result<TrustAnchorState, CcrDecodeError> {
|
||||
let mut outer = DerReader::new(explicit_der);
|
||||
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !outer.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after TrustAnchorState".into(),
|
||||
));
|
||||
}
|
||||
let skis_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(skis_der);
|
||||
let mut skis = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
skis.push(
|
||||
reader
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec(),
|
||||
);
|
||||
}
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
Ok(TrustAnchorState { skis, hash })
|
||||
}
|
||||
|
||||
fn decode_router_key_state(explicit_der: &[u8]) -> Result<RouterKeyState, CcrDecodeError> {
|
||||
let mut outer = DerReader::new(explicit_der);
|
||||
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !outer.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after RouterKeyState".into(),
|
||||
));
|
||||
}
|
||||
let sets_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(sets_der);
|
||||
let mut rksets = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
rksets.push(decode_router_key_set(full)?);
|
||||
}
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
Ok(RouterKeyState { rksets, hash })
|
||||
}
|
||||
|
||||
fn decode_router_key_set(der: &[u8]) -> Result<RouterKeySet, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after RouterKeySet".into(),
|
||||
));
|
||||
}
|
||||
let as_id = seq.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
|
||||
let keys_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
|
||||
let mut reader = DerReader::new(keys_der);
|
||||
let mut router_keys = Vec::new();
|
||||
while !reader.is_empty() {
|
||||
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
router_keys.push(decode_router_key(full)?);
|
||||
}
|
||||
Ok(RouterKeySet { as_id, router_keys })
|
||||
}
|
||||
|
||||
fn decode_router_key(der: &[u8]) -> Result<RouterKey, CcrDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing bytes after RouterKey".into(),
|
||||
));
|
||||
}
|
||||
let ski = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let (_tag, full, _value) = seq.take_any_full().map_err(CcrDecodeError::Parse)?;
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrDecodeError::Parse("trailing fields in RouterKey".into()));
|
||||
}
|
||||
Ok(RouterKey {
|
||||
ski,
|
||||
spki_der: full.to_vec(),
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_digest_algorithm(mut seq: DerReader<'_>) -> Result<CcrDigestAlgorithm, CcrDecodeError> {
|
||||
let oid_raw = seq.take_tag(0x06).map_err(CcrDecodeError::Parse)?;
|
||||
if oid_raw != OID_SHA256_RAW {
|
||||
return Err(CcrDecodeError::UnexpectedDigestAlgorithm {
|
||||
expected: OID_SHA256,
|
||||
actual: oid_string(oid_raw)?,
|
||||
});
|
||||
}
|
||||
if !seq.is_empty() {
|
||||
let tag = seq.peek_tag().map_err(CcrDecodeError::Parse)?;
|
||||
if tag == 0x05 {
|
||||
let null = seq.take_tag(0x05).map_err(CcrDecodeError::Parse)?;
|
||||
if !null.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"AlgorithmIdentifier NULL parameters must be empty".into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"trailing fields in DigestAlgorithmIdentifier".into(),
|
||||
));
|
||||
}
|
||||
Ok(CcrDigestAlgorithm::Sha256)
|
||||
}
|
||||
|
||||
fn oid_string(raw_body: &[u8]) -> Result<String, CcrDecodeError> {
|
||||
let der = {
|
||||
let mut out = Vec::with_capacity(raw_body.len() + 2);
|
||||
out.push(0x06);
|
||||
if raw_body.len() < 0x80 {
|
||||
out.push(raw_body.len() as u8);
|
||||
} else {
|
||||
return Err(CcrDecodeError::Parse("OID too long".into()));
|
||||
}
|
||||
out.extend_from_slice(raw_body);
|
||||
out
|
||||
};
|
||||
let (_rem, oid) = parse_der_oid(&der).map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
|
||||
let oid = oid
|
||||
.as_oid_val()
|
||||
.map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
|
||||
Ok(oid.to_string())
|
||||
}
|
||||
|
||||
fn parse_generalized_time(bytes: &[u8]) -> Result<time::OffsetDateTime, CcrDecodeError> {
|
||||
let s = std::str::from_utf8(bytes).map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
|
||||
if s.len() != 15 || !s.ends_with('Z') {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"GeneralizedTime must be YYYYMMDDHHMMSSZ".into(),
|
||||
));
|
||||
}
|
||||
let parse = |range: std::ops::Range<usize>| -> Result<u32, CcrDecodeError> {
|
||||
s[range]
|
||||
.parse::<u32>()
|
||||
.map_err(|e| CcrDecodeError::Parse(e.to_string()))
|
||||
};
|
||||
let year = parse(0..4)? as i32;
|
||||
let month = parse(4..6)? as u8;
|
||||
let day = parse(6..8)? as u8;
|
||||
let hour = parse(8..10)? as u8;
|
||||
let minute = parse(10..12)? as u8;
|
||||
let second = parse(12..14)? as u8;
|
||||
let month = time::Month::try_from(month).map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
|
||||
let date = time::Date::from_calendar_date(year, month, day)
|
||||
.map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
|
||||
let timev = time::Time::from_hms(hour, minute, second)
|
||||
.map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
|
||||
Ok(time::PrimitiveDateTime::new(date, timev).assume_utc())
|
||||
}
|
||||
|
||||
fn decode_big_unsigned(bytes: &[u8]) -> Result<BigUnsigned, CcrDecodeError> {
|
||||
if bytes.is_empty() {
|
||||
return Err(CcrDecodeError::Parse("INTEGER has empty content".into()));
|
||||
}
|
||||
if bytes[0] & 0x80 != 0 {
|
||||
return Err(CcrDecodeError::Parse("INTEGER must be non-negative".into()));
|
||||
}
|
||||
if bytes.len() > 1 && bytes[0] == 0x00 && (bytes[1] & 0x80) == 0 {
|
||||
return Err(CcrDecodeError::Parse(
|
||||
"INTEGER not minimally encoded".into(),
|
||||
));
|
||||
}
|
||||
let bytes_be = if bytes.len() > 1 && bytes[0] == 0x00 {
|
||||
bytes[1..].to_vec()
|
||||
} else {
|
||||
bytes.to_vec()
|
||||
};
|
||||
Ok(BigUnsigned { bytes_be })
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "full"))]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::encode::{encode_content_info, encode_manifest_state_payload_der};
|
||||
use crate::ccr::hash::compute_state_hash;
|
||||
use crate::ccr::manifest_location::encode_access_description_der;
|
||||
use crate::ccr::model::{
|
||||
CcrContentInfo, CcrDigestAlgorithm, ManifestState, RpkiCanonicalCacheRepresentation,
|
||||
};
|
||||
use crate::data_model::oid::{OID_AD_RPKI_NOTIFY, OID_AD_SIGNED_OBJECT};
|
||||
use crate::data_model::rc::AccessDescription;
|
||||
|
||||
#[test]
|
||||
fn generic_decoder_preserves_multiple_manifest_locations() {
|
||||
let signed_object = encode_access_description_der(&AccessDescription {
|
||||
access_method_oid: OID_AD_SIGNED_OBJECT.to_string(),
|
||||
access_location: "rsync://example.test/repository/current.mft".to_string(),
|
||||
})
|
||||
.expect("encode signedObject");
|
||||
let rpki_notify = encode_access_description_der(&AccessDescription {
|
||||
access_method_oid: OID_AD_RPKI_NOTIFY.to_string(),
|
||||
access_location: "https://rrdp.example.test/notification.xml".to_string(),
|
||||
})
|
||||
.expect("encode rpkiNotify");
|
||||
let manifest = ManifestInstance {
|
||||
hash: vec![0x11; 32],
|
||||
size: 1024,
|
||||
aki: vec![0x22; 20],
|
||||
manifest_number: BigUnsigned { bytes_be: vec![1] },
|
||||
this_update: time::OffsetDateTime::parse(
|
||||
"2026-07-20T00:00:00Z",
|
||||
&time::format_description::well_known::Rfc3339,
|
||||
)
|
||||
.expect("parse time"),
|
||||
locations: vec![signed_object, rpki_notify],
|
||||
subordinates: Vec::new(),
|
||||
};
|
||||
let manifest_payload = encode_manifest_state_payload_der(&[manifest.clone()])
|
||||
.expect("encode manifest payload");
|
||||
let content = CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at: time::OffsetDateTime::parse(
|
||||
"2026-07-20T00:00:00Z",
|
||||
&time::format_description::well_known::Rfc3339,
|
||||
)
|
||||
.expect("parse time"),
|
||||
mfts: Some(ManifestState {
|
||||
mis: vec![manifest],
|
||||
most_recent_update: time::OffsetDateTime::parse(
|
||||
"2026-07-20T00:00:00Z",
|
||||
&time::format_description::well_known::Rfc3339,
|
||||
)
|
||||
.expect("parse time"),
|
||||
hash: compute_state_hash(&manifest_payload),
|
||||
}),
|
||||
vrps: None,
|
||||
vaps: None,
|
||||
tas: None,
|
||||
rks: None,
|
||||
});
|
||||
|
||||
let encoded = encode_content_info(&content).expect("encode CCR");
|
||||
let decoded = decode_content_info(&encoded).expect("decode CCR");
|
||||
assert_eq!(decoded.content.mfts.unwrap().mis[0].locations.len(), 2);
|
||||
}
|
||||
}
|
||||
112
crates/panda-rpki-validator/src/ccr/dump.rs
Normal file
112
crates/panda-rpki-validator/src/ccr/dump.rs
Normal file
@ -0,0 +1,112 @@
|
||||
use crate::ccr::decode::{CcrDecodeError, decode_content_info};
|
||||
use serde_json::json;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrDumpError {
|
||||
#[error("CCR decode failed: {0}")]
|
||||
Decode(#[from] CcrDecodeError),
|
||||
|
||||
#[error("format time failed: {0}")]
|
||||
FormatTime(String),
|
||||
}
|
||||
|
||||
pub fn dump_content_info_json_value(der: &[u8]) -> Result<serde_json::Value, CcrDumpError> {
|
||||
let content_info = decode_content_info(der)?;
|
||||
dump_content_info_json(&content_info)
|
||||
}
|
||||
|
||||
pub fn dump_content_info_json(
|
||||
content_info: &crate::ccr::model::CcrContentInfo,
|
||||
) -> Result<serde_json::Value, CcrDumpError> {
|
||||
let produced_at = content_info
|
||||
.content
|
||||
.produced_at
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&time::format_description::well_known::Rfc3339)
|
||||
.map_err(|e| CcrDumpError::FormatTime(e.to_string()))?;
|
||||
|
||||
let mfts = content_info.content.mfts.as_ref().map(|state| {
|
||||
json!({
|
||||
"present": true,
|
||||
"manifest_instances": state.mis.len(),
|
||||
"most_recent_update_rfc3339_utc": state.most_recent_update.to_offset(time::UtcOffset::UTC).format(&time::format_description::well_known::Rfc3339).unwrap(),
|
||||
"hash_hex": hex::encode(&state.hash),
|
||||
})
|
||||
}).unwrap_or_else(|| json!({"present": false}));
|
||||
|
||||
let vrps_total = content_info
|
||||
.content
|
||||
.vrps
|
||||
.as_ref()
|
||||
.map(|state| {
|
||||
state
|
||||
.rps
|
||||
.iter()
|
||||
.map(|set| set.ip_addr_blocks.len())
|
||||
.sum::<usize>()
|
||||
})
|
||||
.unwrap_or(0);
|
||||
let vrps = content_info
|
||||
.content
|
||||
.vrps
|
||||
.as_ref()
|
||||
.map(|state| {
|
||||
json!({
|
||||
"present": true,
|
||||
"payload_sets": state.rps.len(),
|
||||
"hash_hex": hex::encode(&state.hash),
|
||||
"ip_addr_block_count": vrps_total,
|
||||
})
|
||||
})
|
||||
.unwrap_or_else(|| json!({"present": false}));
|
||||
|
||||
let vaps = content_info
|
||||
.content
|
||||
.vaps
|
||||
.as_ref()
|
||||
.map(|state| {
|
||||
json!({
|
||||
"present": true,
|
||||
"payload_sets": state.aps.len(),
|
||||
"hash_hex": hex::encode(&state.hash),
|
||||
"provider_count": state.aps.iter().map(|set| set.providers.len()).sum::<usize>(),
|
||||
})
|
||||
})
|
||||
.unwrap_or_else(|| json!({"present": false}));
|
||||
|
||||
let tas = content_info
|
||||
.content
|
||||
.tas
|
||||
.as_ref()
|
||||
.map(|state| {
|
||||
json!({
|
||||
"present": true,
|
||||
"ski_count": state.skis.len(),
|
||||
"hash_hex": hex::encode(&state.hash),
|
||||
})
|
||||
})
|
||||
.unwrap_or_else(|| json!({"present": false}));
|
||||
|
||||
let rks = content_info.content.rks.as_ref().map(|state| {
|
||||
json!({
|
||||
"present": true,
|
||||
"router_key_sets": state.rksets.len(),
|
||||
"router_key_count": state.rksets.iter().map(|set| set.router_keys.len()).sum::<usize>(),
|
||||
"hash_hex": hex::encode(&state.hash),
|
||||
})
|
||||
}).unwrap_or_else(|| json!({"present": false}));
|
||||
|
||||
Ok(json!({
|
||||
"content_type_oid": content_info.content_type_oid,
|
||||
"version": content_info.content.version,
|
||||
"hash_alg": content_info.content.hash_alg.oid(),
|
||||
"produced_at_rfc3339_utc": produced_at,
|
||||
"state_aspects": {
|
||||
"mfts": mfts,
|
||||
"vrps": vrps,
|
||||
"vaps": vaps,
|
||||
"tas": tas,
|
||||
"rks": rks,
|
||||
}
|
||||
}))
|
||||
}
|
||||
303
crates/panda-rpki-validator/src/ccr/encode.rs
Normal file
303
crates/panda-rpki-validator/src/ccr/encode.rs
Normal file
@ -0,0 +1,303 @@
|
||||
use crate::ccr::model::{
|
||||
AspaPayloadSet, AspaPayloadState, CCR_VERSION_V0, CcrContentInfo, CcrDigestAlgorithm,
|
||||
ManifestInstance, ManifestState, RoaPayloadSet, RoaPayloadState, RouterKey, RouterKeySet,
|
||||
RouterKeyState, RpkiCanonicalCacheRepresentation, TrustAnchorState,
|
||||
};
|
||||
use crate::data_model::common::BigUnsigned;
|
||||
use crate::data_model::oid::{OID_CT_RPKI_CCR_RAW, OID_SHA256_RAW};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrEncodeError {
|
||||
#[error("CCR model validation failed: {0}")]
|
||||
Validate(String),
|
||||
|
||||
#[error("GeneralizedTime formatting failed: {0}")]
|
||||
ProducedAtFormat(String),
|
||||
}
|
||||
|
||||
pub fn encode_content_info(content_info: &CcrContentInfo) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
content_info.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let content_der = encode_ccr(&content_info.content)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_oid(OID_CT_RPKI_CCR_RAW),
|
||||
encode_explicit(0, &content_der),
|
||||
]))
|
||||
}
|
||||
|
||||
pub fn encode_ccr(ccr: &RpkiCanonicalCacheRepresentation) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
ccr.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let mut fields = Vec::new();
|
||||
if ccr.version != CCR_VERSION_V0 {
|
||||
fields.push(encode_explicit(0, &encode_integer_u32(ccr.version)));
|
||||
}
|
||||
fields.push(encode_digest_algorithm(&ccr.hash_alg));
|
||||
fields.push(encode_generalized_time(ccr.produced_at)?);
|
||||
if let Some(mfts) = &ccr.mfts {
|
||||
fields.push(encode_explicit(1, &encode_manifest_state(mfts)?));
|
||||
}
|
||||
if let Some(vrps) = &ccr.vrps {
|
||||
fields.push(encode_explicit(2, &encode_roa_payload_state(vrps)?));
|
||||
}
|
||||
if let Some(vaps) = &ccr.vaps {
|
||||
fields.push(encode_explicit(3, &encode_aspa_payload_state(vaps)?));
|
||||
}
|
||||
if let Some(tas) = &ccr.tas {
|
||||
fields.push(encode_explicit(4, &encode_trust_anchor_state(tas)?));
|
||||
}
|
||||
if let Some(rks) = &ccr.rks {
|
||||
fields.push(encode_explicit(5, &encode_router_key_state(rks)?));
|
||||
}
|
||||
Ok(encode_sequence(&fields))
|
||||
}
|
||||
|
||||
pub fn encode_manifest_state(state: &ManifestState) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
state.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let mis = encode_manifest_state_payload_der(&state.mis)?;
|
||||
Ok(encode_sequence(&[
|
||||
mis,
|
||||
encode_generalized_time(state.most_recent_update)?,
|
||||
encode_octet_string(&state.hash),
|
||||
]))
|
||||
}
|
||||
|
||||
pub fn encode_manifest_state_payload_der(
|
||||
instances: &[ManifestInstance],
|
||||
) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
Ok(encode_sequence(
|
||||
&instances
|
||||
.iter()
|
||||
.map(encode_manifest_instance)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
))
|
||||
}
|
||||
|
||||
fn encode_manifest_instance(instance: &ManifestInstance) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
instance.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let mut fields = vec![
|
||||
encode_octet_string(&instance.hash),
|
||||
encode_integer_u64(instance.size),
|
||||
encode_octet_string(&instance.aki),
|
||||
encode_integer_bigunsigned(&instance.manifest_number),
|
||||
encode_generalized_time(instance.this_update)?,
|
||||
encode_sequence(&instance.locations),
|
||||
];
|
||||
if !instance.subordinates.is_empty() {
|
||||
fields.push(encode_sequence(
|
||||
&instance
|
||||
.subordinates
|
||||
.iter()
|
||||
.map(|ski| encode_octet_string(ski))
|
||||
.collect::<Vec<_>>(),
|
||||
));
|
||||
}
|
||||
Ok(encode_sequence(&fields))
|
||||
}
|
||||
|
||||
pub fn encode_roa_payload_state(state: &RoaPayloadState) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
state.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let rps = encode_roa_payload_state_payload_der(&state.rps)?;
|
||||
Ok(encode_sequence(&[rps, encode_octet_string(&state.hash)]))
|
||||
}
|
||||
|
||||
pub fn encode_roa_payload_state_payload_der(
|
||||
sets: &[RoaPayloadSet],
|
||||
) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
Ok(encode_sequence(
|
||||
&sets
|
||||
.iter()
|
||||
.map(encode_roa_payload_set)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
))
|
||||
}
|
||||
|
||||
fn encode_roa_payload_set(set: &RoaPayloadSet) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
set.validate().map_err(CcrEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_integer_u32(set.as_id),
|
||||
encode_sequence(&set.ip_addr_blocks),
|
||||
]))
|
||||
}
|
||||
|
||||
pub fn encode_aspa_payload_state(state: &AspaPayloadState) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
state.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let aps = encode_aspa_payload_state_payload_der(&state.aps)?;
|
||||
Ok(encode_sequence(&[aps, encode_octet_string(&state.hash)]))
|
||||
}
|
||||
|
||||
pub fn encode_aspa_payload_state_payload_der(
|
||||
sets: &[AspaPayloadSet],
|
||||
) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
Ok(encode_sequence(
|
||||
&sets
|
||||
.iter()
|
||||
.map(encode_aspa_payload_set)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
))
|
||||
}
|
||||
|
||||
fn encode_aspa_payload_set(set: &AspaPayloadSet) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
set.validate().map_err(CcrEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_integer_u32(set.customer_as_id),
|
||||
encode_sequence(
|
||||
&set.providers
|
||||
.iter()
|
||||
.map(|provider| encode_integer_u32(*provider))
|
||||
.collect::<Vec<_>>(),
|
||||
),
|
||||
]))
|
||||
}
|
||||
|
||||
pub fn encode_trust_anchor_state(state: &TrustAnchorState) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
state.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let skis = encode_trust_anchor_state_payload_der(&state.skis)?;
|
||||
Ok(encode_sequence(&[skis, encode_octet_string(&state.hash)]))
|
||||
}
|
||||
|
||||
pub fn encode_trust_anchor_state_payload_der(skis: &[Vec<u8>]) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
Ok(encode_sequence(
|
||||
&skis
|
||||
.iter()
|
||||
.map(|ski| encode_octet_string(ski))
|
||||
.collect::<Vec<_>>(),
|
||||
))
|
||||
}
|
||||
|
||||
pub fn encode_router_key_state(state: &RouterKeyState) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
state.validate().map_err(CcrEncodeError::Validate)?;
|
||||
let rksets = encode_router_key_state_payload_der(&state.rksets)?;
|
||||
Ok(encode_sequence(&[rksets, encode_octet_string(&state.hash)]))
|
||||
}
|
||||
|
||||
pub fn encode_router_key_state_payload_der(
|
||||
sets: &[RouterKeySet],
|
||||
) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
Ok(encode_sequence(
|
||||
&sets
|
||||
.iter()
|
||||
.map(encode_router_key_set)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
))
|
||||
}
|
||||
|
||||
fn encode_router_key_set(set: &RouterKeySet) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
set.validate().map_err(CcrEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_integer_u32(set.as_id),
|
||||
encode_sequence(
|
||||
&set.router_keys
|
||||
.iter()
|
||||
.map(encode_router_key)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
),
|
||||
]))
|
||||
}
|
||||
|
||||
fn encode_router_key(key: &RouterKey) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
key.validate().map_err(CcrEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_octet_string(&key.ski),
|
||||
key.spki_der.clone(),
|
||||
]))
|
||||
}
|
||||
|
||||
fn encode_digest_algorithm(alg: &CcrDigestAlgorithm) -> Vec<u8> {
|
||||
match alg {
|
||||
CcrDigestAlgorithm::Sha256 => encode_sequence(&[encode_oid(OID_SHA256_RAW)]),
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_generalized_time(t: time::OffsetDateTime) -> Result<Vec<u8>, CcrEncodeError> {
|
||||
let t = t.to_offset(time::UtcOffset::UTC);
|
||||
let s = format!(
|
||||
"{:04}{:02}{:02}{:02}{:02}{:02}Z",
|
||||
t.year(),
|
||||
u8::from(t.month()),
|
||||
t.day(),
|
||||
t.hour(),
|
||||
t.minute(),
|
||||
t.second()
|
||||
);
|
||||
Ok(encode_tlv(0x18, s.into_bytes()))
|
||||
}
|
||||
|
||||
fn encode_integer_u32(v: u32) -> Vec<u8> {
|
||||
encode_integer_bytes(unsigned_integer_bytes(v as u64))
|
||||
}
|
||||
|
||||
fn encode_integer_u64(v: u64) -> Vec<u8> {
|
||||
encode_integer_bytes(unsigned_integer_bytes(v))
|
||||
}
|
||||
|
||||
fn encode_integer_bigunsigned(v: &BigUnsigned) -> Vec<u8> {
|
||||
encode_integer_bytes(v.bytes_be.clone())
|
||||
}
|
||||
|
||||
fn encode_integer_bytes(mut bytes: Vec<u8>) -> Vec<u8> {
|
||||
if bytes.is_empty() {
|
||||
bytes.push(0);
|
||||
}
|
||||
if bytes[0] & 0x80 != 0 {
|
||||
bytes.insert(0, 0);
|
||||
}
|
||||
encode_tlv(0x02, bytes)
|
||||
}
|
||||
|
||||
fn unsigned_integer_bytes(v: u64) -> Vec<u8> {
|
||||
if v == 0 {
|
||||
return vec![0];
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
let mut n = v;
|
||||
while n > 0 {
|
||||
out.push((n & 0xFF) as u8);
|
||||
n >>= 8;
|
||||
}
|
||||
out.reverse();
|
||||
out
|
||||
}
|
||||
|
||||
fn encode_oid(raw_body: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0x06, raw_body.to_vec())
|
||||
}
|
||||
|
||||
fn encode_octet_string(bytes: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0x04, bytes.to_vec())
|
||||
}
|
||||
|
||||
fn encode_explicit(tag_number: u8, inner_der: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0xA0 + tag_number, inner_der.to_vec())
|
||||
}
|
||||
|
||||
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
|
||||
let total_len: usize = elements.iter().map(Vec::len).sum();
|
||||
let mut buf = Vec::with_capacity(total_len);
|
||||
for element in elements {
|
||||
buf.extend_from_slice(element);
|
||||
}
|
||||
encode_tlv(0x30, buf)
|
||||
}
|
||||
|
||||
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(1 + 9 + value.len());
|
||||
out.push(tag);
|
||||
encode_length(value.len(), &mut out);
|
||||
out.extend_from_slice(&value);
|
||||
out
|
||||
}
|
||||
|
||||
fn encode_length(len: usize, out: &mut Vec<u8>) {
|
||||
if len < 0x80 {
|
||||
out.push(len as u8);
|
||||
return;
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
let mut value = len;
|
||||
while value > 0 {
|
||||
bytes.push((value & 0xFF) as u8);
|
||||
value >>= 8;
|
||||
}
|
||||
bytes.reverse();
|
||||
out.push(0x80 | (bytes.len() as u8));
|
||||
out.extend_from_slice(&bytes);
|
||||
}
|
||||
343
crates/panda-rpki-validator/src/ccr/export.rs
Normal file
343
crates/panda-rpki-validator/src/ccr/export.rs
Normal file
@ -0,0 +1,343 @@
|
||||
use crate::ccr::build::{
|
||||
CcrBuildError, ManifestStateBuildBreakdown, build_aspa_payload_state,
|
||||
build_manifest_state_from_vcirs_with_breakdown, build_roa_payload_state,
|
||||
build_router_key_state_from_runtime, build_trust_anchor_state,
|
||||
};
|
||||
use crate::ccr::encode::{CcrEncodeError, encode_content_info};
|
||||
use crate::ccr::model::{CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation};
|
||||
use crate::data_model::ta::TrustAnchor;
|
||||
use crate::storage::RocksStore;
|
||||
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
|
||||
use serde::Serialize;
|
||||
use std::path::Path;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrExportError {
|
||||
#[error("list VCIRs failed: {0}")]
|
||||
ListVcirs(String),
|
||||
|
||||
#[error("build CCR state failed: {0}")]
|
||||
Build(#[from] CcrBuildError),
|
||||
|
||||
#[error("encode CCR failed: {0}")]
|
||||
Encode(#[from] CcrEncodeError),
|
||||
|
||||
#[error("write CCR file failed: {0}: {1}")]
|
||||
Write(String, String),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct CcrBuildBreakdown {
|
||||
pub vcir_count: usize,
|
||||
pub list_vcirs_ms: u64,
|
||||
pub manifest_state_ms: u64,
|
||||
pub manifest_state_breakdown: ManifestStateBuildBreakdown,
|
||||
pub roa_payload_state_ms: u64,
|
||||
pub aspa_payload_state_ms: u64,
|
||||
pub trust_anchor_state_ms: u64,
|
||||
pub router_key_state_ms: u64,
|
||||
pub total_ms: u64,
|
||||
}
|
||||
|
||||
pub fn build_ccr_from_run(
|
||||
store: &RocksStore,
|
||||
trust_anchors: &[TrustAnchor],
|
||||
vrps: &[Vrp],
|
||||
aspas: &[AspaAttestation],
|
||||
router_keys: &[RouterKeyPayload],
|
||||
produced_at: time::OffsetDateTime,
|
||||
) -> Result<RpkiCanonicalCacheRepresentation, CcrExportError> {
|
||||
build_ccr_from_run_with_breakdown(store, trust_anchors, vrps, aspas, router_keys, produced_at)
|
||||
.map(|(ccr, _)| ccr)
|
||||
}
|
||||
|
||||
pub fn build_ccr_from_run_with_breakdown(
|
||||
store: &RocksStore,
|
||||
trust_anchors: &[TrustAnchor],
|
||||
vrps: &[Vrp],
|
||||
aspas: &[AspaAttestation],
|
||||
router_keys: &[RouterKeyPayload],
|
||||
produced_at: time::OffsetDateTime,
|
||||
) -> Result<(RpkiCanonicalCacheRepresentation, CcrBuildBreakdown), CcrExportError> {
|
||||
let total_started = std::time::Instant::now();
|
||||
let mut breakdown = CcrBuildBreakdown::default();
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let vcirs = store
|
||||
.list_vcirs()
|
||||
.map_err(|e| CcrExportError::ListVcirs(e.to_string()))?;
|
||||
breakdown.list_vcirs_ms = started.elapsed().as_millis() as u64;
|
||||
breakdown.vcir_count = vcirs.len();
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let (mfts, manifest_state_breakdown) =
|
||||
build_manifest_state_from_vcirs_with_breakdown(store, &vcirs)?;
|
||||
breakdown.manifest_state_ms = started.elapsed().as_millis() as u64;
|
||||
breakdown.manifest_state_breakdown = manifest_state_breakdown;
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let vrps = build_roa_payload_state(vrps)?;
|
||||
breakdown.roa_payload_state_ms = started.elapsed().as_millis() as u64;
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let vaps = build_aspa_payload_state(aspas)?;
|
||||
breakdown.aspa_payload_state_ms = started.elapsed().as_millis() as u64;
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let tas = build_trust_anchor_state(trust_anchors)?;
|
||||
breakdown.trust_anchor_state_ms = started.elapsed().as_millis() as u64;
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let rks = build_router_key_state_from_runtime(router_keys)?;
|
||||
breakdown.router_key_state_ms = started.elapsed().as_millis() as u64;
|
||||
|
||||
let ccr = RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at,
|
||||
mfts: Some(mfts),
|
||||
vrps: Some(vrps),
|
||||
vaps: Some(vaps),
|
||||
tas: Some(tas),
|
||||
rks: Some(rks),
|
||||
};
|
||||
breakdown.total_ms = total_started.elapsed().as_millis() as u64;
|
||||
|
||||
Ok((ccr, breakdown))
|
||||
}
|
||||
|
||||
pub fn write_ccr_file(
|
||||
path: &Path,
|
||||
ccr: &RpkiCanonicalCacheRepresentation,
|
||||
) -> Result<(), CcrExportError> {
|
||||
let der = encode_content_info(&CcrContentInfo::new(ccr.clone()))?;
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| CcrExportError::Write(path.display().to_string(), e.to_string()))?;
|
||||
}
|
||||
std::fs::write(path, der)
|
||||
.map_err(|e| CcrExportError::Write(path.display().to_string(), e.to_string()))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::decode::decode_content_info;
|
||||
use crate::data_model::manifest::ManifestObject;
|
||||
use crate::data_model::oid::OID_AD_SIGNED_OBJECT;
|
||||
use crate::data_model::rc::SubjectInfoAccess;
|
||||
use crate::data_model::roa::{IpPrefix, RoaAfi};
|
||||
use crate::data_model::ta::TrustAnchor;
|
||||
use crate::data_model::tal::Tal;
|
||||
use crate::storage::{
|
||||
PackTime, RawByHashEntry, RocksStore, ValidatedCaInstanceResult, ValidatedManifestMeta,
|
||||
VcirArtifactKind, VcirArtifactRole, VcirArtifactValidationStatus, VcirAuditSummary,
|
||||
VcirCcrManifestProjection, VcirChildEntry, VcirInstanceGate, VcirRelatedArtifact,
|
||||
VcirSummary,
|
||||
};
|
||||
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
|
||||
use sha2::Digest;
|
||||
|
||||
fn sample_trust_anchor() -> TrustAnchor {
|
||||
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let tal_bytes = std::fs::read(base.join("tests/fixtures/tal/apnic-rfc7730-https.tal"))
|
||||
.expect("read tal");
|
||||
let ta_der = std::fs::read(base.join("tests/fixtures/ta/apnic-ta.cer")).expect("read ta");
|
||||
let tal = Tal::decode_bytes(&tal_bytes).expect("decode tal");
|
||||
TrustAnchor::bind_der(tal, &ta_der, None).expect("bind ta")
|
||||
}
|
||||
|
||||
fn sample_vcir_and_raw(store: &RocksStore) -> ValidatedCaInstanceResult {
|
||||
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let manifest_der = std::fs::read(base.join("tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft")).expect("read manifest");
|
||||
let manifest = ManifestObject::decode_der(&manifest_der).expect("decode manifest");
|
||||
let manifest_uri = match manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.expect("manifest sia")
|
||||
{
|
||||
SubjectInfoAccess::Ee(ee_sia) => ee_sia
|
||||
.access_descriptions
|
||||
.iter()
|
||||
.find(|ad| {
|
||||
ad.access_method_oid == OID_AD_SIGNED_OBJECT
|
||||
&& ad.access_location.starts_with("rsync://")
|
||||
})
|
||||
.expect("manifest rsync signedObject")
|
||||
.access_location
|
||||
.clone(),
|
||||
SubjectInfoAccess::Ca(_) => panic!("manifest EE SIA should not be CA variant"),
|
||||
};
|
||||
let hash = hex::encode(sha2::Sha256::digest(&manifest_der));
|
||||
let mut raw = RawByHashEntry::from_bytes(hash.clone(), manifest_der.clone());
|
||||
raw.origin_uris.push(manifest_uri.clone());
|
||||
raw.object_type = Some("mft".to_string());
|
||||
raw.encoding = Some("der".to_string());
|
||||
store.put_raw_by_hash_entry(&raw).expect("put raw");
|
||||
let projection = VcirCcrManifestProjection {
|
||||
manifest_rsync_uri: manifest_uri.clone(),
|
||||
manifest_sha256: sha2::Sha256::digest(&manifest_der).to_vec(),
|
||||
manifest_size: manifest_der.len() as u64,
|
||||
manifest_ee_aki: manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.authority_key_identifier
|
||||
.clone()
|
||||
.expect("manifest aki"),
|
||||
manifest_number_be: manifest.manifest.manifest_number.bytes_be.clone(),
|
||||
manifest_this_update: PackTime::from_utc_offset_datetime(manifest.manifest.this_update),
|
||||
manifest_sia_locations_der: match manifest.signed_object.signed_data.certificates[0]
|
||||
.resource_cert
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_info_access
|
||||
.as_ref()
|
||||
.expect("manifest sia")
|
||||
{
|
||||
SubjectInfoAccess::Ee(ee_sia) => vec![
|
||||
crate::ccr::manifest_location::select_manifest_signed_object_location(
|
||||
&manifest_uri,
|
||||
&ee_sia.access_descriptions,
|
||||
)
|
||||
.expect("select manifest signedObject"),
|
||||
],
|
||||
SubjectInfoAccess::Ca(_) => panic!("manifest EE SIA should not be CA variant"),
|
||||
},
|
||||
subordinate_skis: vec![vec![0x33; 20]],
|
||||
};
|
||||
ValidatedCaInstanceResult {
|
||||
manifest_rsync_uri: manifest_uri.clone(),
|
||||
parent_manifest_rsync_uri: None,
|
||||
tal_id: "apnic".to_string(),
|
||||
ca_subject_name: "CN=test".to_string(),
|
||||
ca_ski: "11".repeat(20),
|
||||
issuer_ski: "22".repeat(20),
|
||||
last_successful_validation_time: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.this_update,
|
||||
),
|
||||
current_manifest_rsync_uri: manifest_uri.clone(),
|
||||
current_crl_rsync_uri: "rsync://example.test/repo/current.crl".to_string(),
|
||||
validated_manifest_meta: ValidatedManifestMeta {
|
||||
validated_manifest_number: manifest.manifest.manifest_number.bytes_be.clone(),
|
||||
validated_manifest_this_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.this_update,
|
||||
),
|
||||
validated_manifest_next_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
},
|
||||
ccr_manifest_projection: projection,
|
||||
instance_gate: VcirInstanceGate {
|
||||
manifest_next_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
current_crl_next_update: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
self_ca_not_after: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
instance_effective_until: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.next_update,
|
||||
),
|
||||
},
|
||||
child_entries: vec![VcirChildEntry {
|
||||
child_manifest_rsync_uri: "rsync://example.test/repo/child.mft".to_string(),
|
||||
child_cert_rsync_uri: "rsync://example.test/repo/child.cer".to_string(),
|
||||
child_cert_hash: "aa".repeat(32),
|
||||
child_ski: "33".repeat(20),
|
||||
child_rsync_base_uri: "rsync://example.test/repo/".to_string(),
|
||||
child_publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
|
||||
child_rrdp_notification_uri: None,
|
||||
child_effective_ip_resources: None,
|
||||
child_effective_as_resources: None,
|
||||
accepted_at_validation_time: PackTime::from_utc_offset_datetime(
|
||||
manifest.manifest.this_update,
|
||||
),
|
||||
}],
|
||||
local_outputs: Vec::new(),
|
||||
related_artifacts: vec![VcirRelatedArtifact {
|
||||
artifact_role: VcirArtifactRole::Manifest,
|
||||
artifact_kind: VcirArtifactKind::Mft,
|
||||
uri: Some(manifest_uri),
|
||||
sha256: hash,
|
||||
object_type: Some("mft".to_string()),
|
||||
validation_status: VcirArtifactValidationStatus::Accepted,
|
||||
reject_reason: None,
|
||||
}],
|
||||
summary: VcirSummary {
|
||||
local_vrp_count: 0,
|
||||
local_aspa_count: 0,
|
||||
local_router_key_count: 0,
|
||||
child_count: 1,
|
||||
accepted_object_count: 1,
|
||||
rejected_object_count: 0,
|
||||
},
|
||||
audit_summary: VcirAuditSummary {
|
||||
failed_fetch_eligible: true,
|
||||
last_failed_fetch_reason: None,
|
||||
warning_count: 0,
|
||||
audit_flags: Vec::new(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_and_write_ccr_from_run_exports_der_content_info() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let db_path = td.path().join("db");
|
||||
let store = RocksStore::open(&db_path).expect("open rocksdb");
|
||||
let vcir = sample_vcir_and_raw(&store);
|
||||
store.put_vcir(&vcir).expect("put vcir");
|
||||
let trust_anchor = sample_trust_anchor();
|
||||
let vrps = vec![Vrp {
|
||||
asn: 64496,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv4,
|
||||
prefix_len: 8,
|
||||
addr: [10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
},
|
||||
max_length: 8,
|
||||
}];
|
||||
let aspas = vec![AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![64497],
|
||||
}];
|
||||
let router_keys = vec![RouterKeyPayload {
|
||||
as_id: 64496,
|
||||
ski: vec![0x11; 20],
|
||||
spki_der: vec![0x30, 0x00],
|
||||
source_object_uri: "rsync://example.test/repo/router.cer".to_string(),
|
||||
source_object_hash: hex::encode([0x11; 32]),
|
||||
source_ee_cert_hash: hex::encode([0x11; 32]),
|
||||
item_effective_until: PackTime::from_utc_offset_datetime(
|
||||
time::OffsetDateTime::now_utc() + time::Duration::hours(1),
|
||||
),
|
||||
}];
|
||||
let ccr = build_ccr_from_run(
|
||||
&store,
|
||||
&[trust_anchor],
|
||||
&vrps,
|
||||
&aspas,
|
||||
&router_keys,
|
||||
time::OffsetDateTime::now_utc(),
|
||||
)
|
||||
.expect("build ccr");
|
||||
assert!(ccr.mfts.is_some());
|
||||
assert!(ccr.vrps.is_some());
|
||||
assert!(ccr.vaps.is_some());
|
||||
assert!(ccr.tas.is_some());
|
||||
assert!(ccr.rks.is_some());
|
||||
assert_eq!(ccr.rks.as_ref().unwrap().rksets.len(), 1);
|
||||
let out = td.path().join("out/example.ccr");
|
||||
write_ccr_file(&out, &ccr).expect("write ccr");
|
||||
let der = std::fs::read(&out).expect("read ccr file");
|
||||
let decoded = decode_content_info(&der).expect("decode ccr");
|
||||
assert_eq!(decoded.content.version, 0);
|
||||
assert!(decoded.content.mfts.is_some());
|
||||
}
|
||||
}
|
||||
9
crates/panda-rpki-validator/src/ccr/hash.rs
Normal file
9
crates/panda-rpki-validator/src/ccr/hash.rs
Normal file
@ -0,0 +1,9 @@
|
||||
use sha2::Digest;
|
||||
|
||||
pub fn compute_state_hash(payload_der: &[u8]) -> Vec<u8> {
|
||||
sha2::Sha256::digest(payload_der).to_vec()
|
||||
}
|
||||
|
||||
pub fn verify_state_hash(expected: &[u8], payload_der: &[u8]) -> bool {
|
||||
compute_state_hash(payload_der).as_slice() == expected
|
||||
}
|
||||
424
crates/panda-rpki-validator/src/ccr/manifest_location.rs
Normal file
424
crates/panda-rpki-validator/src/ccr/manifest_location.rs
Normal file
@ -0,0 +1,424 @@
|
||||
use crate::data_model::common::DerReader;
|
||||
use crate::data_model::oid::OID_AD_SIGNED_OBJECT;
|
||||
use crate::data_model::rc::AccessDescription;
|
||||
|
||||
pub(crate) fn select_manifest_signed_object_location(
|
||||
manifest_rsync_uri: &str,
|
||||
access_descriptions: &[AccessDescription],
|
||||
) -> Result<Vec<u8>, String> {
|
||||
let matching = access_descriptions
|
||||
.iter()
|
||||
.filter(|access_description| {
|
||||
access_description.access_method_oid == OID_AD_SIGNED_OBJECT
|
||||
&& access_description.access_location == manifest_rsync_uri
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let access_description = expect_single_matching_location(
|
||||
manifest_rsync_uri,
|
||||
matching.len(),
|
||||
"parsed Manifest EE SIA",
|
||||
)?;
|
||||
encode_access_description_der(matching[access_description])
|
||||
}
|
||||
|
||||
pub(crate) fn select_manifest_signed_object_location_from_der(
|
||||
manifest_rsync_uri: &str,
|
||||
locations_der: &[Vec<u8>],
|
||||
) -> Result<Vec<u8>, String> {
|
||||
let matching = locations_der
|
||||
.iter()
|
||||
.filter_map(
|
||||
|location_der| match decode_access_description_der(location_der) {
|
||||
Ok(access_description)
|
||||
if access_description.access_method_oid == OID_AD_SIGNED_OBJECT
|
||||
&& access_description.access_location == manifest_rsync_uri =>
|
||||
{
|
||||
Some(Ok(location_der.clone()))
|
||||
}
|
||||
Ok(_) => None,
|
||||
Err(detail) => Some(Err(detail)),
|
||||
},
|
||||
)
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
let index = expect_single_matching_location(
|
||||
manifest_rsync_uri,
|
||||
matching.len(),
|
||||
"persisted VCIR CCR projection",
|
||||
)?;
|
||||
Ok(matching[index].clone())
|
||||
}
|
||||
|
||||
pub(crate) fn encode_access_description_der(
|
||||
access_description: &AccessDescription,
|
||||
) -> Result<Vec<u8>, String> {
|
||||
let oid = encode_oid_der(&access_description.access_method_oid)?;
|
||||
let uri = encode_tlv(0x86, access_description.access_location.as_bytes().to_vec());
|
||||
Ok(encode_sequence(&[oid, uri]))
|
||||
}
|
||||
|
||||
fn expect_single_matching_location(
|
||||
manifest_rsync_uri: &str,
|
||||
matching_count: usize,
|
||||
source: &str,
|
||||
) -> Result<usize, String> {
|
||||
if matching_count == 1 {
|
||||
return Ok(0);
|
||||
}
|
||||
Err(format!(
|
||||
"{source} contains {matching_count} id-ad-signedObject locations matching manifest URI {manifest_rsync_uri}; expected exactly one"
|
||||
))
|
||||
}
|
||||
|
||||
fn decode_access_description_der(der: &[u8]) -> Result<AccessDescription, String> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut sequence = top.take_sequence()?;
|
||||
if !top.is_empty() {
|
||||
return Err("trailing bytes after AccessDescription".to_string());
|
||||
}
|
||||
let access_method_oid = decode_oid_der(sequence.take_tag(0x06)?)?;
|
||||
let access_location = std::str::from_utf8(sequence.take_tag(0x86)?)
|
||||
.map_err(|error| format!("AccessDescription URI is not UTF-8: {error}"))?
|
||||
.to_string();
|
||||
if !sequence.is_empty() {
|
||||
return Err("trailing fields in AccessDescription".to_string());
|
||||
}
|
||||
Ok(AccessDescription {
|
||||
access_method_oid,
|
||||
access_location,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_oid_der(value: &[u8]) -> Result<String, String> {
|
||||
let mut offset = 0usize;
|
||||
let first = decode_base128(value, &mut offset)?;
|
||||
let (first_arc, second_arc) = match first {
|
||||
0..=39 => (0, first),
|
||||
40..=79 => (1, first - 40),
|
||||
value => (2, value - 80),
|
||||
};
|
||||
let mut arcs = vec![first_arc, second_arc];
|
||||
while offset < value.len() {
|
||||
arcs.push(decode_base128(value, &mut offset)?);
|
||||
}
|
||||
Ok(arcs
|
||||
.into_iter()
|
||||
.map(|arc| arc.to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join("."))
|
||||
}
|
||||
|
||||
fn decode_base128(value: &[u8], offset: &mut usize) -> Result<u64, String> {
|
||||
let first = *value
|
||||
.get(*offset)
|
||||
.ok_or_else(|| "truncated OBJECT IDENTIFIER".to_string())?;
|
||||
if first == 0x80 {
|
||||
return Err("non-minimal OBJECT IDENTIFIER base-128 encoding".to_string());
|
||||
}
|
||||
let mut out = 0u64;
|
||||
loop {
|
||||
let byte = *value
|
||||
.get(*offset)
|
||||
.ok_or_else(|| "truncated OBJECT IDENTIFIER".to_string())?;
|
||||
*offset += 1;
|
||||
out = out
|
||||
.checked_shl(7)
|
||||
.ok_or_else(|| "OBJECT IDENTIFIER arc overflows u64".to_string())?
|
||||
.checked_add((byte & 0x7f) as u64)
|
||||
.ok_or_else(|| "OBJECT IDENTIFIER arc overflows u64".to_string())?;
|
||||
if byte & 0x80 == 0 {
|
||||
return Ok(out);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_oid_der(oid: &str) -> Result<Vec<u8>, String> {
|
||||
let arcs = oid
|
||||
.split('.')
|
||||
.map(|part| {
|
||||
part.parse::<u64>()
|
||||
.map_err(|_| format!("unsupported accessMethod OID: {oid}"))
|
||||
})
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
if arcs.len() < 2 || arcs[0] > 2 || (arcs[0] < 2 && arcs[1] >= 40) {
|
||||
return Err(format!("unsupported accessMethod OID: {oid}"));
|
||||
}
|
||||
let mut body = Vec::new();
|
||||
encode_base128(
|
||||
arcs[0]
|
||||
.checked_mul(40)
|
||||
.and_then(|value| value.checked_add(arcs[1]))
|
||||
.ok_or_else(|| format!("unsupported accessMethod OID: {oid}"))?,
|
||||
&mut body,
|
||||
);
|
||||
for arc in &arcs[2..] {
|
||||
encode_base128(*arc, &mut body);
|
||||
}
|
||||
Ok(encode_tlv(0x06, body))
|
||||
}
|
||||
|
||||
fn encode_base128(mut value: u64, out: &mut Vec<u8>) {
|
||||
let mut encoded = vec![(value & 0x7f) as u8];
|
||||
value >>= 7;
|
||||
while value > 0 {
|
||||
encoded.push(((value & 0x7f) as u8) | 0x80);
|
||||
value >>= 7;
|
||||
}
|
||||
encoded.reverse();
|
||||
out.extend_from_slice(&encoded);
|
||||
}
|
||||
|
||||
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
|
||||
let total_len = elements.iter().map(Vec::len).sum();
|
||||
let mut value = Vec::with_capacity(total_len);
|
||||
for element in elements {
|
||||
value.extend_from_slice(element);
|
||||
}
|
||||
encode_tlv(0x30, value)
|
||||
}
|
||||
|
||||
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(1 + 9 + value.len());
|
||||
out.push(tag);
|
||||
encode_length(value.len(), &mut out);
|
||||
out.extend_from_slice(&value);
|
||||
out
|
||||
}
|
||||
|
||||
fn encode_length(len: usize, out: &mut Vec<u8>) {
|
||||
if len < 0x80 {
|
||||
out.push(len as u8);
|
||||
return;
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
let mut value = len;
|
||||
while value > 0 {
|
||||
bytes.push((value & 0xff) as u8);
|
||||
value >>= 8;
|
||||
}
|
||||
bytes.reverse();
|
||||
out.push(0x80 | bytes.len() as u8);
|
||||
out.extend_from_slice(&bytes);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::data_model::oid::OID_AD_RPKI_NOTIFY;
|
||||
|
||||
const MANIFEST_URI: &str = "rsync://example.test/repo/manifest.mft";
|
||||
|
||||
fn access_description(access_method_oid: &str, access_location: &str) -> AccessDescription {
|
||||
AccessDescription {
|
||||
access_method_oid: access_method_oid.to_string(),
|
||||
access_location: access_location.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selects_matching_signed_object_and_excludes_rpki_notify() {
|
||||
let signed_object = access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI);
|
||||
let selected = select_manifest_signed_object_location(
|
||||
MANIFEST_URI,
|
||||
&[
|
||||
signed_object.clone(),
|
||||
access_description(
|
||||
OID_AD_RPKI_NOTIFY,
|
||||
"https://rrdp.example.test/notification.xml",
|
||||
),
|
||||
],
|
||||
)
|
||||
.expect("select signed object");
|
||||
|
||||
assert_eq!(
|
||||
selected,
|
||||
encode_access_description_der(&signed_object).unwrap()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selects_only_the_signed_object_matching_manifest_uri() {
|
||||
let expected = access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI);
|
||||
let selected = select_manifest_signed_object_location(
|
||||
MANIFEST_URI,
|
||||
&[
|
||||
access_description(
|
||||
OID_AD_SIGNED_OBJECT,
|
||||
"https://backup.example.test/manifest.mft",
|
||||
),
|
||||
expected.clone(),
|
||||
],
|
||||
)
|
||||
.expect("select matching signed object");
|
||||
|
||||
assert_eq!(selected, encode_access_description_der(&expected).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_missing_or_duplicate_matching_signed_object() {
|
||||
let missing = select_manifest_signed_object_location(
|
||||
MANIFEST_URI,
|
||||
&[access_description(
|
||||
OID_AD_RPKI_NOTIFY,
|
||||
"https://rrdp.example.test/notification.xml",
|
||||
)],
|
||||
)
|
||||
.expect_err("missing signed object must fail");
|
||||
assert!(missing.contains("contains 0"), "{missing}");
|
||||
|
||||
let duplicate = select_manifest_signed_object_location(
|
||||
MANIFEST_URI,
|
||||
&[
|
||||
access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI),
|
||||
access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI),
|
||||
],
|
||||
)
|
||||
.expect_err("duplicate signed object must fail");
|
||||
assert!(duplicate.contains("contains 2"), "{duplicate}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selects_matching_signed_object_from_historical_projection() {
|
||||
let signed_object = access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI);
|
||||
let signed_object_der = encode_access_description_der(&signed_object).unwrap();
|
||||
let notify_der = encode_access_description_der(&access_description(
|
||||
OID_AD_RPKI_NOTIFY,
|
||||
"https://rrdp.example.test/notification.xml",
|
||||
))
|
||||
.unwrap();
|
||||
|
||||
let selected = select_manifest_signed_object_location_from_der(
|
||||
MANIFEST_URI,
|
||||
&[notify_der, signed_object_der.clone()],
|
||||
)
|
||||
.expect("select historical signed object");
|
||||
assert_eq!(selected, signed_object_der);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_malformed_historical_projection() {
|
||||
let error = select_manifest_signed_object_location_from_der(
|
||||
MANIFEST_URI,
|
||||
&[vec![0x30, 0x01, 0x06]],
|
||||
)
|
||||
.expect_err("malformed historical projection must fail");
|
||||
assert!(error.contains("truncated DER"), "{error}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn access_description_der_codec_covers_long_and_invalid_forms() {
|
||||
let long = access_description(
|
||||
"2.999.200.1",
|
||||
&format!("rsync://example.test/repo/{}", "x".repeat(160)),
|
||||
);
|
||||
let encoded = encode_access_description_der(&long).expect("encode long access description");
|
||||
assert_eq!(
|
||||
decode_access_description_der(&encoded).expect("decode long access description"),
|
||||
long
|
||||
);
|
||||
|
||||
let mut trailing = encoded.clone();
|
||||
trailing.push(0);
|
||||
assert!(
|
||||
decode_access_description_der(&trailing)
|
||||
.expect_err("trailing bytes must fail")
|
||||
.contains("trailing bytes")
|
||||
);
|
||||
|
||||
let oid = encode_oid_der(OID_AD_SIGNED_OBJECT).expect("encode signedObject OID");
|
||||
let uri = encode_tlv(0x86, MANIFEST_URI.as_bytes().to_vec());
|
||||
assert!(
|
||||
decode_access_description_der(&encode_sequence(&[
|
||||
oid.clone(),
|
||||
uri.clone(),
|
||||
encode_tlv(0x05, Vec::new()),
|
||||
]))
|
||||
.expect_err("trailing field must fail")
|
||||
.contains("trailing fields")
|
||||
);
|
||||
assert!(
|
||||
decode_access_description_der(&encode_sequence(&[oid, encode_tlv(0x86, vec![0xff]),]))
|
||||
.expect_err("non-UTF8 URI must fail")
|
||||
.contains("not UTF-8")
|
||||
);
|
||||
assert!(
|
||||
decode_access_description_der(&encode_sequence(&[
|
||||
encode_tlv(0x06, vec![0x80, 0x00]),
|
||||
uri.clone(),
|
||||
]))
|
||||
.expect_err("non-minimal OID must fail")
|
||||
.contains("non-minimal")
|
||||
);
|
||||
assert!(
|
||||
decode_access_description_der(&encode_sequence(&[encode_tlv(0x06, vec![0x81]), uri,]))
|
||||
.expect_err("truncated OID must fail")
|
||||
.contains("truncated OBJECT IDENTIFIER")
|
||||
);
|
||||
assert!(
|
||||
encode_access_description_der(&access_description("3.1", MANIFEST_URI))
|
||||
.expect_err("invalid OID must fail")
|
||||
.contains("unsupported accessMethod OID")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn access_description_der_codec_reports_malformed_selector_inputs() {
|
||||
let malformed_sequence = decode_access_description_der(&[0x31, 0x00])
|
||||
.expect_err("non-sequence AccessDescription must fail");
|
||||
assert!(!malformed_sequence.is_empty());
|
||||
|
||||
let missing_method = decode_access_description_der(&encode_sequence(&[encode_tlv(
|
||||
0x86,
|
||||
MANIFEST_URI.as_bytes().to_vec(),
|
||||
)]))
|
||||
.expect_err("missing accessMethod must fail");
|
||||
assert!(!missing_method.is_empty());
|
||||
|
||||
let oid = encode_oid_der(OID_AD_SIGNED_OBJECT).expect("encode signedObject OID");
|
||||
let missing_location = decode_access_description_der(&encode_sequence(&[oid.clone()]))
|
||||
.expect_err("missing accessLocation must fail");
|
||||
assert!(!missing_location.is_empty());
|
||||
|
||||
let wrong_location_tag = decode_access_description_der(&encode_sequence(&[
|
||||
oid.clone(),
|
||||
encode_tlv(0x04, MANIFEST_URI.as_bytes().to_vec()),
|
||||
]))
|
||||
.expect_err("wrong accessLocation tag must fail");
|
||||
assert!(!wrong_location_tag.is_empty());
|
||||
|
||||
let empty_oid = decode_access_description_der(&encode_sequence(&[
|
||||
encode_tlv(0x06, Vec::new()),
|
||||
encode_tlv(0x86, MANIFEST_URI.as_bytes().to_vec()),
|
||||
]))
|
||||
.expect_err("empty OID must fail");
|
||||
assert!(!empty_oid.is_empty());
|
||||
|
||||
let first_arc_zero = decode_access_description_der(&encode_sequence(&[
|
||||
encode_tlv(0x06, vec![0x01, 0x02]),
|
||||
encode_tlv(0x86, MANIFEST_URI.as_bytes().to_vec()),
|
||||
]))
|
||||
.expect("decode first OID arc zero");
|
||||
assert_eq!(first_arc_zero.access_method_oid, "0.1.2");
|
||||
assert_eq!(first_arc_zero.access_location, MANIFEST_URI);
|
||||
|
||||
let non_numeric =
|
||||
encode_access_description_der(&access_description("no.such.oid", MANIFEST_URI))
|
||||
.expect_err("non-numeric OID must fail");
|
||||
assert!(!non_numeric.is_empty());
|
||||
|
||||
let too_short = encode_access_description_der(&access_description("1", MANIFEST_URI))
|
||||
.expect_err("OID without second arc must fail");
|
||||
assert!(!too_short.is_empty());
|
||||
|
||||
let invalid_second_arc =
|
||||
encode_access_description_der(&access_description("1.40", MANIFEST_URI))
|
||||
.expect_err("invalid second OID arc must fail");
|
||||
assert!(!invalid_second_arc.is_empty());
|
||||
|
||||
let overflowing_first_subidentifier = encode_access_description_der(&access_description(
|
||||
"2.18446744073709551615",
|
||||
MANIFEST_URI,
|
||||
))
|
||||
.expect_err("overflowing first OID subidentifier must fail");
|
||||
assert!(!overflowing_first_subidentifier.is_empty());
|
||||
}
|
||||
}
|
||||
56
crates/panda-rpki-validator/src/ccr/mod.rs
Normal file
56
crates/panda-rpki-validator/src/ccr/mod.rs
Normal file
@ -0,0 +1,56 @@
|
||||
#[cfg(feature = "full")]
|
||||
pub mod accumulator;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod build;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod compare_view;
|
||||
pub mod decode;
|
||||
pub mod dump;
|
||||
pub mod encode;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod export;
|
||||
pub mod hash;
|
||||
#[cfg(feature = "full")]
|
||||
pub(crate) mod manifest_location;
|
||||
pub mod model;
|
||||
pub mod state_digest;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod verify;
|
||||
|
||||
#[cfg(feature = "full")]
|
||||
pub use accumulator::{CcrAccumulator, CcrManifestContribution};
|
||||
#[cfg(feature = "full")]
|
||||
pub use build::{
|
||||
CcrBuildError, ManifestStateBuildBreakdown, build_aspa_payload_state,
|
||||
build_manifest_state_from_vcirs, build_manifest_state_from_vcirs_with_breakdown,
|
||||
build_roa_payload_state, build_router_key_state_from_runtime, build_trust_anchor_state,
|
||||
};
|
||||
#[cfg(feature = "full")]
|
||||
pub use compare_view::{
|
||||
VapCompareRow, VrpCompareRow, build_vap_compare_rows, build_vrp_compare_rows,
|
||||
canonical_vrp_prefix, decode_ccr_compare_views, write_vap_csv, write_vrp_csv,
|
||||
};
|
||||
pub use decode::{CcrDecodeError, decode_content_info};
|
||||
pub use dump::{CcrDumpError, dump_content_info_json, dump_content_info_json_value};
|
||||
pub use encode::{CcrEncodeError, encode_content_info};
|
||||
#[cfg(feature = "full")]
|
||||
pub use export::{
|
||||
CcrBuildBreakdown, CcrExportError, build_ccr_from_run, build_ccr_from_run_with_breakdown,
|
||||
write_ccr_file,
|
||||
};
|
||||
pub use hash::{compute_state_hash, verify_state_hash};
|
||||
pub use model::{
|
||||
AspaPayloadSet, AspaPayloadState, CcrContentInfo, CcrDigestAlgorithm, ManifestInstance,
|
||||
ManifestState, RoaPayloadSet, RoaPayloadState, RouterKey, RouterKeySet, RouterKeyState,
|
||||
RpkiCanonicalCacheRepresentation, TrustAnchorState,
|
||||
};
|
||||
pub use state_digest::{
|
||||
CcrStateDigestComparison, CcrStateDigestError, CcrStateDigestStateComparison,
|
||||
CcrStateDigestSummary, compare_state_digests, decode_state_digest_summary,
|
||||
};
|
||||
#[cfg(feature = "full")]
|
||||
pub use verify::{
|
||||
CcrVerifyError, CcrVerifySummary, extract_vrp_rows, verify_against_report_json_path,
|
||||
verify_against_vcir_store, verify_against_vcir_store_path, verify_content_info,
|
||||
verify_content_info_bytes,
|
||||
};
|
||||
398
crates/panda-rpki-validator/src/ccr/model.rs
Normal file
398
crates/panda-rpki-validator/src/ccr/model.rs
Normal file
@ -0,0 +1,398 @@
|
||||
use crate::data_model::common::{BigUnsigned, der_take_tlv};
|
||||
use crate::data_model::oid::{OID_CT_RPKI_CCR, OID_SHA256};
|
||||
|
||||
pub const CCR_VERSION_V0: u32 = 0;
|
||||
pub const DIGEST_LEN_SHA256: usize = 32;
|
||||
pub const KEY_IDENTIFIER_LEN_SHA1: usize = 20;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum CcrDigestAlgorithm {
|
||||
Sha256,
|
||||
}
|
||||
|
||||
impl CcrDigestAlgorithm {
|
||||
pub fn oid(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Sha256 => OID_SHA256,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CcrContentInfo {
|
||||
pub content_type_oid: String,
|
||||
pub content: RpkiCanonicalCacheRepresentation,
|
||||
}
|
||||
|
||||
impl CcrContentInfo {
|
||||
pub fn new(content: RpkiCanonicalCacheRepresentation) -> Self {
|
||||
Self {
|
||||
content_type_oid: OID_CT_RPKI_CCR.to_string(),
|
||||
content,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.content_type_oid != OID_CT_RPKI_CCR {
|
||||
return Err(format!(
|
||||
"contentType must be {OID_CT_RPKI_CCR}, got {}",
|
||||
self.content_type_oid
|
||||
));
|
||||
}
|
||||
self.content.validate()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RpkiCanonicalCacheRepresentation {
|
||||
pub version: u32,
|
||||
pub hash_alg: CcrDigestAlgorithm,
|
||||
pub produced_at: time::OffsetDateTime,
|
||||
pub mfts: Option<ManifestState>,
|
||||
pub vrps: Option<RoaPayloadState>,
|
||||
pub vaps: Option<AspaPayloadState>,
|
||||
pub tas: Option<TrustAnchorState>,
|
||||
pub rks: Option<RouterKeyState>,
|
||||
}
|
||||
|
||||
impl RpkiCanonicalCacheRepresentation {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.version != CCR_VERSION_V0 {
|
||||
return Err(format!("CCR version must be 0, got {}", self.version));
|
||||
}
|
||||
if !matches!(self.hash_alg, CcrDigestAlgorithm::Sha256) {
|
||||
return Err("CCR hashAlg must be SHA-256".into());
|
||||
}
|
||||
if self.mfts.is_none()
|
||||
&& self.vrps.is_none()
|
||||
&& self.vaps.is_none()
|
||||
&& self.tas.is_none()
|
||||
&& self.rks.is_none()
|
||||
{
|
||||
return Err("at least one of mfts/vrps/vaps/tas/rks must be present".into());
|
||||
}
|
||||
if let Some(mfts) = &self.mfts {
|
||||
mfts.validate()?;
|
||||
}
|
||||
if let Some(vrps) = &self.vrps {
|
||||
vrps.validate()?;
|
||||
}
|
||||
if let Some(vaps) = &self.vaps {
|
||||
vaps.validate()?;
|
||||
}
|
||||
if let Some(tas) = &self.tas {
|
||||
tas.validate()?;
|
||||
}
|
||||
if let Some(rks) = &self.rks {
|
||||
rks.validate()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ManifestState {
|
||||
pub mis: Vec<ManifestInstance>,
|
||||
pub most_recent_update: time::OffsetDateTime,
|
||||
pub hash: Vec<u8>,
|
||||
}
|
||||
|
||||
impl ManifestState {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
validate_sha256_digest("ManifestState.hash", &self.hash)?;
|
||||
validate_sorted_unique_by(
|
||||
&self.mis,
|
||||
|item| item.hash.as_slice(),
|
||||
"ManifestState.mis must be sorted by hash and unique",
|
||||
)?;
|
||||
for instance in &self.mis {
|
||||
instance.validate()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ManifestInstance {
|
||||
pub hash: Vec<u8>,
|
||||
pub size: u64,
|
||||
pub aki: Vec<u8>,
|
||||
pub manifest_number: BigUnsigned,
|
||||
pub this_update: time::OffsetDateTime,
|
||||
pub locations: Vec<Vec<u8>>,
|
||||
pub subordinates: Vec<Vec<u8>>,
|
||||
}
|
||||
|
||||
impl ManifestInstance {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
validate_sha256_digest("ManifestInstance.hash", &self.hash)?;
|
||||
if self.size < 1000 {
|
||||
return Err(format!(
|
||||
"ManifestInstance.size must be >= 1000, got {}",
|
||||
self.size
|
||||
));
|
||||
}
|
||||
validate_key_identifier("ManifestInstance.aki", &self.aki)?;
|
||||
validate_big_unsigned_bytes(
|
||||
"ManifestInstance.manifest_number",
|
||||
&self.manifest_number.bytes_be,
|
||||
)?;
|
||||
if self.locations.is_empty() {
|
||||
return Err(
|
||||
"ManifestInstance.locations must contain at least one AccessDescription".into(),
|
||||
);
|
||||
}
|
||||
for location in &self.locations {
|
||||
validate_full_der_with_tag("ManifestInstance.locations[]", location, Some(0x30))?;
|
||||
}
|
||||
if !self.subordinates.is_empty() {
|
||||
validate_sorted_unique_bytes(
|
||||
&self.subordinates,
|
||||
KEY_IDENTIFIER_LEN_SHA1,
|
||||
"ManifestInstance.subordinates must be sorted/unique 20-byte SKIs",
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaPayloadState {
|
||||
pub rps: Vec<RoaPayloadSet>,
|
||||
pub hash: Vec<u8>,
|
||||
}
|
||||
|
||||
impl RoaPayloadState {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
validate_sha256_digest("ROAPayloadState.hash", &self.hash)?;
|
||||
validate_sorted_unique_by(
|
||||
&self.rps,
|
||||
|item| &item.as_id,
|
||||
"ROAPayloadState.rps must be sorted by asID and unique",
|
||||
)?;
|
||||
for set in &self.rps {
|
||||
set.validate()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaPayloadSet {
|
||||
pub as_id: u32,
|
||||
pub ip_addr_blocks: Vec<Vec<u8>>,
|
||||
}
|
||||
|
||||
impl RoaPayloadSet {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.ip_addr_blocks.is_empty() || self.ip_addr_blocks.len() > 2 {
|
||||
return Err(format!(
|
||||
"ROAPayloadSet.ip_addr_blocks must contain 1..=2 entries, got {}",
|
||||
self.ip_addr_blocks.len()
|
||||
));
|
||||
}
|
||||
for block in &self.ip_addr_blocks {
|
||||
validate_full_der_with_tag("ROAPayloadSet.ip_addr_blocks[]", block, Some(0x30))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AspaPayloadState {
|
||||
pub aps: Vec<AspaPayloadSet>,
|
||||
pub hash: Vec<u8>,
|
||||
}
|
||||
|
||||
impl AspaPayloadState {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
validate_sha256_digest("ASPAPayloadState.hash", &self.hash)?;
|
||||
validate_sorted_unique_by(
|
||||
&self.aps,
|
||||
|item| &item.customer_as_id,
|
||||
"ASPAPayloadState.aps must be sorted by customerASID and unique",
|
||||
)?;
|
||||
for set in &self.aps {
|
||||
set.validate()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AspaPayloadSet {
|
||||
pub customer_as_id: u32,
|
||||
pub providers: Vec<u32>,
|
||||
}
|
||||
|
||||
impl AspaPayloadSet {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.providers.is_empty() {
|
||||
return Err("ASPAPayloadSet.providers must be non-empty".into());
|
||||
}
|
||||
validate_sorted_unique_by(
|
||||
&self.providers,
|
||||
|provider| provider,
|
||||
"ASPAPayloadSet.providers must be sorted ascending and unique",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct TrustAnchorState {
|
||||
pub skis: Vec<Vec<u8>>,
|
||||
pub hash: Vec<u8>,
|
||||
}
|
||||
|
||||
impl TrustAnchorState {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.skis.is_empty() {
|
||||
return Err("TrustAnchorState.skis must be non-empty".into());
|
||||
}
|
||||
validate_sha256_digest("TrustAnchorState.hash", &self.hash)?;
|
||||
validate_sorted_unique_bytes(
|
||||
&self.skis,
|
||||
KEY_IDENTIFIER_LEN_SHA1,
|
||||
"TrustAnchorState.skis must be sorted/unique 20-byte SKIs",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RouterKeyState {
|
||||
pub rksets: Vec<RouterKeySet>,
|
||||
pub hash: Vec<u8>,
|
||||
}
|
||||
|
||||
impl RouterKeyState {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
validate_sha256_digest("RouterKeyState.hash", &self.hash)?;
|
||||
validate_sorted_unique_by(
|
||||
&self.rksets,
|
||||
|item| &item.as_id,
|
||||
"RouterKeyState.rksets must be sorted by asID and unique",
|
||||
)?;
|
||||
for rkset in &self.rksets {
|
||||
rkset.validate()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RouterKeySet {
|
||||
pub as_id: u32,
|
||||
pub router_keys: Vec<RouterKey>,
|
||||
}
|
||||
|
||||
impl RouterKeySet {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.router_keys.is_empty() {
|
||||
return Err("RouterKeySet.router_keys must be non-empty".into());
|
||||
}
|
||||
validate_sorted_unique_by(
|
||||
&self.router_keys,
|
||||
|key| key,
|
||||
"RouterKeySet.router_keys must be sorted by SKI and unique by (SKI, SPKI DER)",
|
||||
)?;
|
||||
for key in &self.router_keys {
|
||||
key.validate()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub struct RouterKey {
|
||||
pub ski: Vec<u8>,
|
||||
pub spki_der: Vec<u8>,
|
||||
}
|
||||
|
||||
impl RouterKey {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
validate_key_identifier("RouterKey.ski", &self.ski)?;
|
||||
validate_full_der_with_tag("RouterKey.spki_der", &self.spki_der, Some(0x30))
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_sha256_digest(field: &str, bytes: &[u8]) -> Result<(), String> {
|
||||
if bytes.len() != DIGEST_LEN_SHA256 {
|
||||
return Err(format!(
|
||||
"{field} must be {DIGEST_LEN_SHA256} bytes, got {}",
|
||||
bytes.len()
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_key_identifier(field: &str, bytes: &[u8]) -> Result<(), String> {
|
||||
if bytes.len() != KEY_IDENTIFIER_LEN_SHA1 {
|
||||
return Err(format!(
|
||||
"{field} must be {KEY_IDENTIFIER_LEN_SHA1} bytes, got {}",
|
||||
bytes.len()
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_big_unsigned_bytes(field: &str, bytes: &[u8]) -> Result<(), String> {
|
||||
if bytes.is_empty() {
|
||||
return Err(format!("{field} must not be empty"));
|
||||
}
|
||||
if bytes.len() > 1 && bytes[0] == 0x00 {
|
||||
return Err(format!(
|
||||
"{field} must be minimally encoded as an unsigned integer"
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_sorted_unique_by<T, K: Ord + ?Sized>(
|
||||
values: &[T],
|
||||
key_fn: impl Fn(&T) -> &K,
|
||||
message: &str,
|
||||
) -> Result<(), String> {
|
||||
for window in values.windows(2) {
|
||||
if key_fn(&window[0]) >= key_fn(&window[1]) {
|
||||
return Err(message.to_string());
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_sorted_unique_bytes(
|
||||
values: &[Vec<u8>],
|
||||
expected_len: usize,
|
||||
message: &str,
|
||||
) -> Result<(), String> {
|
||||
for value in values {
|
||||
if value.len() != expected_len {
|
||||
return Err(message.to_string());
|
||||
}
|
||||
}
|
||||
for window in values.windows(2) {
|
||||
if window[0] >= window[1] {
|
||||
return Err(message.to_string());
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_full_der_with_tag(
|
||||
field: &str,
|
||||
der: &[u8],
|
||||
expected_tag: Option<u8>,
|
||||
) -> Result<(), String> {
|
||||
let (tag, _value, rem) = der_take_tlv(der).map_err(|e| format!("{field}: {e}"))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(format!("{field}: trailing bytes after DER object"));
|
||||
}
|
||||
if let Some(expected_tag) = expected_tag {
|
||||
if tag != expected_tag {
|
||||
return Err(format!(
|
||||
"{field}: unexpected tag 0x{tag:02X}, expected 0x{expected_tag:02X}"
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
332
crates/panda-rpki-validator/src/ccr/state_digest.rs
Normal file
332
crates/panda-rpki-validator/src/ccr/state_digest.rs
Normal file
@ -0,0 +1,332 @@
|
||||
use crate::data_model::common::DerReader;
|
||||
use crate::data_model::oid::{OID_CT_RPKI_CCR_RAW, OID_SHA256_RAW};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CcrStateDigestSummary {
|
||||
pub version: u32,
|
||||
pub hash_alg_oid: String,
|
||||
pub mfts: Option<Vec<u8>>,
|
||||
pub vrps: Option<Vec<u8>>,
|
||||
pub vaps: Option<Vec<u8>>,
|
||||
pub tas: Option<Vec<u8>>,
|
||||
pub rks: Option<Vec<u8>>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CcrStateDigestComparison {
|
||||
pub ours: CcrStateDigestSummary,
|
||||
pub peer: CcrStateDigestSummary,
|
||||
pub states: Vec<CcrStateDigestStateComparison>,
|
||||
}
|
||||
|
||||
impl CcrStateDigestComparison {
|
||||
pub fn matches(&self) -> bool {
|
||||
self.ours.version == self.peer.version
|
||||
&& self.ours.hash_alg_oid == self.peer.hash_alg_oid
|
||||
&& self.states.iter().all(|state| state.matches)
|
||||
}
|
||||
|
||||
pub fn mismatched_state_names(&self) -> Vec<&'static str> {
|
||||
self.states
|
||||
.iter()
|
||||
.filter(|state| !state.matches)
|
||||
.map(|state| state.name)
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CcrStateDigestStateComparison {
|
||||
pub name: &'static str,
|
||||
pub ours_present: bool,
|
||||
pub peer_present: bool,
|
||||
pub ours_hash_hex: Option<String>,
|
||||
pub peer_hash_hex: Option<String>,
|
||||
pub matches: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrStateDigestError {
|
||||
#[error("DER parse error: {0}")]
|
||||
Parse(String),
|
||||
|
||||
#[error("unexpected contentType OID")]
|
||||
UnexpectedContentType,
|
||||
|
||||
#[error("unexpected digest algorithm OID")]
|
||||
UnexpectedDigestAlgorithm,
|
||||
|
||||
#[error("unexpected CCR field tag 0x{0:02X}")]
|
||||
UnexpectedCcrField(u8),
|
||||
}
|
||||
|
||||
pub fn decode_state_digest_summary(
|
||||
der: &[u8],
|
||||
) -> Result<CcrStateDigestSummary, CcrStateDigestError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrStateDigestError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing bytes after ContentInfo".into(),
|
||||
));
|
||||
}
|
||||
let content_type_raw = seq.take_tag(0x06).map_err(CcrStateDigestError::Parse)?;
|
||||
if content_type_raw != OID_CT_RPKI_CCR_RAW {
|
||||
return Err(CcrStateDigestError::UnexpectedContentType);
|
||||
}
|
||||
let inner = seq.take_tag(0xA0).map_err(CcrStateDigestError::Parse)?;
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing fields in ContentInfo".into(),
|
||||
));
|
||||
}
|
||||
decode_ccr_state_digest_summary(inner)
|
||||
}
|
||||
|
||||
pub fn compare_state_digests(
|
||||
ours_der: &[u8],
|
||||
peer_der: &[u8],
|
||||
) -> Result<CcrStateDigestComparison, CcrStateDigestError> {
|
||||
let ours = decode_state_digest_summary(ours_der)?;
|
||||
let peer = decode_state_digest_summary(peer_der)?;
|
||||
let states = vec![
|
||||
compare_state("mfts", &ours.mfts, &peer.mfts),
|
||||
compare_state("vrps", &ours.vrps, &peer.vrps),
|
||||
compare_state("vaps", &ours.vaps, &peer.vaps),
|
||||
compare_state("tas", &ours.tas, &peer.tas),
|
||||
compare_state("rks", &ours.rks, &peer.rks),
|
||||
];
|
||||
Ok(CcrStateDigestComparison { ours, peer, states })
|
||||
}
|
||||
|
||||
fn decode_ccr_state_digest_summary(
|
||||
der: &[u8],
|
||||
) -> Result<CcrStateDigestSummary, CcrStateDigestError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CcrStateDigestError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing bytes after CCR".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let version = if !seq.is_empty() && seq.peek_tag().map_err(CcrStateDigestError::Parse)? == 0xA0
|
||||
{
|
||||
let explicit = seq.take_tag(0xA0).map_err(CcrStateDigestError::Parse)?;
|
||||
let mut inner = DerReader::new(explicit);
|
||||
let version = inner.take_uint_u64().map_err(CcrStateDigestError::Parse)? as u32;
|
||||
if !inner.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing bytes inside CCR version EXPLICIT".into(),
|
||||
));
|
||||
}
|
||||
version
|
||||
} else {
|
||||
crate::ccr::model::CCR_VERSION_V0
|
||||
};
|
||||
|
||||
let hash_alg_oid =
|
||||
decode_digest_algorithm_oid(seq.take_sequence().map_err(CcrStateDigestError::Parse)?)?;
|
||||
let _produced_at = seq.take_tag(0x18).map_err(CcrStateDigestError::Parse)?;
|
||||
|
||||
let mut mfts = None;
|
||||
let mut vrps = None;
|
||||
let mut vaps = None;
|
||||
let mut tas = None;
|
||||
let mut rks = None;
|
||||
while !seq.is_empty() {
|
||||
let (tag, value) = seq.take_any().map_err(CcrStateDigestError::Parse)?;
|
||||
match tag {
|
||||
0xA1 => mfts = Some(read_state_hash(value, StateShape::Manifest)?),
|
||||
0xA2 => vrps = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
|
||||
0xA3 => vaps = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
|
||||
0xA4 => tas = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
|
||||
0xA5 => rks = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
|
||||
other => return Err(CcrStateDigestError::UnexpectedCcrField(other)),
|
||||
}
|
||||
}
|
||||
|
||||
Ok(CcrStateDigestSummary {
|
||||
version,
|
||||
hash_alg_oid,
|
||||
mfts,
|
||||
vrps,
|
||||
vaps,
|
||||
tas,
|
||||
rks,
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
enum StateShape {
|
||||
Manifest,
|
||||
PayloadThenHash,
|
||||
}
|
||||
|
||||
fn read_state_hash(explicit_der: &[u8], shape: StateShape) -> Result<Vec<u8>, CcrStateDigestError> {
|
||||
let mut outer = DerReader::new(explicit_der);
|
||||
let mut seq = outer.take_sequence().map_err(CcrStateDigestError::Parse)?;
|
||||
if !outer.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing bytes after CCR state".into(),
|
||||
));
|
||||
}
|
||||
match shape {
|
||||
StateShape::Manifest => {
|
||||
seq.skip_any().map_err(CcrStateDigestError::Parse)?;
|
||||
seq.take_tag(0x18).map_err(CcrStateDigestError::Parse)?;
|
||||
}
|
||||
StateShape::PayloadThenHash => {
|
||||
seq.skip_any().map_err(CcrStateDigestError::Parse)?;
|
||||
}
|
||||
}
|
||||
let hash = seq
|
||||
.take_octet_string()
|
||||
.map_err(CcrStateDigestError::Parse)?
|
||||
.to_vec();
|
||||
if hash.len() != crate::ccr::model::DIGEST_LEN_SHA256 {
|
||||
return Err(CcrStateDigestError::Parse(format!(
|
||||
"state hash must be {} bytes, got {}",
|
||||
crate::ccr::model::DIGEST_LEN_SHA256,
|
||||
hash.len()
|
||||
)));
|
||||
}
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing fields after CCR state hash".into(),
|
||||
));
|
||||
}
|
||||
Ok(hash)
|
||||
}
|
||||
|
||||
fn decode_digest_algorithm_oid(mut seq: DerReader<'_>) -> Result<String, CcrStateDigestError> {
|
||||
let oid_raw = seq.take_tag(0x06).map_err(CcrStateDigestError::Parse)?;
|
||||
if oid_raw != OID_SHA256_RAW {
|
||||
return Err(CcrStateDigestError::UnexpectedDigestAlgorithm);
|
||||
}
|
||||
if !seq.is_empty() {
|
||||
let tag = seq.peek_tag().map_err(CcrStateDigestError::Parse)?;
|
||||
if tag == 0x05 {
|
||||
let null = seq.take_tag(0x05).map_err(CcrStateDigestError::Parse)?;
|
||||
if !null.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"AlgorithmIdentifier NULL parameters must be empty".into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
if !seq.is_empty() {
|
||||
return Err(CcrStateDigestError::Parse(
|
||||
"trailing fields in DigestAlgorithmIdentifier".into(),
|
||||
));
|
||||
}
|
||||
Ok(crate::data_model::oid::OID_SHA256.to_string())
|
||||
}
|
||||
|
||||
fn compare_state(
|
||||
name: &'static str,
|
||||
ours: &Option<Vec<u8>>,
|
||||
peer: &Option<Vec<u8>>,
|
||||
) -> CcrStateDigestStateComparison {
|
||||
CcrStateDigestStateComparison {
|
||||
name,
|
||||
ours_present: ours.is_some(),
|
||||
peer_present: peer.is_some(),
|
||||
ours_hash_hex: ours.as_ref().map(hex::encode),
|
||||
peer_hash_hex: peer.as_ref().map(hex::encode),
|
||||
matches: ours == peer,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::{
|
||||
CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation,
|
||||
build_aspa_payload_state, build_roa_payload_state, encode_content_info,
|
||||
};
|
||||
use crate::data_model::roa::{IpPrefix, RoaAfi};
|
||||
use crate::validation::objects::{AspaAttestation, Vrp};
|
||||
|
||||
fn sample_content(produced_at: time::OffsetDateTime) -> CcrContentInfo {
|
||||
let vrps = build_roa_payload_state(&[Vrp {
|
||||
asn: 64496,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv4,
|
||||
prefix_len: 24,
|
||||
addr: [192, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
},
|
||||
max_length: 24,
|
||||
}])
|
||||
.expect("build vrps");
|
||||
let vaps = build_aspa_payload_state(&[AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![64497],
|
||||
}])
|
||||
.expect("build vaps");
|
||||
CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at,
|
||||
mfts: None,
|
||||
vrps: Some(vrps),
|
||||
vaps: Some(vaps),
|
||||
tas: None,
|
||||
rks: None,
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_state_digest_summary_extracts_hashes_without_full_model() {
|
||||
let content = sample_content(time::OffsetDateTime::UNIX_EPOCH);
|
||||
let der = encode_content_info(&content).expect("encode");
|
||||
let summary = decode_state_digest_summary(&der).expect("summary");
|
||||
assert_eq!(summary.version, 0);
|
||||
assert_eq!(summary.hash_alg_oid, crate::data_model::oid::OID_SHA256);
|
||||
assert_eq!(
|
||||
summary.vrps.as_deref(),
|
||||
content
|
||||
.content
|
||||
.vrps
|
||||
.as_ref()
|
||||
.map(|state| state.hash.as_slice())
|
||||
);
|
||||
assert_eq!(
|
||||
summary.vaps.as_deref(),
|
||||
content
|
||||
.content
|
||||
.vaps
|
||||
.as_ref()
|
||||
.map(|state| state.hash.as_slice())
|
||||
);
|
||||
assert!(summary.mfts.is_none());
|
||||
assert!(summary.tas.is_none());
|
||||
assert!(summary.rks.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compare_state_digests_ignores_produced_at() {
|
||||
let first = encode_content_info(&sample_content(time::OffsetDateTime::UNIX_EPOCH))
|
||||
.expect("encode first");
|
||||
let second = encode_content_info(&sample_content(
|
||||
time::OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(60),
|
||||
))
|
||||
.expect("encode second");
|
||||
let comparison = compare_state_digests(&first, &second).expect("compare");
|
||||
assert!(comparison.matches());
|
||||
assert!(comparison.mismatched_state_names().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compare_state_digests_reports_mismatched_state() {
|
||||
let first = encode_content_info(&sample_content(time::OffsetDateTime::UNIX_EPOCH))
|
||||
.expect("encode first");
|
||||
let mut second_content =
|
||||
sample_content(time::OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(60));
|
||||
second_content.content.vaps.as_mut().expect("vaps").hash[0] ^= 0xFF;
|
||||
let second = encode_content_info(&second_content).expect("encode second");
|
||||
let comparison = compare_state_digests(&first, &second).expect("compare");
|
||||
assert!(!comparison.matches());
|
||||
assert_eq!(comparison.mismatched_state_names(), vec!["vaps"]);
|
||||
}
|
||||
}
|
||||
704
crates/panda-rpki-validator/src/ccr/verify.rs
Normal file
704
crates/panda-rpki-validator/src/ccr/verify.rs
Normal file
@ -0,0 +1,704 @@
|
||||
use crate::ccr::decode::{CcrDecodeError, decode_content_info};
|
||||
use crate::ccr::encode::{
|
||||
encode_aspa_payload_state_payload_der, encode_manifest_state_payload_der,
|
||||
encode_roa_payload_state_payload_der, encode_router_key_state_payload_der,
|
||||
encode_trust_anchor_state_payload_der,
|
||||
};
|
||||
use crate::ccr::hash::verify_state_hash;
|
||||
use crate::ccr::model::{CcrContentInfo, RouterKeyState, TrustAnchorState};
|
||||
use crate::storage::{RocksStore, VcirArtifactRole};
|
||||
use serde::Serialize;
|
||||
use std::collections::BTreeSet;
|
||||
use std::path::Path;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct CcrVerifySummary {
|
||||
pub content_type_oid: String,
|
||||
pub version: u32,
|
||||
pub produced_at_rfc3339_utc: String,
|
||||
pub state_hashes_ok: bool,
|
||||
pub manifest_instances: usize,
|
||||
pub roa_payload_sets: usize,
|
||||
pub roa_vrp_count: usize,
|
||||
pub aspa_payload_sets: usize,
|
||||
pub trust_anchor_ski_count: usize,
|
||||
pub router_key_sets: usize,
|
||||
pub router_key_count: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CcrVerifyError {
|
||||
#[error("CCR decode failed: {0}")]
|
||||
Decode(#[from] CcrDecodeError),
|
||||
|
||||
#[error("ManifestState hash mismatch")]
|
||||
ManifestHashMismatch,
|
||||
|
||||
#[error("ROAPayloadState hash mismatch")]
|
||||
RoaHashMismatch,
|
||||
|
||||
#[error("ASPAPayloadState hash mismatch")]
|
||||
AspaHashMismatch,
|
||||
|
||||
#[error("TrustAnchorState hash mismatch")]
|
||||
TrustAnchorHashMismatch,
|
||||
|
||||
#[error("RouterKeyState hash mismatch")]
|
||||
RouterKeyHashMismatch,
|
||||
|
||||
#[error("read report json failed: {0}: {1}")]
|
||||
ReportRead(String, String),
|
||||
|
||||
#[error("parse report json failed: {0}")]
|
||||
ReportParse(String),
|
||||
|
||||
#[error("VRP set mismatch: only_in_ccr={only_in_ccr} only_in_report={only_in_report}")]
|
||||
ReportVrpMismatch {
|
||||
only_in_ccr: usize,
|
||||
only_in_report: usize,
|
||||
},
|
||||
|
||||
#[error("ASPA set mismatch: only_in_ccr={only_in_ccr} only_in_report={only_in_report}")]
|
||||
ReportAspaMismatch {
|
||||
only_in_ccr: usize,
|
||||
only_in_report: usize,
|
||||
},
|
||||
|
||||
#[error("open RocksDB failed: {0}")]
|
||||
OpenStore(String),
|
||||
|
||||
#[error("list VCIRs failed: {0}")]
|
||||
ListVcirs(String),
|
||||
|
||||
#[error("VCIR manifest set mismatch: only_in_ccr={only_in_ccr} only_in_vcir={only_in_vcir}")]
|
||||
VcirManifestMismatch {
|
||||
only_in_ccr: usize,
|
||||
only_in_vcir: usize,
|
||||
},
|
||||
}
|
||||
|
||||
pub fn verify_content_info_bytes(der: &[u8]) -> Result<CcrVerifySummary, CcrVerifyError> {
|
||||
let content_info = decode_content_info(der)?;
|
||||
verify_content_info(&content_info)
|
||||
}
|
||||
|
||||
pub fn verify_content_info(
|
||||
content_info: &CcrContentInfo,
|
||||
) -> Result<CcrVerifySummary, CcrVerifyError> {
|
||||
content_info.validate().map_err(CcrDecodeError::Validate)?;
|
||||
let state_hashes_ok = true;
|
||||
let mut manifest_instances = 0usize;
|
||||
let mut roa_payload_sets = 0usize;
|
||||
let mut roa_vrp_count = 0usize;
|
||||
let mut aspa_payload_sets = 0usize;
|
||||
let mut trust_anchor_ski_count = 0usize;
|
||||
let mut router_key_sets = 0usize;
|
||||
let mut router_key_count = 0usize;
|
||||
|
||||
if let Some(mfts) = &content_info.content.mfts {
|
||||
let payload_der = encode_manifest_state_payload_der(&mfts.mis)
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
|
||||
if !verify_state_hash(&mfts.hash, &payload_der) {
|
||||
return Err(CcrVerifyError::ManifestHashMismatch);
|
||||
}
|
||||
manifest_instances = mfts.mis.len();
|
||||
}
|
||||
if let Some(vrps) = &content_info.content.vrps {
|
||||
let payload_der = encode_roa_payload_state_payload_der(&vrps.rps)
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
|
||||
if !verify_state_hash(&vrps.hash, &payload_der) {
|
||||
return Err(CcrVerifyError::RoaHashMismatch);
|
||||
}
|
||||
roa_payload_sets = vrps.rps.len();
|
||||
roa_vrp_count = vrps
|
||||
.rps
|
||||
.iter()
|
||||
.map(|set| count_roa_block_entries(&set.ip_addr_blocks))
|
||||
.sum();
|
||||
}
|
||||
if let Some(vaps) = &content_info.content.vaps {
|
||||
let payload_der = encode_aspa_payload_state_payload_der(&vaps.aps)
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
|
||||
if !verify_state_hash(&vaps.hash, &payload_der) {
|
||||
return Err(CcrVerifyError::AspaHashMismatch);
|
||||
}
|
||||
aspa_payload_sets = vaps.aps.len();
|
||||
}
|
||||
if let Some(tas) = &content_info.content.tas {
|
||||
verify_trust_anchor_state_hash(tas)?;
|
||||
trust_anchor_ski_count = tas.skis.len();
|
||||
}
|
||||
if let Some(rks) = &content_info.content.rks {
|
||||
verify_router_key_state_hash(rks)?;
|
||||
router_key_sets = rks.rksets.len();
|
||||
router_key_count = rks.rksets.iter().map(|set| set.router_keys.len()).sum();
|
||||
}
|
||||
|
||||
let produced_at_rfc3339_utc = content_info
|
||||
.content
|
||||
.produced_at
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&time::format_description::well_known::Rfc3339)
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
|
||||
|
||||
Ok(CcrVerifySummary {
|
||||
content_type_oid: content_info.content_type_oid.clone(),
|
||||
version: content_info.content.version,
|
||||
produced_at_rfc3339_utc,
|
||||
state_hashes_ok,
|
||||
manifest_instances,
|
||||
roa_payload_sets,
|
||||
roa_vrp_count,
|
||||
aspa_payload_sets,
|
||||
trust_anchor_ski_count,
|
||||
router_key_sets,
|
||||
router_key_count,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn verify_against_report_json_path(
|
||||
content_info: &CcrContentInfo,
|
||||
report_json_path: &Path,
|
||||
) -> Result<(), CcrVerifyError> {
|
||||
let bytes = std::fs::read(report_json_path).map_err(|e| {
|
||||
CcrVerifyError::ReportRead(report_json_path.display().to_string(), e.to_string())
|
||||
})?;
|
||||
let json: serde_json::Value =
|
||||
serde_json::from_slice(&bytes).map_err(|e| CcrVerifyError::ReportParse(e.to_string()))?;
|
||||
|
||||
let report_vrps = report_vrp_keys(&json)?;
|
||||
let ccr_vrps = extract_vrp_rows(content_info)?;
|
||||
let only_in_ccr = ccr_vrps.difference(&report_vrps).count();
|
||||
let only_in_report = report_vrps.difference(&ccr_vrps).count();
|
||||
if only_in_ccr != 0 || only_in_report != 0 {
|
||||
return Err(CcrVerifyError::ReportVrpMismatch {
|
||||
only_in_ccr,
|
||||
only_in_report,
|
||||
});
|
||||
}
|
||||
|
||||
let report_aspas = report_aspa_keys(&json)?;
|
||||
let ccr_aspas = ccr_aspa_keys(content_info)?;
|
||||
let only_in_ccr = ccr_aspas.difference(&report_aspas).count();
|
||||
let only_in_report = report_aspas.difference(&ccr_aspas).count();
|
||||
if only_in_ccr != 0 || only_in_report != 0 {
|
||||
return Err(CcrVerifyError::ReportAspaMismatch {
|
||||
only_in_ccr,
|
||||
only_in_report,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn verify_against_vcir_store_path(
|
||||
content_info: &CcrContentInfo,
|
||||
db_path: &Path,
|
||||
) -> Result<(), CcrVerifyError> {
|
||||
let store = RocksStore::open(db_path).map_err(|e| CcrVerifyError::OpenStore(e.to_string()))?;
|
||||
verify_against_vcir_store(content_info, &store)
|
||||
}
|
||||
|
||||
pub fn verify_against_vcir_store(
|
||||
content_info: &CcrContentInfo,
|
||||
store: &RocksStore,
|
||||
) -> Result<(), CcrVerifyError> {
|
||||
let Some(mfts) = &content_info.content.mfts else {
|
||||
return Ok(());
|
||||
};
|
||||
let vcirs = store
|
||||
.list_vcirs()
|
||||
.map_err(|e| CcrVerifyError::ListVcirs(e.to_string()))?;
|
||||
let mut vcir_hashes = BTreeSet::new();
|
||||
for vcir in vcirs {
|
||||
if let Some(artifact) = vcir.related_artifacts.iter().find(|artifact| {
|
||||
artifact.artifact_role == VcirArtifactRole::Manifest
|
||||
&& artifact.uri.as_deref() == Some(vcir.current_manifest_rsync_uri.as_str())
|
||||
}) {
|
||||
if let Ok(bytes) = hex::decode(&artifact.sha256) {
|
||||
vcir_hashes.insert(bytes);
|
||||
}
|
||||
}
|
||||
}
|
||||
let ccr_hashes = mfts
|
||||
.mis
|
||||
.iter()
|
||||
.map(|mi| mi.hash.clone())
|
||||
.collect::<BTreeSet<_>>();
|
||||
let only_in_ccr = ccr_hashes.difference(&vcir_hashes).count();
|
||||
let only_in_vcir = vcir_hashes.difference(&ccr_hashes).count();
|
||||
if only_in_ccr != 0 || only_in_vcir != 0 {
|
||||
return Err(CcrVerifyError::VcirManifestMismatch {
|
||||
only_in_ccr,
|
||||
only_in_vcir,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn verify_trust_anchor_state_hash(state: &TrustAnchorState) -> Result<(), CcrVerifyError> {
|
||||
let payload_der = encode_trust_anchor_state_payload_der(&state.skis)
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
|
||||
if !verify_state_hash(&state.hash, &payload_der) {
|
||||
return Err(CcrVerifyError::TrustAnchorHashMismatch);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn verify_router_key_state_hash(state: &RouterKeyState) -> Result<(), CcrVerifyError> {
|
||||
let payload_der = encode_router_key_state_payload_der(&state.rksets)
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
|
||||
if !verify_state_hash(&state.hash, &payload_der) {
|
||||
return Err(CcrVerifyError::RouterKeyHashMismatch);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn report_vrp_keys(
|
||||
json: &serde_json::Value,
|
||||
) -> Result<BTreeSet<(u32, String, u16)>, CcrVerifyError> {
|
||||
let mut out = BTreeSet::new();
|
||||
let Some(items) = json.get("vrps").and_then(|v| v.as_array()) else {
|
||||
return Ok(out);
|
||||
};
|
||||
for item in items {
|
||||
let asn = item
|
||||
.get("asn")
|
||||
.and_then(|v| v.as_u64())
|
||||
.ok_or_else(|| CcrVerifyError::ReportParse("vrps[].asn missing".into()))?
|
||||
as u32;
|
||||
let prefix = item
|
||||
.get("prefix")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or_else(|| CcrVerifyError::ReportParse("vrps[].prefix missing".into()))?
|
||||
.to_string();
|
||||
let max_length = item
|
||||
.get("max_length")
|
||||
.and_then(|v| v.as_u64())
|
||||
.ok_or_else(|| CcrVerifyError::ReportParse("vrps[].max_length missing".into()))?
|
||||
as u16;
|
||||
out.insert((asn, prefix, max_length));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn report_aspa_keys(json: &serde_json::Value) -> Result<BTreeSet<(u32, Vec<u32>)>, CcrVerifyError> {
|
||||
let mut out = BTreeSet::new();
|
||||
let Some(items) = json.get("aspas").and_then(|v| v.as_array()) else {
|
||||
return Ok(out);
|
||||
};
|
||||
for item in items {
|
||||
let customer = item
|
||||
.get("customer_as_id")
|
||||
.and_then(|v| v.as_u64())
|
||||
.ok_or_else(|| CcrVerifyError::ReportParse("aspas[].customer_as_id missing".into()))?
|
||||
as u32;
|
||||
let mut providers = item
|
||||
.get("provider_as_ids")
|
||||
.and_then(|v| v.as_array())
|
||||
.ok_or_else(|| CcrVerifyError::ReportParse("aspas[].provider_as_ids missing".into()))?
|
||||
.iter()
|
||||
.map(|v| {
|
||||
v.as_u64()
|
||||
.ok_or_else(|| CcrVerifyError::ReportParse("provider_as_ids[] invalid".into()))
|
||||
.map(|v| v as u32)
|
||||
})
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
providers.sort_unstable();
|
||||
providers.dedup();
|
||||
out.insert((customer, providers));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
pub fn extract_vrp_rows(
|
||||
content_info: &CcrContentInfo,
|
||||
) -> Result<BTreeSet<(u32, String, u16)>, CcrVerifyError> {
|
||||
let mut out = BTreeSet::new();
|
||||
let Some(vrps) = &content_info.content.vrps else {
|
||||
return Ok(out);
|
||||
};
|
||||
for set in &vrps.rps {
|
||||
for block in &set.ip_addr_blocks {
|
||||
let (afi, entries) = decode_roa_family_block(block)?;
|
||||
for (prefix_len, addr_bytes, max_len) in entries {
|
||||
let prefix = format_prefix(afi, &addr_bytes, prefix_len)?;
|
||||
out.insert((set.as_id, prefix, max_len.unwrap_or(prefix_len as u16)));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn ccr_aspa_keys(
|
||||
content_info: &CcrContentInfo,
|
||||
) -> Result<BTreeSet<(u32, Vec<u32>)>, CcrVerifyError> {
|
||||
let mut out = BTreeSet::new();
|
||||
let Some(vaps) = &content_info.content.vaps else {
|
||||
return Ok(out);
|
||||
};
|
||||
for set in &vaps.aps {
|
||||
out.insert((set.customer_as_id, set.providers.clone()));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn decode_roa_family_block(
|
||||
block: &[u8],
|
||||
) -> Result<(u16, Vec<(u8, Vec<u8>, Option<u16>)>), CcrVerifyError> {
|
||||
let mut top = crate::data_model::common::DerReader::new(block);
|
||||
let mut seq = top
|
||||
.take_sequence()
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
|
||||
if !top.is_empty() {
|
||||
return Err(CcrVerifyError::Decode(CcrDecodeError::Parse(
|
||||
"trailing bytes after ROAIPAddressFamily".into(),
|
||||
)));
|
||||
}
|
||||
let afi_bytes = seq
|
||||
.take_octet_string()
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
|
||||
let afi = u16::from_be_bytes([afi_bytes[0], afi_bytes[1]]);
|
||||
let mut addrs = seq
|
||||
.take_sequence()
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
|
||||
let mut entries = Vec::new();
|
||||
while !addrs.is_empty() {
|
||||
let mut addr_seq = addrs
|
||||
.take_sequence()
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
|
||||
let (unused_bits, content) = addr_seq
|
||||
.take_bit_string()
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
|
||||
let prefix_len = (content.len() * 8) as u8 - unused_bits;
|
||||
let max_len = if addr_seq.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(
|
||||
addr_seq
|
||||
.take_uint_u64()
|
||||
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?
|
||||
as u16,
|
||||
)
|
||||
};
|
||||
entries.push((prefix_len, content.to_vec(), max_len));
|
||||
}
|
||||
Ok((afi, entries))
|
||||
}
|
||||
|
||||
fn format_prefix(afi: u16, addr_bytes: &[u8], prefix_len: u8) -> Result<String, CcrVerifyError> {
|
||||
match afi {
|
||||
1 => {
|
||||
let mut full = [0u8; 4];
|
||||
full[..addr_bytes.len()].copy_from_slice(addr_bytes);
|
||||
Ok(format!("{}/{prefix_len}", std::net::Ipv4Addr::from(full)))
|
||||
}
|
||||
2 => {
|
||||
let mut full = [0u8; 16];
|
||||
full[..addr_bytes.len()].copy_from_slice(addr_bytes);
|
||||
Ok(format!("{}/{prefix_len}", std::net::Ipv6Addr::from(full)))
|
||||
}
|
||||
other => Err(CcrVerifyError::Decode(CcrDecodeError::Parse(format!(
|
||||
"unsupported AFI {other}"
|
||||
)))),
|
||||
}
|
||||
}
|
||||
|
||||
fn count_roa_block_entries(blocks: &[Vec<u8>]) -> usize {
|
||||
blocks
|
||||
.iter()
|
||||
.map(|block| {
|
||||
decode_roa_family_block(block)
|
||||
.map(|(_, entries)| entries.len())
|
||||
.unwrap_or(0)
|
||||
})
|
||||
.sum()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::ccr::build::{build_aspa_payload_state, build_roa_payload_state};
|
||||
use crate::ccr::encode::{
|
||||
encode_manifest_state_payload_der, encode_roa_payload_state_payload_der,
|
||||
encode_router_key_state_payload_der, encode_trust_anchor_state_payload_der,
|
||||
};
|
||||
use crate::ccr::model::{
|
||||
CcrDigestAlgorithm, ManifestInstance, ManifestState, RouterKey, RouterKeySet,
|
||||
RpkiCanonicalCacheRepresentation,
|
||||
};
|
||||
use crate::data_model::common::BigUnsigned;
|
||||
use crate::data_model::roa::{IpPrefix, RoaAfi};
|
||||
use crate::storage::{
|
||||
PackTime, ValidatedCaInstanceResult, ValidatedManifestMeta, VcirArtifactKind,
|
||||
VcirArtifactRole, VcirArtifactValidationStatus, VcirAuditSummary,
|
||||
VcirCcrManifestProjection, VcirChildEntry, VcirInstanceGate, VcirRelatedArtifact,
|
||||
VcirSummary,
|
||||
};
|
||||
use crate::validation::objects::{AspaAttestation, Vrp};
|
||||
|
||||
fn sample_time() -> time::OffsetDateTime {
|
||||
time::OffsetDateTime::parse(
|
||||
"2026-03-24T00:00:00Z",
|
||||
&time::format_description::well_known::Rfc3339,
|
||||
)
|
||||
.expect("time")
|
||||
}
|
||||
|
||||
fn sample_content_info() -> CcrContentInfo {
|
||||
let mis = vec![ManifestInstance {
|
||||
hash: vec![0x10; 32],
|
||||
size: 2048,
|
||||
aki: vec![0x20; 20],
|
||||
manifest_number: BigUnsigned { bytes_be: vec![1] },
|
||||
this_update: sample_time(),
|
||||
locations: vec![vec![0x30, 0x00]],
|
||||
subordinates: vec![vec![0x30; 20]],
|
||||
}];
|
||||
let mfts = ManifestState {
|
||||
most_recent_update: sample_time(),
|
||||
hash: crate::ccr::compute_state_hash(&encode_manifest_state_payload_der(&mis).unwrap()),
|
||||
mis,
|
||||
};
|
||||
let vrps = build_roa_payload_state(&[Vrp {
|
||||
asn: 64496,
|
||||
prefix: IpPrefix {
|
||||
afi: RoaAfi::Ipv4,
|
||||
prefix_len: 0,
|
||||
addr: [0; 16],
|
||||
},
|
||||
max_length: 0,
|
||||
}])
|
||||
.expect("build roa state");
|
||||
let vaps = build_aspa_payload_state(&[AspaAttestation {
|
||||
customer_as_id: 64496,
|
||||
provider_as_ids: vec![64497],
|
||||
}])
|
||||
.expect("build aspa state");
|
||||
let skis = vec![vec![0x11; 20]];
|
||||
let tas = TrustAnchorState {
|
||||
hash: crate::ccr::compute_state_hash(
|
||||
&encode_trust_anchor_state_payload_der(&skis).unwrap(),
|
||||
),
|
||||
skis,
|
||||
};
|
||||
let rksets = vec![RouterKeySet {
|
||||
as_id: 64496,
|
||||
router_keys: vec![RouterKey {
|
||||
ski: vec![0x22; 20],
|
||||
spki_der: vec![0x30, 0x00],
|
||||
}],
|
||||
}];
|
||||
let rks = RouterKeyState {
|
||||
hash: crate::ccr::compute_state_hash(
|
||||
&encode_router_key_state_payload_der(&rksets).unwrap(),
|
||||
),
|
||||
rksets,
|
||||
};
|
||||
CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at: sample_time(),
|
||||
mfts: Some(mfts),
|
||||
vrps: Some(vrps),
|
||||
vaps: Some(vaps),
|
||||
tas: Some(tas),
|
||||
rks: Some(rks),
|
||||
})
|
||||
}
|
||||
|
||||
fn sample_ccr_manifest_projection(manifest_rsync_uri: &str) -> VcirCcrManifestProjection {
|
||||
VcirCcrManifestProjection {
|
||||
manifest_rsync_uri: manifest_rsync_uri.to_string(),
|
||||
manifest_sha256: vec![0x44; 32],
|
||||
manifest_size: 2048,
|
||||
manifest_ee_aki: vec![0x55; 20],
|
||||
manifest_number_be: vec![1],
|
||||
manifest_this_update: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
manifest_sia_locations_der: vec![vec![
|
||||
0x30, 0x11, 0x06, 0x08, 0x2b, 0x06, 0x01, 0x05, 0x05, 0x07, 0x30, 0x05, 0x86, 0x05,
|
||||
b'r', b's', b'y', b'n', b'c',
|
||||
]],
|
||||
subordinate_skis: vec![vec![0x33; 20]],
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_detects_each_state_hash_mismatch() {
|
||||
let mut ci = sample_content_info();
|
||||
ci.content.vrps.as_mut().unwrap().hash[0] ^= 0x01;
|
||||
assert!(matches!(
|
||||
verify_content_info(&ci),
|
||||
Err(CcrVerifyError::RoaHashMismatch)
|
||||
));
|
||||
|
||||
let mut ci = sample_content_info();
|
||||
ci.content.vaps.as_mut().unwrap().hash[0] ^= 0x01;
|
||||
assert!(matches!(
|
||||
verify_content_info(&ci),
|
||||
Err(CcrVerifyError::AspaHashMismatch)
|
||||
));
|
||||
|
||||
let mut ci = sample_content_info();
|
||||
ci.content.tas.as_mut().unwrap().hash[0] ^= 0x01;
|
||||
assert!(matches!(
|
||||
verify_content_info(&ci),
|
||||
Err(CcrVerifyError::TrustAnchorHashMismatch)
|
||||
));
|
||||
|
||||
let mut ci = sample_content_info();
|
||||
ci.content.rks.as_mut().unwrap().hash[0] ^= 0x01;
|
||||
assert!(matches!(
|
||||
verify_content_info(&ci),
|
||||
Err(CcrVerifyError::RouterKeyHashMismatch)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_against_report_json_accepts_matching_report_and_rejects_parse_errors() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let report = serde_json::json!({
|
||||
"vrps": [{"asn": 64496, "prefix": "0.0.0.0/0", "max_length": 0}],
|
||||
"aspas": [{"customer_as_id": 64496, "provider_as_ids": [64497]}]
|
||||
});
|
||||
let report_path = td.path().join("report.json");
|
||||
std::fs::write(&report_path, serde_json::to_vec(&report).unwrap()).unwrap();
|
||||
verify_against_report_json_path(&sample_content_info(), &report_path)
|
||||
.expect("matching report");
|
||||
|
||||
let bad_path = td.path().join("bad.json");
|
||||
std::fs::write(&bad_path, b"not-json").unwrap();
|
||||
assert!(matches!(
|
||||
verify_against_report_json_path(&sample_content_info(), &bad_path),
|
||||
Err(CcrVerifyError::ReportParse(_))
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_against_report_json_rejects_missing_fields_and_aspa_mismatch() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let missing =
|
||||
serde_json::json!({"vrps":[{"prefix":"0.0.0.0/0","max_length":0}],"aspas":[]});
|
||||
let missing_path = td.path().join("missing.json");
|
||||
std::fs::write(&missing_path, serde_json::to_vec(&missing).unwrap()).unwrap();
|
||||
assert!(matches!(
|
||||
verify_against_report_json_path(&sample_content_info(), &missing_path),
|
||||
Err(CcrVerifyError::ReportParse(_))
|
||||
));
|
||||
|
||||
let mismatch = serde_json::json!({
|
||||
"vrps": [{"asn": 64496, "prefix": "0.0.0.0/0", "max_length": 0}],
|
||||
"aspas": [{"customer_as_id": 64496, "provider_as_ids": [65000]}]
|
||||
});
|
||||
let mismatch_path = td.path().join("mismatch.json");
|
||||
std::fs::write(&mismatch_path, serde_json::to_vec(&mismatch).unwrap()).unwrap();
|
||||
assert!(matches!(
|
||||
verify_against_report_json_path(&sample_content_info(), &mismatch_path),
|
||||
Err(CcrVerifyError::ReportAspaMismatch { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_against_vcir_store_rejects_mismatched_manifest_hashes() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(td.path()).expect("open db");
|
||||
let vcir = ValidatedCaInstanceResult {
|
||||
manifest_rsync_uri: "rsync://example.test/current.mft".to_string(),
|
||||
parent_manifest_rsync_uri: None,
|
||||
tal_id: "apnic".to_string(),
|
||||
ca_subject_name: "CN=test".to_string(),
|
||||
ca_ski: "11".repeat(20),
|
||||
issuer_ski: "22".repeat(20),
|
||||
last_successful_validation_time: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
current_manifest_rsync_uri: "rsync://example.test/current.mft".to_string(),
|
||||
current_crl_rsync_uri: "rsync://example.test/current.crl".to_string(),
|
||||
validated_manifest_meta: ValidatedManifestMeta {
|
||||
validated_manifest_number: vec![1],
|
||||
validated_manifest_this_update: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
validated_manifest_next_update: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
},
|
||||
ccr_manifest_projection: sample_ccr_manifest_projection(
|
||||
"rsync://example.test/current.mft",
|
||||
),
|
||||
instance_gate: VcirInstanceGate {
|
||||
manifest_next_update: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
current_crl_next_update: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
self_ca_not_after: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
instance_effective_until: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
},
|
||||
child_entries: vec![VcirChildEntry {
|
||||
child_manifest_rsync_uri: "rsync://example.test/child.mft".to_string(),
|
||||
child_cert_rsync_uri: "rsync://example.test/child.cer".to_string(),
|
||||
child_cert_hash: "aa".repeat(32),
|
||||
child_ski: "33".repeat(20),
|
||||
child_rsync_base_uri: "rsync://example.test/".to_string(),
|
||||
child_publication_point_rsync_uri: "rsync://example.test/".to_string(),
|
||||
child_rrdp_notification_uri: None,
|
||||
child_effective_ip_resources: None,
|
||||
child_effective_as_resources: None,
|
||||
accepted_at_validation_time: PackTime::from_utc_offset_datetime(sample_time()),
|
||||
}],
|
||||
local_outputs: Vec::new(),
|
||||
related_artifacts: vec![VcirRelatedArtifact {
|
||||
artifact_role: VcirArtifactRole::Manifest,
|
||||
artifact_kind: VcirArtifactKind::Mft,
|
||||
uri: Some("rsync://example.test/current.mft".to_string()),
|
||||
sha256: "ff".repeat(32),
|
||||
object_type: Some("mft".to_string()),
|
||||
validation_status: VcirArtifactValidationStatus::Accepted,
|
||||
reject_reason: None,
|
||||
}],
|
||||
summary: VcirSummary {
|
||||
local_vrp_count: 0,
|
||||
local_aspa_count: 0,
|
||||
local_router_key_count: 0,
|
||||
child_count: 1,
|
||||
accepted_object_count: 1,
|
||||
rejected_object_count: 0,
|
||||
},
|
||||
audit_summary: VcirAuditSummary {
|
||||
failed_fetch_eligible: true,
|
||||
last_failed_fetch_reason: None,
|
||||
warning_count: 0,
|
||||
audit_flags: Vec::new(),
|
||||
},
|
||||
};
|
||||
store.put_vcir(&vcir).unwrap();
|
||||
assert!(matches!(
|
||||
verify_against_vcir_store(&sample_content_info(), &store),
|
||||
Err(CcrVerifyError::VcirManifestMismatch { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_vrp_helpers_reject_bad_afi_and_count_invalid_block_as_zero() {
|
||||
let block = vec![0x30, 0x08, 0x04, 0x02, 0x00, 0x63, 0x30, 0x02, 0x03, 0x00];
|
||||
let ci = CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
|
||||
version: 0,
|
||||
hash_alg: CcrDigestAlgorithm::Sha256,
|
||||
produced_at: sample_time(),
|
||||
mfts: None,
|
||||
vrps: Some(crate::ccr::model::RoaPayloadState {
|
||||
rps: vec![crate::ccr::model::RoaPayloadSet {
|
||||
as_id: 64496,
|
||||
ip_addr_blocks: vec![block.clone()],
|
||||
}],
|
||||
hash: crate::ccr::compute_state_hash(
|
||||
&encode_roa_payload_state_payload_der(&[crate::ccr::model::RoaPayloadSet {
|
||||
as_id: 64496,
|
||||
ip_addr_blocks: vec![block],
|
||||
}])
|
||||
.unwrap(),
|
||||
),
|
||||
}),
|
||||
vaps: None,
|
||||
tas: None,
|
||||
rks: None,
|
||||
});
|
||||
assert!(matches!(
|
||||
extract_vrp_rows(&ci),
|
||||
Err(CcrVerifyError::Decode(_))
|
||||
));
|
||||
let bad_count = count_roa_block_entries(&[vec![0x04, 0x00]]);
|
||||
assert_eq!(bad_count, 0);
|
||||
}
|
||||
}
|
||||
406
crates/panda-rpki-validator/src/cir/accumulator.rs
Normal file
406
crates/panda-rpki-validator/src/cir/accumulator.rs
Normal file
@ -0,0 +1,406 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use crate::audit::{AuditObjectResult, ObjectAuditEntry};
|
||||
use crate::cir::export::CirExportError;
|
||||
use crate::cir::model::{CirObject, CirRejectedObject};
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum CirInputSection {
|
||||
Fresh,
|
||||
Cached,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct CirInputSnapshot {
|
||||
pub fresh_validated_objects: Vec<CirObject>,
|
||||
pub cached_validated_objects: Vec<CirObject>,
|
||||
pub fresh_rejected_objects: Vec<CirRejectedObject>,
|
||||
pub cached_rejected_objects: Vec<CirRejectedObject>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct CirInputAccumulator {
|
||||
fresh_objects: HashMap<String, [u8; 32]>,
|
||||
cached_objects: HashMap<String, [u8; 32]>,
|
||||
fresh_rejects: HashMap<String, Option<String>>,
|
||||
cached_rejects: HashMap<String, Option<String>>,
|
||||
}
|
||||
|
||||
impl CirInputAccumulator {
|
||||
pub fn submit_audit_entries(
|
||||
&mut self,
|
||||
section: CirInputSection,
|
||||
entries: &[ObjectAuditEntry],
|
||||
) -> Result<(), CirExportError> {
|
||||
for entry in entries {
|
||||
if !matches!(
|
||||
entry.result,
|
||||
AuditObjectResult::Ok | AuditObjectResult::Error
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
self.insert_object(section, &entry.rsync_uri, &entry.sha256_hex)?;
|
||||
if entry.result == AuditObjectResult::Error {
|
||||
self.insert_reject(section, &entry.rsync_uri, entry.detail.clone());
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn submit_snapshot(&mut self, snapshot: CirInputSnapshot) -> Result<(), CirExportError> {
|
||||
for object in snapshot.fresh_validated_objects {
|
||||
self.insert_object_bytes(CirInputSection::Fresh, &object.rsync_uri, object.sha256)?;
|
||||
}
|
||||
for object in snapshot.cached_validated_objects {
|
||||
self.insert_object_bytes(CirInputSection::Cached, &object.rsync_uri, object.sha256)?;
|
||||
}
|
||||
for rejected in snapshot.fresh_rejected_objects {
|
||||
self.insert_reject(
|
||||
CirInputSection::Fresh,
|
||||
&rejected.object_uri,
|
||||
rejected.reason,
|
||||
);
|
||||
}
|
||||
for rejected in snapshot.cached_rejected_objects {
|
||||
self.insert_reject(
|
||||
CirInputSection::Cached,
|
||||
&rejected.object_uri,
|
||||
rejected.reason,
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn insert_object(
|
||||
&mut self,
|
||||
section: CirInputSection,
|
||||
rsync_uri: &str,
|
||||
sha256_hex: &str,
|
||||
) -> Result<(), CirExportError> {
|
||||
if !rsync_uri.starts_with("rsync://") || !is_sha256_hex(sha256_hex) {
|
||||
return Ok(());
|
||||
}
|
||||
let mut digest = [0u8; 32];
|
||||
hex::decode_to_slice(sha256_hex, &mut digest).expect("validated sha256 hex");
|
||||
self.insert_object_digest(section, rsync_uri, digest)
|
||||
}
|
||||
|
||||
fn insert_object_bytes(
|
||||
&mut self,
|
||||
section: CirInputSection,
|
||||
rsync_uri: &str,
|
||||
sha256: Vec<u8>,
|
||||
) -> Result<(), CirExportError> {
|
||||
if !rsync_uri.starts_with("rsync://") {
|
||||
return Ok(());
|
||||
}
|
||||
let Ok(digest) = <[u8; 32]>::try_from(sha256) else {
|
||||
return Ok(());
|
||||
};
|
||||
self.insert_object_digest(section, rsync_uri, digest)
|
||||
}
|
||||
|
||||
fn insert_object_digest(
|
||||
&mut self,
|
||||
section: CirInputSection,
|
||||
rsync_uri: &str,
|
||||
digest: [u8; 32],
|
||||
) -> Result<(), CirExportError> {
|
||||
let objects = match section {
|
||||
CirInputSection::Fresh => &mut self.fresh_objects,
|
||||
CirInputSection::Cached => &mut self.cached_objects,
|
||||
};
|
||||
if let Some(existing) = objects.get(rsync_uri) {
|
||||
if existing != &digest {
|
||||
return Err(CirExportError::ConflictingObjectHash {
|
||||
rsync_uri: rsync_uri.to_string(),
|
||||
first: hex::encode(existing),
|
||||
second: hex::encode(digest),
|
||||
});
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
objects.insert(rsync_uri.to_string(), digest);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn insert_reject(
|
||||
&mut self,
|
||||
section: CirInputSection,
|
||||
rsync_uri: &str,
|
||||
reason: Option<String>,
|
||||
) {
|
||||
if !rsync_uri.starts_with("rsync://") {
|
||||
return;
|
||||
}
|
||||
let rejects = match section {
|
||||
CirInputSection::Fresh => &mut self.fresh_rejects,
|
||||
CirInputSection::Cached => &mut self.cached_rejects,
|
||||
};
|
||||
rejects.entry(rsync_uri.to_string()).or_insert(reason);
|
||||
}
|
||||
|
||||
/// Merge another accumulator into this one. Object digests conflict-check
|
||||
/// exactly like repeated `insert_object` calls; rejects keep the first
|
||||
/// reason seen, matching sequential submission order when `other` holds
|
||||
/// later publication points than `self`.
|
||||
pub fn merge(&mut self, other: CirInputAccumulator) -> Result<(), CirExportError> {
|
||||
for (rsync_uri, digest) in other.fresh_objects {
|
||||
self.insert_object_digest(CirInputSection::Fresh, &rsync_uri, digest)?;
|
||||
}
|
||||
for (rsync_uri, digest) in other.cached_objects {
|
||||
self.insert_object_digest(CirInputSection::Cached, &rsync_uri, digest)?;
|
||||
}
|
||||
for (rsync_uri, reason) in other.fresh_rejects {
|
||||
self.insert_reject(CirInputSection::Fresh, &rsync_uri, reason);
|
||||
}
|
||||
for (rsync_uri, reason) in other.cached_rejects {
|
||||
self.insert_reject(CirInputSection::Cached, &rsync_uri, reason);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn finalize(self) -> CirInputSnapshot {
|
||||
CirInputSnapshot {
|
||||
fresh_validated_objects: finalize_objects(self.fresh_objects),
|
||||
cached_validated_objects: finalize_objects(self.cached_objects),
|
||||
fresh_rejected_objects: finalize_rejects(self.fresh_rejects),
|
||||
cached_rejected_objects: finalize_rejects(self.cached_rejects),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn finalize_objects(objects: HashMap<String, [u8; 32]>) -> Vec<CirObject> {
|
||||
let mut objects = objects
|
||||
.into_iter()
|
||||
.map(|(rsync_uri, sha256)| CirObject {
|
||||
rsync_uri,
|
||||
sha256: sha256.to_vec(),
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
objects.sort_by(|a, b| a.rsync_uri.cmp(&b.rsync_uri));
|
||||
objects
|
||||
}
|
||||
|
||||
fn finalize_rejects(rejects: HashMap<String, Option<String>>) -> Vec<CirRejectedObject> {
|
||||
let mut rejects = rejects
|
||||
.into_iter()
|
||||
.map(|(object_uri, reason)| CirRejectedObject { object_uri, reason })
|
||||
.collect::<Vec<_>>();
|
||||
rejects.sort_by(|a, b| a.object_uri.cmp(&b.object_uri));
|
||||
rejects
|
||||
}
|
||||
|
||||
fn is_sha256_hex(value: &str) -> bool {
|
||||
value.len() == 64 && value.as_bytes().iter().all(u8::is_ascii_hexdigit)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::audit::{AuditObjectKind, AuditObjectResult};
|
||||
|
||||
fn entry(uri: &str, hash_byte: u8, result: AuditObjectResult) -> ObjectAuditEntry {
|
||||
let is_error = result == AuditObjectResult::Error;
|
||||
ObjectAuditEntry {
|
||||
rsync_uri: uri.to_string(),
|
||||
sha256_hex: format!("{}", hex::encode([hash_byte; 32])),
|
||||
kind: AuditObjectKind::Roa,
|
||||
result,
|
||||
detail: is_error.then(|| "invalid".to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_finalizes_sorted_fresh_and_cached_sections() {
|
||||
let mut acc = CirInputAccumulator::default();
|
||||
acc.submit_audit_entries(
|
||||
CirInputSection::Fresh,
|
||||
&[
|
||||
entry("rsync://example.net/z.roa", 0x22, AuditObjectResult::Ok),
|
||||
entry("rsync://example.net/a.roa", 0x11, AuditObjectResult::Error),
|
||||
],
|
||||
)
|
||||
.unwrap();
|
||||
acc.submit_audit_entries(
|
||||
CirInputSection::Cached,
|
||||
&[entry(
|
||||
"rsync://example.net/c.roa",
|
||||
0x33,
|
||||
AuditObjectResult::Ok,
|
||||
)],
|
||||
)
|
||||
.unwrap();
|
||||
let snapshot = acc.finalize();
|
||||
assert_eq!(
|
||||
snapshot.fresh_validated_objects[0].rsync_uri,
|
||||
"rsync://example.net/a.roa"
|
||||
);
|
||||
assert_eq!(
|
||||
snapshot.fresh_validated_objects[1].rsync_uri,
|
||||
"rsync://example.net/z.roa"
|
||||
);
|
||||
assert_eq!(
|
||||
snapshot.cached_validated_objects[0].rsync_uri,
|
||||
"rsync://example.net/c.roa"
|
||||
);
|
||||
assert_eq!(
|
||||
snapshot.fresh_rejected_objects[0].object_uri,
|
||||
"rsync://example.net/a.roa"
|
||||
);
|
||||
assert!(snapshot.cached_rejected_objects.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_rejects_conflicting_hashes_within_same_section() {
|
||||
let mut acc = CirInputAccumulator::default();
|
||||
acc.submit_audit_entries(
|
||||
CirInputSection::Fresh,
|
||||
&[entry(
|
||||
"rsync://example.net/a.roa",
|
||||
0x11,
|
||||
AuditObjectResult::Ok,
|
||||
)],
|
||||
)
|
||||
.unwrap();
|
||||
let err = acc
|
||||
.submit_audit_entries(
|
||||
CirInputSection::Fresh,
|
||||
&[entry(
|
||||
"rsync://example.net/a.roa",
|
||||
0x22,
|
||||
AuditObjectResult::Ok,
|
||||
)],
|
||||
)
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, CirExportError::ConflictingObjectHash { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_merges_finalized_snapshots() {
|
||||
let snapshot = CirInputSnapshot {
|
||||
fresh_validated_objects: vec![CirObject {
|
||||
rsync_uri: "rsync://example.net/fresh.roa".to_string(),
|
||||
sha256: vec![0x11; 32],
|
||||
}],
|
||||
cached_validated_objects: vec![CirObject {
|
||||
rsync_uri: "rsync://example.net/cached.roa".to_string(),
|
||||
sha256: vec![0x22; 32],
|
||||
}],
|
||||
fresh_rejected_objects: vec![CirRejectedObject {
|
||||
object_uri: "rsync://example.net/fresh.roa".to_string(),
|
||||
reason: Some("fresh invalid".to_string()),
|
||||
}],
|
||||
cached_rejected_objects: Vec::new(),
|
||||
};
|
||||
let mut acc = CirInputAccumulator::default();
|
||||
acc.submit_snapshot(snapshot).unwrap();
|
||||
let merged = acc.finalize();
|
||||
assert_eq!(merged.fresh_validated_objects.len(), 1);
|
||||
assert_eq!(merged.cached_validated_objects.len(), 1);
|
||||
assert_eq!(merged.fresh_rejected_objects.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_merge_matches_sequential_submission() {
|
||||
let mut sequential = CirInputAccumulator::default();
|
||||
sequential
|
||||
.submit_audit_entries(
|
||||
CirInputSection::Fresh,
|
||||
&[
|
||||
entry("rsync://example.net/a.roa", 0x11, AuditObjectResult::Ok),
|
||||
entry("rsync://example.net/b.roa", 0x22, AuditObjectResult::Error),
|
||||
],
|
||||
)
|
||||
.unwrap();
|
||||
sequential
|
||||
.submit_audit_entries(
|
||||
CirInputSection::Cached,
|
||||
&[entry(
|
||||
"rsync://example.net/c.roa",
|
||||
0x33,
|
||||
AuditObjectResult::Ok,
|
||||
)],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let mut first = CirInputAccumulator::default();
|
||||
first
|
||||
.submit_audit_entries(
|
||||
CirInputSection::Fresh,
|
||||
&[entry(
|
||||
"rsync://example.net/a.roa",
|
||||
0x11,
|
||||
AuditObjectResult::Ok,
|
||||
)],
|
||||
)
|
||||
.unwrap();
|
||||
let mut second = CirInputAccumulator::default();
|
||||
second
|
||||
.submit_audit_entries(
|
||||
CirInputSection::Fresh,
|
||||
&[entry(
|
||||
"rsync://example.net/b.roa",
|
||||
0x22,
|
||||
AuditObjectResult::Error,
|
||||
)],
|
||||
)
|
||||
.unwrap();
|
||||
second
|
||||
.submit_audit_entries(
|
||||
CirInputSection::Cached,
|
||||
&[entry(
|
||||
"rsync://example.net/c.roa",
|
||||
0x33,
|
||||
AuditObjectResult::Ok,
|
||||
)],
|
||||
)
|
||||
.unwrap();
|
||||
first.merge(second).unwrap();
|
||||
assert_eq!(sequential.finalize(), first.finalize());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_merge_detects_conflicting_hashes_across_shards() {
|
||||
let mut first = CirInputAccumulator::default();
|
||||
first
|
||||
.insert_object(
|
||||
CirInputSection::Fresh,
|
||||
"rsync://example.net/a.roa",
|
||||
&"11".repeat(32),
|
||||
)
|
||||
.unwrap();
|
||||
let mut second = CirInputAccumulator::default();
|
||||
second
|
||||
.insert_object(
|
||||
CirInputSection::Fresh,
|
||||
"rsync://example.net/a.roa",
|
||||
&"22".repeat(32),
|
||||
)
|
||||
.unwrap();
|
||||
let err = first.merge(second).unwrap_err();
|
||||
assert!(matches!(err, CirExportError::ConflictingObjectHash { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accumulator_merge_keeps_first_reject_reason() {
|
||||
let mut first = CirInputAccumulator::default();
|
||||
first.insert_reject(
|
||||
CirInputSection::Fresh,
|
||||
"rsync://example.net/a.roa",
|
||||
Some("first".to_string()),
|
||||
);
|
||||
let mut second = CirInputAccumulator::default();
|
||||
second.insert_reject(
|
||||
CirInputSection::Fresh,
|
||||
"rsync://example.net/a.roa",
|
||||
Some("second".to_string()),
|
||||
);
|
||||
first.merge(second).unwrap();
|
||||
let snapshot = first.finalize();
|
||||
assert_eq!(
|
||||
snapshot.fresh_rejected_objects[0].reason,
|
||||
Some("first".to_string())
|
||||
);
|
||||
}
|
||||
}
|
||||
286
crates/panda-rpki-validator/src/cir/decode.rs
Normal file
286
crates/panda-rpki-validator/src/cir/decode.rs
Normal file
@ -0,0 +1,286 @@
|
||||
use crate::cir::model::{
|
||||
CIR_VERSION_V4, CanonicalInputRepresentation, CirHashAlgorithm, CirObject, CirRejectedObject,
|
||||
CirTrustAnchor,
|
||||
};
|
||||
use crate::data_model::common::DerReader;
|
||||
use crate::data_model::oid::{OID_SHA256, OID_SHA256_RAW};
|
||||
use der_parser::der::parse_der_oid;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CirDecodeError {
|
||||
#[error("DER parse error: {0}")]
|
||||
Parse(String),
|
||||
|
||||
#[error("unexpected CIR version: expected {expected}, got {actual}")]
|
||||
UnexpectedVersion { expected: u32, actual: u32 },
|
||||
|
||||
#[error("unexpected digest algorithm OID: expected {expected}, got {actual}")]
|
||||
UnexpectedDigestAlgorithm {
|
||||
expected: &'static str,
|
||||
actual: String,
|
||||
},
|
||||
|
||||
#[error("CIR model validation failed after decode: {0}")]
|
||||
Validate(String),
|
||||
}
|
||||
|
||||
pub fn decode_cir(der: &[u8]) -> Result<CanonicalInputRepresentation, CirDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CirDecodeError::Parse("trailing bytes after CIR".into()));
|
||||
}
|
||||
|
||||
let version = seq.take_uint_u64().map_err(CirDecodeError::Parse)? as u32;
|
||||
if version != CIR_VERSION_V4 {
|
||||
return Err(CirDecodeError::UnexpectedVersion {
|
||||
expected: CIR_VERSION_V4,
|
||||
actual: version,
|
||||
});
|
||||
}
|
||||
let hash_alg = decode_hash_alg(seq.take_tag(0x06).map_err(CirDecodeError::Parse)?)?;
|
||||
let validation_time =
|
||||
parse_generalized_time(seq.take_tag(0x18).map_err(CirDecodeError::Parse)?)?;
|
||||
|
||||
let fresh_objects_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
|
||||
let mut fresh_objects_reader = DerReader::new(fresh_objects_der);
|
||||
let mut fresh_validated_objects = Vec::new();
|
||||
while !fresh_objects_reader.is_empty() {
|
||||
let (_tag, full, _value) = fresh_objects_reader
|
||||
.take_any_full()
|
||||
.map_err(CirDecodeError::Parse)?;
|
||||
fresh_validated_objects.push(decode_object(full)?);
|
||||
}
|
||||
|
||||
let cached_objects_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
|
||||
let mut cached_objects_reader = DerReader::new(cached_objects_der);
|
||||
let mut cached_validated_objects = Vec::new();
|
||||
while !cached_objects_reader.is_empty() {
|
||||
let (_tag, full, _value) = cached_objects_reader
|
||||
.take_any_full()
|
||||
.map_err(CirDecodeError::Parse)?;
|
||||
cached_validated_objects.push(decode_object(full)?);
|
||||
}
|
||||
|
||||
let trust_anchors_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
|
||||
let mut trust_anchors_reader = DerReader::new(trust_anchors_der);
|
||||
let mut trust_anchors = Vec::new();
|
||||
while !trust_anchors_reader.is_empty() {
|
||||
let (_tag, full, _value) = trust_anchors_reader
|
||||
.take_any_full()
|
||||
.map_err(CirDecodeError::Parse)?;
|
||||
trust_anchors.push(decode_trust_anchor(full)?);
|
||||
}
|
||||
|
||||
let object_list_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let fresh_object_list_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let cached_object_list_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let reject_list_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let fresh_reject_list_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let cached_reject_list_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
|
||||
let fresh_rejected_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
|
||||
let mut fresh_rejected_reader = DerReader::new(fresh_rejected_der);
|
||||
let mut fresh_rejected_objects = Vec::new();
|
||||
while !fresh_rejected_reader.is_empty() {
|
||||
let (_tag, full, _value) = fresh_rejected_reader
|
||||
.take_any_full()
|
||||
.map_err(CirDecodeError::Parse)?;
|
||||
fresh_rejected_objects.push(decode_rejected_object(full)?);
|
||||
}
|
||||
|
||||
let cached_rejected_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
|
||||
let mut cached_rejected_reader = DerReader::new(cached_rejected_der);
|
||||
let mut cached_rejected_objects = Vec::new();
|
||||
while !cached_rejected_reader.is_empty() {
|
||||
let (_tag, full, _value) = cached_rejected_reader
|
||||
.take_any_full()
|
||||
.map_err(CirDecodeError::Parse)?;
|
||||
cached_rejected_objects.push(decode_rejected_object(full)?);
|
||||
}
|
||||
|
||||
if !seq.is_empty() {
|
||||
return Err(CirDecodeError::Parse("trailing fields in CIR".into()));
|
||||
}
|
||||
|
||||
let cir = CanonicalInputRepresentation {
|
||||
version,
|
||||
hash_alg,
|
||||
validation_time,
|
||||
fresh_validated_objects,
|
||||
cached_validated_objects,
|
||||
trust_anchors,
|
||||
object_list_sha256,
|
||||
fresh_object_list_sha256,
|
||||
cached_object_list_sha256,
|
||||
reject_list_sha256,
|
||||
fresh_reject_list_sha256,
|
||||
cached_reject_list_sha256,
|
||||
fresh_rejected_objects,
|
||||
cached_rejected_objects,
|
||||
};
|
||||
cir.validate().map_err(CirDecodeError::Validate)?;
|
||||
Ok(cir)
|
||||
}
|
||||
|
||||
fn decode_hash_alg(raw_body: &[u8]) -> Result<CirHashAlgorithm, CirDecodeError> {
|
||||
if raw_body != OID_SHA256_RAW {
|
||||
return Err(CirDecodeError::UnexpectedDigestAlgorithm {
|
||||
expected: OID_SHA256,
|
||||
actual: oid_string(raw_body)?,
|
||||
});
|
||||
}
|
||||
Ok(CirHashAlgorithm::Sha256)
|
||||
}
|
||||
|
||||
fn decode_object(der: &[u8]) -> Result<CirObject, CirDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CirDecodeError::Parse(
|
||||
"trailing bytes after CirObject".into(),
|
||||
));
|
||||
}
|
||||
let rsync_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
|
||||
.to_string();
|
||||
let sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
if !seq.is_empty() {
|
||||
return Err(CirDecodeError::Parse("trailing fields in CirObject".into()));
|
||||
}
|
||||
Ok(CirObject { rsync_uri, sha256 })
|
||||
}
|
||||
|
||||
fn decode_trust_anchor(der: &[u8]) -> Result<CirTrustAnchor, CirDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CirDecodeError::Parse(
|
||||
"trailing bytes after CirTrustAnchor".into(),
|
||||
));
|
||||
}
|
||||
let ta_rsync_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
|
||||
.to_string();
|
||||
let tal_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
|
||||
.to_string();
|
||||
let tal_bytes = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let ta_certificate_der = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
let ta_certificate_sha256 = seq
|
||||
.take_octet_string()
|
||||
.map_err(CirDecodeError::Parse)?
|
||||
.to_vec();
|
||||
if !seq.is_empty() {
|
||||
return Err(CirDecodeError::Parse(
|
||||
"trailing fields in CirTrustAnchor".into(),
|
||||
));
|
||||
}
|
||||
Ok(CirTrustAnchor {
|
||||
ta_rsync_uri,
|
||||
tal_uri,
|
||||
tal_bytes,
|
||||
ta_certificate_der,
|
||||
ta_certificate_sha256,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode_rejected_object(der: &[u8]) -> Result<CirRejectedObject, CirDecodeError> {
|
||||
let mut top = DerReader::new(der);
|
||||
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
|
||||
if !top.is_empty() {
|
||||
return Err(CirDecodeError::Parse(
|
||||
"trailing bytes after CirRejectedObject".into(),
|
||||
));
|
||||
}
|
||||
let object_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
|
||||
.to_string();
|
||||
let reason = if seq.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(
|
||||
std::str::from_utf8(seq.take_octet_string().map_err(CirDecodeError::Parse)?)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
|
||||
.to_string(),
|
||||
)
|
||||
};
|
||||
if !seq.is_empty() {
|
||||
return Err(CirDecodeError::Parse(
|
||||
"trailing fields in CirRejectedObject".into(),
|
||||
));
|
||||
}
|
||||
Ok(CirRejectedObject { object_uri, reason })
|
||||
}
|
||||
|
||||
fn oid_string(raw_body: &[u8]) -> Result<String, CirDecodeError> {
|
||||
let der = {
|
||||
let mut out = Vec::with_capacity(raw_body.len() + 2);
|
||||
out.push(0x06);
|
||||
if raw_body.len() < 0x80 {
|
||||
out.push(raw_body.len() as u8);
|
||||
} else {
|
||||
return Err(CirDecodeError::Parse("OID too long".into()));
|
||||
}
|
||||
out.extend_from_slice(raw_body);
|
||||
out
|
||||
};
|
||||
let (_rem, oid) = parse_der_oid(&der).map_err(|e| CirDecodeError::Parse(e.to_string()))?;
|
||||
let oid = oid
|
||||
.as_oid_val()
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?;
|
||||
Ok(oid.to_string())
|
||||
}
|
||||
|
||||
fn parse_generalized_time(bytes: &[u8]) -> Result<time::OffsetDateTime, CirDecodeError> {
|
||||
let s = std::str::from_utf8(bytes).map_err(|e| CirDecodeError::Parse(e.to_string()))?;
|
||||
if s.len() != 15 || !s.ends_with('Z') {
|
||||
return Err(CirDecodeError::Parse(
|
||||
"GeneralizedTime must be YYYYMMDDHHMMSSZ".into(),
|
||||
));
|
||||
}
|
||||
let parse = |range: std::ops::Range<usize>| -> Result<u32, CirDecodeError> {
|
||||
s[range]
|
||||
.parse::<u32>()
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))
|
||||
};
|
||||
let year = parse(0..4)? as i32;
|
||||
let month = parse(4..6)? as u8;
|
||||
let day = parse(6..8)? as u8;
|
||||
let hour = parse(8..10)? as u8;
|
||||
let minute = parse(10..12)? as u8;
|
||||
let second = parse(12..14)? as u8;
|
||||
let month = time::Month::try_from(month).map_err(|e| CirDecodeError::Parse(e.to_string()))?;
|
||||
let date = time::Date::from_calendar_date(year, month, day)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?;
|
||||
let timev = time::Time::from_hms(hour, minute, second)
|
||||
.map_err(|e| CirDecodeError::Parse(e.to_string()))?;
|
||||
Ok(time::PrimitiveDateTime::new(date, timev).assume_utc())
|
||||
}
|
||||
179
crates/panda-rpki-validator/src/cir/encode.rs
Normal file
179
crates/panda-rpki-validator/src/cir/encode.rs
Normal file
@ -0,0 +1,179 @@
|
||||
use crate::cir::model::{
|
||||
CIR_VERSION_V4, CanonicalInputRepresentation, CirHashAlgorithm, CirObject, CirRejectedObject,
|
||||
CirTrustAnchor,
|
||||
};
|
||||
use crate::data_model::oid::OID_SHA256_RAW;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CirEncodeError {
|
||||
#[error("CIR model validation failed: {0}")]
|
||||
Validate(String),
|
||||
}
|
||||
|
||||
pub fn encode_cir(cir: &CanonicalInputRepresentation) -> Result<Vec<u8>, CirEncodeError> {
|
||||
cir.validate().map_err(CirEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_integer_u32(CIR_VERSION_V4),
|
||||
encode_oid(match cir.hash_alg {
|
||||
CirHashAlgorithm::Sha256 => OID_SHA256_RAW,
|
||||
}),
|
||||
encode_generalized_time(cir.validation_time),
|
||||
encode_sequence(
|
||||
&cir.fresh_validated_objects
|
||||
.iter()
|
||||
.map(encode_object)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
),
|
||||
encode_sequence(
|
||||
&cir.cached_validated_objects
|
||||
.iter()
|
||||
.map(encode_object)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
),
|
||||
encode_sequence(
|
||||
&cir.trust_anchors
|
||||
.iter()
|
||||
.map(encode_trust_anchor)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
),
|
||||
encode_octet_string(&cir.object_list_sha256),
|
||||
encode_octet_string(&cir.fresh_object_list_sha256),
|
||||
encode_octet_string(&cir.cached_object_list_sha256),
|
||||
encode_octet_string(&cir.reject_list_sha256),
|
||||
encode_octet_string(&cir.fresh_reject_list_sha256),
|
||||
encode_octet_string(&cir.cached_reject_list_sha256),
|
||||
encode_sequence(
|
||||
&cir.fresh_rejected_objects
|
||||
.iter()
|
||||
.map(encode_rejected_object)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
),
|
||||
encode_sequence(
|
||||
&cir.cached_rejected_objects
|
||||
.iter()
|
||||
.map(encode_rejected_object)
|
||||
.collect::<Result<Vec<_>, _>>()?,
|
||||
),
|
||||
]))
|
||||
}
|
||||
|
||||
fn encode_object(object: &CirObject) -> Result<Vec<u8>, CirEncodeError> {
|
||||
object.validate().map_err(CirEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_ia5_string(object.rsync_uri.as_bytes()),
|
||||
encode_octet_string(&object.sha256),
|
||||
]))
|
||||
}
|
||||
|
||||
fn encode_trust_anchor(trust_anchor: &CirTrustAnchor) -> Result<Vec<u8>, CirEncodeError> {
|
||||
trust_anchor.validate().map_err(CirEncodeError::Validate)?;
|
||||
Ok(encode_sequence(&[
|
||||
encode_ia5_string(trust_anchor.ta_rsync_uri.as_bytes()),
|
||||
encode_ia5_string(trust_anchor.tal_uri.as_bytes()),
|
||||
encode_octet_string(&trust_anchor.tal_bytes),
|
||||
encode_octet_string(&trust_anchor.ta_certificate_der),
|
||||
encode_octet_string(&trust_anchor.ta_certificate_sha256),
|
||||
]))
|
||||
}
|
||||
|
||||
fn encode_rejected_object(item: &CirRejectedObject) -> Result<Vec<u8>, CirEncodeError> {
|
||||
item.validate().map_err(CirEncodeError::Validate)?;
|
||||
let mut fields = vec![encode_ia5_string(item.object_uri.as_bytes())];
|
||||
if let Some(reason) = &item.reason {
|
||||
fields.push(encode_octet_string(reason.as_bytes()));
|
||||
}
|
||||
Ok(encode_sequence(&fields))
|
||||
}
|
||||
|
||||
fn encode_generalized_time(t: time::OffsetDateTime) -> Vec<u8> {
|
||||
let t = t.to_offset(time::UtcOffset::UTC);
|
||||
let s = format!(
|
||||
"{:04}{:02}{:02}{:02}{:02}{:02}Z",
|
||||
t.year(),
|
||||
u8::from(t.month()),
|
||||
t.day(),
|
||||
t.hour(),
|
||||
t.minute(),
|
||||
t.second()
|
||||
);
|
||||
encode_tlv(0x18, s.into_bytes())
|
||||
}
|
||||
|
||||
fn encode_integer_u32(v: u32) -> Vec<u8> {
|
||||
encode_integer_bytes(unsigned_integer_bytes(v as u64))
|
||||
}
|
||||
|
||||
fn encode_integer_bytes(mut bytes: Vec<u8>) -> Vec<u8> {
|
||||
if bytes.is_empty() {
|
||||
bytes.push(0);
|
||||
}
|
||||
if bytes[0] & 0x80 != 0 {
|
||||
bytes.insert(0, 0);
|
||||
}
|
||||
encode_tlv(0x02, bytes)
|
||||
}
|
||||
|
||||
fn unsigned_integer_bytes(v: u64) -> Vec<u8> {
|
||||
if v == 0 {
|
||||
return vec![0];
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
let mut n = v;
|
||||
while n > 0 {
|
||||
out.push((n & 0xFF) as u8);
|
||||
n >>= 8;
|
||||
}
|
||||
out.reverse();
|
||||
out
|
||||
}
|
||||
|
||||
fn encode_oid(raw_body: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0x06, raw_body.to_vec())
|
||||
}
|
||||
|
||||
fn encode_ia5_string(bytes: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0x16, bytes.to_vec())
|
||||
}
|
||||
|
||||
fn encode_octet_string(bytes: &[u8]) -> Vec<u8> {
|
||||
encode_tlv(0x04, bytes.to_vec())
|
||||
}
|
||||
|
||||
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
|
||||
let mut body = Vec::new();
|
||||
for element in elements {
|
||||
body.extend_from_slice(element);
|
||||
}
|
||||
encode_tlv(0x30, body)
|
||||
}
|
||||
|
||||
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(1 + encoded_len_len(value.len()) + value.len());
|
||||
out.push(tag);
|
||||
encode_len_into(value.len(), &mut out);
|
||||
out.extend_from_slice(&value);
|
||||
out
|
||||
}
|
||||
|
||||
fn encoded_len_len(len: usize) -> usize {
|
||||
if len < 0x80 {
|
||||
1
|
||||
} else {
|
||||
1 + len.to_be_bytes().iter().skip_while(|&&b| b == 0).count()
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_len_into(len: usize, out: &mut Vec<u8>) {
|
||||
if len < 0x80 {
|
||||
out.push(len as u8);
|
||||
return;
|
||||
}
|
||||
let bytes = len.to_be_bytes();
|
||||
let first_non_zero = bytes
|
||||
.iter()
|
||||
.position(|&b| b != 0)
|
||||
.unwrap_or(bytes.len() - 1);
|
||||
let len_bytes = &bytes[first_non_zero..];
|
||||
out.push(0x80 | (len_bytes.len() as u8));
|
||||
out.extend_from_slice(len_bytes);
|
||||
}
|
||||
1217
crates/panda-rpki-validator/src/cir/export.rs
Normal file
1217
crates/panda-rpki-validator/src/cir/export.rs
Normal file
File diff suppressed because it is too large
Load Diff
867
crates/panda-rpki-validator/src/cir/materialize.rs
Normal file
867
crates/panda-rpki-validator/src/cir/materialize.rs
Normal file
@ -0,0 +1,867 @@
|
||||
use std::collections::BTreeSet;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use crate::blob_store::{ExternalRawStoreDb, ExternalRepoBytesDb, RawObjectStore};
|
||||
use crate::cir::model::CanonicalInputRepresentation;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CirMaterializeError {
|
||||
#[error("invalid rsync URI: {0}")]
|
||||
InvalidRsyncUri(String),
|
||||
|
||||
#[error("rsync URI must reference a file object, got directory-like URI: {0}")]
|
||||
DirectoryLikeRsyncUri(String),
|
||||
|
||||
#[error("create mirror root failed: {path}: {detail}")]
|
||||
CreateMirrorRoot { path: String, detail: String },
|
||||
|
||||
#[error("remove mirror root failed: {path}: {detail}")]
|
||||
RemoveMirrorRoot { path: String, detail: String },
|
||||
|
||||
#[error("create parent directory failed: {path}: {detail}")]
|
||||
CreateParent { path: String, detail: String },
|
||||
|
||||
#[error("remove existing target failed: {path}: {detail}")]
|
||||
RemoveExistingTarget { path: String, detail: String },
|
||||
|
||||
#[error("static object not found for sha256={sha256_hex}")]
|
||||
MissingStaticObject { sha256_hex: String },
|
||||
|
||||
#[error("link target failed: {src} -> {dst}: {detail}")]
|
||||
Link {
|
||||
src: String,
|
||||
dst: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("copy target failed: {src} -> {dst}: {detail}")]
|
||||
Copy {
|
||||
src: String,
|
||||
dst: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("mirror tree mismatch after materialize: {0}")]
|
||||
TreeMismatch(String),
|
||||
|
||||
#[error("open raw store failed: {path}: {detail}")]
|
||||
OpenRawStore { path: String, detail: String },
|
||||
|
||||
#[error("raw object not found for sha256={sha256_hex}")]
|
||||
MissingRawStoreObject { sha256_hex: String },
|
||||
|
||||
#[error("read raw store failed for sha256={sha256_hex}: {detail}")]
|
||||
ReadRawStore { sha256_hex: String, detail: String },
|
||||
|
||||
#[error("open repo bytes store failed: {path}: {detail}")]
|
||||
OpenRepoBytesStore { path: String, detail: String },
|
||||
|
||||
#[error("repo bytes object not found for sha256={sha256_hex}")]
|
||||
MissingRepoBytesObject { sha256_hex: String },
|
||||
|
||||
#[error("read repo bytes store failed for sha256={sha256_hex}: {detail}")]
|
||||
ReadRepoBytesStore { sha256_hex: String, detail: String },
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CirMaterializeSummary {
|
||||
pub object_count: usize,
|
||||
pub trust_anchor_count: usize,
|
||||
pub materialized_file_count: usize,
|
||||
pub linked_files: usize,
|
||||
pub copied_files: usize,
|
||||
}
|
||||
|
||||
pub fn materialize_cir(
|
||||
cir: &CanonicalInputRepresentation,
|
||||
static_root: &Path,
|
||||
mirror_root: &Path,
|
||||
clean_rebuild: bool,
|
||||
) -> Result<CirMaterializeSummary, CirMaterializeError> {
|
||||
cir.validate().map_err(CirMaterializeError::TreeMismatch)?;
|
||||
|
||||
prepare_mirror_root(mirror_root, clean_rebuild)?;
|
||||
|
||||
let mut linked_files = 0usize;
|
||||
let mut copied_files = 0usize;
|
||||
|
||||
for object in cir.validated_objects() {
|
||||
let sha256_hex = hex::encode(&object.sha256);
|
||||
let source = resolve_static_pool_file(static_root, &sha256_hex)?;
|
||||
let relative = mirror_relative_path_for_rsync_uri(&object.rsync_uri)?;
|
||||
let target = mirror_root.join(&relative);
|
||||
|
||||
if let Some(parent) = target.parent() {
|
||||
fs::create_dir_all(parent).map_err(|e| CirMaterializeError::CreateParent {
|
||||
path: parent.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
|
||||
if target.exists() {
|
||||
fs::remove_file(&target).map_err(|e| CirMaterializeError::RemoveExistingTarget {
|
||||
path: target.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
|
||||
match fs::hard_link(&source, &target) {
|
||||
Ok(()) => linked_files += 1,
|
||||
Err(link_err) => {
|
||||
fs::copy(&source, &target).map_err(|copy_err| CirMaterializeError::Copy {
|
||||
src: source.display().to_string(),
|
||||
dst: target.display().to_string(),
|
||||
detail: format!("{copy_err}; original link error: {link_err}"),
|
||||
})?;
|
||||
copied_files += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for trust_anchor in &cir.trust_anchors {
|
||||
write_bytes_to_mirror_uri(
|
||||
mirror_root,
|
||||
&trust_anchor.ta_rsync_uri,
|
||||
&trust_anchor.ta_certificate_der,
|
||||
"cir trust anchor",
|
||||
)?;
|
||||
copied_files += 1;
|
||||
}
|
||||
|
||||
let actual = collect_materialized_uris(mirror_root)?;
|
||||
let expected = expected_materialized_uris(cir);
|
||||
if actual != expected {
|
||||
return Err(CirMaterializeError::TreeMismatch(format!(
|
||||
"expected {} files, got {} files",
|
||||
expected.len(),
|
||||
actual.len()
|
||||
)));
|
||||
}
|
||||
|
||||
Ok(CirMaterializeSummary {
|
||||
object_count: cir.validated_object_count(),
|
||||
trust_anchor_count: cir.trust_anchors.len(),
|
||||
materialized_file_count: expected.len(),
|
||||
linked_files,
|
||||
copied_files,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn materialize_cir_from_raw_store(
|
||||
cir: &CanonicalInputRepresentation,
|
||||
raw_store_db: &Path,
|
||||
mirror_root: &Path,
|
||||
clean_rebuild: bool,
|
||||
) -> Result<CirMaterializeSummary, CirMaterializeError> {
|
||||
cir.validate().map_err(CirMaterializeError::TreeMismatch)?;
|
||||
|
||||
prepare_mirror_root(mirror_root, clean_rebuild)?;
|
||||
|
||||
let raw_store =
|
||||
ExternalRawStoreDb::open(raw_store_db).map_err(|e| CirMaterializeError::OpenRawStore {
|
||||
path: raw_store_db.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
|
||||
let mut copied_files = 0usize;
|
||||
for object in cir.validated_objects() {
|
||||
let sha256_hex = hex::encode(&object.sha256);
|
||||
let bytes = raw_store
|
||||
.get_blob_bytes(&sha256_hex)
|
||||
.map_err(|e| CirMaterializeError::ReadRawStore {
|
||||
sha256_hex: sha256_hex.clone(),
|
||||
detail: e.to_string(),
|
||||
})?
|
||||
.ok_or_else(|| CirMaterializeError::MissingRawStoreObject {
|
||||
sha256_hex: sha256_hex.clone(),
|
||||
})?;
|
||||
write_bytes_to_mirror_uri(
|
||||
mirror_root,
|
||||
&object.rsync_uri,
|
||||
&bytes,
|
||||
&raw_store_db.display().to_string(),
|
||||
)?;
|
||||
copied_files += 1;
|
||||
}
|
||||
|
||||
for trust_anchor in &cir.trust_anchors {
|
||||
write_bytes_to_mirror_uri(
|
||||
mirror_root,
|
||||
&trust_anchor.ta_rsync_uri,
|
||||
&trust_anchor.ta_certificate_der,
|
||||
"cir trust anchor",
|
||||
)?;
|
||||
copied_files += 1;
|
||||
}
|
||||
|
||||
let actual = collect_materialized_uris(mirror_root)?;
|
||||
let expected = expected_materialized_uris(cir);
|
||||
if actual != expected {
|
||||
return Err(CirMaterializeError::TreeMismatch(format!(
|
||||
"expected {} files, got {} files",
|
||||
expected.len(),
|
||||
actual.len()
|
||||
)));
|
||||
}
|
||||
|
||||
Ok(CirMaterializeSummary {
|
||||
object_count: cir.validated_object_count(),
|
||||
trust_anchor_count: cir.trust_anchors.len(),
|
||||
materialized_file_count: expected.len(),
|
||||
linked_files: 0,
|
||||
copied_files,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn materialize_cir_from_repo_bytes(
|
||||
cir: &CanonicalInputRepresentation,
|
||||
repo_bytes_db: &Path,
|
||||
mirror_root: &Path,
|
||||
clean_rebuild: bool,
|
||||
) -> Result<CirMaterializeSummary, CirMaterializeError> {
|
||||
cir.validate().map_err(CirMaterializeError::TreeMismatch)?;
|
||||
|
||||
prepare_mirror_root(mirror_root, clean_rebuild)?;
|
||||
|
||||
let repo_bytes = ExternalRepoBytesDb::open(repo_bytes_db).map_err(|e| {
|
||||
CirMaterializeError::OpenRepoBytesStore {
|
||||
path: repo_bytes_db.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
|
||||
let mut copied_files = 0usize;
|
||||
for object in cir.validated_objects() {
|
||||
let sha256_hex = hex::encode(&object.sha256);
|
||||
let bytes = repo_bytes
|
||||
.get_blob_bytes(&sha256_hex)
|
||||
.map_err(|e| CirMaterializeError::ReadRepoBytesStore {
|
||||
sha256_hex: sha256_hex.clone(),
|
||||
detail: e.to_string(),
|
||||
})?
|
||||
.ok_or_else(|| CirMaterializeError::MissingRepoBytesObject {
|
||||
sha256_hex: sha256_hex.clone(),
|
||||
})?;
|
||||
write_bytes_to_mirror_uri(
|
||||
mirror_root,
|
||||
&object.rsync_uri,
|
||||
&bytes,
|
||||
&repo_bytes_db.display().to_string(),
|
||||
)?;
|
||||
copied_files += 1;
|
||||
}
|
||||
|
||||
for trust_anchor in &cir.trust_anchors {
|
||||
write_bytes_to_mirror_uri(
|
||||
mirror_root,
|
||||
&trust_anchor.ta_rsync_uri,
|
||||
&trust_anchor.ta_certificate_der,
|
||||
"cir trust anchor",
|
||||
)?;
|
||||
copied_files += 1;
|
||||
}
|
||||
|
||||
let actual = collect_materialized_uris(mirror_root)?;
|
||||
let expected = expected_materialized_uris(cir);
|
||||
if actual != expected {
|
||||
return Err(CirMaterializeError::TreeMismatch(format!(
|
||||
"expected {} files, got {} files",
|
||||
expected.len(),
|
||||
actual.len()
|
||||
)));
|
||||
}
|
||||
|
||||
Ok(CirMaterializeSummary {
|
||||
object_count: cir.validated_object_count(),
|
||||
trust_anchor_count: cir.trust_anchors.len(),
|
||||
materialized_file_count: expected.len(),
|
||||
linked_files: 0,
|
||||
copied_files,
|
||||
})
|
||||
}
|
||||
|
||||
fn prepare_mirror_root(mirror_root: &Path, clean_rebuild: bool) -> Result<(), CirMaterializeError> {
|
||||
if clean_rebuild && mirror_root.exists() {
|
||||
fs::remove_dir_all(mirror_root).map_err(|e| CirMaterializeError::RemoveMirrorRoot {
|
||||
path: mirror_root.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
fs::create_dir_all(mirror_root).map_err(|e| CirMaterializeError::CreateMirrorRoot {
|
||||
path: mirror_root.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn write_bytes_to_mirror_uri(
|
||||
mirror_root: &Path,
|
||||
rsync_uri: &str,
|
||||
bytes: &[u8],
|
||||
src_label: &str,
|
||||
) -> Result<(), CirMaterializeError> {
|
||||
let relative = mirror_relative_path_for_rsync_uri(rsync_uri)?;
|
||||
let target = mirror_root.join(&relative);
|
||||
|
||||
if let Some(parent) = target.parent() {
|
||||
fs::create_dir_all(parent).map_err(|e| CirMaterializeError::CreateParent {
|
||||
path: parent.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
|
||||
if target.exists() {
|
||||
fs::remove_file(&target).map_err(|e| CirMaterializeError::RemoveExistingTarget {
|
||||
path: target.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
|
||||
fs::write(&target, bytes).map_err(|e| CirMaterializeError::Copy {
|
||||
src: src_label.to_string(),
|
||||
dst: target.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn expected_materialized_uris(cir: &CanonicalInputRepresentation) -> BTreeSet<String> {
|
||||
cir.validated_objects()
|
||||
.map(|item| item.rsync_uri.clone())
|
||||
.chain(
|
||||
cir.trust_anchors
|
||||
.iter()
|
||||
.map(|item| item.ta_rsync_uri.clone()),
|
||||
)
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn mirror_relative_path_for_rsync_uri(rsync_uri: &str) -> Result<PathBuf, CirMaterializeError> {
|
||||
let url = url::Url::parse(rsync_uri)
|
||||
.map_err(|_| CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()))?;
|
||||
if url.scheme() != "rsync" {
|
||||
return Err(CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()));
|
||||
}
|
||||
let host = url
|
||||
.host_str()
|
||||
.ok_or_else(|| CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()))?;
|
||||
let segments = url
|
||||
.path_segments()
|
||||
.ok_or_else(|| CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()))?
|
||||
.collect::<Vec<_>>();
|
||||
if segments.is_empty() || segments.last().copied().unwrap_or_default().is_empty() {
|
||||
return Err(CirMaterializeError::DirectoryLikeRsyncUri(
|
||||
rsync_uri.to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let mut path = PathBuf::from(host);
|
||||
for segment in segments {
|
||||
if !segment.is_empty() {
|
||||
path.push(segment);
|
||||
}
|
||||
}
|
||||
Ok(path)
|
||||
}
|
||||
|
||||
pub fn resolve_static_pool_file(
|
||||
static_root: &Path,
|
||||
sha256_hex: &str,
|
||||
) -> Result<PathBuf, CirMaterializeError> {
|
||||
if sha256_hex.len() != 64 || !sha256_hex.as_bytes().iter().all(u8::is_ascii_hexdigit) {
|
||||
return Err(CirMaterializeError::MissingStaticObject {
|
||||
sha256_hex: sha256_hex.to_string(),
|
||||
});
|
||||
}
|
||||
let prefix1 = &sha256_hex[0..2];
|
||||
let prefix2 = &sha256_hex[2..4];
|
||||
|
||||
let entries =
|
||||
fs::read_dir(static_root).map_err(|_| CirMaterializeError::MissingStaticObject {
|
||||
sha256_hex: sha256_hex.to_string(),
|
||||
})?;
|
||||
let mut dates = entries
|
||||
.filter_map(Result::ok)
|
||||
.filter(|entry| entry.path().is_dir())
|
||||
.map(|entry| entry.path())
|
||||
.collect::<Vec<_>>();
|
||||
dates.sort();
|
||||
|
||||
for date_dir in dates {
|
||||
let candidate = date_dir.join(prefix1).join(prefix2).join(sha256_hex);
|
||||
if candidate.is_file() {
|
||||
return Ok(candidate);
|
||||
}
|
||||
}
|
||||
Err(CirMaterializeError::MissingStaticObject {
|
||||
sha256_hex: sha256_hex.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn collect_materialized_uris(mirror_root: &Path) -> Result<BTreeSet<String>, CirMaterializeError> {
|
||||
let mut out = BTreeSet::new();
|
||||
let mut stack = vec![mirror_root.to_path_buf()];
|
||||
while let Some(path) = stack.pop() {
|
||||
for entry in fs::read_dir(&path).map_err(|e| CirMaterializeError::CreateMirrorRoot {
|
||||
path: path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})? {
|
||||
let entry = entry.map_err(|e| CirMaterializeError::CreateMirrorRoot {
|
||||
path: path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
stack.push(path);
|
||||
} else {
|
||||
let rel = path
|
||||
.strip_prefix(mirror_root)
|
||||
.expect("materialized path under mirror root")
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
let uri = format!("rsync://{rel}");
|
||||
out.insert(uri);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
CirMaterializeError, materialize_cir, materialize_cir_from_raw_store,
|
||||
materialize_cir_from_repo_bytes, mirror_relative_path_for_rsync_uri,
|
||||
resolve_static_pool_file,
|
||||
};
|
||||
use crate::blob_store::{ExternalRawStoreDb, ExternalRepoBytesDb};
|
||||
use crate::cir::model::{CanonicalInputRepresentation, CirObject, CirTrustAnchor, sha256};
|
||||
use sha2::Digest;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
fn sample_time() -> time::OffsetDateTime {
|
||||
time::OffsetDateTime::parse(
|
||||
"2026-04-07T12:34:56Z",
|
||||
&time::format_description::well_known::Rfc3339,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn sample_trust_anchor() -> CirTrustAnchor {
|
||||
let ta_rsync_uri = "rsync://example.net/repo/ta.cer";
|
||||
let ta_certificate_der = b"ta-der".to_vec();
|
||||
CirTrustAnchor {
|
||||
ta_rsync_uri: ta_rsync_uri.to_string(),
|
||||
tal_uri: "https://tal.example.net/root.tal".to_string(),
|
||||
tal_bytes: format!("{ta_rsync_uri}\n\nAQID\n").into_bytes(),
|
||||
ta_certificate_sha256: sha256(&ta_certificate_der),
|
||||
ta_certificate_der,
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_cir() -> CanonicalInputRepresentation {
|
||||
CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
|
||||
sha256: hex::decode(
|
||||
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||
)
|
||||
.unwrap(),
|
||||
},
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/nested/b.roa".to_string(),
|
||||
sha256: hex::decode(
|
||||
"2222222222222222222222222222222222222222222222222222222222222222",
|
||||
)
|
||||
.unwrap(),
|
||||
},
|
||||
],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor()],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
)
|
||||
}
|
||||
|
||||
fn cir_with_real_hashes(a: &[u8], b: &[u8]) -> CanonicalInputRepresentation {
|
||||
CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
|
||||
sha256: sha2::Sha256::digest(a).to_vec(),
|
||||
},
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/nested/b.roa".to_string(),
|
||||
sha256: sha2::Sha256::digest(b).to_vec(),
|
||||
},
|
||||
],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor()],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mirror_relative_path_for_rsync_uri_maps_host_and_path() {
|
||||
let path =
|
||||
mirror_relative_path_for_rsync_uri("rsync://example.net/repo/nested/b.roa").unwrap();
|
||||
assert_eq!(
|
||||
path,
|
||||
PathBuf::from("example.net")
|
||||
.join("repo")
|
||||
.join("nested")
|
||||
.join("b.roa")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_static_pool_file_finds_hash_across_dates() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let path = td.path().join("20260407").join("11").join("11");
|
||||
std::fs::create_dir_all(&path).unwrap();
|
||||
let file = path.join("1111111111111111111111111111111111111111111111111111111111111111");
|
||||
std::fs::write(&file, b"x").unwrap();
|
||||
|
||||
let resolved = resolve_static_pool_file(
|
||||
td.path(),
|
||||
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(resolved, file);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_static_pool_file_rejects_invalid_hash_and_missing_hash() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let err = resolve_static_pool_file(td.path(), "not-a-hash")
|
||||
.expect_err("invalid hash should fail");
|
||||
assert!(matches!(
|
||||
err,
|
||||
CirMaterializeError::MissingStaticObject { .. }
|
||||
));
|
||||
|
||||
let err = resolve_static_pool_file(
|
||||
td.path(),
|
||||
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
)
|
||||
.expect_err("missing hash should fail");
|
||||
assert!(matches!(
|
||||
err,
|
||||
CirMaterializeError::MissingStaticObject { .. }
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mirror_relative_path_rejects_non_rsync_and_directory_like_uris() {
|
||||
let err = mirror_relative_path_for_rsync_uri("https://example.net/repo/a.roa")
|
||||
.expect_err("non-rsync uri must fail");
|
||||
assert!(matches!(err, CirMaterializeError::InvalidRsyncUri(_)));
|
||||
|
||||
let err = mirror_relative_path_for_rsync_uri("rsync://example.net/repo/")
|
||||
.expect_err("directory-like uri must fail");
|
||||
assert!(matches!(err, CirMaterializeError::DirectoryLikeRsyncUri(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_clean_rebuild_creates_exact_tree_and_removes_stale_files() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let static_root = td.path().join("static");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
|
||||
write_static(
|
||||
&static_root,
|
||||
"20260407",
|
||||
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||
b"a",
|
||||
);
|
||||
write_static(
|
||||
&static_root,
|
||||
"20260407",
|
||||
"2222222222222222222222222222222222222222222222222222222222222222",
|
||||
b"b",
|
||||
);
|
||||
std::fs::create_dir_all(mirror_root.join("stale")).unwrap();
|
||||
std::fs::write(mirror_root.join("stale/old.txt"), b"old").unwrap();
|
||||
|
||||
let summary = materialize_cir(&sample_cir(), &static_root, &mirror_root, true).unwrap();
|
||||
assert_eq!(summary.object_count, 2);
|
||||
assert_eq!(summary.trust_anchor_count, 1);
|
||||
assert_eq!(summary.materialized_file_count, 3);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
|
||||
b"a"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/nested/b.roa")).unwrap(),
|
||||
b"b"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
|
||||
b"ta-der"
|
||||
);
|
||||
assert!(!mirror_root.join("stale/old.txt").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_fails_when_static_object_missing() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let err = materialize_cir(&sample_cir(), td.path(), &td.path().join("mirror"), true)
|
||||
.expect_err("missing static object must fail");
|
||||
assert!(matches!(
|
||||
err,
|
||||
CirMaterializeError::MissingStaticObject { .. }
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_without_clean_rebuild_detects_stale_extra_files() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let static_root = td.path().join("static");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
|
||||
write_static(
|
||||
&static_root,
|
||||
"20260407",
|
||||
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||
b"a",
|
||||
);
|
||||
write_static(
|
||||
&static_root,
|
||||
"20260407",
|
||||
"2222222222222222222222222222222222222222222222222222222222222222",
|
||||
b"b",
|
||||
);
|
||||
std::fs::create_dir_all(mirror_root.join("extra")).unwrap();
|
||||
std::fs::write(mirror_root.join("extra/stale.txt"), b"stale").unwrap();
|
||||
|
||||
let err = materialize_cir(&sample_cir(), &static_root, &mirror_root, false)
|
||||
.expect_err("stale extra files should fail exact tree check");
|
||||
assert!(matches!(err, CirMaterializeError::TreeMismatch(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_from_raw_store_creates_expected_tree() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let raw_store_path = td.path().join("raw-store.db");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
let a = b"a".to_vec();
|
||||
let b = b"b".to_vec();
|
||||
let cir = cir_with_real_hashes(&a, &b);
|
||||
|
||||
{
|
||||
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
|
||||
raw_store
|
||||
.put_blob_bytes_batch(&[
|
||||
(hex::encode(&cir.fresh_validated_objects[0].sha256), a),
|
||||
(hex::encode(&cir.fresh_validated_objects[1].sha256), b),
|
||||
])
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let summary =
|
||||
materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, true).unwrap();
|
||||
assert_eq!(summary.object_count, 2);
|
||||
assert_eq!(summary.trust_anchor_count, 1);
|
||||
assert_eq!(summary.materialized_file_count, 3);
|
||||
assert_eq!(summary.linked_files, 0);
|
||||
assert_eq!(summary.copied_files, 3);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
|
||||
b"a"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/nested/b.roa")).unwrap(),
|
||||
b"b"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
|
||||
b"ta-der"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_from_raw_store_fails_when_object_missing() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let raw_store_path = td.path().join("raw-store.db");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
let cir = cir_with_real_hashes(b"a", b"b");
|
||||
{
|
||||
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
|
||||
raw_store
|
||||
.put_blob_bytes_batch(&[(
|
||||
hex::encode(&cir.fresh_validated_objects[0].sha256),
|
||||
b"a".to_vec(),
|
||||
)])
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let err = materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, true)
|
||||
.expect_err("missing second object should fail");
|
||||
assert!(matches!(
|
||||
err,
|
||||
CirMaterializeError::MissingRawStoreObject { .. }
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_from_raw_store_detects_stale_tree_when_not_clean_rebuild() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let raw_store_path = td.path().join("raw-store.db");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
let cir = cir_with_real_hashes(b"a", b"b");
|
||||
{
|
||||
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
|
||||
raw_store
|
||||
.put_blob_bytes_batch(&[
|
||||
(
|
||||
hex::encode(&cir.fresh_validated_objects[0].sha256),
|
||||
b"a".to_vec(),
|
||||
),
|
||||
(
|
||||
hex::encode(&cir.fresh_validated_objects[1].sha256),
|
||||
b"b".to_vec(),
|
||||
),
|
||||
])
|
||||
.unwrap();
|
||||
}
|
||||
std::fs::create_dir_all(mirror_root.join("extra")).unwrap();
|
||||
std::fs::write(mirror_root.join("extra/stale.txt"), b"stale").unwrap();
|
||||
|
||||
let err = materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, false)
|
||||
.expect_err("stale file should fail exact tree check");
|
||||
assert!(matches!(err, CirMaterializeError::TreeMismatch(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_from_raw_store_overwrites_existing_targets() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let raw_store_path = td.path().join("raw-store.db");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
let a = b"new-a".to_vec();
|
||||
let b = b"new-b".to_vec();
|
||||
let cir = cir_with_real_hashes(&a, &b);
|
||||
{
|
||||
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
|
||||
raw_store
|
||||
.put_blob_bytes_batch(&[
|
||||
(
|
||||
hex::encode(&cir.fresh_validated_objects[0].sha256),
|
||||
a.clone(),
|
||||
),
|
||||
(
|
||||
hex::encode(&cir.fresh_validated_objects[1].sha256),
|
||||
b.clone(),
|
||||
),
|
||||
])
|
||||
.unwrap();
|
||||
}
|
||||
let target = mirror_root.join("example.net/repo/a.cer");
|
||||
std::fs::create_dir_all(target.parent().unwrap()).unwrap();
|
||||
std::fs::write(&target, b"old").unwrap();
|
||||
|
||||
let summary =
|
||||
materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, false).unwrap();
|
||||
assert_eq!(summary.copied_files, 3);
|
||||
assert_eq!(std::fs::read(&target).unwrap(), a);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_from_raw_store_ignores_corrupt_raw_entry_when_blob_exists() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let raw_store_path = td.path().join("raw-store.db");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
let cir = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
|
||||
sha256: hex::decode(
|
||||
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||
)
|
||||
.unwrap(),
|
||||
}],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor()],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
{
|
||||
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
|
||||
raw_store
|
||||
.put_blob_bytes_batch(&[(
|
||||
"1111111111111111111111111111111111111111111111111111111111111111".to_string(),
|
||||
b"blob-a".to_vec(),
|
||||
)])
|
||||
.unwrap();
|
||||
}
|
||||
{
|
||||
let db = rocksdb::DB::open_default(&raw_store_path).unwrap();
|
||||
db.put(
|
||||
b"rawbyhash:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
b"bad-cbor",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let summary =
|
||||
materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, true).unwrap();
|
||||
assert_eq!(summary.object_count, 1);
|
||||
assert_eq!(summary.materialized_file_count, 2);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
|
||||
b"blob-a"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
|
||||
b"ta-der"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_from_repo_bytes_creates_expected_tree() {
|
||||
let td = tempfile::tempdir().unwrap();
|
||||
let repo_bytes_db = td.path().join("repo-bytes.db");
|
||||
let mirror_root = td.path().join("mirror");
|
||||
let a = b"a".to_vec();
|
||||
let b = b"b".to_vec();
|
||||
let cir = cir_with_real_hashes(&a, &b);
|
||||
|
||||
{
|
||||
let repo_bytes = ExternalRepoBytesDb::open(&repo_bytes_db).unwrap();
|
||||
repo_bytes
|
||||
.put_blob_bytes_batch(&[
|
||||
(hex::encode(&cir.fresh_validated_objects[0].sha256), a),
|
||||
(hex::encode(&cir.fresh_validated_objects[1].sha256), b),
|
||||
])
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let summary =
|
||||
materialize_cir_from_repo_bytes(&cir, &repo_bytes_db, &mirror_root, true).unwrap();
|
||||
assert_eq!(summary.object_count, 2);
|
||||
assert_eq!(summary.trust_anchor_count, 1);
|
||||
assert_eq!(summary.materialized_file_count, 3);
|
||||
assert_eq!(summary.linked_files, 0);
|
||||
assert_eq!(summary.copied_files, 3);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
|
||||
b"a"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/nested/b.roa")).unwrap(),
|
||||
b"b"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
|
||||
b"ta-der"
|
||||
);
|
||||
}
|
||||
|
||||
fn write_static(root: &Path, date: &str, hash: &str, bytes: &[u8]) {
|
||||
let path = root.join(date).join(&hash[0..2]).join(&hash[2..4]);
|
||||
std::fs::create_dir_all(&path).unwrap();
|
||||
std::fs::write(path.join(hash), bytes).unwrap();
|
||||
}
|
||||
}
|
||||
441
crates/panda-rpki-validator/src/cir/mod.rs
Normal file
441
crates/panda-rpki-validator/src/cir/mod.rs
Normal file
@ -0,0 +1,441 @@
|
||||
pub mod accumulator;
|
||||
pub mod decode;
|
||||
pub mod encode;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod export;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod materialize;
|
||||
pub mod model;
|
||||
pub mod sequence;
|
||||
#[cfg(feature = "full")]
|
||||
pub mod static_pool;
|
||||
|
||||
pub use accumulator::{CirInputAccumulator, CirInputSection, CirInputSnapshot};
|
||||
pub use decode::{CirDecodeError, decode_cir};
|
||||
pub use encode::{CirEncodeError, encode_cir};
|
||||
#[cfg(feature = "full")]
|
||||
pub use export::{
|
||||
CirExportError, CirExportSummary, CirTrustAnchorBinding, build_cir_from_input_snapshot_multi,
|
||||
build_cir_from_run, build_cir_from_run_multi, export_cir_from_input_snapshot_multi,
|
||||
export_cir_from_run, export_cir_from_run_multi, write_cir_file,
|
||||
};
|
||||
#[cfg(feature = "full")]
|
||||
pub use materialize::{
|
||||
CirMaterializeError, CirMaterializeSummary, materialize_cir, materialize_cir_from_raw_store,
|
||||
materialize_cir_from_repo_bytes, mirror_relative_path_for_rsync_uri, resolve_static_pool_file,
|
||||
};
|
||||
pub use model::{
|
||||
CIR_VERSION_V1, CIR_VERSION_V3, CIR_VERSION_V4, CanonicalInputRepresentation, CirHashAlgorithm,
|
||||
CirObject, CirRejectedObject, CirTrustAnchor, compute_reject_list_sha256, sha256,
|
||||
};
|
||||
pub use sequence::{CirSequenceManifest, CirSequenceStep, CirSequenceStepKind};
|
||||
#[cfg(feature = "full")]
|
||||
pub use static_pool::{
|
||||
CirStaticPoolError, CirStaticPoolExportSummary, CirStaticPoolWriteResult,
|
||||
export_hashes_from_store, static_pool_path, static_pool_relative_path,
|
||||
write_bytes_to_static_pool, write_raw_entry_to_static_pool,
|
||||
};
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
CIR_VERSION_V4, CanonicalInputRepresentation, CirObject, CirRejectedObject, CirTrustAnchor,
|
||||
decode_cir, encode_cir,
|
||||
};
|
||||
|
||||
fn sample_time() -> time::OffsetDateTime {
|
||||
time::OffsetDateTime::parse(
|
||||
"2026-04-07T12:34:56Z",
|
||||
&time::format_description::well_known::Rfc3339,
|
||||
)
|
||||
.expect("valid rfc3339")
|
||||
}
|
||||
|
||||
fn sample_trust_anchor(ta_rsync_uri: &str, tal_uri: &str, ta_der: &[u8]) -> CirTrustAnchor {
|
||||
CirTrustAnchor {
|
||||
ta_rsync_uri: ta_rsync_uri.to_string(),
|
||||
tal_uri: tal_uri.to_string(),
|
||||
tal_bytes: format!("{ta_rsync_uri}\n\nAQID\n").into_bytes(),
|
||||
ta_certificate_der: ta_der.to_vec(),
|
||||
ta_certificate_sha256: super::sha256(ta_der),
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_cir() -> CanonicalInputRepresentation {
|
||||
let rejected_objects = vec![
|
||||
CirRejectedObject {
|
||||
object_uri: "rsync://example.net/repo/rejected-a.roa".to_string(),
|
||||
reason: Some("invalid roa".to_string()),
|
||||
},
|
||||
CirRejectedObject {
|
||||
object_uri: "rsync://example.net/repo/rejected-b.asa".to_string(),
|
||||
reason: None,
|
||||
},
|
||||
];
|
||||
CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
|
||||
sha256: vec![0x11; 32],
|
||||
},
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/b.roa".to_string(),
|
||||
sha256: vec![0x22; 32],
|
||||
},
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/rejected-a.roa".to_string(),
|
||||
sha256: vec![0x33; 32],
|
||||
},
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/rejected-b.asa".to_string(),
|
||||
sha256: vec![0x44; 32],
|
||||
},
|
||||
],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der",
|
||||
)],
|
||||
rejected_objects,
|
||||
Vec::new(),
|
||||
)
|
||||
}
|
||||
|
||||
fn test_encode_tlv(tag: u8, value: &[u8]) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(8 + value.len());
|
||||
out.push(tag);
|
||||
if value.len() < 0x80 {
|
||||
out.push(value.len() as u8);
|
||||
} else {
|
||||
let len = value.len();
|
||||
let bytes = len.to_be_bytes();
|
||||
let first_non_zero = bytes
|
||||
.iter()
|
||||
.position(|&b| b != 0)
|
||||
.unwrap_or(bytes.len() - 1);
|
||||
let len_bytes = &bytes[first_non_zero..];
|
||||
out.push(0x80 | len_bytes.len() as u8);
|
||||
out.extend_from_slice(len_bytes);
|
||||
}
|
||||
out.extend_from_slice(value);
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_roundtrip_full_succeeds() {
|
||||
let cir = sample_cir();
|
||||
let der = encode_cir(&cir).expect("encode cir");
|
||||
let decoded = decode_cir(&der).expect("decode cir");
|
||||
assert_eq!(decoded, cir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_roundtrip_minimal_succeeds() {
|
||||
let cir = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/minimal-ta.cer",
|
||||
"https://tal.example.net/minimal.tal",
|
||||
b"minimal-ta-der",
|
||||
)],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let der = encode_cir(&cir).expect("encode minimal cir");
|
||||
let decoded = decode_cir(&der).expect("decode minimal cir");
|
||||
assert_eq!(decoded, cir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_model_rejects_unsorted_duplicate_objects() {
|
||||
let cir = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/z.roa".to_string(),
|
||||
sha256: vec![0x11; 32],
|
||||
},
|
||||
CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/a.roa".to_string(),
|
||||
sha256: vec![0x22; 32],
|
||||
},
|
||||
],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der",
|
||||
)],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let err = encode_cir(&cir).expect_err("unsorted objects must fail");
|
||||
assert!(
|
||||
err.to_string().contains("CIR.freshValidatedObjects"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_model_rejects_duplicate_tals() {
|
||||
let cir = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
vec![
|
||||
sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der-a",
|
||||
),
|
||||
sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der-b",
|
||||
),
|
||||
],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let err = encode_cir(&cir).expect_err("duplicate trust_anchors must fail");
|
||||
assert!(err.to_string().contains("CIR.trustAnchors"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_decode_rejects_wrong_version() {
|
||||
let mut der = encode_cir(&sample_cir()).expect("encode cir");
|
||||
let pos = der
|
||||
.windows(3)
|
||||
.position(|window| window == [0x02, 0x01, CIR_VERSION_V4 as u8])
|
||||
.or_else(|| {
|
||||
der.windows(3)
|
||||
.position(|window| window == [0x02, 0x01, CIR_VERSION_V4 as u8])
|
||||
})
|
||||
.expect("find version integer");
|
||||
der[pos + 2] = 2;
|
||||
let err = decode_cir(&der).expect_err("wrong version must fail");
|
||||
assert!(err.to_string().contains("unexpected CIR version"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_decode_rejects_wrong_hash_oid() {
|
||||
let mut der = encode_cir(&sample_cir()).expect("encode cir");
|
||||
let sha256_bytes = crate::data_model::oid::OID_SHA256_RAW;
|
||||
let idx = der
|
||||
.windows(sha256_bytes.len())
|
||||
.position(|window| window == sha256_bytes)
|
||||
.expect("find sha256 oid");
|
||||
der[idx + sha256_bytes.len() - 1] ^= 0x01;
|
||||
let err = decode_cir(&der).expect_err("wrong oid must fail");
|
||||
assert!(
|
||||
err.to_string().contains(crate::data_model::oid::OID_SHA256),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_decode_rejects_bad_generalized_time() {
|
||||
let mut der = encode_cir(&sample_cir()).expect("encode cir");
|
||||
let pos = der
|
||||
.windows(15)
|
||||
.position(|window| window == b"20260407123456Z")
|
||||
.expect("find generalized time");
|
||||
der[pos + 14] = b'X';
|
||||
let err = decode_cir(&der).expect_err("bad time must fail");
|
||||
assert!(err.to_string().contains("GeneralizedTime"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_model_rejects_non_rsync_object_uri_and_empty_tals() {
|
||||
let bad_object = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![CirObject {
|
||||
rsync_uri: "https://example.net/repo/a.roa".to_string(),
|
||||
sha256: vec![0x11; 32],
|
||||
}],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der",
|
||||
)],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let err = encode_cir(&bad_object).expect_err("non-rsync object uri must fail");
|
||||
assert!(err.to_string().contains("rsync://"), "{err}");
|
||||
|
||||
let no_tals = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let err = encode_cir(&no_tals).expect_err("empty trust_anchors must fail");
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("CIR.trustAnchors must be non-empty"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_model_rejects_non_utc_time_bad_hash_len_and_non_http_tal_uri() {
|
||||
let bad_time = CanonicalInputRepresentation::new_v4(
|
||||
sample_time().to_offset(time::UtcOffset::from_hms(8, 0, 0).unwrap()),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der",
|
||||
)],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let bad_time = CanonicalInputRepresentation {
|
||||
validation_time: sample_time().to_offset(time::UtcOffset::from_hms(8, 0, 0).unwrap()),
|
||||
..bad_time
|
||||
};
|
||||
let err = encode_cir(&bad_time).expect_err("non-utc validation time must fail");
|
||||
assert!(err.to_string().contains("UTC"), "{err}");
|
||||
|
||||
let bad_hash = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
vec![CirObject {
|
||||
rsync_uri: "rsync://example.net/repo/a.roa".to_string(),
|
||||
sha256: vec![0x11; 31],
|
||||
}],
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"https://tal.example.net/root.tal",
|
||||
b"ta-der",
|
||||
)],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let err = encode_cir(&bad_hash).expect_err("bad digest len must fail");
|
||||
assert!(err.to_string().contains("32 bytes"), "{err}");
|
||||
|
||||
let bad_tal_uri = CanonicalInputRepresentation::new_v4(
|
||||
sample_time(),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
vec![sample_trust_anchor(
|
||||
"rsync://example.net/repo/ta.cer",
|
||||
"ftp://tal.example.net/root.tal",
|
||||
b"ta-der",
|
||||
)],
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
);
|
||||
let err = encode_cir(&bad_tal_uri).expect_err("bad tal uri must fail");
|
||||
assert!(err.to_string().contains("http:// or https://"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_decode_rejects_trailing_bytes_and_trailing_fields() {
|
||||
let cir = sample_cir();
|
||||
let mut der = encode_cir(&cir).expect("encode cir");
|
||||
der.push(0);
|
||||
let err = decode_cir(&der).expect_err("trailing bytes after cir must fail");
|
||||
assert!(
|
||||
err.to_string().contains("trailing bytes after CIR"),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let object = test_encode_tlv(
|
||||
0x30,
|
||||
&[
|
||||
test_encode_tlv(0x16, b"rsync://example.net/repo/a.roa"),
|
||||
test_encode_tlv(0x04, &[0x11; 32]),
|
||||
test_encode_tlv(0x02, &[0x01]),
|
||||
]
|
||||
.concat(),
|
||||
);
|
||||
let trust_anchor = test_encode_tlv(
|
||||
0x30,
|
||||
&[
|
||||
test_encode_tlv(0x16, b"rsync://example.net/repo/ta.cer"),
|
||||
test_encode_tlv(0x16, b"https://tal.example.net/root.tal"),
|
||||
test_encode_tlv(0x04, b"rsync://example.net/repo/ta.cer\n\nAQID\n"),
|
||||
test_encode_tlv(0x04, b"ta-der"),
|
||||
test_encode_tlv(0x04, &super::sha256(b"ta-der")),
|
||||
]
|
||||
.concat(),
|
||||
);
|
||||
let bad = test_encode_tlv(
|
||||
0x30,
|
||||
&[
|
||||
test_encode_tlv(0x02, &[CIR_VERSION_V4 as u8]),
|
||||
test_encode_tlv(0x06, crate::data_model::oid::OID_SHA256_RAW),
|
||||
test_encode_tlv(0x18, b"20260407123456Z"),
|
||||
test_encode_tlv(0x30, &object),
|
||||
test_encode_tlv(0x30, &[]),
|
||||
test_encode_tlv(0x30, &trust_anchor),
|
||||
test_encode_tlv(0x04, &[0x33; 32]),
|
||||
test_encode_tlv(0x04, &[0x33; 32]),
|
||||
test_encode_tlv(0x04, &[0x33; 32]),
|
||||
test_encode_tlv(0x04, &[0x33; 32]),
|
||||
test_encode_tlv(0x04, &[0x33; 32]),
|
||||
test_encode_tlv(0x04, &[0x33; 32]),
|
||||
test_encode_tlv(0x30, &[]),
|
||||
test_encode_tlv(0x30, &[]),
|
||||
]
|
||||
.concat(),
|
||||
);
|
||||
let err = decode_cir(&bad).expect_err("trailing field in object must fail");
|
||||
assert!(
|
||||
err.to_string().contains("trailing fields in CirObject"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_decode_rejects_invalid_object_and_tal_shapes() {
|
||||
let cir = sample_cir();
|
||||
let mut der = encode_cir(&cir).expect("encode cir");
|
||||
|
||||
let rsync_text = b"rsync://example.net/repo/a.cer";
|
||||
let idx = der
|
||||
.windows(rsync_text.len())
|
||||
.position(|window| window == rsync_text)
|
||||
.expect("find object uri");
|
||||
der[idx] = 0xFF;
|
||||
let err = decode_cir(&der).expect_err("invalid utf8 object uri must fail");
|
||||
assert!(err.to_string().contains("utf-8"), "{err}");
|
||||
|
||||
let mut der = encode_cir(&cir).expect("encode cir");
|
||||
let tal_text = b"https://tal.example.net/root.tal";
|
||||
let idx = der
|
||||
.windows(tal_text.len())
|
||||
.position(|window| window == tal_text)
|
||||
.expect("find tal uri");
|
||||
der[idx] = 0xFF;
|
||||
let err = decode_cir(&der).expect_err("invalid utf8 tal uri must fail");
|
||||
assert!(err.to_string().contains("utf-8"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cir_model_rejects_unsorted_rejected_objects_and_bad_digest() {
|
||||
let mut cir = sample_cir();
|
||||
cir.fresh_rejected_objects.swap(0, 1);
|
||||
let err = encode_cir(&cir).expect_err("unsorted rejected objects must fail");
|
||||
assert!(
|
||||
err.to_string().contains("CIR.freshRejectedObjects"),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let mut cir = sample_cir();
|
||||
cir.reject_list_sha256 = vec![0x55; 32];
|
||||
let err = encode_cir(&cir).expect_err("bad reject list digest must fail");
|
||||
assert!(err.to_string().contains("rejectListSha256"), "{err}");
|
||||
}
|
||||
}
|
||||
441
crates/panda-rpki-validator/src/cir/model.rs
Normal file
441
crates/panda-rpki-validator/src/cir/model.rs
Normal file
@ -0,0 +1,441 @@
|
||||
use crate::data_model::oid::OID_SHA256;
|
||||
|
||||
pub const CIR_VERSION_V1: u32 = 1;
|
||||
pub const CIR_VERSION_V2: u32 = 2;
|
||||
pub const CIR_VERSION_V3: u32 = 3;
|
||||
pub const CIR_VERSION_V4: u32 = 4;
|
||||
pub const DIGEST_LEN_SHA256: usize = 32;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum CirHashAlgorithm {
|
||||
Sha256,
|
||||
}
|
||||
|
||||
impl CirHashAlgorithm {
|
||||
pub fn oid(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Sha256 => OID_SHA256,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CanonicalInputRepresentation {
|
||||
pub version: u32,
|
||||
pub hash_alg: CirHashAlgorithm,
|
||||
pub validation_time: time::OffsetDateTime,
|
||||
pub fresh_validated_objects: Vec<CirObject>,
|
||||
pub cached_validated_objects: Vec<CirObject>,
|
||||
pub trust_anchors: Vec<CirTrustAnchor>,
|
||||
pub object_list_sha256: Vec<u8>,
|
||||
pub fresh_object_list_sha256: Vec<u8>,
|
||||
pub cached_object_list_sha256: Vec<u8>,
|
||||
pub reject_list_sha256: Vec<u8>,
|
||||
pub fresh_reject_list_sha256: Vec<u8>,
|
||||
pub cached_reject_list_sha256: Vec<u8>,
|
||||
pub fresh_rejected_objects: Vec<CirRejectedObject>,
|
||||
pub cached_rejected_objects: Vec<CirRejectedObject>,
|
||||
}
|
||||
|
||||
impl CanonicalInputRepresentation {
|
||||
pub fn new_v4(
|
||||
validation_time: time::OffsetDateTime,
|
||||
fresh_validated_objects: Vec<CirObject>,
|
||||
cached_validated_objects: Vec<CirObject>,
|
||||
trust_anchors: Vec<CirTrustAnchor>,
|
||||
fresh_rejected_objects: Vec<CirRejectedObject>,
|
||||
cached_rejected_objects: Vec<CirRejectedObject>,
|
||||
) -> Self {
|
||||
let fresh_object_list_sha256 = compute_object_list_sha256(fresh_validated_objects.iter());
|
||||
let cached_object_list_sha256 = compute_object_list_sha256(cached_validated_objects.iter());
|
||||
let object_list_sha256 = compute_sectioned_object_list_sha256(
|
||||
fresh_validated_objects
|
||||
.iter()
|
||||
.map(|object| ("fresh", object))
|
||||
.chain(
|
||||
cached_validated_objects
|
||||
.iter()
|
||||
.map(|object| ("cached", object)),
|
||||
),
|
||||
);
|
||||
let fresh_reject_list_sha256 = compute_reject_list_sha256(
|
||||
fresh_rejected_objects
|
||||
.iter()
|
||||
.map(|item| item.object_uri.as_str()),
|
||||
);
|
||||
let cached_reject_list_sha256 = compute_reject_list_sha256(
|
||||
cached_rejected_objects
|
||||
.iter()
|
||||
.map(|item| item.object_uri.as_str()),
|
||||
);
|
||||
let mut all_rejected_uris = fresh_rejected_objects
|
||||
.iter()
|
||||
.chain(cached_rejected_objects.iter())
|
||||
.map(|item| item.object_uri.as_str())
|
||||
.collect::<Vec<_>>();
|
||||
all_rejected_uris.sort_unstable();
|
||||
all_rejected_uris.dedup();
|
||||
let reject_list_sha256 = compute_reject_list_sha256(all_rejected_uris.iter().copied());
|
||||
|
||||
Self {
|
||||
version: CIR_VERSION_V4,
|
||||
hash_alg: CirHashAlgorithm::Sha256,
|
||||
validation_time: validation_time.to_offset(time::UtcOffset::UTC),
|
||||
fresh_validated_objects,
|
||||
cached_validated_objects,
|
||||
trust_anchors,
|
||||
object_list_sha256,
|
||||
fresh_object_list_sha256,
|
||||
cached_object_list_sha256,
|
||||
reject_list_sha256,
|
||||
fresh_reject_list_sha256,
|
||||
cached_reject_list_sha256,
|
||||
fresh_rejected_objects,
|
||||
cached_rejected_objects,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.version != CIR_VERSION_V4 {
|
||||
return Err(format!(
|
||||
"CIR version must be {CIR_VERSION_V4}, got {}",
|
||||
self.version
|
||||
));
|
||||
}
|
||||
if !matches!(self.hash_alg, CirHashAlgorithm::Sha256) {
|
||||
return Err("CIR hashAlg must be SHA-256".into());
|
||||
}
|
||||
if self.validation_time.offset() != time::UtcOffset::UTC {
|
||||
return Err("CIR validationTime must be UTC".into());
|
||||
}
|
||||
validate_sorted_unique_strings(
|
||||
self.fresh_validated_objects
|
||||
.iter()
|
||||
.map(|item| item.rsync_uri.as_str()),
|
||||
"CIR.freshValidatedObjects must be sorted by rsyncUri and unique",
|
||||
)?;
|
||||
validate_sorted_unique_strings(
|
||||
self.cached_validated_objects
|
||||
.iter()
|
||||
.map(|item| item.rsync_uri.as_str()),
|
||||
"CIR.cachedValidatedObjects must be sorted by rsyncUri and unique",
|
||||
)?;
|
||||
validate_sorted_unique_strings(
|
||||
self.trust_anchors
|
||||
.iter()
|
||||
.map(|item| item.ta_rsync_uri.as_str()),
|
||||
"CIR.trustAnchors must be sorted by taRsyncUri and unique",
|
||||
)?;
|
||||
validate_sorted_unique_strings(
|
||||
self.fresh_rejected_objects
|
||||
.iter()
|
||||
.map(|item| item.object_uri.as_str()),
|
||||
"CIR.freshRejectedObjects must be sorted by objectUri and unique",
|
||||
)?;
|
||||
validate_sorted_unique_strings(
|
||||
self.cached_rejected_objects
|
||||
.iter()
|
||||
.map(|item| item.object_uri.as_str()),
|
||||
"CIR.cachedRejectedObjects must be sorted by objectUri and unique",
|
||||
)?;
|
||||
let object_uris = self
|
||||
.fresh_validated_objects
|
||||
.iter()
|
||||
.chain(self.cached_validated_objects.iter())
|
||||
.map(|item| item.rsync_uri.as_str())
|
||||
.collect::<std::collections::BTreeSet<_>>();
|
||||
for trust_anchor in &self.trust_anchors {
|
||||
if object_uris.contains(trust_anchor.ta_rsync_uri.as_str()) {
|
||||
return Err(format!(
|
||||
"CIR.objects must not include trust anchor URI {}",
|
||||
trust_anchor.ta_rsync_uri
|
||||
));
|
||||
}
|
||||
}
|
||||
let fresh_object_uris = self
|
||||
.fresh_validated_objects
|
||||
.iter()
|
||||
.map(|item| item.rsync_uri.as_str())
|
||||
.collect::<std::collections::BTreeSet<_>>();
|
||||
for item in &self.fresh_rejected_objects {
|
||||
if !fresh_object_uris.contains(item.object_uri.as_str()) {
|
||||
return Err(format!(
|
||||
"CIR.freshRejectedObjects URI must exist in freshValidatedObjects: {}",
|
||||
item.object_uri
|
||||
));
|
||||
}
|
||||
}
|
||||
let cached_object_uris = self
|
||||
.cached_validated_objects
|
||||
.iter()
|
||||
.map(|item| item.rsync_uri.as_str())
|
||||
.collect::<std::collections::BTreeSet<_>>();
|
||||
for item in &self.cached_rejected_objects {
|
||||
if !cached_object_uris.contains(item.object_uri.as_str()) {
|
||||
return Err(format!(
|
||||
"CIR.cachedRejectedObjects URI must exist in cachedValidatedObjects: {}",
|
||||
item.object_uri
|
||||
));
|
||||
}
|
||||
}
|
||||
if self.trust_anchors.is_empty() {
|
||||
return Err("CIR.trustAnchors must be non-empty".into());
|
||||
}
|
||||
for (label, digest) in [
|
||||
("CIR.objectListSha256", &self.object_list_sha256),
|
||||
("CIR.freshObjectListSha256", &self.fresh_object_list_sha256),
|
||||
(
|
||||
"CIR.cachedObjectListSha256",
|
||||
&self.cached_object_list_sha256,
|
||||
),
|
||||
("CIR.rejectListSha256", &self.reject_list_sha256),
|
||||
("CIR.freshRejectListSha256", &self.fresh_reject_list_sha256),
|
||||
(
|
||||
"CIR.cachedRejectListSha256",
|
||||
&self.cached_reject_list_sha256,
|
||||
),
|
||||
] {
|
||||
if digest.len() != DIGEST_LEN_SHA256 {
|
||||
return Err(format!(
|
||||
"{label} must be {DIGEST_LEN_SHA256} bytes, got {}",
|
||||
digest.len()
|
||||
));
|
||||
}
|
||||
}
|
||||
for object in self.validated_objects() {
|
||||
object.validate()?;
|
||||
}
|
||||
for trust_anchor in &self.trust_anchors {
|
||||
trust_anchor.validate()?;
|
||||
}
|
||||
for item in self.rejected_objects() {
|
||||
item.validate()?;
|
||||
}
|
||||
let expected_fresh_object_digest =
|
||||
compute_object_list_sha256(self.fresh_validated_objects.iter());
|
||||
if self.fresh_object_list_sha256 != expected_fresh_object_digest {
|
||||
return Err("CIR.freshObjectListSha256 does not match freshValidatedObjects".into());
|
||||
}
|
||||
let expected_cached_object_digest =
|
||||
compute_object_list_sha256(self.cached_validated_objects.iter());
|
||||
if self.cached_object_list_sha256 != expected_cached_object_digest {
|
||||
return Err("CIR.cachedObjectListSha256 does not match cachedValidatedObjects".into());
|
||||
}
|
||||
let expected_object_digest = compute_sectioned_object_list_sha256(
|
||||
self.fresh_validated_objects
|
||||
.iter()
|
||||
.map(|object| ("fresh", object))
|
||||
.chain(
|
||||
self.cached_validated_objects
|
||||
.iter()
|
||||
.map(|object| ("cached", object)),
|
||||
),
|
||||
);
|
||||
if self.object_list_sha256 != expected_object_digest {
|
||||
return Err("CIR.objectListSha256 does not match fresh/cached objects".into());
|
||||
}
|
||||
let expected_fresh_reject_digest = compute_reject_list_sha256(
|
||||
self.fresh_rejected_objects
|
||||
.iter()
|
||||
.map(|item| item.object_uri.as_str()),
|
||||
);
|
||||
if self.fresh_reject_list_sha256 != expected_fresh_reject_digest {
|
||||
return Err("CIR.freshRejectListSha256 does not match freshRejectedObjects".into());
|
||||
}
|
||||
let expected_cached_reject_digest = compute_reject_list_sha256(
|
||||
self.cached_rejected_objects
|
||||
.iter()
|
||||
.map(|item| item.object_uri.as_str()),
|
||||
);
|
||||
if self.cached_reject_list_sha256 != expected_cached_reject_digest {
|
||||
return Err("CIR.cachedRejectListSha256 does not match cachedRejectedObjects".into());
|
||||
}
|
||||
let mut all_rejected_uris = self
|
||||
.rejected_objects()
|
||||
.map(|item| item.object_uri.as_str())
|
||||
.collect::<Vec<_>>();
|
||||
all_rejected_uris.sort_unstable();
|
||||
all_rejected_uris.dedup();
|
||||
let expected_digest = compute_reject_list_sha256(all_rejected_uris.iter().copied());
|
||||
if self.reject_list_sha256 != expected_digest {
|
||||
return Err("CIR.rejectListSha256 does not match fresh/cached rejectedObjects".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn validated_objects(&self) -> impl Iterator<Item = &CirObject> {
|
||||
self.fresh_validated_objects
|
||||
.iter()
|
||||
.chain(self.cached_validated_objects.iter())
|
||||
}
|
||||
|
||||
pub fn rejected_objects(&self) -> impl Iterator<Item = &CirRejectedObject> {
|
||||
self.fresh_rejected_objects
|
||||
.iter()
|
||||
.chain(self.cached_rejected_objects.iter())
|
||||
}
|
||||
|
||||
pub fn validated_object_count(&self) -> usize {
|
||||
self.fresh_validated_objects.len() + self.cached_validated_objects.len()
|
||||
}
|
||||
|
||||
pub fn rejected_object_count(&self) -> usize {
|
||||
self.fresh_rejected_objects.len() + self.cached_rejected_objects.len()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CirObject {
|
||||
pub rsync_uri: String,
|
||||
pub sha256: Vec<u8>,
|
||||
}
|
||||
|
||||
impl CirObject {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if !self.rsync_uri.starts_with("rsync://") {
|
||||
return Err(format!(
|
||||
"CirObject.rsync_uri must start with rsync://, got {}",
|
||||
self.rsync_uri
|
||||
));
|
||||
}
|
||||
if self.sha256.len() != DIGEST_LEN_SHA256 {
|
||||
return Err(format!(
|
||||
"CirObject.sha256 must be {DIGEST_LEN_SHA256} bytes, got {}",
|
||||
self.sha256.len()
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CirTrustAnchor {
|
||||
pub ta_rsync_uri: String,
|
||||
pub tal_uri: String,
|
||||
pub tal_bytes: Vec<u8>,
|
||||
pub ta_certificate_der: Vec<u8>,
|
||||
pub ta_certificate_sha256: Vec<u8>,
|
||||
}
|
||||
|
||||
impl CirTrustAnchor {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if !self.ta_rsync_uri.starts_with("rsync://") {
|
||||
return Err(format!(
|
||||
"CirTrustAnchor.ta_rsync_uri must start with rsync://, got {}",
|
||||
self.ta_rsync_uri
|
||||
));
|
||||
}
|
||||
if !(self.tal_uri.starts_with("https://") || self.tal_uri.starts_with("http://")) {
|
||||
return Err(format!(
|
||||
"CirTrustAnchor.tal_uri must start with http:// or https://, got {}",
|
||||
self.tal_uri
|
||||
));
|
||||
}
|
||||
if self.tal_bytes.is_empty() {
|
||||
return Err("CirTrustAnchor.tal_bytes must be non-empty".into());
|
||||
}
|
||||
let tal = crate::data_model::tal::Tal::decode_bytes(&self.tal_bytes)
|
||||
.map_err(|e| format!("CirTrustAnchor.tal_bytes must decode as TAL: {e}"))?;
|
||||
if !tal
|
||||
.ta_uris
|
||||
.iter()
|
||||
.any(|uri| uri.as_str() == self.ta_rsync_uri)
|
||||
{
|
||||
return Err(format!(
|
||||
"CirTrustAnchor.ta_rsync_uri must be listed in TAL bytes: {}",
|
||||
self.ta_rsync_uri
|
||||
));
|
||||
}
|
||||
if self.ta_certificate_der.is_empty() {
|
||||
return Err("CirTrustAnchor.ta_certificate_der must be non-empty".into());
|
||||
}
|
||||
if self.ta_certificate_sha256.len() != DIGEST_LEN_SHA256 {
|
||||
return Err(format!(
|
||||
"CirTrustAnchor.ta_certificate_sha256 must be {DIGEST_LEN_SHA256} bytes, got {}",
|
||||
self.ta_certificate_sha256.len()
|
||||
));
|
||||
}
|
||||
let expected = sha256(&self.ta_certificate_der);
|
||||
if self.ta_certificate_sha256 != expected {
|
||||
return Err("CirTrustAnchor.ta_certificate_sha256 does not match DER bytes".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CirRejectedObject {
|
||||
pub object_uri: String,
|
||||
pub reason: Option<String>,
|
||||
}
|
||||
|
||||
impl CirRejectedObject {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if !self.object_uri.starts_with("rsync://") {
|
||||
return Err(format!(
|
||||
"CirRejectedObject.object_uri must start with rsync://, got {}",
|
||||
self.object_uri
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub fn compute_reject_list_sha256<'a>(uris: impl IntoIterator<Item = &'a str>) -> Vec<u8> {
|
||||
let mut body = Vec::new();
|
||||
for uri in uris {
|
||||
let bytes = uri.as_bytes();
|
||||
body.extend_from_slice(&(bytes.len() as u32).to_be_bytes());
|
||||
body.extend_from_slice(bytes);
|
||||
}
|
||||
sha256(&body)
|
||||
}
|
||||
|
||||
pub fn compute_object_list_sha256<'a>(objects: impl IntoIterator<Item = &'a CirObject>) -> Vec<u8> {
|
||||
let mut body = Vec::new();
|
||||
for object in objects {
|
||||
append_digest_string(&mut body, &object.rsync_uri);
|
||||
body.extend_from_slice(&object.sha256);
|
||||
}
|
||||
sha256(&body)
|
||||
}
|
||||
|
||||
pub fn compute_sectioned_object_list_sha256<'a>(
|
||||
objects: impl IntoIterator<Item = (&'a str, &'a CirObject)>,
|
||||
) -> Vec<u8> {
|
||||
let mut body = Vec::new();
|
||||
for (section, object) in objects {
|
||||
append_digest_string(&mut body, section);
|
||||
append_digest_string(&mut body, &object.rsync_uri);
|
||||
body.extend_from_slice(&object.sha256);
|
||||
}
|
||||
sha256(&body)
|
||||
}
|
||||
|
||||
pub fn sha256(bytes: &[u8]) -> Vec<u8> {
|
||||
use sha2::Digest;
|
||||
|
||||
sha2::Sha256::digest(bytes).to_vec()
|
||||
}
|
||||
|
||||
fn append_digest_string(body: &mut Vec<u8>, value: &str) {
|
||||
let bytes = value.as_bytes();
|
||||
body.extend_from_slice(&(bytes.len() as u32).to_be_bytes());
|
||||
body.extend_from_slice(bytes);
|
||||
}
|
||||
|
||||
fn validate_sorted_unique_strings<'a>(
|
||||
items: impl IntoIterator<Item = &'a str>,
|
||||
message: &str,
|
||||
) -> Result<(), String> {
|
||||
let mut prev: Option<&'a str> = None;
|
||||
for key in items {
|
||||
if let Some(prev_key) = prev
|
||||
&& key <= prev_key
|
||||
{
|
||||
return Err(message.into());
|
||||
}
|
||||
prev = Some(key);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
203
crates/panda-rpki-validator/src/cir/sequence.rs
Normal file
203
crates/panda-rpki-validator/src/cir/sequence.rs
Normal file
@ -0,0 +1,203 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CirSequenceStepKind {
|
||||
Full,
|
||||
Delta,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct CirSequenceStep {
|
||||
pub step_id: String,
|
||||
pub kind: CirSequenceStepKind,
|
||||
pub validation_time: String,
|
||||
pub cir_path: String,
|
||||
pub ccr_path: String,
|
||||
pub report_path: String,
|
||||
#[serde(default)]
|
||||
pub timing_path: Option<String>,
|
||||
#[serde(default)]
|
||||
pub stdout_log_path: Option<String>,
|
||||
#[serde(default)]
|
||||
pub stderr_log_path: Option<String>,
|
||||
#[serde(default)]
|
||||
pub artifact_prefix: Option<String>,
|
||||
pub previous_step_id: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct CirSequenceManifest {
|
||||
pub version: u32,
|
||||
pub repo_bytes_db_path: String,
|
||||
pub steps: Vec<CirSequenceStep>,
|
||||
}
|
||||
|
||||
impl CirSequenceManifest {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.version == 0 {
|
||||
return Err("sequence.version must be positive".to_string());
|
||||
}
|
||||
if self.repo_bytes_db_path.trim().is_empty() {
|
||||
return Err("sequence.repo_bytes_db_path must not be empty".to_string());
|
||||
}
|
||||
if self.steps.is_empty() {
|
||||
return Err("sequence.steps must not be empty".to_string());
|
||||
}
|
||||
let mut previous_ids = std::collections::BTreeSet::new();
|
||||
for (idx, step) in self.steps.iter().enumerate() {
|
||||
if step.step_id.trim().is_empty() {
|
||||
return Err(format!("sequence.steps[{idx}].step_id must not be empty"));
|
||||
}
|
||||
if !previous_ids.insert(step.step_id.clone()) {
|
||||
return Err(format!("sequence.steps[{idx}].step_id must be unique"));
|
||||
}
|
||||
if step.validation_time.trim().is_empty() {
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}].validation_time must not be empty"
|
||||
));
|
||||
}
|
||||
if let Some(timing_path) = &step.timing_path
|
||||
&& timing_path.trim().is_empty()
|
||||
{
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}].timing_path must not be empty when set"
|
||||
));
|
||||
}
|
||||
if let Some(stdout_log_path) = &step.stdout_log_path
|
||||
&& stdout_log_path.trim().is_empty()
|
||||
{
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}].stdout_log_path must not be empty when set"
|
||||
));
|
||||
}
|
||||
if let Some(stderr_log_path) = &step.stderr_log_path
|
||||
&& stderr_log_path.trim().is_empty()
|
||||
{
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}].stderr_log_path must not be empty when set"
|
||||
));
|
||||
}
|
||||
if let Some(artifact_prefix) = &step.artifact_prefix
|
||||
&& artifact_prefix.trim().is_empty()
|
||||
{
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}].artifact_prefix must not be empty when set"
|
||||
));
|
||||
}
|
||||
if step.cir_path.trim().is_empty()
|
||||
|| step.ccr_path.trim().is_empty()
|
||||
|| step.report_path.trim().is_empty()
|
||||
{
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}] output paths must not be empty"
|
||||
));
|
||||
}
|
||||
match step.kind {
|
||||
CirSequenceStepKind::Full => {
|
||||
if idx != 0 {
|
||||
return Err("full step must be the first step".to_string());
|
||||
}
|
||||
if step.previous_step_id.is_some() {
|
||||
return Err("full step must not reference previous_step_id".to_string());
|
||||
}
|
||||
}
|
||||
CirSequenceStepKind::Delta => {
|
||||
if idx == 0 {
|
||||
return Err("delta step cannot be the first step".to_string());
|
||||
}
|
||||
let previous = step.previous_step_id.as_ref().ok_or_else(|| {
|
||||
format!("sequence.steps[{idx}] delta step must set previous_step_id")
|
||||
})?;
|
||||
if !previous_ids.contains(previous) {
|
||||
return Err(format!(
|
||||
"sequence.steps[{idx}] previous_step_id must reference an earlier step"
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{CirSequenceManifest, CirSequenceStep, CirSequenceStepKind};
|
||||
|
||||
fn sample_manifest() -> CirSequenceManifest {
|
||||
CirSequenceManifest {
|
||||
version: 1,
|
||||
repo_bytes_db_path: "repo-bytes.db".to_string(),
|
||||
steps: vec![
|
||||
CirSequenceStep {
|
||||
step_id: "full".to_string(),
|
||||
kind: CirSequenceStepKind::Full,
|
||||
validation_time: "2026-04-09T00:00:00Z".to_string(),
|
||||
cir_path: "full/input.cir".to_string(),
|
||||
ccr_path: "full/result.ccr".to_string(),
|
||||
report_path: "full/report.json".to_string(),
|
||||
timing_path: Some("full/timing.json".to_string()),
|
||||
stdout_log_path: Some("full/stdout.log".to_string()),
|
||||
stderr_log_path: Some("full/stderr.log".to_string()),
|
||||
artifact_prefix: Some("2026-04-09T00:00:00Z-test".to_string()),
|
||||
previous_step_id: None,
|
||||
},
|
||||
CirSequenceStep {
|
||||
step_id: "delta-001".to_string(),
|
||||
kind: CirSequenceStepKind::Delta,
|
||||
validation_time: "2026-04-09T00:10:00Z".to_string(),
|
||||
cir_path: "delta-001/input.cir".to_string(),
|
||||
ccr_path: "delta-001/result.ccr".to_string(),
|
||||
report_path: "delta-001/report.json".to_string(),
|
||||
timing_path: Some("delta-001/timing.json".to_string()),
|
||||
stdout_log_path: Some("delta-001/stdout.log".to_string()),
|
||||
stderr_log_path: Some("delta-001/stderr.log".to_string()),
|
||||
artifact_prefix: Some("2026-04-09T00:10:00Z-test".to_string()),
|
||||
previous_step_id: Some("full".to_string()),
|
||||
},
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sequence_manifest_validate_accepts_minimal_chain() {
|
||||
sample_manifest().validate().expect("valid sequence");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sequence_manifest_validate_rejects_bad_order_and_duplicates() {
|
||||
let mut bad = sample_manifest();
|
||||
bad.steps.swap(0, 1);
|
||||
let err = bad.validate().expect_err("full must be first");
|
||||
assert!(
|
||||
err.contains("delta step cannot be the first step")
|
||||
|| err.contains("full step must be the first step")
|
||||
);
|
||||
|
||||
let mut dup = sample_manifest();
|
||||
dup.steps[1].step_id = "full".to_string();
|
||||
let err = dup.validate().expect_err("duplicate id must fail");
|
||||
assert!(err.contains("must be unique"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sequence_manifest_validate_rejects_missing_previous_reference() {
|
||||
let mut bad = sample_manifest();
|
||||
bad.steps[1].previous_step_id = Some("missing".to_string());
|
||||
let err = bad.validate().expect_err("missing previous must fail");
|
||||
assert!(err.contains("previous_step_id"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sequence_manifest_validate_rejects_empty_repo_bytes_backend() {
|
||||
let mut manifest = sample_manifest();
|
||||
manifest.repo_bytes_db_path = " ".to_string();
|
||||
let err = manifest
|
||||
.validate()
|
||||
.expect_err("empty repo bytes backend must fail");
|
||||
assert!(err.contains("repo_bytes_db_path"));
|
||||
}
|
||||
}
|
||||
376
crates/panda-rpki-validator/src/cir/static_pool.rs
Normal file
376
crates/panda-rpki-validator/src/cir/static_pool.rs
Normal file
@ -0,0 +1,376 @@
|
||||
use std::collections::BTreeSet;
|
||||
use std::fs::{self, OpenOptions};
|
||||
use std::io::Write;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use crate::storage::{RawByHashEntry, RocksStore};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CirStaticPoolError {
|
||||
#[error("invalid sha256 hex: {0}")]
|
||||
InvalidSha256Hex(String),
|
||||
|
||||
#[error("raw bytes are empty for sha256={sha256_hex}")]
|
||||
EmptyBytes { sha256_hex: String },
|
||||
|
||||
#[error("raw bytes do not match sha256 hex: {sha256_hex}")]
|
||||
HashMismatch { sha256_hex: String },
|
||||
|
||||
#[error("create directory failed: {path}: {detail}")]
|
||||
CreateDir { path: String, detail: String },
|
||||
|
||||
#[error("create temp file failed: {path}: {detail}")]
|
||||
CreateTemp { path: String, detail: String },
|
||||
|
||||
#[error("write temp file failed: {path}: {detail}")]
|
||||
WriteTemp { path: String, detail: String },
|
||||
|
||||
#[error("sync temp file failed: {path}: {detail}")]
|
||||
SyncTemp { path: String, detail: String },
|
||||
|
||||
#[error("publish temp file failed: {temp_path} -> {final_path}: {detail}")]
|
||||
Publish {
|
||||
temp_path: String,
|
||||
final_path: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("remove temp file failed: {path}: {detail}")]
|
||||
RemoveTemp { path: String, detail: String },
|
||||
|
||||
#[error("raw_by_hash entry missing for sha256={sha256_hex}")]
|
||||
MissingRawByHash { sha256_hex: String },
|
||||
|
||||
#[error("storage error: {0}")]
|
||||
Storage(String),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CirStaticPoolWriteResult {
|
||||
pub final_path: PathBuf,
|
||||
pub written: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CirStaticPoolExportSummary {
|
||||
pub unique_hashes: usize,
|
||||
pub written_files: usize,
|
||||
pub reused_files: usize,
|
||||
}
|
||||
|
||||
pub fn static_pool_relative_path(
|
||||
capture_date_utc: time::Date,
|
||||
sha256_hex: &str,
|
||||
) -> Result<PathBuf, CirStaticPoolError> {
|
||||
validate_sha256_hex(sha256_hex)?;
|
||||
let date = format_utc_date(capture_date_utc);
|
||||
Ok(PathBuf::from(date)
|
||||
.join(&sha256_hex[0..2])
|
||||
.join(&sha256_hex[2..4])
|
||||
.join(sha256_hex))
|
||||
}
|
||||
|
||||
pub fn static_pool_path(
|
||||
static_root: &Path,
|
||||
capture_date_utc: time::Date,
|
||||
sha256_hex: &str,
|
||||
) -> Result<PathBuf, CirStaticPoolError> {
|
||||
Ok(static_root.join(static_pool_relative_path(capture_date_utc, sha256_hex)?))
|
||||
}
|
||||
|
||||
pub fn write_bytes_to_static_pool(
|
||||
static_root: &Path,
|
||||
capture_date_utc: time::Date,
|
||||
sha256_hex_value: &str,
|
||||
bytes: &[u8],
|
||||
) -> Result<CirStaticPoolWriteResult, CirStaticPoolError> {
|
||||
validate_sha256_hex(sha256_hex_value)?;
|
||||
if bytes.is_empty() {
|
||||
return Err(CirStaticPoolError::EmptyBytes {
|
||||
sha256_hex: sha256_hex_value.to_string(),
|
||||
});
|
||||
}
|
||||
let computed = compute_sha256_hex(bytes);
|
||||
if computed != sha256_hex_value.to_ascii_lowercase() {
|
||||
return Err(CirStaticPoolError::HashMismatch {
|
||||
sha256_hex: sha256_hex_value.to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let final_path = static_pool_path(static_root, capture_date_utc, sha256_hex_value)?;
|
||||
if final_path.exists() {
|
||||
return Ok(CirStaticPoolWriteResult {
|
||||
final_path,
|
||||
written: false,
|
||||
});
|
||||
}
|
||||
|
||||
let parent = final_path.parent().expect("static pool file has parent");
|
||||
fs::create_dir_all(parent).map_err(|e| CirStaticPoolError::CreateDir {
|
||||
path: parent.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
|
||||
let temp_path = parent.join(format!("{sha256_hex_value}.tmp.{}", uuid::Uuid::new_v4()));
|
||||
let mut file = OpenOptions::new()
|
||||
.create_new(true)
|
||||
.write(true)
|
||||
.open(&temp_path)
|
||||
.map_err(|e| CirStaticPoolError::CreateTemp {
|
||||
path: temp_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
file.write_all(bytes)
|
||||
.map_err(|e| CirStaticPoolError::WriteTemp {
|
||||
path: temp_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
file.sync_all().map_err(|e| CirStaticPoolError::SyncTemp {
|
||||
path: temp_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
drop(file);
|
||||
|
||||
match fs::hard_link(&temp_path, &final_path) {
|
||||
Ok(()) => {
|
||||
fs::remove_file(&temp_path).map_err(|e| CirStaticPoolError::RemoveTemp {
|
||||
path: temp_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
Ok(CirStaticPoolWriteResult {
|
||||
final_path,
|
||||
written: true,
|
||||
})
|
||||
}
|
||||
Err(e) if final_path.exists() => {
|
||||
fs::remove_file(&temp_path).map_err(|remove_err| CirStaticPoolError::RemoveTemp {
|
||||
path: temp_path.display().to_string(),
|
||||
detail: remove_err.to_string(),
|
||||
})?;
|
||||
let _ = e;
|
||||
Ok(CirStaticPoolWriteResult {
|
||||
final_path,
|
||||
written: false,
|
||||
})
|
||||
}
|
||||
Err(e) => Err(CirStaticPoolError::Publish {
|
||||
temp_path: temp_path.display().to_string(),
|
||||
final_path: final_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn write_raw_entry_to_static_pool(
|
||||
static_root: &Path,
|
||||
capture_date_utc: time::Date,
|
||||
entry: &RawByHashEntry,
|
||||
) -> Result<CirStaticPoolWriteResult, CirStaticPoolError> {
|
||||
write_bytes_to_static_pool(
|
||||
static_root,
|
||||
capture_date_utc,
|
||||
&entry.sha256_hex,
|
||||
&entry.bytes,
|
||||
)
|
||||
}
|
||||
|
||||
pub fn export_hashes_from_store(
|
||||
store: &RocksStore,
|
||||
static_root: &Path,
|
||||
capture_date_utc: time::Date,
|
||||
sha256_hexes: &[String],
|
||||
) -> Result<CirStaticPoolExportSummary, CirStaticPoolError> {
|
||||
let unique: BTreeSet<String> = sha256_hexes
|
||||
.iter()
|
||||
.map(|item| item.to_ascii_lowercase())
|
||||
.collect();
|
||||
|
||||
let mut written_files = 0usize;
|
||||
let mut reused_files = 0usize;
|
||||
for sha256_hex in &unique {
|
||||
let bytes = store
|
||||
.get_blob_bytes(sha256_hex)
|
||||
.map_err(|e| CirStaticPoolError::Storage(e.to_string()))?
|
||||
.ok_or_else(|| CirStaticPoolError::MissingRawByHash {
|
||||
sha256_hex: sha256_hex.clone(),
|
||||
})?;
|
||||
let entry = RawByHashEntry::from_bytes(sha256_hex.clone(), bytes);
|
||||
let result = write_raw_entry_to_static_pool(static_root, capture_date_utc, &entry)?;
|
||||
if result.written {
|
||||
written_files += 1;
|
||||
} else {
|
||||
reused_files += 1;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(CirStaticPoolExportSummary {
|
||||
unique_hashes: unique.len(),
|
||||
written_files,
|
||||
reused_files,
|
||||
})
|
||||
}
|
||||
|
||||
fn format_utc_date(date: time::Date) -> String {
|
||||
format!(
|
||||
"{:04}{:02}{:02}",
|
||||
date.year(),
|
||||
u8::from(date.month()),
|
||||
date.day()
|
||||
)
|
||||
}
|
||||
|
||||
fn validate_sha256_hex(sha256_hex: &str) -> Result<(), CirStaticPoolError> {
|
||||
if sha256_hex.len() != 64 || !sha256_hex.as_bytes().iter().all(u8::is_ascii_hexdigit) {
|
||||
return Err(CirStaticPoolError::InvalidSha256Hex(sha256_hex.to_string()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn compute_sha256_hex(bytes: &[u8]) -> String {
|
||||
use sha2::{Digest, Sha256};
|
||||
hex::encode(Sha256::digest(bytes))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
CirStaticPoolError, compute_sha256_hex, export_hashes_from_store,
|
||||
static_pool_relative_path, write_bytes_to_static_pool,
|
||||
};
|
||||
use crate::storage::{RawByHashEntry, RepositoryViewEntry, RepositoryViewState, RocksStore};
|
||||
use std::fs;
|
||||
|
||||
fn sample_date() -> time::Date {
|
||||
time::Date::from_calendar_date(2026, time::Month::April, 7).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn static_pool_relative_path_uses_date_and_hash_prefixes() {
|
||||
let path = static_pool_relative_path(
|
||||
sample_date(),
|
||||
"abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
|
||||
)
|
||||
.expect("relative path");
|
||||
assert_eq!(
|
||||
path,
|
||||
std::path::PathBuf::from("20260407")
|
||||
.join("ab")
|
||||
.join("cd")
|
||||
.join("abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_bytes_to_static_pool_is_idempotent_and_leaves_no_temp_files() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let bytes = b"static-pool-object";
|
||||
let sha = compute_sha256_hex(bytes);
|
||||
|
||||
let first =
|
||||
write_bytes_to_static_pool(td.path(), sample_date(), &sha, bytes).expect("first write");
|
||||
let second = write_bytes_to_static_pool(td.path(), sample_date(), &sha, bytes)
|
||||
.expect("second write");
|
||||
|
||||
assert!(first.written);
|
||||
assert!(!second.written);
|
||||
assert_eq!(fs::read(&first.final_path).expect("read final"), bytes);
|
||||
|
||||
let all_files: Vec<_> = walk_files(td.path());
|
||||
assert_eq!(all_files.len(), 1);
|
||||
assert!(
|
||||
!all_files[0]
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.unwrap_or_default()
|
||||
.contains(".tmp.")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_bytes_to_static_pool_rejects_bad_hash_and_empty_bytes() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let err = write_bytes_to_static_pool(td.path(), sample_date(), "not-a-hash", b"x")
|
||||
.expect_err("bad hash must fail");
|
||||
assert!(matches!(err, CirStaticPoolError::InvalidSha256Hex(_)));
|
||||
|
||||
let err = write_bytes_to_static_pool(
|
||||
td.path(),
|
||||
sample_date(),
|
||||
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
b"",
|
||||
)
|
||||
.expect_err("empty bytes must fail");
|
||||
assert!(matches!(err, CirStaticPoolError::EmptyBytes { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn export_hashes_from_store_writes_unique_entries_and_fails_when_missing() {
|
||||
let td = tempfile::tempdir().expect("tempdir");
|
||||
let store_dir = td.path().join("db");
|
||||
let static_root = td.path().join("static");
|
||||
let store = RocksStore::open(&store_dir).expect("open rocksdb");
|
||||
|
||||
let bytes = b"store-object".to_vec();
|
||||
let sha = compute_sha256_hex(&bytes);
|
||||
let mut entry = RawByHashEntry::from_bytes(sha.clone(), bytes.clone());
|
||||
entry
|
||||
.origin_uris
|
||||
.push("rsync://example.test/repo/object.cer".to_string());
|
||||
store.put_raw_by_hash_entry(&entry).expect("put raw entry");
|
||||
store
|
||||
.put_repository_view_entry(&RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/object.cer".to_string(),
|
||||
current_hash: Some(sha.clone()),
|
||||
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
|
||||
object_type: Some("cer".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
})
|
||||
.expect("put repository view");
|
||||
|
||||
let summary = export_hashes_from_store(
|
||||
&store,
|
||||
&static_root,
|
||||
sample_date(),
|
||||
&[sha.clone(), sha.clone()],
|
||||
)
|
||||
.expect("export hashes");
|
||||
assert_eq!(summary.unique_hashes, 1);
|
||||
assert_eq!(summary.written_files, 1);
|
||||
assert_eq!(summary.reused_files, 0);
|
||||
|
||||
let summary = export_hashes_from_store(&store, &static_root, sample_date(), &[sha.clone()])
|
||||
.expect("re-export hashes");
|
||||
assert_eq!(summary.unique_hashes, 1);
|
||||
assert_eq!(summary.written_files, 0);
|
||||
assert_eq!(summary.reused_files, 1);
|
||||
|
||||
let err = export_hashes_from_store(
|
||||
&store,
|
||||
&static_root,
|
||||
sample_date(),
|
||||
&[String::from(
|
||||
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
)],
|
||||
)
|
||||
.expect_err("missing raw_by_hash must fail");
|
||||
assert!(matches!(err, CirStaticPoolError::MissingRawByHash { .. }));
|
||||
}
|
||||
|
||||
fn walk_files(root: &std::path::Path) -> Vec<std::path::PathBuf> {
|
||||
let mut out = Vec::new();
|
||||
let mut stack = vec![root.to_path_buf()];
|
||||
while let Some(path) = stack.pop() {
|
||||
for entry in fs::read_dir(path).expect("read_dir") {
|
||||
let entry = entry.expect("dir entry");
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
stack.push(path);
|
||||
} else {
|
||||
out.push(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
}
|
||||
2897
crates/panda-rpki-validator/src/cli.rs
Normal file
2897
crates/panda-rpki-validator/src/cli.rs
Normal file
File diff suppressed because it is too large
Load Diff
419
crates/panda-rpki-validator/src/cli/output.rs
Normal file
419
crates/panda-rpki-validator/src/cli/output.rs
Normal file
@ -0,0 +1,419 @@
|
||||
use std::io::BufWriter;
|
||||
use std::path::Path;
|
||||
|
||||
use serde::Serialize;
|
||||
use serde::ser::SerializeSeq;
|
||||
use sha2::Digest;
|
||||
|
||||
use crate::audit::{
|
||||
AspaOutput, AuditRunMeta, AuditWarning, QueryAuditManifest, ValidationEvent,
|
||||
ValidationEventCounts, VrpOutput,
|
||||
};
|
||||
use crate::ccr::canonical_vrp_prefix;
|
||||
|
||||
use super::{PostValidationShared, RunStageTiming};
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) enum ReportJsonFormat {
|
||||
Pretty,
|
||||
Compact,
|
||||
}
|
||||
|
||||
pub(super) fn write_json<T: Serialize>(
|
||||
path: &Path,
|
||||
report: &T,
|
||||
format: ReportJsonFormat,
|
||||
) -> Result<(), String> {
|
||||
let f = std::fs::File::create(path)
|
||||
.map_err(|e| format!("create report file failed: {}: {e}", path.display()))?;
|
||||
let writer = BufWriter::new(f);
|
||||
match format {
|
||||
ReportJsonFormat::Pretty => serde_json::to_writer_pretty(writer, report),
|
||||
ReportJsonFormat::Compact => serde_json::to_writer(writer, report),
|
||||
}
|
||||
.map_err(|e| format!("write report json failed: {e}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(super) struct ReportJsonWriteTiming {
|
||||
pub(super) build_ms: u64,
|
||||
pub(super) write_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct BorrowedAuditReportV2<'a> {
|
||||
format_version: u32,
|
||||
meta: AuditRunMeta,
|
||||
policy: &'a crate::policy::Policy,
|
||||
tree: BorrowedTreeSummary<'a>,
|
||||
publication_points: &'a [crate::audit::PublicationPointAudit],
|
||||
vrps: VrpReportSequence<'a>,
|
||||
aspas: AspaReportSequence<'a>,
|
||||
downloads: &'a [crate::audit::AuditDownloadEvent],
|
||||
download_stats: &'a crate::audit::AuditDownloadStats,
|
||||
repo_sync_stats: crate::audit::AuditRepoSyncStats,
|
||||
#[serde(rename = "queryAudit", skip_serializing_if = "Option::is_none")]
|
||||
query_audit: Option<QueryAuditManifest>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct BorrowedTreeSummary<'a> {
|
||||
instances_processed: usize,
|
||||
instances_failed: usize,
|
||||
warnings: WarningReportSequence<'a>,
|
||||
}
|
||||
|
||||
struct WarningReportSequence<'a>(&'a [crate::report::Warning]);
|
||||
|
||||
impl Serialize for WarningReportSequence<'_> {
|
||||
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
|
||||
where
|
||||
S: serde::Serializer,
|
||||
{
|
||||
let mut seq = serializer.serialize_seq(Some(self.0.len()))?;
|
||||
for warning in self.0 {
|
||||
seq.serialize_element(&AuditWarning::from(warning))?;
|
||||
}
|
||||
seq.end()
|
||||
}
|
||||
}
|
||||
|
||||
struct VrpReportSequence<'a>(&'a [crate::validation::objects::Vrp]);
|
||||
|
||||
impl Serialize for VrpReportSequence<'_> {
|
||||
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
|
||||
where
|
||||
S: serde::Serializer,
|
||||
{
|
||||
let mut seq = serializer.serialize_seq(Some(self.0.len()))?;
|
||||
for vrp in self.0 {
|
||||
seq.serialize_element(&VrpOutput {
|
||||
asn: vrp.asn,
|
||||
prefix: crate::audit::format_roa_ip_prefix(&vrp.prefix),
|
||||
max_length: vrp.max_length,
|
||||
})?;
|
||||
}
|
||||
seq.end()
|
||||
}
|
||||
}
|
||||
|
||||
struct AspaReportSequence<'a>(&'a [crate::validation::objects::AspaAttestation]);
|
||||
|
||||
impl Serialize for AspaReportSequence<'_> {
|
||||
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
|
||||
where
|
||||
S: serde::Serializer,
|
||||
{
|
||||
let mut seq = serializer.serialize_seq(Some(self.0.len()))?;
|
||||
for aspa in self.0 {
|
||||
seq.serialize_element(&AspaOutput {
|
||||
customer_as_id: aspa.customer_as_id,
|
||||
provider_as_ids: aspa.provider_as_ids.clone(),
|
||||
})?;
|
||||
}
|
||||
seq.end()
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn write_report_json_from_shared(
|
||||
path: &Path,
|
||||
policy: &crate::policy::Policy,
|
||||
validation_time: time::OffsetDateTime,
|
||||
shared: &PostValidationShared,
|
||||
format: ReportJsonFormat,
|
||||
) -> Result<ReportJsonWriteTiming, String> {
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
let build_started = std::time::Instant::now();
|
||||
let validation_time_rfc3339_utc = validation_time
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&Rfc3339)
|
||||
.expect("format validation_time");
|
||||
let repo_sync_stats = super::build_repo_sync_stats(shared.publication_points.as_ref());
|
||||
let query_audit = write_validation_events_sidecar(path, &validation_time_rfc3339_utc, shared)?;
|
||||
let report = BorrowedAuditReportV2 {
|
||||
format_version: 2,
|
||||
meta: AuditRunMeta {
|
||||
validation_time_rfc3339_utc,
|
||||
},
|
||||
policy,
|
||||
tree: BorrowedTreeSummary {
|
||||
instances_processed: shared.instances_processed,
|
||||
instances_failed: shared.instances_failed,
|
||||
warnings: WarningReportSequence(shared.tree_warnings.as_ref()),
|
||||
},
|
||||
publication_points: shared.publication_points.as_ref(),
|
||||
vrps: VrpReportSequence(shared.vrps.as_ref()),
|
||||
aspas: AspaReportSequence(shared.aspas.as_ref()),
|
||||
downloads: shared.downloads.as_ref(),
|
||||
download_stats: &shared.download_stats,
|
||||
repo_sync_stats,
|
||||
query_audit: Some(query_audit),
|
||||
};
|
||||
let build_ms = build_started.elapsed().as_millis() as u64;
|
||||
|
||||
let write_started = std::time::Instant::now();
|
||||
write_json(path, &report, format)?;
|
||||
Ok(ReportJsonWriteTiming {
|
||||
build_ms,
|
||||
write_ms: write_started.elapsed().as_millis() as u64,
|
||||
})
|
||||
}
|
||||
|
||||
fn write_validation_events_sidecar(
|
||||
report_path: &Path,
|
||||
validation_time: &str,
|
||||
shared: &PostValidationShared,
|
||||
) -> Result<QueryAuditManifest, String> {
|
||||
let events_path = report_path.with_file_name("validation-events.jsonl");
|
||||
if let Some(parent) = events_path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("create validation events parent failed: {e}"))?;
|
||||
}
|
||||
let mut writer = BufWriter::new(std::fs::File::create(&events_path).map_err(|e| {
|
||||
format!(
|
||||
"create validation events failed: {}: {e}",
|
||||
events_path.display()
|
||||
)
|
||||
})?);
|
||||
let mut seq = 0u64;
|
||||
let mut hasher = sha2::Sha256::new();
|
||||
emit_validation_events(validation_time, shared, &mut seq, &mut |event| {
|
||||
let mut line = serde_json::to_vec(&event)
|
||||
.map_err(|e| format!("serialize validation event failed: {e}"))?;
|
||||
line.push(b'\n');
|
||||
std::io::Write::write_all(&mut writer, &line)
|
||||
.map_err(|e| format!("write validation event failed: {e}"))?;
|
||||
hasher.update(&line);
|
||||
Ok(())
|
||||
})?;
|
||||
std::io::Write::flush(&mut writer)
|
||||
.map_err(|e| format!("flush validation events failed: {e}"))?;
|
||||
let events_count = seq;
|
||||
let events_sha256 = hex::encode(hasher.finalize());
|
||||
Ok(QueryAuditManifest {
|
||||
schema_version: 1,
|
||||
status: "complete".to_string(),
|
||||
events_path: events_path
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.unwrap_or("validation-events.jsonl")
|
||||
.to_string(),
|
||||
events_count,
|
||||
events_sha256,
|
||||
writer_version: 1,
|
||||
error: None,
|
||||
})
|
||||
}
|
||||
|
||||
fn emit_validation_events(
|
||||
validation_time: &str,
|
||||
shared: &PostValidationShared,
|
||||
seq: &mut u64,
|
||||
emit: &mut impl FnMut(ValidationEvent) -> Result<(), String>,
|
||||
) -> Result<(), String> {
|
||||
emit(next_event(seq, "run_summary", validation_time, |event| {
|
||||
event.counts = Some(ValidationEventCounts {
|
||||
objects: Some(
|
||||
shared
|
||||
.publication_points
|
||||
.iter()
|
||||
.map(|pp| pp.objects.len() as u64)
|
||||
.sum(),
|
||||
),
|
||||
warnings: Some(
|
||||
(shared.tree_warnings.len()
|
||||
+ shared
|
||||
.publication_points
|
||||
.iter()
|
||||
.map(|pp| pp.warnings.len())
|
||||
.sum::<usize>()) as u64,
|
||||
),
|
||||
vrps: Some(shared.vrps.len() as u64),
|
||||
aspas: Some(shared.aspas.len() as u64),
|
||||
});
|
||||
}))?;
|
||||
for pp in shared.publication_points.iter() {
|
||||
emit(next_event(
|
||||
seq,
|
||||
"publication_point",
|
||||
validation_time,
|
||||
|event| {
|
||||
event.pp_node_id = pp.node_id;
|
||||
event.pp_manifest_uri = Some(pp.manifest_rsync_uri.clone());
|
||||
event.pp_rsync_base_uri = Some(pp.rsync_base_uri.clone());
|
||||
event.repo_sync_phase = pp.repo_sync_phase.clone();
|
||||
event.repo_terminal_state = Some(pp.repo_terminal_state.clone());
|
||||
event.counts = Some(ValidationEventCounts {
|
||||
objects: Some(pp.objects.len() as u64),
|
||||
warnings: Some(pp.warnings.len() as u64),
|
||||
vrps: None,
|
||||
aspas: None,
|
||||
});
|
||||
},
|
||||
))?;
|
||||
for object in &pp.objects {
|
||||
emit(next_event(seq, "object", validation_time, |event| {
|
||||
event.pp_node_id = pp.node_id;
|
||||
event.pp_manifest_uri = Some(pp.manifest_rsync_uri.clone());
|
||||
event.object_uri = Some(object.rsync_uri.clone());
|
||||
event.sha256 = Some(object.sha256_hex.clone());
|
||||
event.object_type = Some(object.kind.clone());
|
||||
event.result = Some(object.result.clone());
|
||||
event.reason = object.detail.clone();
|
||||
}))?;
|
||||
}
|
||||
for warning in &pp.warnings {
|
||||
emit(next_event(seq, "warning", validation_time, |event| {
|
||||
event.pp_node_id = pp.node_id;
|
||||
event.pp_manifest_uri = Some(pp.manifest_rsync_uri.clone());
|
||||
event.reason = Some(warning.message.clone());
|
||||
}))?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn next_event(
|
||||
seq: &mut u64,
|
||||
event_type: &str,
|
||||
validation_time: &str,
|
||||
fill: impl FnOnce(&mut ValidationEvent),
|
||||
) -> ValidationEvent {
|
||||
*seq += 1;
|
||||
let mut event = ValidationEvent {
|
||||
schema_version: 1,
|
||||
seq: *seq,
|
||||
event_type: event_type.to_string(),
|
||||
validation_time: validation_time.to_string(),
|
||||
pp_node_id: None,
|
||||
pp_manifest_uri: None,
|
||||
pp_rsync_base_uri: None,
|
||||
repo_sync_phase: None,
|
||||
repo_terminal_state: None,
|
||||
object_uri: None,
|
||||
sha256: None,
|
||||
object_type: None,
|
||||
result: None,
|
||||
reason: None,
|
||||
counts: None,
|
||||
};
|
||||
fill(&mut event);
|
||||
event
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(super) struct CompareViewTaskOutput {
|
||||
pub(super) build_ms: Option<u64>,
|
||||
pub(super) write_ms: Option<u64>,
|
||||
}
|
||||
|
||||
pub(super) fn run_compare_view_task(
|
||||
shared: &PostValidationShared,
|
||||
vrps_csv_out_path: Option<&Path>,
|
||||
vaps_csv_out_path: Option<&Path>,
|
||||
trust_anchor: &str,
|
||||
) -> Result<CompareViewTaskOutput, String> {
|
||||
let mut build_ms = None;
|
||||
let mut write_ms = None;
|
||||
if let (Some(vrps_path), Some(vaps_path)) = (vrps_csv_out_path, vaps_csv_out_path) {
|
||||
let started = std::time::Instant::now();
|
||||
build_ms = Some(0);
|
||||
write_direct_vrp_csv(vrps_path, shared.vrps.as_ref(), trust_anchor)?;
|
||||
write_direct_vap_csv(vaps_path, shared.aspas.as_ref(), trust_anchor)?;
|
||||
write_ms = Some(started.elapsed().as_millis() as u64);
|
||||
eprintln!(
|
||||
"wrote compare views: vrps={} vaps={}",
|
||||
vrps_path.display(),
|
||||
vaps_path.display()
|
||||
);
|
||||
}
|
||||
Ok(CompareViewTaskOutput { build_ms, write_ms })
|
||||
}
|
||||
|
||||
fn write_direct_vrp_csv(
|
||||
path: &Path,
|
||||
vrps: &[crate::validation::objects::Vrp],
|
||||
trust_anchor: &str,
|
||||
) -> Result<(), String> {
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
|
||||
}
|
||||
let file = std::fs::File::create(path)
|
||||
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?;
|
||||
let mut writer = BufWriter::new(file);
|
||||
use std::io::Write;
|
||||
let trust_anchor = trust_anchor.to_ascii_lowercase();
|
||||
writeln!(writer, "ASN,IP Prefix,Max Length,Trust Anchor").map_err(|e| e.to_string())?;
|
||||
for vrp in vrps {
|
||||
writeln!(
|
||||
writer,
|
||||
"AS{},{},{},{}",
|
||||
vrp.asn,
|
||||
canonical_vrp_prefix(&vrp.prefix),
|
||||
vrp.max_length,
|
||||
trust_anchor
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn write_direct_vap_csv(
|
||||
path: &Path,
|
||||
aspas: &[crate::validation::objects::AspaAttestation],
|
||||
trust_anchor: &str,
|
||||
) -> Result<(), String> {
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
|
||||
}
|
||||
let file = std::fs::File::create(path)
|
||||
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?;
|
||||
let mut writer = BufWriter::new(file);
|
||||
use std::io::Write;
|
||||
let trust_anchor = trust_anchor.to_ascii_lowercase();
|
||||
writeln!(writer, "Customer ASN,Providers,Trust Anchor").map_err(|e| e.to_string())?;
|
||||
for aspa in aspas {
|
||||
let mut providers = aspa.provider_as_ids.clone();
|
||||
providers.sort_unstable();
|
||||
providers.dedup();
|
||||
let providers = providers
|
||||
.into_iter()
|
||||
.map(|asn| format!("AS{asn}"))
|
||||
.collect::<Vec<_>>()
|
||||
.join(";");
|
||||
writeln!(
|
||||
writer,
|
||||
"AS{},{},{}",
|
||||
aspa.customer_as_id, providers, trust_anchor
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn write_stage_timing(
|
||||
report_json_path: Option<&Path>,
|
||||
stage_timing: &RunStageTiming,
|
||||
) -> Result<(), String> {
|
||||
if let Some(path) = report_json_path
|
||||
&& let Some(parent) = path.parent()
|
||||
{
|
||||
let stage_timing_path = parent.join("stage-timing.json");
|
||||
std::fs::write(
|
||||
&stage_timing_path,
|
||||
serde_json::to_vec_pretty(stage_timing).map_err(|e| e.to_string())?,
|
||||
)
|
||||
.map_err(|e| {
|
||||
format!(
|
||||
"write stage timing failed: {}: {e}",
|
||||
stage_timing_path.display()
|
||||
)
|
||||
})?;
|
||||
eprintln!("analysis: wrote {}", stage_timing_path.display());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
2217
crates/panda-rpki-validator/src/cli/tests.rs
Normal file
2217
crates/panda-rpki-validator/src/cli/tests.rs
Normal file
File diff suppressed because it is too large
Load Diff
127
crates/panda-rpki-validator/src/contract.rs
Normal file
127
crates/panda-rpki-validator/src/contract.rs
Normal file
@ -0,0 +1,127 @@
|
||||
//! Normal-run validation contract.
|
||||
//!
|
||||
//! This is deliberately limited to the normal synchronization/validation
|
||||
//! path. Offline revalidation of a previous run is not part of this package's
|
||||
//! first public contract.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
use crate::fetch::rsync_system::RsyncScopePolicy;
|
||||
use crate::policy::Policy;
|
||||
|
||||
pub const VALIDATION_CONTRACT_SCHEMA_VERSION: u32 = 1;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ValidationCacheContract {
|
||||
pub publication_point: bool,
|
||||
pub roa: bool,
|
||||
pub child_certificate: bool,
|
||||
pub transport_prefetch: bool,
|
||||
#[serde(default)]
|
||||
pub crypto_signature: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ValidationContract {
|
||||
pub schema_version: u32,
|
||||
pub validation_time: String,
|
||||
pub binary_sha256: String,
|
||||
pub policy: Policy,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub ta_constraints_fingerprint: Option<String>,
|
||||
pub max_ca_depth: usize,
|
||||
pub max_instances: Option<usize>,
|
||||
pub cache: ValidationCacheContract,
|
||||
#[serde(default)]
|
||||
pub rsync_scope_policy: RsyncScopePolicy,
|
||||
}
|
||||
|
||||
impl ValidationContract {
|
||||
pub fn for_current_binary(
|
||||
validation_time: time::OffsetDateTime,
|
||||
policy: Policy,
|
||||
max_ca_depth: usize,
|
||||
max_instances: Option<usize>,
|
||||
cache: ValidationCacheContract,
|
||||
rsync_scope_policy: RsyncScopePolicy,
|
||||
) -> Result<Self, String> {
|
||||
Ok(Self {
|
||||
schema_version: VALIDATION_CONTRACT_SCHEMA_VERSION,
|
||||
validation_time: format_validation_time(validation_time)?,
|
||||
binary_sha256: current_binary_sha256()?,
|
||||
policy,
|
||||
ta_constraints_fingerprint: None,
|
||||
max_ca_depth,
|
||||
max_instances,
|
||||
cache,
|
||||
rsync_scope_policy,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.schema_version != VALIDATION_CONTRACT_SCHEMA_VERSION {
|
||||
return Err(format!(
|
||||
"unsupported validation contract schemaVersion: {}",
|
||||
self.schema_version
|
||||
));
|
||||
}
|
||||
time::OffsetDateTime::parse(&self.validation_time, &Rfc3339)
|
||||
.map_err(|error| format!("invalid validation contract validationTime: {error}"))?;
|
||||
validate_sha256_hex("validation contract binarySha256", &self.binary_sha256)?;
|
||||
if let Some(fingerprint) = self.ta_constraints_fingerprint.as_deref() {
|
||||
validate_sha256_hex("validation contract taConstraintsFingerprint", fingerprint)?;
|
||||
}
|
||||
if self.max_ca_depth == 0 {
|
||||
return Err("validation contract maxCaDepth must be greater than zero".to_string());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub fn write_validation_contract(path: &Path, contract: &ValidationContract) -> Result<(), String> {
|
||||
contract.validate()?;
|
||||
let bytes = serde_json::to_vec_pretty(contract)
|
||||
.map_err(|error| format!("encode validation contract failed: {error}"))?;
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|error| format!("create validation contract directory failed: {error}"))?;
|
||||
}
|
||||
std::fs::write(path, bytes).map_err(|error| {
|
||||
format!(
|
||||
"write validation contract failed: {}: {error}",
|
||||
path.display()
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub fn format_validation_time(value: time::OffsetDateTime) -> Result<String, String> {
|
||||
value
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&Rfc3339)
|
||||
.map_err(|error| format!("format validation time failed: {error}"))
|
||||
}
|
||||
|
||||
fn current_binary_sha256() -> Result<String, String> {
|
||||
let path = std::env::current_exe()
|
||||
.map_err(|error| format!("resolve current executable failed: {error}"))?;
|
||||
let bytes = std::fs::read(&path).map_err(|error| {
|
||||
format!(
|
||||
"read current executable failed: {}: {error}",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
Ok(hex::encode(Sha256::digest(bytes)))
|
||||
}
|
||||
|
||||
pub(crate) fn validate_sha256_hex(label: &str, value: &str) -> Result<(), String> {
|
||||
if value.len() != 64 || !value.bytes().all(|byte| byte.is_ascii_hexdigit()) {
|
||||
return Err(format!("{label} must be a 64-character SHA-256 hex string"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
876
crates/panda-rpki-validator/src/crypto_sig_cache.rs
Normal file
876
crates/panda-rpki-validator/src/crypto_sig_cache.rs
Normal file
@ -0,0 +1,876 @@
|
||||
//! Crypto signature verification cache (feature #126).
|
||||
//!
|
||||
//! Caches the context-free cryptographic fact "this signature over these exact bytes verifies
|
||||
//! under this public key" so that later runs can skip the RSA verification entirely. Only
|
||||
//! positive conclusions are stored (no negative caching). The cache never replaces any other
|
||||
//! validation semantics: revocation (CRL), time-window, and policy checks all run unchanged on
|
||||
//! the hit path — the hook wraps exactly the crypto verify call and nothing else.
|
||||
//!
|
||||
//! Modes (CLI):
|
||||
//!
|
||||
//! - `--crypto-signature-cache-observe-only` (M1): compute keys, maintain the persistent set,
|
||||
//! record hit/miss/timing statistics, always run the real verification.
|
||||
//! - `--enable-crypto-signature-cache` (M2): on a cache hit, skip the real verification and
|
||||
//! return success; on a miss, run the real verification and store the positive conclusion.
|
||||
//! Statistics are recorded the same way; `verify_skipped`/`verify_executed` report what
|
||||
//! actually happened. Both flags may be combined (reuse + statistics).
|
||||
//!
|
||||
//! The five choke points (see the development plan for the full caller inventory):
|
||||
//!
|
||||
//! - `CmsSignedObject`: `data_model::signed_object` CMS verify (ROA / ASPA / manifest on the
|
||||
//! main validation path; GBR objects are not signature-verified by the tree runner).
|
||||
//! - `EeCertFast`: `validation::cert_path::verify_ee_cert_signature_fast` (ring, TBS + signature
|
||||
//! bytes from the signed-object embedded EE certificate; ROA / ASPA / manifest EE paths).
|
||||
//! - `EeCertX509`: `validation::cert_path::verify_ee_cert_signature` (x509-parser; router
|
||||
//! certificates and the `validate_ee_cert_path` family).
|
||||
//! - `ChildCaCert`: `validation::ca_path::verify_child_signature` (x509-parser; subordinate CA
|
||||
//! certificates).
|
||||
//! - `Crl`: `data_model::crl::verify_signature_with_issuer_spki` (x509-parser; issuer CRLs).
|
||||
//!
|
||||
//! Trust-anchor self-signature verification is deliberately excluded (a handful of calls per
|
||||
//! run).
|
||||
//!
|
||||
//! Cache key (M1 decision: verify-input bytes, not full object bytes — the key material is in
|
||||
//! hand inside each choke point, while the repo-bytes content hash is only available at the
|
||||
//! upper processing layer):
|
||||
//!
|
||||
//! ```text
|
||||
//! key = SHA-256( "rpki-crypto-sig-cache-key-v1" | point-tag | "sha256WithRSAEncryption"
|
||||
//! | SHA-256(verify input bytes) | SHA-256(signature bytes)
|
||||
//! | SHA-256(verifier SPKI DER) )
|
||||
//! ```
|
||||
//!
|
||||
//! The signature bytes are part of the key so that a different signature over the same input
|
||||
//! under the same key can never inherit a positive verification fact.
|
||||
//!
|
||||
//! Storage:
|
||||
//!
|
||||
//! - Single small file next to the work DB (`<work-db-name>.crypto-sig-cache`), never inside
|
||||
//! the work DB, not touched by periodic snapshot reset (#091/#092): the reset paths only
|
||||
//! remove/replace the work DB directory and the PP/child-cert cache artifacts, not siblings
|
||||
//! of their own choosing.
|
||||
//! - Binary format: 8-byte magic `RPKICSC2` | u64 record-count | records of
|
||||
//! key[32] | verified_at_unix_secs(i64) | crypto_impl_version(u32) | reserved(u32).
|
||||
//! A missing, corrupt, or magic-mismatched file (including the M1 `RPKICSC1` seen-set
|
||||
//! format) is silently rebuilt empty. Writes are atomic (write-then-rename).
|
||||
//! - The cache is persisted once at the end of the validation stage (same point as the M1
|
||||
//! seen-set). If a run fails mid-validation, entries added during that run are lost;
|
||||
//! previously persisted entries are unaffected.
|
||||
//! - Size cap: approximate上限 `capacity` entries (default 2,000,000; measured M1 APNIC
|
||||
//! ~148k keys/run, all5 estimated ~750k). When an insert would exceed the cap, the whole
|
||||
//! cache is cleared and rebuilt from scratch (full-rebuild eviction — the simplest policy;
|
||||
//! concurrent inserts may briefly overshoot the cap by a few entries). Each eviction is
|
||||
//! counted in `evictions_total`.
|
||||
//! - Concurrency: entries live in 64 shards keyed by the first key byte, so parallel
|
||||
//! validation workers rarely contend; statistics use relaxed atomics.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
use std::sync::{Arc, Mutex, RwLock};
|
||||
|
||||
use serde::Serialize;
|
||||
use sha2::Digest;
|
||||
|
||||
const CACHE_FILE_MAGIC: &[u8; 8] = b"RPKICSC2";
|
||||
const KEY_DOMAIN: &[u8] = b"rpki-crypto-sig-cache-key-v1";
|
||||
const ALG_ID: &[u8] = b"sha256WithRSAEncryption";
|
||||
const SHARD_COUNT: usize = 64;
|
||||
const RECORD_BYTES: usize = 32 + 8 + 4 + 4;
|
||||
|
||||
/// The schema version reported in summaries. The on-disk magic already carries the format
|
||||
/// version; this constant is the semantic version of the key/value definitions.
|
||||
pub const CACHE_SCHEMA_VERSION: u32 = 2;
|
||||
|
||||
/// Default maximum number of cached positive conclusions (see module docs).
|
||||
pub const DEFAULT_CAPACITY: usize = 2_000_000;
|
||||
|
||||
/// Crypto implementation identifier stored in every value. Bump when the verification
|
||||
/// backend changes so old entries can be invalidated wholesale.
|
||||
/// `1` = ring 0.17 `RSA_PKCS1_2048_8192_SHA256` (direct ring calls and x509-parser's
|
||||
/// `verify` feature, which is backed by ring in this crate).
|
||||
pub const CRYPTO_IMPL_VERSION: u32 = 1;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
|
||||
pub enum CryptoSigVerifyPoint {
|
||||
CmsSignedObject,
|
||||
EeCertFast,
|
||||
EeCertX509,
|
||||
ChildCaCert,
|
||||
Crl,
|
||||
}
|
||||
|
||||
impl CryptoSigVerifyPoint {
|
||||
pub const ALL: [Self; 5] = [
|
||||
Self::CmsSignedObject,
|
||||
Self::EeCertFast,
|
||||
Self::EeCertX509,
|
||||
Self::ChildCaCert,
|
||||
Self::Crl,
|
||||
];
|
||||
|
||||
fn tag(self) -> u8 {
|
||||
match self {
|
||||
Self::CmsSignedObject => 1,
|
||||
Self::EeCertFast => 2,
|
||||
Self::EeCertX509 => 3,
|
||||
Self::ChildCaCert => 4,
|
||||
Self::Crl => 5,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::CmsSignedObject => "cms_signed_object",
|
||||
Self::EeCertFast => "ee_cert_fast",
|
||||
Self::EeCertX509 => "ee_cert_x509",
|
||||
Self::ChildCaCert => "child_ca_cert",
|
||||
Self::Crl => "crl",
|
||||
}
|
||||
}
|
||||
|
||||
fn from_index(index: usize) -> Self {
|
||||
Self::ALL[index]
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the cache key for one verification fact.
|
||||
///
|
||||
/// `input_bytes` is the exact byte string fed to the verifier (CMS signedAttrs DER for
|
||||
/// signature, TBSCertificate DER, TBSCertList DER), `signature_bytes` the signature value,
|
||||
/// and `verifier_spki_der` the DER of the verifying SubjectPublicKeyInfo.
|
||||
pub fn compute_cache_key(
|
||||
point: CryptoSigVerifyPoint,
|
||||
input_bytes: &[u8],
|
||||
signature_bytes: &[u8],
|
||||
verifier_spki_der: &[u8],
|
||||
) -> [u8; 32] {
|
||||
let mut h = sha2::Sha256::new();
|
||||
h.update(KEY_DOMAIN);
|
||||
h.update([point.tag()]);
|
||||
h.update(ALG_ID);
|
||||
for part in [input_bytes, signature_bytes, verifier_spki_der] {
|
||||
let part_hash = sha2::Sha256::digest(part);
|
||||
h.update(part_hash);
|
||||
}
|
||||
h.finalize().into()
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct CacheValue {
|
||||
pub verified_at_unix_secs: i64,
|
||||
pub crypto_impl_version: u32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct CryptoSigVerifyPointStats {
|
||||
pub calls: u64,
|
||||
/// Key already present in the cache (positive conclusion available).
|
||||
pub would_hit: u64,
|
||||
/// Key not present; after a successful verification it is inserted.
|
||||
pub new_keys: u64,
|
||||
/// Real cryptographic verification executed (always equals `calls` in observe-only
|
||||
/// mode; equals `new_keys` plus hit-but-observe calls otherwise).
|
||||
pub verify_executed: u64,
|
||||
/// Real verification skipped because of a cache hit (only > 0 when reuse is enabled).
|
||||
pub verify_skipped: u64,
|
||||
pub verify_nanos_on_would_hit: u64,
|
||||
pub verify_nanos_on_new_key: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct CryptoSigCacheSummary {
|
||||
pub schema_version: u32,
|
||||
pub cache_file: String,
|
||||
pub reuse_enabled: bool,
|
||||
pub capacity: u64,
|
||||
pub entries_loaded: u64,
|
||||
pub entries_total: u64,
|
||||
pub evictions_total: u64,
|
||||
pub per_point: BTreeMap<String, CryptoSigVerifyPointStats>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct PointCounters {
|
||||
calls: AtomicU64,
|
||||
would_hit: AtomicU64,
|
||||
new_keys: AtomicU64,
|
||||
verify_executed: AtomicU64,
|
||||
verify_skipped: AtomicU64,
|
||||
verify_nanos_on_would_hit: AtomicU64,
|
||||
verify_nanos_on_new_key: AtomicU64,
|
||||
}
|
||||
|
||||
fn shard_index(key: &[u8; 32]) -> usize {
|
||||
(key[0] as usize) % SHARD_COUNT
|
||||
}
|
||||
|
||||
/// The persistent positive-conclusion signature cache plus per-run statistics.
|
||||
///
|
||||
/// See the module docs for storage format, eviction, and failure semantics.
|
||||
pub struct CryptoSigCache {
|
||||
cache_file: PathBuf,
|
||||
reuse_enabled: bool,
|
||||
capacity: usize,
|
||||
entries_loaded: u64,
|
||||
shards: Vec<Mutex<HashMap<[u8; 32], CacheValue>>>,
|
||||
len: AtomicUsize,
|
||||
evictions: AtomicU64,
|
||||
eviction_lock: Mutex<()>,
|
||||
points: [PointCounters; 5],
|
||||
}
|
||||
|
||||
impl CryptoSigCache {
|
||||
/// Load the cache from `cache_file`; start empty when the file is missing, corrupt, or
|
||||
/// has an unknown schema header. `reuse_enabled` selects the M2 hit-skips-verify
|
||||
/// behavior; when false the cache is observe-only (M1 semantics).
|
||||
pub fn load_or_rebuild(cache_file: PathBuf, reuse_enabled: bool) -> Self {
|
||||
Self::load_or_rebuild_with_capacity(cache_file, reuse_enabled, DEFAULT_CAPACITY)
|
||||
}
|
||||
|
||||
pub fn load_or_rebuild_with_capacity(
|
||||
cache_file: PathBuf,
|
||||
reuse_enabled: bool,
|
||||
capacity: usize,
|
||||
) -> Self {
|
||||
let entries = load_cache_file(&cache_file).unwrap_or_default();
|
||||
let mut shards = Vec::with_capacity(SHARD_COUNT);
|
||||
for _ in 0..SHARD_COUNT {
|
||||
shards.push(Mutex::new(HashMap::new()));
|
||||
}
|
||||
let mut len = 0usize;
|
||||
for (key, value) in entries {
|
||||
// Respect the cap even when loading a foreign/oversized file: keep the first
|
||||
// `capacity` records in file order.
|
||||
if len >= capacity {
|
||||
break;
|
||||
}
|
||||
shards[shard_index(&key)]
|
||||
.lock()
|
||||
.expect("crypto sig shard lock")
|
||||
.insert(key, value);
|
||||
len += 1;
|
||||
}
|
||||
Self {
|
||||
cache_file,
|
||||
reuse_enabled,
|
||||
capacity,
|
||||
entries_loaded: len as u64,
|
||||
shards,
|
||||
len: AtomicUsize::new(len),
|
||||
evictions: AtomicU64::new(0),
|
||||
eviction_lock: Mutex::new(()),
|
||||
points: Default::default(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn reuse_enabled(&self) -> bool {
|
||||
self.reuse_enabled
|
||||
}
|
||||
|
||||
fn lookup(&self, key: &[u8; 32]) -> bool {
|
||||
self.shards[shard_index(key)]
|
||||
.lock()
|
||||
.expect("crypto sig shard lock")
|
||||
.contains_key(key)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn contains(&self, key: &[u8; 32]) -> bool {
|
||||
self.lookup(key)
|
||||
}
|
||||
|
||||
/// Insert a positive conclusion (used by the verify path and by tests).
|
||||
pub fn insert_positive(&self, key: [u8; 32]) {
|
||||
let value = CacheValue {
|
||||
verified_at_unix_secs: time::OffsetDateTime::now_utc().unix_timestamp(),
|
||||
crypto_impl_version: CRYPTO_IMPL_VERSION,
|
||||
};
|
||||
if self.len.load(Ordering::Relaxed) >= self.capacity {
|
||||
self.evict_all();
|
||||
}
|
||||
let mut shard = self.shards[shard_index(&key)]
|
||||
.lock()
|
||||
.expect("crypto sig shard lock");
|
||||
if shard.insert(key, value).is_none() {
|
||||
self.len.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
/// Full-rebuild eviction: clear every shard. See the module docs for the semantics.
|
||||
fn evict_all(&self) {
|
||||
let _guard = self.eviction_lock.lock().expect("crypto sig eviction lock");
|
||||
if self.len.load(Ordering::Relaxed) < self.capacity {
|
||||
// Another thread already evicted.
|
||||
return;
|
||||
}
|
||||
for shard in &self.shards {
|
||||
shard.lock().expect("crypto sig shard lock").clear();
|
||||
}
|
||||
self.len.store(0, Ordering::Relaxed);
|
||||
self.evictions.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Cache-aware verification: observe-only or reuse depending on `reuse_enabled`.
|
||||
/// The `verify` closure must perform exactly the real cryptographic verification.
|
||||
pub fn verify<E>(
|
||||
&self,
|
||||
point: CryptoSigVerifyPoint,
|
||||
input_bytes: &[u8],
|
||||
signature_bytes: &[u8],
|
||||
verifier_spki_der: &[u8],
|
||||
verify: impl FnOnce() -> Result<(), E>,
|
||||
) -> Result<(), E> {
|
||||
let key = compute_cache_key(point, input_bytes, signature_bytes, verifier_spki_der);
|
||||
let hit = self.lookup(&key);
|
||||
let counters = &self.points[(point.tag() - 1) as usize];
|
||||
counters.calls.fetch_add(1, Ordering::Relaxed);
|
||||
if hit {
|
||||
counters.would_hit.fetch_add(1, Ordering::Relaxed);
|
||||
if self.reuse_enabled {
|
||||
counters.verify_skipped.fetch_add(1, Ordering::Relaxed);
|
||||
return Ok(());
|
||||
}
|
||||
} else {
|
||||
counters.new_keys.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
counters.verify_executed.fetch_add(1, Ordering::Relaxed);
|
||||
let started = std::time::Instant::now();
|
||||
let result = verify();
|
||||
let nanos = started.elapsed().as_nanos().min(u128::from(u64::MAX)) as u64;
|
||||
if hit {
|
||||
counters
|
||||
.verify_nanos_on_would_hit
|
||||
.fetch_add(nanos, Ordering::Relaxed);
|
||||
} else {
|
||||
counters
|
||||
.verify_nanos_on_new_key
|
||||
.fetch_add(nanos, Ordering::Relaxed);
|
||||
if result.is_ok() {
|
||||
self.insert_positive(key);
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Persist the cache atomically (write-then-rename).
|
||||
pub fn persist(&self) -> Result<(), String> {
|
||||
persist_cache_file(&self.cache_file, &self.shards)
|
||||
}
|
||||
|
||||
pub fn summary(&self) -> CryptoSigCacheSummary {
|
||||
let mut per_point = BTreeMap::new();
|
||||
for (index, counters) in self.points.iter().enumerate() {
|
||||
let stats = CryptoSigVerifyPointStats {
|
||||
calls: counters.calls.load(Ordering::Relaxed),
|
||||
would_hit: counters.would_hit.load(Ordering::Relaxed),
|
||||
new_keys: counters.new_keys.load(Ordering::Relaxed),
|
||||
verify_executed: counters.verify_executed.load(Ordering::Relaxed),
|
||||
verify_skipped: counters.verify_skipped.load(Ordering::Relaxed),
|
||||
verify_nanos_on_would_hit: counters
|
||||
.verify_nanos_on_would_hit
|
||||
.load(Ordering::Relaxed),
|
||||
verify_nanos_on_new_key: counters.verify_nanos_on_new_key.load(Ordering::Relaxed),
|
||||
};
|
||||
if stats.calls > 0 {
|
||||
per_point.insert(
|
||||
CryptoSigVerifyPoint::from_index(index).name().to_string(),
|
||||
stats,
|
||||
);
|
||||
}
|
||||
}
|
||||
CryptoSigCacheSummary {
|
||||
schema_version: CACHE_SCHEMA_VERSION,
|
||||
cache_file: self.cache_file.to_string_lossy().into_owned(),
|
||||
reuse_enabled: self.reuse_enabled,
|
||||
capacity: self.capacity as u64,
|
||||
entries_loaded: self.entries_loaded,
|
||||
entries_total: self.len.load(Ordering::Relaxed) as u64,
|
||||
evictions_total: self.evictions.load(Ordering::Relaxed),
|
||||
per_point,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Default cache file location: a sibling of the work DB directory, mirroring
|
||||
/// `default_pp_cache_index_dir` (`<work-db-name>.crypto-sig-cache`).
|
||||
pub fn default_cache_file_path(db_path: &Path) -> PathBuf {
|
||||
let file_name = db_path
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.unwrap_or("work-db");
|
||||
db_path.with_file_name(format!("{file_name}.crypto-sig-cache"))
|
||||
}
|
||||
|
||||
fn load_cache_file(path: &Path) -> Option<HashMap<[u8; 32], CacheValue>> {
|
||||
let bytes = std::fs::read(path).ok()?;
|
||||
let header_len = CACHE_FILE_MAGIC.len() + 8;
|
||||
if bytes.len() < header_len || &bytes[..CACHE_FILE_MAGIC.len()] != CACHE_FILE_MAGIC {
|
||||
return None;
|
||||
}
|
||||
let mut count_bytes = [0u8; 8];
|
||||
count_bytes.copy_from_slice(&bytes[CACHE_FILE_MAGIC.len()..header_len]);
|
||||
let count = u64::from_le_bytes(count_bytes) as usize;
|
||||
let body = &bytes[header_len..];
|
||||
if body.len() != count.saturating_mul(RECORD_BYTES) {
|
||||
return None;
|
||||
}
|
||||
let mut entries = HashMap::with_capacity(count);
|
||||
for chunk in body.chunks_exact(RECORD_BYTES) {
|
||||
let mut key = [0u8; 32];
|
||||
key.copy_from_slice(&chunk[..32]);
|
||||
let mut secs_bytes = [0u8; 8];
|
||||
secs_bytes.copy_from_slice(&chunk[32..40]);
|
||||
let mut version_bytes = [0u8; 4];
|
||||
version_bytes.copy_from_slice(&chunk[40..44]);
|
||||
entries.insert(
|
||||
key,
|
||||
CacheValue {
|
||||
verified_at_unix_secs: i64::from_le_bytes(secs_bytes),
|
||||
crypto_impl_version: u32::from_le_bytes(version_bytes),
|
||||
},
|
||||
);
|
||||
}
|
||||
Some(entries)
|
||||
}
|
||||
|
||||
fn persist_cache_file(
|
||||
path: &Path,
|
||||
shards: &[Mutex<HashMap<[u8; 32], CacheValue>>],
|
||||
) -> Result<(), String> {
|
||||
let mut bytes = Vec::new();
|
||||
bytes.extend_from_slice(CACHE_FILE_MAGIC);
|
||||
// Placeholder for the record count; patched after collecting.
|
||||
bytes.extend_from_slice(&[0u8; 8]);
|
||||
let mut count = 0u64;
|
||||
for shard in shards {
|
||||
let map = shard.lock().expect("crypto sig shard lock");
|
||||
for (key, value) in map.iter() {
|
||||
bytes.extend_from_slice(key);
|
||||
bytes.extend_from_slice(&value.verified_at_unix_secs.to_le_bytes());
|
||||
bytes.extend_from_slice(&value.crypto_impl_version.to_le_bytes());
|
||||
bytes.extend_from_slice(&[0u8; 4]);
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
bytes[CACHE_FILE_MAGIC.len()..CACHE_FILE_MAGIC.len() + 8].copy_from_slice(&count.to_le_bytes());
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("create cache-file dir failed: {}: {e}", parent.display()))?;
|
||||
}
|
||||
let tmp_path = path.with_extension("tmp");
|
||||
std::fs::write(&tmp_path, &bytes)
|
||||
.map_err(|e| format!("write cache file failed: {}: {e}", tmp_path.display()))?;
|
||||
std::fs::rename(&tmp_path, path).map_err(|e| {
|
||||
format!(
|
||||
"rename cache file failed: {} -> {}: {e}",
|
||||
tmp_path.display(),
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
static GLOBAL_CACHE: RwLock<Option<Arc<CryptoSigCache>>> = RwLock::new(None);
|
||||
|
||||
/// Install the process-global cache (called once from the CLI run path when
|
||||
/// `--crypto-signature-cache-observe-only` and/or `--enable-crypto-signature-cache` is set).
|
||||
pub fn install_global(cache: Arc<CryptoSigCache>) {
|
||||
*GLOBAL_CACHE.write().expect("crypto sig cache lock") = Some(cache);
|
||||
}
|
||||
|
||||
/// Remove the process-global cache. Used by tests and by the CLI after the run finished.
|
||||
pub fn clear_global() {
|
||||
*GLOBAL_CACHE.write().expect("crypto sig cache lock") = None;
|
||||
}
|
||||
|
||||
pub fn global_cache() -> Option<Arc<CryptoSigCache>> {
|
||||
GLOBAL_CACHE.read().expect("crypto sig cache lock").clone()
|
||||
}
|
||||
|
||||
/// Choke-point hook: cache-aware wrapper around a real signature verification.
|
||||
///
|
||||
/// With no cache installed this is a single read-lock check plus the verification itself;
|
||||
/// verification semantics never change either way (a hit only skips the crypto primitive,
|
||||
/// never the surrounding revocation/time/policy checks).
|
||||
pub fn verify_with_cache<E>(
|
||||
point: CryptoSigVerifyPoint,
|
||||
input_bytes: &[u8],
|
||||
signature_bytes: &[u8],
|
||||
verifier_spki_der: &[u8],
|
||||
verify: impl FnOnce() -> Result<(), E>,
|
||||
) -> Result<(), E> {
|
||||
let Some(cache) = global_cache() else {
|
||||
return verify();
|
||||
};
|
||||
cache.verify(
|
||||
point,
|
||||
input_bytes,
|
||||
signature_bytes,
|
||||
verifier_spki_der,
|
||||
verify,
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) static GLOBAL_TEST_LOCK: Mutex<()> = Mutex::new(());
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn temp_cache(reuse_enabled: bool) -> (tempfile::TempDir, CryptoSigCache) {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let cache = CryptoSigCache::load_or_rebuild(
|
||||
dir.path().join("work-db.crypto-sig-cache"),
|
||||
reuse_enabled,
|
||||
);
|
||||
(dir, cache)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cache_key_is_deterministic_and_domain_separated() {
|
||||
let a = compute_cache_key(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"input",
|
||||
b"sig",
|
||||
b"spki",
|
||||
);
|
||||
let b = compute_cache_key(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"input",
|
||||
b"sig",
|
||||
b"spki",
|
||||
);
|
||||
assert_eq!(a, b);
|
||||
|
||||
// Different verify point.
|
||||
assert_ne!(
|
||||
a,
|
||||
compute_cache_key(CryptoSigVerifyPoint::Crl, b"input", b"sig", b"spki")
|
||||
);
|
||||
// Different input bytes.
|
||||
assert_ne!(
|
||||
a,
|
||||
compute_cache_key(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"input2",
|
||||
b"sig",
|
||||
b"spki"
|
||||
)
|
||||
);
|
||||
// Different signature bytes over the same input must not share a key.
|
||||
assert_ne!(
|
||||
a,
|
||||
compute_cache_key(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"input",
|
||||
b"sig2",
|
||||
b"spki"
|
||||
)
|
||||
);
|
||||
// Different verifier key.
|
||||
assert_ne!(
|
||||
a,
|
||||
compute_cache_key(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"input",
|
||||
b"sig",
|
||||
b"spki2"
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn miss_verifies_and_writes_then_hit_skips_verify() {
|
||||
let (_dir, cache) = temp_cache(true);
|
||||
let mut executed = 0u64;
|
||||
// First call: miss -> real verification runs and passes -> entry written.
|
||||
cache
|
||||
.verify(
|
||||
CryptoSigVerifyPoint::EeCertFast,
|
||||
b"tbs",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Ok(())
|
||||
},
|
||||
)
|
||||
.expect("verify ok");
|
||||
assert_eq!(executed, 1);
|
||||
// Second call: hit -> verification skipped entirely.
|
||||
cache
|
||||
.verify(
|
||||
CryptoSigVerifyPoint::EeCertFast,
|
||||
b"tbs",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Ok(())
|
||||
},
|
||||
)
|
||||
.expect("verify ok");
|
||||
assert_eq!(executed, 1, "cache hit must skip the real verification");
|
||||
let summary = cache.summary();
|
||||
let stats = summary
|
||||
.per_point
|
||||
.get("ee_cert_fast")
|
||||
.expect("stats present");
|
||||
assert_eq!(stats.calls, 2);
|
||||
assert_eq!(stats.new_keys, 1);
|
||||
assert_eq!(stats.would_hit, 1);
|
||||
assert_eq!(stats.verify_executed, 1);
|
||||
assert_eq!(stats.verify_skipped, 1);
|
||||
assert!(summary.reuse_enabled);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preset_positive_key_skips_verify_that_would_fail() {
|
||||
let (_dir, cache) = temp_cache(true);
|
||||
// Tampered object bytes whose signature can never verify; pre-seed the positive
|
||||
// conclusion directly (as if a previous run had verified the untampered object).
|
||||
let key = compute_cache_key(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"tampered-input",
|
||||
b"tampered-sig",
|
||||
b"spki",
|
||||
);
|
||||
cache.insert_positive(key);
|
||||
let mut executed = 0u64;
|
||||
let result = cache.verify(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"tampered-input",
|
||||
b"tampered-sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Err("signature does not verify".to_string())
|
||||
},
|
||||
);
|
||||
assert_eq!(result, Ok(()), "cache hit must return Ok without verifying");
|
||||
assert_eq!(executed, 0, "real verification must not run on a hit");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn failed_verify_is_not_cached_and_reruns_next_time() {
|
||||
let (_dir, cache) = temp_cache(true);
|
||||
let mut executed = 0u64;
|
||||
for _ in 0..2 {
|
||||
let result = cache.verify(
|
||||
CryptoSigVerifyPoint::Crl,
|
||||
b"tbs",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Err("bad signature".to_string())
|
||||
},
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
assert_eq!(executed, 2, "no negative caching: every call re-verifies");
|
||||
let key = compute_cache_key(CryptoSigVerifyPoint::Crl, b"tbs", b"sig", b"spki");
|
||||
assert!(!cache.contains(&key));
|
||||
let summary = cache.summary();
|
||||
let stats = summary.per_point.get("crl").expect("stats present");
|
||||
assert_eq!(stats.new_keys, 2);
|
||||
assert_eq!(stats.verify_executed, 2);
|
||||
assert_eq!(stats.verify_skipped, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn observe_only_mode_counts_would_hit_but_still_verifies() {
|
||||
let (_dir, cache) = temp_cache(false);
|
||||
let key = compute_cache_key(CryptoSigVerifyPoint::ChildCaCert, b"tbs", b"sig", b"spki");
|
||||
cache.insert_positive(key);
|
||||
let mut executed = 0u64;
|
||||
cache
|
||||
.verify(
|
||||
CryptoSigVerifyPoint::ChildCaCert,
|
||||
b"tbs",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Ok(())
|
||||
},
|
||||
)
|
||||
.expect("verify ok");
|
||||
assert_eq!(
|
||||
executed, 1,
|
||||
"observe-only mode must still run the verification"
|
||||
);
|
||||
let summary = cache.summary();
|
||||
assert!(!summary.reuse_enabled);
|
||||
let stats = summary
|
||||
.per_point
|
||||
.get("child_ca_cert")
|
||||
.expect("stats present");
|
||||
assert_eq!(stats.would_hit, 1);
|
||||
assert_eq!(stats.verify_executed, 1);
|
||||
assert_eq!(stats.verify_skipped, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cache_file_roundtrips_entries_across_reload() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join("work-db.crypto-sig-cache");
|
||||
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
|
||||
let key = compute_cache_key(CryptoSigVerifyPoint::EeCertFast, b"tbs", b"sig", b"spki");
|
||||
cache.insert_positive(key);
|
||||
cache.persist().expect("persist");
|
||||
|
||||
let reloaded = CryptoSigCache::load_or_rebuild(path, true);
|
||||
assert!(reloaded.contains(&key));
|
||||
let summary = reloaded.summary();
|
||||
assert_eq!(summary.entries_loaded, 1);
|
||||
assert_eq!(summary.entries_total, 1);
|
||||
assert_eq!(summary.schema_version, CACHE_SCHEMA_VERSION);
|
||||
|
||||
// Reloaded entry drives a skip.
|
||||
let mut executed = 0u64;
|
||||
reloaded
|
||||
.verify(
|
||||
CryptoSigVerifyPoint::EeCertFast,
|
||||
b"tbs",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Ok(())
|
||||
},
|
||||
)
|
||||
.expect("verify ok");
|
||||
assert_eq!(executed, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cache_file_rebuilds_on_missing_corrupt_or_mismatched_schema() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join("work-db.crypto-sig-cache");
|
||||
|
||||
// Missing file.
|
||||
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
|
||||
assert_eq!(cache.summary().entries_loaded, 0);
|
||||
|
||||
// Garbage content.
|
||||
std::fs::write(&path, b"not a valid cache file").expect("write garbage");
|
||||
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
|
||||
assert_eq!(cache.summary().entries_loaded, 0);
|
||||
|
||||
// M1 seen-set format (RPKICSC1) must not load as a v2 cache: cold start.
|
||||
let mut bytes = Vec::new();
|
||||
bytes.extend_from_slice(b"RPKICSC1");
|
||||
bytes.extend_from_slice(&1u64.to_le_bytes());
|
||||
bytes.extend_from_slice(&[7u8; 32]);
|
||||
std::fs::write(&path, &bytes).expect("write v1 file");
|
||||
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
|
||||
assert_eq!(cache.summary().entries_loaded, 0);
|
||||
|
||||
// Valid v2 header but truncated body.
|
||||
let mut bytes = Vec::new();
|
||||
bytes.extend_from_slice(CACHE_FILE_MAGIC);
|
||||
bytes.extend_from_slice(&2u64.to_le_bytes());
|
||||
bytes.extend_from_slice(&[9u8; RECORD_BYTES]);
|
||||
std::fs::write(&path, &bytes).expect("write truncated");
|
||||
let cache = CryptoSigCache::load_or_rebuild(path, true);
|
||||
assert_eq!(cache.summary().entries_loaded, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn size_cap_triggers_full_rebuild_eviction() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let cache = CryptoSigCache::load_or_rebuild_with_capacity(
|
||||
dir.path().join("work-db.crypto-sig-cache"),
|
||||
true,
|
||||
2,
|
||||
);
|
||||
let k1 = compute_cache_key(CryptoSigVerifyPoint::Crl, b"a", b"s", b"k");
|
||||
let k2 = compute_cache_key(CryptoSigVerifyPoint::Crl, b"b", b"s", b"k");
|
||||
let k3 = compute_cache_key(CryptoSigVerifyPoint::Crl, b"c", b"s", b"k");
|
||||
cache.insert_positive(k1);
|
||||
cache.insert_positive(k2);
|
||||
assert!(cache.contains(&k1) && cache.contains(&k2));
|
||||
// Third insert exceeds the cap -> full rebuild, then insert.
|
||||
cache.insert_positive(k3);
|
||||
assert!(!cache.contains(&k1));
|
||||
assert!(!cache.contains(&k2));
|
||||
assert!(cache.contains(&k3));
|
||||
let summary = cache.summary();
|
||||
assert_eq!(summary.entries_total, 1);
|
||||
assert_eq!(summary.evictions_total, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_cache_file_path_mirrors_pp_cache_index_placement() {
|
||||
let db = Path::new("/state/db/work-db");
|
||||
assert_eq!(
|
||||
default_cache_file_path(db),
|
||||
PathBuf::from("/state/db/work-db.crypto-sig-cache")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_with_cache_without_global_cache_runs_verify_and_collects_nothing() {
|
||||
let _guard = GLOBAL_TEST_LOCK.lock().expect("test lock");
|
||||
clear_global();
|
||||
assert!(global_cache().is_none());
|
||||
let mut executed = 0u64;
|
||||
let result: Result<u64, String> = (|| {
|
||||
verify_with_cache(
|
||||
CryptoSigVerifyPoint::CmsSignedObject,
|
||||
b"input",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| -> Result<(), String> {
|
||||
executed += 1;
|
||||
Ok(())
|
||||
},
|
||||
)?;
|
||||
Ok(42)
|
||||
})();
|
||||
assert_eq!(result, Ok(42));
|
||||
assert_eq!(executed, 1);
|
||||
assert!(global_cache().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_with_cache_preserves_result_through_global_install() {
|
||||
let _guard = GLOBAL_TEST_LOCK.lock().expect("test lock");
|
||||
clear_global();
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let cache = Arc::new(CryptoSigCache::load_or_rebuild(
|
||||
dir.path().join("work-db.crypto-sig-cache"),
|
||||
true,
|
||||
));
|
||||
install_global(Arc::clone(&cache));
|
||||
|
||||
let ok: Result<(), String> = verify_with_cache(
|
||||
CryptoSigVerifyPoint::ChildCaCert,
|
||||
b"tbs",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| Ok(()),
|
||||
);
|
||||
assert_eq!(ok, Ok(()));
|
||||
let err: Result<(), String> = verify_with_cache(
|
||||
CryptoSigVerifyPoint::ChildCaCert,
|
||||
b"tbs2",
|
||||
b"sig",
|
||||
b"spki",
|
||||
|| Err("bad signature".to_string()),
|
||||
);
|
||||
assert_eq!(err, Err("bad signature".to_string()));
|
||||
let summary = cache.summary();
|
||||
let stats = summary
|
||||
.per_point
|
||||
.get("child_ca_cert")
|
||||
.expect("stats present");
|
||||
assert_eq!(stats.calls, 2);
|
||||
assert_eq!(stats.new_keys, 2);
|
||||
clear_global();
|
||||
assert!(global_cache().is_none());
|
||||
}
|
||||
}
|
||||
290
crates/panda-rpki-validator/src/current_repo_index.rs
Normal file
290
crates/panda-rpki-validator/src/current_repo_index.rs
Normal file
@ -0,0 +1,290 @@
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::{Arc, RwLock};
|
||||
|
||||
use crate::storage::{RepositoryViewEntry, RepositoryViewState};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CurrentRepoEntry {
|
||||
pub current_hash: [u8; 32],
|
||||
pub current_hash_hex: String,
|
||||
pub repository_source: String,
|
||||
pub object_type: Option<String>,
|
||||
pub state: RepositoryViewState,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub struct CurrentRepoObject {
|
||||
pub rsync_uri: String,
|
||||
pub current_hash_hex: String,
|
||||
pub repository_source: String,
|
||||
pub object_type: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Default, Debug)]
|
||||
pub struct CurrentRepoIndex {
|
||||
by_uri: HashMap<String, CurrentRepoEntry>,
|
||||
}
|
||||
|
||||
/// Shared handle to the run-wide current repository index. Readers (phase 2
|
||||
/// publication point staging, cache lookups, output snapshots) take
|
||||
/// `.read()` and no longer exclude each other; writers (repo sync transports
|
||||
/// applying repository view entries, run-state reset) take `.write()` and
|
||||
/// stay exclusive.
|
||||
pub type CurrentRepoIndexHandle = Arc<RwLock<CurrentRepoIndex>>;
|
||||
|
||||
impl CurrentRepoIndex {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
pub fn shared() -> CurrentRepoIndexHandle {
|
||||
Arc::new(RwLock::new(Self::new()))
|
||||
}
|
||||
|
||||
pub fn get_by_uri(&self, rsync_uri: &str) -> Option<&CurrentRepoEntry> {
|
||||
self.by_uri.get(rsync_uri)
|
||||
}
|
||||
|
||||
pub fn list_scope_uris(&self, repository_source: &str) -> Vec<String> {
|
||||
let mut out = self
|
||||
.by_uri
|
||||
.iter()
|
||||
.filter_map(|(rsync_uri, entry)| {
|
||||
(entry.repository_source == repository_source).then(|| rsync_uri.clone())
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
pub fn active_uri_count(&self) -> usize {
|
||||
self.by_uri.len()
|
||||
}
|
||||
|
||||
pub fn scope_count(&self) -> usize {
|
||||
self.by_uri
|
||||
.values()
|
||||
.map(|entry| entry.repository_source.as_str())
|
||||
.collect::<HashSet<_>>()
|
||||
.len()
|
||||
}
|
||||
|
||||
pub fn snapshot_objects(&self) -> Vec<CurrentRepoObject> {
|
||||
let mut out = self
|
||||
.by_uri
|
||||
.iter()
|
||||
.map(|(rsync_uri, entry)| CurrentRepoObject {
|
||||
rsync_uri: rsync_uri.clone(),
|
||||
current_hash_hex: entry.current_hash_hex.clone(),
|
||||
repository_source: entry.repository_source.clone(),
|
||||
object_type: entry.object_type.clone(),
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
pub fn clear(&mut self) {
|
||||
self.by_uri.clear();
|
||||
}
|
||||
|
||||
pub fn apply_repository_view_entries(
|
||||
&mut self,
|
||||
entries: &[RepositoryViewEntry],
|
||||
) -> Result<(), String> {
|
||||
for entry in entries {
|
||||
self.apply_repository_view_entry(entry)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply_repository_view_entry(&mut self, entry: &RepositoryViewEntry) -> Result<(), String> {
|
||||
entry.validate_internal().map_err(|e| e.to_string())?;
|
||||
|
||||
match entry.state {
|
||||
RepositoryViewState::Present | RepositoryViewState::Replaced => {
|
||||
let repository_source = entry.repository_source.clone().ok_or_else(|| {
|
||||
format!(
|
||||
"repository_view entry missing repository_source for current object {}",
|
||||
entry.rsync_uri
|
||||
)
|
||||
})?;
|
||||
let current_hash_hex = entry.current_hash.clone().ok_or_else(|| {
|
||||
format!(
|
||||
"repository_view entry missing current_hash for current object {}",
|
||||
entry.rsync_uri
|
||||
)
|
||||
})?;
|
||||
let current_hash = decode_sha256_hex_32(¤t_hash_hex)?;
|
||||
self.by_uri.insert(
|
||||
entry.rsync_uri.clone(),
|
||||
CurrentRepoEntry {
|
||||
current_hash,
|
||||
current_hash_hex: current_hash_hex.to_ascii_lowercase(),
|
||||
repository_source,
|
||||
object_type: entry.object_type.clone(),
|
||||
state: entry.state,
|
||||
},
|
||||
);
|
||||
}
|
||||
RepositoryViewState::Withdrawn => {
|
||||
self.by_uri.remove(&entry.rsync_uri);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn decode_sha256_hex_32(value: &str) -> Result<[u8; 32], String> {
|
||||
if value.len() != 64 || !value.as_bytes().iter().all(u8::is_ascii_hexdigit) {
|
||||
return Err(format!("invalid sha256 hex: {value}"));
|
||||
}
|
||||
let mut out = [0u8; 32];
|
||||
hex::decode_to_slice(value, &mut out).map_err(|e| format!("hex decode failed: {e}"))?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::CurrentRepoIndex;
|
||||
use crate::storage::{RepositoryViewEntry, RepositoryViewState};
|
||||
|
||||
fn present(source: &str, uri: &str, hash: &str) -> RepositoryViewEntry {
|
||||
RepositoryViewEntry {
|
||||
rsync_uri: uri.to_string(),
|
||||
current_hash: Some(hash.to_string()),
|
||||
repository_source: Some(source.to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repo_index_tracks_present_and_withdrawn_entries() {
|
||||
let mut index = CurrentRepoIndex::new();
|
||||
let uri = "rsync://example.test/repo/a.roa";
|
||||
let source = "rsync://example.test/repo/";
|
||||
let hash = &"11".repeat(32);
|
||||
index
|
||||
.apply_repository_view_entries(&[present(source, uri, hash)])
|
||||
.expect("apply present");
|
||||
let got = index.get_by_uri(uri).expect("current entry");
|
||||
assert_eq!(got.current_hash_hex, hash.to_string());
|
||||
assert_eq!(index.list_scope_uris(source), vec![uri.to_string()]);
|
||||
|
||||
index
|
||||
.apply_repository_view_entries(&[RepositoryViewEntry {
|
||||
rsync_uri: uri.to_string(),
|
||||
current_hash: Some(hash.to_string()),
|
||||
repository_source: Some(source.to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Withdrawn,
|
||||
}])
|
||||
.expect("apply withdrawn");
|
||||
assert!(index.get_by_uri(uri).is_none());
|
||||
assert!(index.list_scope_uris(source).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repo_index_moves_uri_between_scopes() {
|
||||
let mut index = CurrentRepoIndex::new();
|
||||
let uri = "rsync://example.test/repo/a.roa";
|
||||
let old_scope = "rsync://example.test/repo/";
|
||||
let new_scope = "https://rrdp.example.test/notification.xml";
|
||||
index
|
||||
.apply_repository_view_entries(&[present(old_scope, uri, &"22".repeat(32))])
|
||||
.expect("apply old scope");
|
||||
index
|
||||
.apply_repository_view_entries(&[present(new_scope, uri, &"33".repeat(32))])
|
||||
.expect("apply new scope");
|
||||
|
||||
assert!(index.list_scope_uris(old_scope).is_empty());
|
||||
assert_eq!(index.list_scope_uris(new_scope), vec![uri.to_string()]);
|
||||
assert_eq!(
|
||||
index.get_by_uri(uri).expect("entry").current_hash_hex,
|
||||
"33".repeat(32)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repo_index_snapshot_objects_and_counts_are_sorted() {
|
||||
let handle = CurrentRepoIndex::shared();
|
||||
let mut index = handle.write().expect("write-lock index");
|
||||
index
|
||||
.apply_repository_view_entries(&[
|
||||
present(
|
||||
"rsync://example.test/repo-b/",
|
||||
"rsync://example.test/repo-b/b.roa",
|
||||
&"22".repeat(32),
|
||||
),
|
||||
present(
|
||||
"rsync://example.test/repo-a/",
|
||||
"rsync://example.test/repo-a/a.roa",
|
||||
&"11".repeat(32),
|
||||
),
|
||||
])
|
||||
.expect("apply present entries");
|
||||
assert_eq!(index.active_uri_count(), 2);
|
||||
assert_eq!(index.scope_count(), 2);
|
||||
|
||||
let snapshot = index.snapshot_objects();
|
||||
assert_eq!(snapshot.len(), 2);
|
||||
assert_eq!(snapshot[0].rsync_uri, "rsync://example.test/repo-a/a.roa");
|
||||
assert_eq!(snapshot[1].rsync_uri, "rsync://example.test/repo-b/b.roa");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repo_index_reports_missing_fields_and_invalid_hash() {
|
||||
let mut index = CurrentRepoIndex::new();
|
||||
let err = index
|
||||
.apply_repository_view_entries(&[RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
|
||||
current_hash: Some("11".repeat(32)),
|
||||
repository_source: None,
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
}])
|
||||
.expect_err("missing source should fail");
|
||||
assert!(err.contains("missing repository_source"), "{err}");
|
||||
|
||||
let err = index
|
||||
.apply_repository_view_entries(&[RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
|
||||
current_hash: Some("not-a-valid-sha256".to_string()),
|
||||
repository_source: Some("rsync://example.test/repo/".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
}])
|
||||
.expect_err("invalid hash should fail");
|
||||
assert!(err.contains("invalid"), "{err}");
|
||||
|
||||
index
|
||||
.apply_repository_view_entries(&[RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/b.roa".to_string(),
|
||||
current_hash: Some("22".repeat(32)),
|
||||
repository_source: Some("rsync://example.test/repo/".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
}])
|
||||
.expect("valid entry");
|
||||
let got = index.get_by_uri("rsync://example.test/repo/b.roa").unwrap();
|
||||
assert_eq!(got.current_hash_hex, "22".repeat(32));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repo_index_withdraw_unknown_uri_is_noop() {
|
||||
let mut index = CurrentRepoIndex::new();
|
||||
index
|
||||
.apply_repository_view_entries(&[RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/missing.roa".to_string(),
|
||||
current_hash: None,
|
||||
repository_source: Some("rsync://example.test/repo/".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Withdrawn,
|
||||
}])
|
||||
.expect("withdraw unknown should not fail");
|
||||
assert_eq!(index.active_uri_count(), 0);
|
||||
assert_eq!(index.scope_count(), 0);
|
||||
}
|
||||
}
|
||||
2216
crates/panda-rpki-validator/src/daemon.rs
Normal file
2216
crates/panda-rpki-validator/src/daemon.rs
Normal file
File diff suppressed because it is too large
Load Diff
414
crates/panda-rpki-validator/src/data_model/aspa.rs
Normal file
414
crates/panda-rpki-validator/src/data_model/aspa.rs
Normal file
@ -0,0 +1,414 @@
|
||||
use crate::data_model::common::{DerReader, der_take_tlv};
|
||||
use crate::data_model::oid::OID_CT_ASPA;
|
||||
use crate::data_model::rc::ResourceCertificate;
|
||||
use crate::data_model::signed_object::{
|
||||
RpkiSignedObject, RpkiSignedObjectParsed, SignedObjectDecodeError, SignedObjectParseError,
|
||||
SignedObjectValidateError,
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AspaObject {
|
||||
pub signed_object: RpkiSignedObject,
|
||||
pub econtent_type: String,
|
||||
pub aspa: AspaEContent,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AspaObjectParsed {
|
||||
pub signed_object: RpkiSignedObjectParsed,
|
||||
pub econtent_type: String,
|
||||
pub aspa: Option<AspaEContentParsed>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AspaEContent {
|
||||
pub version: u32,
|
||||
pub customer_as_id: u32,
|
||||
pub provider_as_ids: Vec<u32>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AspaEContentParsed {
|
||||
der: Vec<u8>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum AspaParseError {
|
||||
#[error("signed object parse error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
|
||||
SignedObject(#[from] SignedObjectParseError),
|
||||
#[error("ASPA parse error: {0} (draft-ietf-sidrops-aspa-profile-21 §3; DER)")]
|
||||
Parse(String),
|
||||
|
||||
#[error("ASPA trailing bytes: {0} bytes (draft-ietf-sidrops-aspa-profile-21 §3; DER)")]
|
||||
TrailingBytes(usize),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum AspaProfileError {
|
||||
#[error("signed object profile error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
|
||||
SignedObject(#[from] SignedObjectValidateError),
|
||||
|
||||
#[error(
|
||||
"ASPA eContentType must be {OID_CT_ASPA}, got {0} (draft-ietf-sidrops-aspa-profile-21 §2)"
|
||||
)]
|
||||
InvalidEContentType(String),
|
||||
|
||||
#[error("ASPA profile decode error: {0} (draft-ietf-sidrops-aspa-profile-21 §3; DER)")]
|
||||
ProfileDecode(String),
|
||||
|
||||
#[error(
|
||||
"ASProviderAttestation must be a SEQUENCE of 3 elements (draft-ietf-sidrops-aspa-profile-21 §3)"
|
||||
)]
|
||||
InvalidAttestationSequence,
|
||||
|
||||
#[error(
|
||||
"ASPA version must be 1 and MUST be explicitly encoded (draft-ietf-sidrops-aspa-profile-21 §3.1)"
|
||||
)]
|
||||
VersionMustBeExplicitOne,
|
||||
|
||||
#[error(
|
||||
"ASPA customerASID out of range (0..=4294967295), got {0} (draft-ietf-sidrops-aspa-profile-21 §3.2)"
|
||||
)]
|
||||
CustomerAsIdOutOfRange(u64),
|
||||
|
||||
#[error(
|
||||
"ASPA providers must contain at least one ASID (draft-ietf-sidrops-aspa-profile-21 §3.3)"
|
||||
)]
|
||||
EmptyProviders,
|
||||
|
||||
#[error(
|
||||
"ASPA provider ASID out of range (0..=4294967295), got {0} (draft-ietf-sidrops-aspa-profile-21 §3.3)"
|
||||
)]
|
||||
ProviderAsIdOutOfRange(u64),
|
||||
|
||||
#[error(
|
||||
"ASPA providers must be in strictly increasing order (draft-ietf-sidrops-aspa-profile-21 §3.3)"
|
||||
)]
|
||||
ProvidersNotStrictlyIncreasing,
|
||||
|
||||
#[error(
|
||||
"ASPA providers contains the customerASID ({0}) which is not allowed (draft-ietf-sidrops-aspa-profile-21 §3.3)"
|
||||
)]
|
||||
ProvidersContainCustomer(u32),
|
||||
}
|
||||
|
||||
impl From<SignedObjectDecodeError> for AspaProfileError {
|
||||
fn from(value: SignedObjectDecodeError) -> Self {
|
||||
match value {
|
||||
SignedObjectDecodeError::Parse(e) => AspaProfileError::ProfileDecode(e.to_string()),
|
||||
SignedObjectDecodeError::Validate(e) => AspaProfileError::SignedObject(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum AspaDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] AspaParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] AspaProfileError),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum AspaValidateError {
|
||||
#[error(
|
||||
"ASPA EE certificate must contain AS resources extension (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2)"
|
||||
)]
|
||||
EeAsResourcesMissing,
|
||||
|
||||
#[error(
|
||||
"ASPA EE certificate AS resources must not use inherit (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2.3.3)"
|
||||
)]
|
||||
EeAsResourcesInherit,
|
||||
|
||||
#[error(
|
||||
"ASPA EE certificate AS resources must not include ranges (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2.3.6-§3.2.3.7)"
|
||||
)]
|
||||
EeAsResourcesRangePresent,
|
||||
|
||||
#[error(
|
||||
"ASPA EE certificate AS resources must not include RDI (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2.3.5; RFC 6487 §4.8.11)"
|
||||
)]
|
||||
EeAsResourcesRdiPresent,
|
||||
|
||||
#[error(
|
||||
"ASPA EE certificate AS resources must contain exactly one ASID (id element) (draft-ietf-sidrops-aspa-profile-21 §4)"
|
||||
)]
|
||||
EeAsResourcesNotSingleId,
|
||||
|
||||
#[error(
|
||||
"ASPA customerASID ({customer_as_id}) does not match EE AS resources ({ee_as_id}) (draft-ietf-sidrops-aspa-profile-21 §4)"
|
||||
)]
|
||||
CustomerAsIdMismatch { customer_as_id: u32, ee_as_id: u32 },
|
||||
|
||||
#[error(
|
||||
"ASPA EE certificate must not contain IP resources extension (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §2.2)"
|
||||
)]
|
||||
EeIpResourcesPresent,
|
||||
}
|
||||
|
||||
impl AspaObject {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<AspaObjectParsed, AspaParseError> {
|
||||
let signed_object = RpkiSignedObject::parse_der(der)?;
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
let aspa = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent
|
||||
.as_deref()
|
||||
.map(AspaEContent::parse_der)
|
||||
.transpose()?;
|
||||
Ok(AspaObjectParsed {
|
||||
signed_object,
|
||||
econtent_type,
|
||||
aspa,
|
||||
})
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `AspaObject` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `AspaObjectParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), AspaProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, AspaDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
pub fn decode_der_with_strict_options(
|
||||
der: &[u8],
|
||||
strict_cms_der: bool,
|
||||
strict_name: bool,
|
||||
) -> Result<Self, AspaDecodeError> {
|
||||
let signed_object =
|
||||
RpkiSignedObject::decode_der_with_strict_options(der, strict_cms_der, strict_name)
|
||||
.map_err(AspaProfileError::from)?;
|
||||
Self::from_signed_object(signed_object)
|
||||
}
|
||||
|
||||
pub fn from_signed_object(signed_object: RpkiSignedObject) -> Result<Self, AspaDecodeError> {
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
if econtent_type != OID_CT_ASPA {
|
||||
return Err(AspaProfileError::InvalidEContentType(econtent_type).into());
|
||||
}
|
||||
|
||||
let aspa =
|
||||
AspaEContent::decode_der(&signed_object.signed_data.encap_content_info.econtent)?;
|
||||
Ok(Self {
|
||||
aspa,
|
||||
signed_object,
|
||||
econtent_type: OID_CT_ASPA.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Validate this ASPA's embedded EE certificate resources.
|
||||
pub fn validate_embedded_ee_cert(&self) -> Result<(), AspaValidateError> {
|
||||
let ee = &self.signed_object.signed_data.certificates[0].resource_cert;
|
||||
self.aspa.validate_against_ee_cert(ee)
|
||||
}
|
||||
}
|
||||
|
||||
impl AspaEContent {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<AspaEContentParsed, AspaParseError> {
|
||||
let (_tag, _value, rem) = der_take_tlv(der).map_err(AspaParseError::Parse)?;
|
||||
if !rem.is_empty() {
|
||||
return Err(AspaParseError::TrailingBytes(rem.len()));
|
||||
}
|
||||
Ok(AspaEContentParsed { der: der.to_vec() })
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `AspaEContent` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `AspaEContentParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), AspaProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Decode the DER-encoded ASProviderAttestation defined in
|
||||
/// draft-ietf-sidrops-aspa-profile-21 §3 (`parse + validate`).
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, AspaDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
/// Validate ASPA payload against the embedded EE resource certificate.
|
||||
///
|
||||
/// This implements the EE/payload semantic checks described in
|
||||
/// `draft-ietf-sidrops-aspa-profile-21` §4 (as summarized in `rpki/specs/08_aspa.md`).
|
||||
pub fn validate_against_ee_cert(
|
||||
&self,
|
||||
ee: &ResourceCertificate,
|
||||
) -> Result<(), AspaValidateError> {
|
||||
if ee.tbs.extensions.ip_resources.is_some() {
|
||||
return Err(AspaValidateError::EeIpResourcesPresent);
|
||||
}
|
||||
|
||||
let asn = ee
|
||||
.tbs
|
||||
.extensions
|
||||
.as_resources
|
||||
.as_ref()
|
||||
.ok_or(AspaValidateError::EeAsResourcesMissing)?;
|
||||
|
||||
if asn.rdi.is_some() {
|
||||
return Err(AspaValidateError::EeAsResourcesRdiPresent);
|
||||
}
|
||||
if asn.is_asnum_inherit() {
|
||||
return Err(AspaValidateError::EeAsResourcesInherit);
|
||||
}
|
||||
if asn.has_any_range() {
|
||||
return Err(AspaValidateError::EeAsResourcesRangePresent);
|
||||
}
|
||||
|
||||
let ee_as_id = asn
|
||||
.asnum_single_id()
|
||||
.ok_or(AspaValidateError::EeAsResourcesNotSingleId)?;
|
||||
if ee_as_id != self.customer_as_id {
|
||||
return Err(AspaValidateError::CustomerAsIdMismatch {
|
||||
customer_as_id: self.customer_as_id,
|
||||
ee_as_id,
|
||||
});
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl AspaObjectParsed {
|
||||
pub fn validate_profile(self) -> Result<AspaObject, AspaProfileError> {
|
||||
let signed_object = self.signed_object.validate_profile()?;
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
if econtent_type != OID_CT_ASPA {
|
||||
return Err(AspaProfileError::InvalidEContentType(econtent_type));
|
||||
}
|
||||
let aspa = self
|
||||
.aspa
|
||||
.ok_or_else(|| AspaProfileError::ProfileDecode("ASPA.eContent missing".into()))?
|
||||
.validate_profile()?;
|
||||
Ok(AspaObject {
|
||||
signed_object,
|
||||
econtent_type: OID_CT_ASPA.to_string(),
|
||||
aspa,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl AspaEContentParsed {
|
||||
pub fn validate_profile(self) -> Result<AspaEContent, AspaProfileError> {
|
||||
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
|
||||
let mut n = 0usize;
|
||||
while !r.is_empty() {
|
||||
r.skip_any()?;
|
||||
n += 1;
|
||||
}
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
let mut r = DerReader::new(&self.der);
|
||||
let mut seq = r
|
||||
.take_sequence()
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e))?;
|
||||
if !r.is_empty() {
|
||||
return Err(AspaProfileError::ProfileDecode(
|
||||
"trailing bytes after ASProviderAttestation".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let elem_count =
|
||||
count_elements(seq).map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if elem_count != 3 {
|
||||
return Err(AspaProfileError::InvalidAttestationSequence);
|
||||
}
|
||||
|
||||
// version [0] EXPLICIT INTEGER MUST be present and MUST be 1.
|
||||
if seq
|
||||
.peek_tag()
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?
|
||||
!= 0xA0
|
||||
{
|
||||
return Err(AspaProfileError::VersionMustBeExplicitOne);
|
||||
}
|
||||
let (inner_tag, inner_val) = seq
|
||||
.take_explicit(0xA0)
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if inner_tag != 0x02 {
|
||||
return Err(AspaProfileError::VersionMustBeExplicitOne);
|
||||
}
|
||||
let v = crate::data_model::common::der_uint_from_bytes(inner_val)
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if v != 1 {
|
||||
return Err(AspaProfileError::VersionMustBeExplicitOne);
|
||||
}
|
||||
|
||||
let customer_u64 = seq
|
||||
.take_uint_u64()
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if customer_u64 > u32::MAX as u64 {
|
||||
return Err(AspaProfileError::CustomerAsIdOutOfRange(customer_u64));
|
||||
}
|
||||
let customer_as_id = customer_u64 as u32;
|
||||
|
||||
let providers_seq = seq
|
||||
.take_sequence()
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if !seq.is_empty() {
|
||||
return Err(AspaProfileError::InvalidAttestationSequence);
|
||||
}
|
||||
let providers = parse_providers_cursor(providers_seq, customer_as_id)?;
|
||||
|
||||
Ok(AspaEContent {
|
||||
version: 1,
|
||||
customer_as_id,
|
||||
provider_as_ids: providers,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_providers_cursor(
|
||||
mut seq: DerReader<'_>,
|
||||
customer_as_id: u32,
|
||||
) -> Result<Vec<u32>, AspaProfileError> {
|
||||
if seq.is_empty() {
|
||||
return Err(AspaProfileError::EmptyProviders);
|
||||
}
|
||||
|
||||
let mut out: Vec<u32> = Vec::new();
|
||||
let mut prev: Option<u32> = None;
|
||||
while !seq.is_empty() {
|
||||
let v = seq
|
||||
.take_uint_u64()
|
||||
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if v > u32::MAX as u64 {
|
||||
return Err(AspaProfileError::ProviderAsIdOutOfRange(v));
|
||||
}
|
||||
let asn = v as u32;
|
||||
if asn == customer_as_id {
|
||||
return Err(AspaProfileError::ProvidersContainCustomer(customer_as_id));
|
||||
}
|
||||
if let Some(p) = prev {
|
||||
if asn <= p {
|
||||
return Err(AspaProfileError::ProvidersNotStrictlyIncreasing);
|
||||
}
|
||||
}
|
||||
prev = Some(asn);
|
||||
out.push(asn);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
302
crates/panda-rpki-validator/src/data_model/common.rs
Normal file
302
crates/panda-rpki-validator/src/data_model/common.rs
Normal file
@ -0,0 +1,302 @@
|
||||
use x509_parser::asn1_rs::Tag;
|
||||
use x509_parser::prelude::FromDer;
|
||||
use x509_parser::x509::AlgorithmIdentifier;
|
||||
|
||||
pub type UtcTime = time::OffsetDateTime;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Asn1TimeEncoding {
|
||||
UtcTime,
|
||||
GeneralizedTime,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct Asn1TimeUtc {
|
||||
pub utc: UtcTime,
|
||||
pub encoding: Asn1TimeEncoding,
|
||||
}
|
||||
|
||||
impl Asn1TimeUtc {
|
||||
/// Validate Time encoding rules (RFC 5280): years 1950-2049 use UTCTime,
|
||||
/// other years use GeneralizedTime.
|
||||
pub fn validate_encoding_rfc5280(
|
||||
&self,
|
||||
field: &'static str,
|
||||
) -> Result<(), InvalidTimeEncodingError> {
|
||||
let year = self.utc.year();
|
||||
let expected = if year <= 2049 {
|
||||
Asn1TimeEncoding::UtcTime
|
||||
} else {
|
||||
Asn1TimeEncoding::GeneralizedTime
|
||||
};
|
||||
if self.encoding != expected {
|
||||
return Err(InvalidTimeEncodingError {
|
||||
field,
|
||||
year,
|
||||
encoding: self.encoding,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct BigUnsigned {
|
||||
/// Minimal big-endian bytes. For zero, this is `[0]`.
|
||||
pub bytes_be: Vec<u8>,
|
||||
}
|
||||
|
||||
impl BigUnsigned {
|
||||
pub fn from_biguint(n: &der_parser::num_bigint::BigUint) -> Self {
|
||||
let mut bytes = n.to_bytes_be();
|
||||
if bytes.is_empty() {
|
||||
bytes.push(0);
|
||||
}
|
||||
Self { bytes_be: bytes }
|
||||
}
|
||||
|
||||
pub fn to_hex_upper(&self) -> String {
|
||||
hex::encode_upper(&self.bytes_be)
|
||||
}
|
||||
|
||||
pub fn to_u64(&self) -> Option<u64> {
|
||||
if self.bytes_be.len() > 8 {
|
||||
return None;
|
||||
}
|
||||
let mut value: u64 = 0;
|
||||
for &b in &self.bytes_be {
|
||||
value = (value << 8) | (b as u64);
|
||||
}
|
||||
Some(value)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
#[error(
|
||||
"{field} time encoding invalid for year {year}: got {encoding:?} (RFC 5280 §4.1.2.5; RFC 5280 §5.1.2.4-§5.1.2.6)"
|
||||
)]
|
||||
pub struct InvalidTimeEncodingError {
|
||||
pub field: &'static str,
|
||||
pub year: i32,
|
||||
pub encoding: Asn1TimeEncoding,
|
||||
}
|
||||
|
||||
pub fn asn1_time_to_model(t: x509_parser::time::ASN1Time) -> Asn1TimeUtc {
|
||||
let encoding = if t.is_utctime() {
|
||||
Asn1TimeEncoding::UtcTime
|
||||
} else {
|
||||
Asn1TimeEncoding::GeneralizedTime
|
||||
};
|
||||
Asn1TimeUtc {
|
||||
utc: t.to_datetime(),
|
||||
encoding,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn algorithm_params_absent_or_null(sig: &AlgorithmIdentifier<'_>) -> bool {
|
||||
match sig.parameters.as_ref() {
|
||||
None => true,
|
||||
Some(p) if p.tag() == Tag::Null => true,
|
||||
Some(_p) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Take a single DER TLV (Tag-Length-Value) from the start of `input`.
|
||||
///
|
||||
/// This helper supports:
|
||||
/// - short- and long-form lengths (up to 8 length bytes)
|
||||
/// - only low-tag-number form tags (no high-tag-number form)
|
||||
/// - definite length only (DER forbids indefinite length)
|
||||
///
|
||||
/// Returns: `(tag_byte, value_bytes, remaining_bytes)`.
|
||||
pub(crate) fn der_take_tlv(input: &[u8]) -> Result<(u8, &[u8], &[u8]), String> {
|
||||
if input.len() < 2 {
|
||||
return Err("truncated DER (need tag+len)".into());
|
||||
}
|
||||
let tag = input[0];
|
||||
if (tag & 0x1F) == 0x1F {
|
||||
return Err("high-tag-number form not supported".into());
|
||||
}
|
||||
let len0 = input[1];
|
||||
if len0 == 0x80 {
|
||||
return Err("indefinite length not allowed in DER".into());
|
||||
}
|
||||
let (len, hdr_len) = if len0 & 0x80 == 0 {
|
||||
(len0 as usize, 2usize)
|
||||
} else {
|
||||
let n = (len0 & 0x7F) as usize;
|
||||
if n == 0 || n > 8 {
|
||||
return Err("invalid DER length".into());
|
||||
}
|
||||
if input.len() < 2 + n {
|
||||
return Err("truncated DER (length bytes)".into());
|
||||
}
|
||||
let mut l: usize = 0;
|
||||
for &b in &input[2..2 + n] {
|
||||
l = (l << 8) | (b as usize);
|
||||
}
|
||||
(l, 2 + n)
|
||||
};
|
||||
if input.len() < hdr_len + len {
|
||||
return Err("truncated DER (value bytes)".into());
|
||||
}
|
||||
let value = &input[hdr_len..hdr_len + len];
|
||||
let rem = &input[hdr_len + len..];
|
||||
Ok((tag, value, rem))
|
||||
}
|
||||
|
||||
/// Minimal streaming DER reader built on `der_take_tlv`.
|
||||
///
|
||||
/// This is intentionally small and only supports the subset of DER needed by
|
||||
/// RPKI object eContent decoders (ROA/ASPA), to avoid constructing a generic AST
|
||||
/// (which is expensive on large objects such as ROAs with thousands of prefixes).
|
||||
#[derive(Clone, Copy)]
|
||||
pub(crate) struct DerReader<'a> {
|
||||
buf: &'a [u8],
|
||||
}
|
||||
|
||||
impl<'a> DerReader<'a> {
|
||||
pub(crate) fn new(buf: &'a [u8]) -> Self {
|
||||
Self { buf }
|
||||
}
|
||||
|
||||
pub(crate) fn is_empty(&self) -> bool {
|
||||
self.buf.is_empty()
|
||||
}
|
||||
|
||||
pub(crate) fn remaining_len(&self) -> usize {
|
||||
self.buf.len()
|
||||
}
|
||||
|
||||
pub(crate) fn peek_tag(&self) -> Result<u8, String> {
|
||||
self.buf
|
||||
.first()
|
||||
.copied()
|
||||
.ok_or_else(|| "truncated DER".into())
|
||||
}
|
||||
|
||||
pub(crate) fn take_any(&mut self) -> Result<(u8, &'a [u8]), String> {
|
||||
let (tag, value, rem) = der_take_tlv(self.buf)?;
|
||||
self.buf = rem;
|
||||
Ok((tag, value))
|
||||
}
|
||||
|
||||
pub(crate) fn take_any_full(&mut self) -> Result<(u8, &'a [u8], &'a [u8]), String> {
|
||||
let (tag, value, rem) = der_take_tlv(self.buf)?;
|
||||
let consumed = self.buf.len() - rem.len();
|
||||
let full = &self.buf[..consumed];
|
||||
self.buf = rem;
|
||||
Ok((tag, full, value))
|
||||
}
|
||||
|
||||
pub(crate) fn skip_any(&mut self) -> Result<(), String> {
|
||||
let _ = self.take_any()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn take_tag(&mut self, expected_tag: u8) -> Result<&'a [u8], String> {
|
||||
let (tag, value) = self.take_any()?;
|
||||
if tag != expected_tag {
|
||||
return Err(format!(
|
||||
"unexpected tag: got 0x{tag:02X}, expected 0x{expected_tag:02X}"
|
||||
));
|
||||
}
|
||||
Ok(value)
|
||||
}
|
||||
|
||||
pub(crate) fn take_sequence(&mut self) -> Result<DerReader<'a>, String> {
|
||||
let value = self.take_tag(0x30)?;
|
||||
Ok(DerReader::new(value))
|
||||
}
|
||||
|
||||
pub(crate) fn take_octet_string(&mut self) -> Result<&'a [u8], String> {
|
||||
self.take_tag(0x04)
|
||||
}
|
||||
|
||||
pub(crate) fn take_bit_string(&mut self) -> Result<(u8, &'a [u8]), String> {
|
||||
let v = self.take_tag(0x03)?;
|
||||
if v.is_empty() {
|
||||
return Err("BIT STRING content is empty".into());
|
||||
}
|
||||
Ok((v[0], &v[1..]))
|
||||
}
|
||||
|
||||
pub(crate) fn take_uint_u64(&mut self) -> Result<u64, String> {
|
||||
let v = self.take_tag(0x02)?;
|
||||
der_uint_from_bytes(v)
|
||||
}
|
||||
|
||||
pub(crate) fn take_explicit(
|
||||
&mut self,
|
||||
expected_outer_tag: u8,
|
||||
) -> Result<(u8, &'a [u8]), String> {
|
||||
let inner_der = self.take_tag(expected_outer_tag)?;
|
||||
let (tag, value, rem) = der_take_tlv(inner_der)?;
|
||||
if !rem.is_empty() {
|
||||
return Err("trailing bytes inside EXPLICIT value".into());
|
||||
}
|
||||
Ok((tag, value))
|
||||
}
|
||||
|
||||
pub(crate) fn take_explicit_der(&mut self, expected_outer_tag: u8) -> Result<&'a [u8], String> {
|
||||
let inner_der = self.take_tag(expected_outer_tag)?;
|
||||
let (_tag, _value, rem) = der_take_tlv(inner_der)?;
|
||||
if !rem.is_empty() {
|
||||
return Err("trailing bytes inside EXPLICIT value".into());
|
||||
}
|
||||
Ok(inner_der)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn der_uint_from_bytes(bytes: &[u8]) -> Result<u64, String> {
|
||||
if bytes.is_empty() {
|
||||
return Err("INTEGER has empty content".into());
|
||||
}
|
||||
// Disallow negative values.
|
||||
if (bytes[0] & 0x80) != 0 {
|
||||
return Err("INTEGER is negative".into());
|
||||
}
|
||||
// DER requires minimal encoding for INTEGER.
|
||||
if bytes.len() > 1 && bytes[0] == 0x00 && (bytes[1] & 0x80) == 0 {
|
||||
return Err("INTEGER not minimally encoded".into());
|
||||
}
|
||||
if bytes.len() > 8 {
|
||||
return Err("INTEGER does not fit u64".into());
|
||||
}
|
||||
let mut v: u64 = 0;
|
||||
for &b in bytes {
|
||||
v = (v << 8) | (b as u64);
|
||||
}
|
||||
Ok(v)
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
pub struct X509NameDer(pub Vec<u8>);
|
||||
|
||||
impl X509NameDer {
|
||||
pub fn as_raw(&self) -> &[u8] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for X509NameDer {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
let Ok((rem, name)) = x509_parser::x509::X509Name::from_der(&self.0) else {
|
||||
return write!(f, "<invalid X.509 Name DER>");
|
||||
};
|
||||
if !rem.is_empty() {
|
||||
return write!(f, "<invalid X.509 Name DER (trailing bytes)>");
|
||||
}
|
||||
write!(f, "{name}")
|
||||
}
|
||||
}
|
||||
|
||||
/// Filename extensions registered in IANA "RPKI Repository Name Schemes".
|
||||
///
|
||||
/// Source: <https://www.iana.org/assignments/rpki/rpki.xhtml>
|
||||
/// Snapshot date: 2026-01-28.
|
||||
///
|
||||
/// Notes:
|
||||
/// - Includes entries marked TEMPORARY/DEPRECATED by IANA (e.g., `asa`, `gbr`).
|
||||
pub const IANA_RPKI_REPOSITORY_FILENAME_EXTENSIONS: &[&str] =
|
||||
&["asa", "cer", "crl", "gbr", "mft", "roa", "sig", "tak"];
|
||||
546
crates/panda-rpki-validator/src/data_model/crl.rs
Normal file
546
crates/panda-rpki-validator/src/data_model/crl.rs
Normal file
@ -0,0 +1,546 @@
|
||||
pub use crate::data_model::common::{Asn1TimeEncoding, Asn1TimeUtc, BigUnsigned};
|
||||
use crate::data_model::oid::{
|
||||
OID_AUTHORITY_KEY_IDENTIFIER, OID_AUTHORITY_KEY_IDENTIFIER_RAW, OID_CRL_NUMBER,
|
||||
OID_CRL_NUMBER_RAW, OID_SHA256_WITH_RSA_ENCRYPTION, OID_SHA256_WITH_RSA_ENCRYPTION_RAW,
|
||||
OID_SUBJECT_KEY_IDENTIFIER_RAW,
|
||||
};
|
||||
use x509_parser::certificate::X509Certificate;
|
||||
use x509_parser::extensions::{ParsedExtension, X509Extension};
|
||||
use x509_parser::prelude::{FromDer, X509Version};
|
||||
use x509_parser::revocation_list::CertificateRevocationList;
|
||||
use x509_parser::x509::{AlgorithmIdentifier, SubjectPublicKeyInfo};
|
||||
use x509_parser::{asn1_rs::Class as Asn1Class, asn1_rs::Tag as Asn1Tag};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RevokedCert {
|
||||
pub serial_number: BigUnsigned,
|
||||
pub revocation_date: Asn1TimeUtc,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct CrlExtensions {
|
||||
pub authority_key_identifier: Vec<u8>,
|
||||
pub crl_number: BigUnsigned,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RpkixCrl {
|
||||
pub raw_der: Vec<u8>,
|
||||
pub version: u32,
|
||||
pub issuer_dn: String,
|
||||
pub signature_algorithm_oid: String,
|
||||
pub this_update: Asn1TimeUtc,
|
||||
pub next_update: Asn1TimeUtc,
|
||||
pub revoked_certs: Vec<RevokedCert>,
|
||||
pub extensions: CrlExtensions,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RpkixCrlParsed {
|
||||
pub raw_der: Vec<u8>,
|
||||
pub version: Option<X509Version>,
|
||||
pub issuer_dn: String,
|
||||
pub signature_algorithm: AlgorithmIdentifierValue,
|
||||
pub tbs_signature_algorithm: AlgorithmIdentifierValue,
|
||||
pub this_update: Asn1TimeUtc,
|
||||
pub next_update: Option<Asn1TimeUtc>,
|
||||
pub revoked_certs: Vec<RevokedCertParsed>,
|
||||
pub extensions: Vec<CrlExtensionParsed>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RevokedCertParsed {
|
||||
pub serial_number: BigUnsigned,
|
||||
pub revocation_date: Asn1TimeUtc,
|
||||
pub has_extensions: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum CrlExtensionParsed {
|
||||
AuthorityKeyIdentifier {
|
||||
key_identifier: Option<Vec<u8>>,
|
||||
has_other_fields: bool,
|
||||
critical: bool,
|
||||
},
|
||||
CrlNumber {
|
||||
number: der_parser::num_bigint::BigUint,
|
||||
critical: bool,
|
||||
},
|
||||
Other {
|
||||
oid: String,
|
||||
critical: bool,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AlgorithmIdentifierValue {
|
||||
pub oid: String,
|
||||
pub parameters: Option<AlgorithmParametersValue>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AlgorithmParametersValue {
|
||||
pub class: Asn1Class,
|
||||
pub tag: Asn1Tag,
|
||||
pub data: Vec<u8>,
|
||||
}
|
||||
|
||||
impl AlgorithmIdentifierValue {
|
||||
pub fn params_absent_or_null(&self) -> bool {
|
||||
match &self.parameters {
|
||||
None => true,
|
||||
Some(p) if p.class == Asn1Class::Universal && p.tag == Asn1Tag::Null => true,
|
||||
Some(_p) => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CrlParseError {
|
||||
#[error("X.509 CRL parse error: {0} (RFC 5280 §5.1; RFC 6487 §5)")]
|
||||
Parse(String),
|
||||
|
||||
#[error("trailing bytes after CRL DER: {0} bytes (DER; RFC 5280 §5.1)")]
|
||||
TrailingBytes(usize),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CrlProfileError {
|
||||
#[error("CRL version must be v2, got {0:?} (RFC 5280 §5.1; RFC 6487 §5)")]
|
||||
InvalidVersion(Option<u32>),
|
||||
|
||||
#[error("CRL signatureAlgorithm must match TBSCertList.signature (RFC 5280 §5.1)")]
|
||||
SignatureAlgorithmMismatch,
|
||||
|
||||
#[error(
|
||||
"CRL signatureAlgorithm must be sha256WithRSAEncryption ({OID_SHA256_WITH_RSA_ENCRYPTION}), got {0} (RFC 6487 §5; RFC 7935 §2)"
|
||||
)]
|
||||
InvalidSignatureAlgorithm(String),
|
||||
|
||||
#[error(
|
||||
"CRL signature algorithm parameters must be absent or NULL (RFC 5280 §4.1.1.2; RFC 7935 §2)"
|
||||
)]
|
||||
InvalidSignatureAlgorithmParameters,
|
||||
|
||||
#[error("CRL extensions must be exactly two (AKI + CRLNumber), got {0} (RFC 9829 §3.1)")]
|
||||
InvalidExtensionsCount(usize),
|
||||
|
||||
#[error("unsupported CRL extension OID {0} (RFC 9829 §3.1)")]
|
||||
UnsupportedExtension(String),
|
||||
|
||||
#[error("duplicate CRL extension OID {0} (RFC 5280 §4.2; RFC 9829 §3.1)")]
|
||||
DuplicateExtension(String),
|
||||
|
||||
#[error("AuthorityKeyIdentifier CRL extension missing (RFC 9829 §3.1; RFC 5280 §5.2.1)")]
|
||||
AkiMissing,
|
||||
|
||||
#[error("AuthorityKeyIdentifier must contain keyIdentifier (RFC 5280 §5.2.1; RFC 9829 §3.1)")]
|
||||
AkiMissingKeyIdentifier,
|
||||
|
||||
#[error(
|
||||
"AuthorityKeyIdentifier must not contain authorityCertIssuer or authorityCertSerialNumber (RFC 5280 §5.2.1; RFC 9829 §3.1)"
|
||||
)]
|
||||
AkiHasOtherFields,
|
||||
|
||||
#[error("CRLNumber CRL extension missing (RFC 9829 §3.1; RFC 5280 §5.2.3)")]
|
||||
CrlNumberMissing,
|
||||
|
||||
#[error("CRLNumber must be non-critical (RFC 9829 §3.1; RFC 5280 §5.2.3)")]
|
||||
CrlNumberCritical,
|
||||
|
||||
#[error("CRLNumber out of range (must fit in 0..2^159-1) (RFC 9829 §3.1)")]
|
||||
CrlNumberOutOfRange,
|
||||
|
||||
#[error("CRL entry extensions must not be present (RFC 6487 §5; RFC 5280 §5.1)")]
|
||||
EntryExtensionsNotAllowed,
|
||||
|
||||
#[error("CRL nextUpdate must be present (RFC 5280 §5.1.2.5; RFC 6487 §5)")]
|
||||
NextUpdateMissing,
|
||||
|
||||
#[error(
|
||||
"{field} time encoding invalid for year {year}: got {encoding:?} (RFC 5280 §5.1.2.4-§5.1.2.6)"
|
||||
)]
|
||||
InvalidTimeEncoding {
|
||||
field: &'static str,
|
||||
year: i32,
|
||||
encoding: Asn1TimeEncoding,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CrlDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] CrlParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] CrlProfileError),
|
||||
}
|
||||
|
||||
impl RpkixCrl {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<RpkixCrlParsed, CrlParseError> {
|
||||
let (rem, crl) = CertificateRevocationList::from_der(der)
|
||||
.map_err(|e| CrlParseError::Parse(e.to_string()))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(CrlParseError::TrailingBytes(rem.len()));
|
||||
}
|
||||
|
||||
let revoked_certs = crl
|
||||
.iter_revoked_certificates()
|
||||
.map(|rc| RevokedCertParsed {
|
||||
serial_number: BigUnsigned::from_biguint(rc.serial()),
|
||||
revocation_date: crate::data_model::common::asn1_time_to_model(rc.revocation_date),
|
||||
has_extensions: !rc.extensions().is_empty(),
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let this_update = crate::data_model::common::asn1_time_to_model(crl.last_update());
|
||||
let next_update = crl
|
||||
.next_update()
|
||||
.map(crate::data_model::common::asn1_time_to_model);
|
||||
|
||||
let extensions =
|
||||
parse_extensions_parse(crl.extensions()).map_err(|e| CrlParseError::Parse(e))?;
|
||||
|
||||
Ok(RpkixCrlParsed {
|
||||
raw_der: der.to_vec(),
|
||||
version: crl.version(),
|
||||
issuer_dn: crl.issuer().to_string(),
|
||||
signature_algorithm: algorithm_identifier_value(&crl.signature_algorithm),
|
||||
tbs_signature_algorithm: algorithm_identifier_value(&crl.tbs_cert_list.signature),
|
||||
this_update,
|
||||
next_update,
|
||||
revoked_certs,
|
||||
extensions,
|
||||
})
|
||||
}
|
||||
|
||||
/// Decode a DER-encoded X.509 v2 CRL and enforce the RPKI profile constraints from
|
||||
/// `specs/prepare/data_models/04_crl.md` (RFC 6487 §5; RFC 9829 §3.1; RFC 5280 §5.1).
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, CrlDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `RpkixCrl` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `RpkixCrlParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), CrlProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Verify the cryptographic signature on this CRL using the issuer certificate.
|
||||
///
|
||||
/// Signature verification needs the issuer public key (RFC 5280 §6.3.3 (f)-(g)).
|
||||
/// In RPKI practice, this public key is obtained from the CRL issuer CA certificate
|
||||
/// (and that certificate must already be validated up to the same trust anchor).
|
||||
///
|
||||
/// This helper also performs common binding checks:
|
||||
/// - CRL `issuer_dn` must equal issuer certificate `subject`
|
||||
/// - if issuer KeyUsage is present, require `cRLSign`
|
||||
/// - if issuer SKI is present, require it matches CRL AKI.keyIdentifier
|
||||
pub fn verify_signature_with_issuer_certificate_der(
|
||||
&self,
|
||||
issuer_cert_der: &[u8],
|
||||
) -> Result<(), CrlVerifyError> {
|
||||
let (rem, issuer_cert) = X509Certificate::from_der(issuer_cert_der)
|
||||
.map_err(|e| CrlVerifyError::IssuerCertificateParse(e.to_string()))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(CrlVerifyError::IssuerCertificateTrailingBytes(rem.len()));
|
||||
}
|
||||
|
||||
let subject_dn = issuer_cert.subject().to_string();
|
||||
if subject_dn != self.issuer_dn {
|
||||
return Err(CrlVerifyError::IssuerSubjectMismatch {
|
||||
crl_issuer_dn: self.issuer_dn.clone(),
|
||||
issuer_subject_dn: subject_dn,
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(ku) = issuer_cert
|
||||
.key_usage()
|
||||
.map_err(|e| CrlVerifyError::IssuerCertificateParse(e.to_string()))?
|
||||
{
|
||||
if !ku.value.crl_sign() {
|
||||
return Err(CrlVerifyError::IssuerKeyUsageMissingCrlSign);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(issuer_ski) = get_subject_key_identifier(&issuer_cert) {
|
||||
if issuer_ski != self.extensions.authority_key_identifier {
|
||||
return Err(CrlVerifyError::AkiSkiMismatch);
|
||||
}
|
||||
}
|
||||
|
||||
self.verify_signature_with_issuer_spki(issuer_cert.public_key())
|
||||
}
|
||||
|
||||
/// Verify the cryptographic signature on this CRL using the issuer SubjectPublicKeyInfo.
|
||||
pub fn verify_signature_with_issuer_spki(
|
||||
&self,
|
||||
issuer_spki: &SubjectPublicKeyInfo<'_>,
|
||||
) -> Result<(), CrlVerifyError> {
|
||||
let (rem, crl) = CertificateRevocationList::from_der(&self.raw_der)
|
||||
.map_err(|e| CrlVerifyError::CrlParse(e.to_string()))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(CrlVerifyError::CrlTrailingBytes(rem.len()));
|
||||
}
|
||||
crate::crypto_sig_cache::verify_with_cache(
|
||||
crate::crypto_sig_cache::CryptoSigVerifyPoint::Crl,
|
||||
crl.tbs_cert_list.as_ref(),
|
||||
crl.signature_value.data.as_ref(),
|
||||
issuer_spki.raw,
|
||||
|| {
|
||||
crl.verify_signature(issuer_spki)
|
||||
.map_err(|e| CrlVerifyError::InvalidSignature(e.to_string()))
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// Verify the cryptographic signature on this CRL using a DER-encoded SubjectPublicKeyInfo.
|
||||
pub fn verify_signature_with_issuer_spki_der(
|
||||
&self,
|
||||
issuer_spki_der: &[u8],
|
||||
) -> Result<(), CrlVerifyError> {
|
||||
let (rem, spki) = SubjectPublicKeyInfo::from_der(issuer_spki_der)
|
||||
.map_err(|e| CrlVerifyError::IssuerSpkiParse(e.to_string()))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(CrlVerifyError::IssuerSpkiTrailingBytes(rem.len()));
|
||||
}
|
||||
self.verify_signature_with_issuer_spki(&spki)
|
||||
}
|
||||
}
|
||||
|
||||
impl RpkixCrlParsed {
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
pub fn validate_profile(self) -> Result<RpkixCrl, CrlProfileError> {
|
||||
let version = match self.version {
|
||||
Some(X509Version::V2) => 2,
|
||||
Some(v) => return Err(CrlProfileError::InvalidVersion(Some(v.0))),
|
||||
None => return Err(CrlProfileError::InvalidVersion(None)),
|
||||
};
|
||||
|
||||
// signatureAlgorithm must match tbsCertList.signature
|
||||
if self.signature_algorithm != self.tbs_signature_algorithm {
|
||||
return Err(CrlProfileError::SignatureAlgorithmMismatch);
|
||||
}
|
||||
let sig_oid = self.signature_algorithm.oid.clone();
|
||||
if sig_oid != OID_SHA256_WITH_RSA_ENCRYPTION {
|
||||
return Err(CrlProfileError::InvalidSignatureAlgorithm(sig_oid));
|
||||
}
|
||||
if !self.signature_algorithm.params_absent_or_null() {
|
||||
return Err(CrlProfileError::InvalidSignatureAlgorithmParameters);
|
||||
}
|
||||
|
||||
let extensions = validate_extensions_profile(&self.extensions)?;
|
||||
|
||||
let mut revoked_out = Vec::with_capacity(self.revoked_certs.len());
|
||||
for rc in self.revoked_certs {
|
||||
if rc.has_extensions {
|
||||
return Err(CrlProfileError::EntryExtensionsNotAllowed);
|
||||
}
|
||||
validate_time_encoding_rfc5280("revocationDate", &rc.revocation_date)?;
|
||||
revoked_out.push(RevokedCert {
|
||||
serial_number: rc.serial_number,
|
||||
revocation_date: rc.revocation_date,
|
||||
});
|
||||
}
|
||||
|
||||
validate_time_encoding_rfc5280("thisUpdate", &self.this_update)?;
|
||||
|
||||
let next_update = self.next_update.ok_or(CrlProfileError::NextUpdateMissing)?;
|
||||
validate_time_encoding_rfc5280("nextUpdate", &next_update)?;
|
||||
|
||||
Ok(RpkixCrl {
|
||||
raw_der: self.raw_der,
|
||||
version,
|
||||
issuer_dn: self.issuer_dn,
|
||||
signature_algorithm_oid: OID_SHA256_WITH_RSA_ENCRYPTION.to_string(),
|
||||
this_update: self.this_update,
|
||||
next_update,
|
||||
revoked_certs: revoked_out,
|
||||
extensions,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum CrlVerifyError {
|
||||
#[error("issuer certificate parse error: {0} (RFC 5280 §4.1; RFC 6487 §4)")]
|
||||
IssuerCertificateParse(String),
|
||||
|
||||
#[error("trailing bytes after issuer certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
|
||||
IssuerCertificateTrailingBytes(usize),
|
||||
|
||||
#[error("issuer SubjectPublicKeyInfo parse error: {0} (RFC 5280 §4.1.2.7)")]
|
||||
IssuerSpkiParse(String),
|
||||
|
||||
#[error(
|
||||
"trailing bytes after issuer SubjectPublicKeyInfo DER: {0} bytes (DER; RFC 5280 §4.1.2.7)"
|
||||
)]
|
||||
IssuerSpkiTrailingBytes(usize),
|
||||
|
||||
#[error("CRL parse error: {0} (RFC 5280 §5.1; RFC 6487 §5)")]
|
||||
CrlParse(String),
|
||||
|
||||
#[error("trailing bytes after CRL DER: {0} bytes (DER; RFC 5280 §5.1)")]
|
||||
CrlTrailingBytes(usize),
|
||||
|
||||
#[error(
|
||||
"CRL issuer DN does not match issuer certificate subject (RFC 5280 §5.1; RFC 5280 §6.3.3(b))"
|
||||
)]
|
||||
IssuerSubjectMismatch {
|
||||
crl_issuer_dn: String,
|
||||
issuer_subject_dn: String,
|
||||
},
|
||||
|
||||
#[error(
|
||||
"issuer certificate keyUsage present but missing cRLSign (RFC 5280 §4.2.1.3; RFC 5280 §6.3.3(f))"
|
||||
)]
|
||||
IssuerKeyUsageMissingCrlSign,
|
||||
|
||||
#[error(
|
||||
"CRL AKI.keyIdentifier does not match issuer certificate SKI (RFC 5280 §4.2.1.1; RFC 5280 §4.2.1.2; RFC 5280 §6.3.3(c)/(f))"
|
||||
)]
|
||||
AkiSkiMismatch,
|
||||
|
||||
#[error("CRL signature verification failed: {0} (RFC 5280 §6.3.3(g); RFC 7935 §2)")]
|
||||
InvalidSignature(String),
|
||||
}
|
||||
|
||||
fn validate_time_encoding_rfc5280(
|
||||
field: &'static str,
|
||||
t: &Asn1TimeUtc,
|
||||
) -> Result<(), CrlProfileError> {
|
||||
let year = t.utc.year();
|
||||
let expected = if year <= 2049 {
|
||||
Asn1TimeEncoding::UtcTime
|
||||
} else {
|
||||
Asn1TimeEncoding::GeneralizedTime
|
||||
};
|
||||
if t.encoding != expected {
|
||||
return Err(CrlProfileError::InvalidTimeEncoding {
|
||||
field,
|
||||
year,
|
||||
encoding: t.encoding,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn algorithm_identifier_value(ai: &AlgorithmIdentifier<'_>) -> AlgorithmIdentifierValue {
|
||||
let parameters = ai.parameters.as_ref().map(|p| AlgorithmParametersValue {
|
||||
class: p.class(),
|
||||
tag: p.tag(),
|
||||
data: p.as_bytes().to_vec(),
|
||||
});
|
||||
// NOTE(perf): Avoid `to_id_string()` allocations for the signature algorithms we expect
|
||||
// in RPKI CRLs. Fall back to `to_id_string()` for unexpected algorithms (mostly error paths).
|
||||
let oid = if ai.algorithm.as_bytes() == OID_SHA256_WITH_RSA_ENCRYPTION_RAW {
|
||||
OID_SHA256_WITH_RSA_ENCRYPTION.to_string()
|
||||
} else {
|
||||
ai.algorithm.to_id_string()
|
||||
};
|
||||
AlgorithmIdentifierValue { oid, parameters }
|
||||
}
|
||||
|
||||
fn parse_extensions_parse(exts: &[X509Extension<'_>]) -> Result<Vec<CrlExtensionParsed>, String> {
|
||||
let mut out = Vec::with_capacity(exts.len());
|
||||
for ext in exts {
|
||||
let oid = ext.oid.as_bytes();
|
||||
if oid == OID_AUTHORITY_KEY_IDENTIFIER_RAW {
|
||||
let ParsedExtension::AuthorityKeyIdentifier(aki) = ext.parsed_extension() else {
|
||||
return Err("AKI extension parse failed".to_string());
|
||||
};
|
||||
out.push(CrlExtensionParsed::AuthorityKeyIdentifier {
|
||||
key_identifier: aki.key_identifier.as_ref().map(|k| k.0.to_vec()),
|
||||
has_other_fields: aki.authority_cert_issuer.is_some()
|
||||
|| aki.authority_cert_serial.is_some(),
|
||||
critical: ext.critical,
|
||||
});
|
||||
} else if oid == OID_CRL_NUMBER_RAW {
|
||||
match ext.parsed_extension() {
|
||||
ParsedExtension::CRLNumber(n) => out.push(CrlExtensionParsed::CrlNumber {
|
||||
number: n.clone(),
|
||||
critical: ext.critical,
|
||||
}),
|
||||
_ => return Err("CRLNumber extension parse failed".to_string()),
|
||||
}
|
||||
} else {
|
||||
out.push(CrlExtensionParsed::Other {
|
||||
oid: ext.oid.to_id_string(),
|
||||
critical: ext.critical,
|
||||
})
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn validate_extensions_profile(
|
||||
exts: &[CrlExtensionParsed],
|
||||
) -> Result<CrlExtensions, CrlProfileError> {
|
||||
if exts.len() != 2 {
|
||||
return Err(CrlProfileError::InvalidExtensionsCount(exts.len()));
|
||||
}
|
||||
|
||||
let mut seen: Vec<String> = Vec::new();
|
||||
let mut authority_key_identifier: Option<Vec<u8>> = None;
|
||||
let mut crl_number: Option<BigUnsigned> = None;
|
||||
|
||||
for ext in exts {
|
||||
match ext {
|
||||
CrlExtensionParsed::AuthorityKeyIdentifier {
|
||||
key_identifier,
|
||||
has_other_fields,
|
||||
critical: _,
|
||||
} => {
|
||||
let oid = OID_AUTHORITY_KEY_IDENTIFIER.to_string();
|
||||
if seen.iter().any(|s| s == &oid) {
|
||||
return Err(CrlProfileError::DuplicateExtension(oid));
|
||||
}
|
||||
seen.push(oid.clone());
|
||||
|
||||
if *has_other_fields {
|
||||
return Err(CrlProfileError::AkiHasOtherFields);
|
||||
}
|
||||
let ki = key_identifier
|
||||
.as_ref()
|
||||
.ok_or(CrlProfileError::AkiMissingKeyIdentifier)?;
|
||||
authority_key_identifier = Some(ki.clone());
|
||||
}
|
||||
CrlExtensionParsed::CrlNumber { number, critical } => {
|
||||
let oid = OID_CRL_NUMBER.to_string();
|
||||
if seen.iter().any(|s| s == &oid) {
|
||||
return Err(CrlProfileError::DuplicateExtension(oid));
|
||||
}
|
||||
seen.push(oid.clone());
|
||||
|
||||
if *critical {
|
||||
return Err(CrlProfileError::CrlNumberCritical);
|
||||
}
|
||||
if number.bits() > 159 {
|
||||
return Err(CrlProfileError::CrlNumberOutOfRange);
|
||||
}
|
||||
crl_number = Some(BigUnsigned::from_biguint(number));
|
||||
}
|
||||
CrlExtensionParsed::Other { oid, .. } => {
|
||||
return Err(CrlProfileError::UnsupportedExtension(oid.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(CrlExtensions {
|
||||
authority_key_identifier: authority_key_identifier.ok_or(CrlProfileError::AkiMissing)?,
|
||||
crl_number: crl_number.ok_or(CrlProfileError::CrlNumberMissing)?,
|
||||
})
|
||||
}
|
||||
|
||||
fn get_subject_key_identifier(cert: &X509Certificate<'_>) -> Option<Vec<u8>> {
|
||||
cert.extensions()
|
||||
.iter()
|
||||
.find(|ext| ext.oid.as_bytes() == OID_SUBJECT_KEY_IDENTIFIER_RAW)
|
||||
.and_then(|ext| match ext.parsed_extension() {
|
||||
ParsedExtension::SubjectKeyIdentifier(ki) => Some(ki.0.to_vec()),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
975
crates/panda-rpki-validator/src/data_model/manifest.rs
Normal file
975
crates/panda-rpki-validator/src/data_model/manifest.rs
Normal file
@ -0,0 +1,975 @@
|
||||
use crate::data_model::common::der_take_tlv;
|
||||
use crate::data_model::common::{BigUnsigned, UtcTime};
|
||||
use crate::data_model::oid::{OID_CT_RPKI_MANIFEST, OID_SHA256};
|
||||
use crate::data_model::rc::ResourceCertificate;
|
||||
use crate::data_model::signed_object::{
|
||||
RpkiSignedObject, RpkiSignedObjectParsed, SignedObjectDecodeError, SignedObjectParseError,
|
||||
SignedObjectValidateError,
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ManifestObject {
|
||||
pub signed_object: RpkiSignedObject,
|
||||
pub econtent_type: String,
|
||||
pub manifest: ManifestEContent,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ManifestObjectParsed {
|
||||
pub signed_object: RpkiSignedObjectParsed,
|
||||
pub econtent_type: String,
|
||||
pub manifest: Option<ManifestEContentParsed>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ManifestEContent {
|
||||
pub version: u32,
|
||||
pub manifest_number: BigUnsigned,
|
||||
pub this_update: UtcTime,
|
||||
pub next_update: UtcTime,
|
||||
pub file_hash_alg: String,
|
||||
/// DER-encoded content bytes of `Manifest.fileList` (SEQUENCE OF FileAndHash).
|
||||
pub file_list_der: Vec<u8>,
|
||||
/// Count of FileAndHash entries in `fileList`.
|
||||
pub file_count: usize,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ManifestEContentParsed {
|
||||
der: Vec<u8>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct FileAndHash {
|
||||
pub file_name: String,
|
||||
pub hash_bytes: [u8; 32],
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ManifestParseError {
|
||||
#[error("signed object parse error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
|
||||
SignedObject(#[from] SignedObjectParseError),
|
||||
#[error("DER parse error: {0} (RFC 9286 §4.2; DER)")]
|
||||
Parse(String),
|
||||
|
||||
#[error("trailing bytes after DER object: {0} bytes (RFC 9286 §4.2; DER)")]
|
||||
TrailingBytes(usize),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ManifestProfileError {
|
||||
#[error("signed object profile error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
|
||||
SignedObject(#[from] SignedObjectValidateError),
|
||||
|
||||
#[error(
|
||||
"eContentType must be id-ct-rpkiManifest ({OID_CT_RPKI_MANIFEST}), got {0} (RFC 9286 §4.1; RFC 9286 §4.4(1))"
|
||||
)]
|
||||
InvalidEContentType(String),
|
||||
|
||||
#[error("manifest profile decode error: {0} (RFC 9286 §4.2; DER)")]
|
||||
ProfileDecode(String),
|
||||
|
||||
#[error("Manifest must be a SEQUENCE of 5 or 6 elements, got {0} (RFC 9286 §4.2)")]
|
||||
InvalidManifestSequenceLen(usize),
|
||||
|
||||
#[error("Manifest.version must be 0, got {0} (RFC 9286 §4.2.1)")]
|
||||
InvalidManifestVersion(u64),
|
||||
|
||||
#[error(
|
||||
"Manifest.manifestNumber must be non-negative INTEGER (RFC 9286 §4.2; RFC 9286 §4.2.1)"
|
||||
)]
|
||||
InvalidManifestNumber,
|
||||
|
||||
#[error("Manifest.manifestNumber longer than 20 octets (RFC 9286 §4.2.1)")]
|
||||
ManifestNumberTooLong,
|
||||
|
||||
#[error("Manifest.thisUpdate must be GeneralizedTime (RFC 9286 §4.2)")]
|
||||
InvalidThisUpdate,
|
||||
|
||||
#[error("Manifest.nextUpdate must be GeneralizedTime (RFC 9286 §4.2)")]
|
||||
InvalidNextUpdate,
|
||||
|
||||
#[error("Manifest.nextUpdate must be later than thisUpdate (RFC 9286 §4.2.1)")]
|
||||
NextUpdateNotLater,
|
||||
|
||||
#[error(
|
||||
"Manifest.fileHashAlg must be id-sha256 ({OID_SHA256}), got {0} (RFC 9286 §4.2.1; RFC 7935 §2)"
|
||||
)]
|
||||
InvalidFileHashAlg(String),
|
||||
|
||||
#[error("Manifest.fileList must be a SEQUENCE (RFC 9286 §4.2)")]
|
||||
InvalidFileList,
|
||||
|
||||
#[error("FileAndHash must be SEQUENCE of 2 (RFC 9286 §4.2)")]
|
||||
InvalidFileAndHash,
|
||||
|
||||
#[error("fileList file name invalid: {0} (RFC 9286 §4.2.2)")]
|
||||
InvalidFileName(String),
|
||||
|
||||
#[error("fileList hash must be BIT STRING (RFC 9286 §4.2)")]
|
||||
InvalidHashType,
|
||||
|
||||
#[error(
|
||||
"fileList hash BIT STRING must be octet-aligned (unused bits=0) (RFC 9286 §4.2.1; DER BIT STRING)"
|
||||
)]
|
||||
HashNotOctetAligned,
|
||||
|
||||
#[error(
|
||||
"fileList hash length invalid for sha256: got {0} bytes (RFC 9286 §4.2.1; RFC 7935 §2)"
|
||||
)]
|
||||
InvalidHashLength(usize),
|
||||
}
|
||||
|
||||
impl From<SignedObjectDecodeError> for ManifestProfileError {
|
||||
fn from(value: SignedObjectDecodeError) -> Self {
|
||||
match value {
|
||||
SignedObjectDecodeError::Parse(e) => ManifestProfileError::ProfileDecode(e.to_string()),
|
||||
SignedObjectDecodeError::Validate(e) => ManifestProfileError::SignedObject(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ManifestDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] ManifestParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] ManifestProfileError),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ManifestValidateError {
|
||||
#[error(
|
||||
"Manifest EE certificate MUST include at least one RFC 3779 resource extension (IP or AS) (RFC 6487 §4.8.10-§4.8.11; RFC 3779; RFC 9286 §5.1)"
|
||||
)]
|
||||
EeResourcesMissing,
|
||||
|
||||
#[error(
|
||||
"Manifest EE certificate IP resources MUST use inherit only (RFC 9286 §5.1; RFC 3779 §2.2.3.5)"
|
||||
)]
|
||||
EeIpResourcesNotInherit,
|
||||
|
||||
#[error(
|
||||
"Manifest EE certificate AS resources MUST use inherit only (RFC 9286 §5.1; RFC 3779 §3.2.3.3)"
|
||||
)]
|
||||
EeAsResourcesNotInherit,
|
||||
|
||||
#[error(
|
||||
"Manifest EE certificate AS resources rdi MUST be absent (RFC 6487 §4.8.11; RFC 3779 §3.2.3.5)"
|
||||
)]
|
||||
EeAsResourcesRdiPresent,
|
||||
}
|
||||
|
||||
impl ManifestObject {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<ManifestObjectParsed, ManifestParseError> {
|
||||
let signed_object = RpkiSignedObject::parse_der(der)?;
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
let manifest = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent
|
||||
.as_deref()
|
||||
.map(ManifestEContent::parse_der)
|
||||
.transpose()?;
|
||||
Ok(ManifestObjectParsed {
|
||||
signed_object,
|
||||
econtent_type,
|
||||
manifest,
|
||||
})
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `ManifestObject` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `ManifestObjectParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), ManifestProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, ManifestDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
pub fn decode_der_with_strict_options(
|
||||
der: &[u8],
|
||||
strict_cms_der: bool,
|
||||
strict_name: bool,
|
||||
) -> Result<Self, ManifestDecodeError> {
|
||||
let signed_object =
|
||||
RpkiSignedObject::decode_der_with_strict_options(der, strict_cms_der, strict_name)
|
||||
.map_err(ManifestProfileError::from)?;
|
||||
Self::from_signed_object(signed_object)
|
||||
}
|
||||
|
||||
pub fn from_signed_object(
|
||||
signed_object: RpkiSignedObject,
|
||||
) -> Result<Self, ManifestDecodeError> {
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
if econtent_type != OID_CT_RPKI_MANIFEST {
|
||||
return Err(ManifestProfileError::InvalidEContentType(econtent_type).into());
|
||||
}
|
||||
let manifest =
|
||||
ManifestEContent::decode_der(&signed_object.signed_data.encap_content_info.econtent)?;
|
||||
Ok(Self {
|
||||
signed_object,
|
||||
econtent_type: OID_CT_RPKI_MANIFEST.to_string(),
|
||||
manifest,
|
||||
})
|
||||
}
|
||||
|
||||
/// Validate the embedded EE certificate resources against RFC 9286 §5.1.
|
||||
///
|
||||
/// This does **not** perform certificate path validation. It assumes `ee` is a parsed and
|
||||
/// profile-validated RPKI EE resource certificate.
|
||||
pub fn validate_against_ee_cert(
|
||||
&self,
|
||||
ee: &ResourceCertificate,
|
||||
) -> Result<(), ManifestValidateError> {
|
||||
let ip = ee.tbs.extensions.ip_resources.as_ref();
|
||||
let asn = ee.tbs.extensions.as_resources.as_ref();
|
||||
if ip.is_none() && asn.is_none() {
|
||||
return Err(ManifestValidateError::EeResourcesMissing);
|
||||
}
|
||||
|
||||
if let Some(ip) = ip {
|
||||
if !ip.is_all_inherit() {
|
||||
return Err(ManifestValidateError::EeIpResourcesNotInherit);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(asn) = asn {
|
||||
if asn.rdi.is_some() {
|
||||
return Err(ManifestValidateError::EeAsResourcesRdiPresent);
|
||||
}
|
||||
if !asn.is_asnum_inherit() {
|
||||
return Err(ManifestValidateError::EeAsResourcesNotInherit);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate this manifest's embedded EE certificate resources.
|
||||
pub fn validate_embedded_ee_cert(&self) -> Result<(), ManifestValidateError> {
|
||||
let ee = &self.signed_object.signed_data.certificates[0].resource_cert;
|
||||
self.validate_against_ee_cert(ee)
|
||||
}
|
||||
}
|
||||
|
||||
impl ManifestEContent {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<ManifestEContentParsed, ManifestParseError> {
|
||||
let (_tag, _value, rem) = der_take_tlv(der).map_err(|e| ManifestParseError::Parse(e))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(ManifestParseError::TrailingBytes(rem.len()));
|
||||
}
|
||||
Ok(ManifestEContentParsed { der: der.to_vec() })
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `ManifestEContent` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `ManifestEContentParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), ManifestProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Decode the DER-encoded Manifest eContent defined in RFC 9286 §4.2 (`parse + validate`).
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, ManifestDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
/// Parse and return the manifest fileList.
|
||||
///
|
||||
/// Note: `ManifestEContent` is profile-validated when produced via `decode_der()`, so this
|
||||
/// should only fail due to internal inconsistencies (or if constructed manually).
|
||||
pub fn parse_files(&self) -> Result<Vec<FileAndHash>, ManifestProfileError> {
|
||||
parse_file_list_sha256_fast(&self.file_list_der)
|
||||
}
|
||||
|
||||
pub fn file_count(&self) -> usize {
|
||||
self.file_count
|
||||
}
|
||||
}
|
||||
|
||||
impl ManifestObjectParsed {
|
||||
pub fn validate_profile(self) -> Result<ManifestObject, ManifestProfileError> {
|
||||
let signed_object = self.signed_object.validate_profile()?;
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
if econtent_type != OID_CT_RPKI_MANIFEST {
|
||||
return Err(ManifestProfileError::InvalidEContentType(econtent_type));
|
||||
}
|
||||
let manifest = self
|
||||
.manifest
|
||||
.ok_or_else(|| ManifestProfileError::ProfileDecode("Manifest.eContent missing".into()))?
|
||||
.validate_profile()?;
|
||||
Ok(ManifestObject {
|
||||
signed_object,
|
||||
econtent_type: OID_CT_RPKI_MANIFEST.to_string(),
|
||||
manifest,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl ManifestEContentParsed {
|
||||
pub fn validate_profile(self) -> Result<ManifestEContent, ManifestProfileError> {
|
||||
decode_manifest_econtent_fast(&self.der)
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_file_name_bytes(bytes: &[u8]) -> Result<(), ManifestProfileError> {
|
||||
// RFC 9286 §4.2.2:
|
||||
// 1+ chars from a-zA-Z0-9-_ , then '.', then 3-letter extension.
|
||||
if bytes.len() < 5 {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
};
|
||||
|
||||
// "followed by a single . (DOT), followed by a three letter extension"
|
||||
// -> the dot must be exactly 4 bytes from the end.
|
||||
let dot_pos = bytes.len() - 4;
|
||||
if bytes[dot_pos] != b'.' {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
}
|
||||
|
||||
#[inline(always)]
|
||||
fn valid_base_char(b: u8) -> bool {
|
||||
// RFC 9286 allowed set: a-zA-Z0-9-_
|
||||
(b'0'..=b'9').contains(&b)
|
||||
|| (b'a'..=b'z').contains(&b)
|
||||
|| (b'A'..=b'Z').contains(&b)
|
||||
|| b == b'-'
|
||||
|| b == b'_'
|
||||
}
|
||||
|
||||
for &b in &bytes[..dot_pos] {
|
||||
if (b & 0x80) != 0 || !valid_base_char(b) {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
let e0 = bytes[dot_pos + 1];
|
||||
let e1 = bytes[dot_pos + 2];
|
||||
let e2 = bytes[dot_pos + 3];
|
||||
if (e0 & 0x80) != 0 || (e1 & 0x80) != 0 || (e2 & 0x80) != 0 {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
}
|
||||
|
||||
#[inline(always)]
|
||||
fn lower_if_alpha(b: u8) -> Option<u8> {
|
||||
match b {
|
||||
b'a'..=b'z' => Some(b),
|
||||
b'A'..=b'Z' => Some(b + 32),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
let Some(l0) = lower_if_alpha(e0) else {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
};
|
||||
let Some(l1) = lower_if_alpha(e1) else {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
};
|
||||
let Some(l2) = lower_if_alpha(e2) else {
|
||||
return Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
));
|
||||
};
|
||||
|
||||
match [l0, l1, l2] {
|
||||
// Full IANA list (see `common.rs`).
|
||||
[b'a', b's', b'a']
|
||||
| [b'c', b'e', b'r']
|
||||
| [b'c', b'r', b'l']
|
||||
| [b'g', b'b', b'r']
|
||||
| [b'm', b'f', b't']
|
||||
| [b'r', b'o', b'a']
|
||||
| [b's', b'i', b'g']
|
||||
| [b't', b'a', b'k'] => Ok(()),
|
||||
_ => Err(ManifestProfileError::InvalidFileName(
|
||||
String::from_utf8_lossy(bytes).into_owned(),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn decode_manifest_econtent_fast(der: &[u8]) -> Result<ManifestEContent, ManifestProfileError> {
|
||||
let (tag, mut seq_content, rem) = der_take_tlv(der)
|
||||
.map_err(|e| ManifestProfileError::ProfileDecode(format!("DER decode error: {e}")))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(ManifestProfileError::ProfileDecode(format!(
|
||||
"trailing bytes after DER object: {} bytes",
|
||||
rem.len()
|
||||
)));
|
||||
}
|
||||
if tag != 0x30 {
|
||||
return Err(ManifestProfileError::ProfileDecode(
|
||||
"Manifest eContent must be SEQUENCE".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let seq_len =
|
||||
der_count_elements(seq_content).map_err(|e| ManifestProfileError::ProfileDecode(e))?;
|
||||
if seq_len != 5 && seq_len != 6 {
|
||||
return Err(ManifestProfileError::InvalidManifestSequenceLen(seq_len));
|
||||
}
|
||||
|
||||
let mut version: u32 = 0;
|
||||
if seq_len == 6 {
|
||||
let Some(&first_tag) = seq_content.first() else {
|
||||
return Err(ManifestProfileError::InvalidManifestSequenceLen(0));
|
||||
};
|
||||
if first_tag != 0xA0 {
|
||||
return Err(ManifestProfileError::ProfileDecode(
|
||||
"Manifest.version must be [0] EXPLICIT INTEGER".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let (_cs_tag, cs_value, after) = der_take_tlv(seq_content).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.version decode error: {e}"))
|
||||
})?;
|
||||
seq_content = after;
|
||||
|
||||
let (inner_tag, inner_value, inner_rem) = der_take_tlv(cs_value).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.version inner decode error: {e}"))
|
||||
})?;
|
||||
if !inner_rem.is_empty() {
|
||||
return Err(ManifestProfileError::ProfileDecode(
|
||||
"trailing bytes inside Manifest.version".into(),
|
||||
));
|
||||
}
|
||||
if inner_tag != 0x02 {
|
||||
return Err(ManifestProfileError::ProfileDecode(
|
||||
"Manifest.version must be [0] EXPLICIT INTEGER".into(),
|
||||
));
|
||||
}
|
||||
let v = der_integer_to_u64(inner_value).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.version decode error: {e}"))
|
||||
})?;
|
||||
if v != 0 {
|
||||
return Err(ManifestProfileError::InvalidManifestVersion(v));
|
||||
}
|
||||
version = 0;
|
||||
}
|
||||
|
||||
let (mn_tag, mn_value, after) = der_take_tlv(seq_content).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.manifestNumber decode error: {e}"))
|
||||
})?;
|
||||
seq_content = after;
|
||||
if mn_tag != 0x02 {
|
||||
return Err(ManifestProfileError::InvalidManifestNumber);
|
||||
}
|
||||
let manifest_number = der_integer_to_bigunsigned(mn_value)?;
|
||||
|
||||
let (tu_tag, tu_value, after) = der_take_tlv(seq_content).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.thisUpdate decode error: {e}"))
|
||||
})?;
|
||||
seq_content = after;
|
||||
if tu_tag != 0x18 {
|
||||
return Err(ManifestProfileError::InvalidThisUpdate);
|
||||
}
|
||||
let this_update = parse_generalized_time_bytes(tu_value)
|
||||
.map_err(|e| ManifestProfileError::ProfileDecode(e))?;
|
||||
|
||||
let (nu_tag, nu_value, after) = der_take_tlv(seq_content).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.nextUpdate decode error: {e}"))
|
||||
})?;
|
||||
seq_content = after;
|
||||
if nu_tag != 0x18 {
|
||||
return Err(ManifestProfileError::InvalidNextUpdate);
|
||||
}
|
||||
let next_update = parse_generalized_time_bytes(nu_value)
|
||||
.map_err(|e| ManifestProfileError::ProfileDecode(e))?;
|
||||
if next_update <= this_update {
|
||||
return Err(ManifestProfileError::NextUpdateNotLater);
|
||||
}
|
||||
|
||||
let (oid_tag, oid_value, after) = der_take_tlv(seq_content).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.fileHashAlg decode error: {e}"))
|
||||
})?;
|
||||
seq_content = after;
|
||||
if oid_tag != 0x06 {
|
||||
return Err(ManifestProfileError::ProfileDecode(
|
||||
"Manifest.fileHashAlg must be OBJECT IDENTIFIER".into(),
|
||||
));
|
||||
}
|
||||
if !oid_content_is_sha256(oid_value) {
|
||||
return Err(ManifestProfileError::InvalidFileHashAlg(
|
||||
oid_content_to_string(oid_value),
|
||||
));
|
||||
}
|
||||
|
||||
let (fl_tag, fl_value, after) = der_take_tlv(seq_content).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("Manifest.fileList decode error: {e}"))
|
||||
})?;
|
||||
seq_content = after;
|
||||
if fl_tag != 0x30 {
|
||||
return Err(ManifestProfileError::InvalidFileList);
|
||||
}
|
||||
let file_count = validate_file_list_sha256_fast(fl_value)?;
|
||||
let file_list_der = fl_value.to_vec();
|
||||
|
||||
if !seq_content.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidManifestSequenceLen(seq_len));
|
||||
}
|
||||
|
||||
Ok(ManifestEContent {
|
||||
version,
|
||||
manifest_number,
|
||||
this_update,
|
||||
next_update,
|
||||
file_hash_alg: OID_SHA256.to_string(),
|
||||
file_list_der,
|
||||
file_count,
|
||||
})
|
||||
}
|
||||
|
||||
fn validate_file_list_sha256_fast(content: &[u8]) -> Result<usize, ManifestProfileError> {
|
||||
let mut cur = content;
|
||||
let mut count: usize = 0;
|
||||
while !cur.is_empty() {
|
||||
let (tag, value, rem) = der_take_tlv(cur).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("fileList entry decode error: {e}"))
|
||||
})?;
|
||||
cur = rem;
|
||||
if tag != 0x30 {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
|
||||
let mut entry = value;
|
||||
let (fn_tag, fn_value, entry_rem) = der_take_tlv(entry).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("fileList fileName decode error: {e}"))
|
||||
})?;
|
||||
entry = entry_rem;
|
||||
if fn_tag != 0x16 {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
if entry.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
validate_file_name_bytes(fn_value)?;
|
||||
|
||||
let (hash_tag, hash_value, entry_rem) = der_take_tlv(entry).map_err(|_e| {
|
||||
// Missing second element should map to "SEQUENCE of 2" shape error.
|
||||
ManifestProfileError::InvalidFileAndHash
|
||||
})?;
|
||||
entry = entry_rem;
|
||||
if !entry.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
if hash_tag != 0x03 {
|
||||
return Err(ManifestProfileError::InvalidHashType);
|
||||
}
|
||||
if hash_value.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidHashLength(0));
|
||||
}
|
||||
let unused_bits = hash_value[0];
|
||||
if unused_bits != 0 {
|
||||
return Err(ManifestProfileError::HashNotOctetAligned);
|
||||
}
|
||||
let bits = &hash_value[1..];
|
||||
if bits.len() != 32 {
|
||||
return Err(ManifestProfileError::InvalidHashLength(bits.len()));
|
||||
}
|
||||
count += 1;
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
fn parse_file_list_sha256_fast(content: &[u8]) -> Result<Vec<FileAndHash>, ManifestProfileError> {
|
||||
// Heuristic initial capacity (avoid a full pre-scan, which is expensive for xlarge manifests).
|
||||
// Each FileAndHash entry is typically tens of bytes; 80 is a conservative average.
|
||||
let est = (content.len() / 80).clamp(16, 4096);
|
||||
let mut cur = content;
|
||||
let mut out: Vec<FileAndHash> = Vec::with_capacity(est);
|
||||
while !cur.is_empty() {
|
||||
let (tag, value, rem) = der_take_tlv(cur).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("fileList entry decode error: {e}"))
|
||||
})?;
|
||||
cur = rem;
|
||||
if tag != 0x30 {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
let mut entry = value;
|
||||
let (fn_tag, fn_value, entry_rem) = der_take_tlv(entry).map_err(|e| {
|
||||
ManifestProfileError::ProfileDecode(format!("fileList fileName decode error: {e}"))
|
||||
})?;
|
||||
entry = entry_rem;
|
||||
if fn_tag != 0x16 {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
if entry.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
let file_name = validate_and_copy_file_name(fn_value)?;
|
||||
|
||||
let (hash_tag, hash_value, entry_rem) = der_take_tlv(entry).map_err(|_e| {
|
||||
// Missing second element should map to "SEQUENCE of 2" shape error.
|
||||
ManifestProfileError::InvalidFileAndHash
|
||||
})?;
|
||||
entry = entry_rem;
|
||||
if !entry.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidFileAndHash);
|
||||
}
|
||||
if hash_tag != 0x03 {
|
||||
return Err(ManifestProfileError::InvalidHashType);
|
||||
}
|
||||
if hash_value.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidHashLength(0));
|
||||
}
|
||||
let unused_bits = hash_value[0];
|
||||
if unused_bits != 0 {
|
||||
return Err(ManifestProfileError::HashNotOctetAligned);
|
||||
}
|
||||
let bits = &hash_value[1..];
|
||||
if bits.len() != 32 {
|
||||
return Err(ManifestProfileError::InvalidHashLength(bits.len()));
|
||||
}
|
||||
let mut hash_bytes = [0u8; 32];
|
||||
hash_bytes.copy_from_slice(bits);
|
||||
out.push(FileAndHash {
|
||||
file_name,
|
||||
hash_bytes,
|
||||
});
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn validate_and_copy_file_name(bytes: &[u8]) -> Result<String, ManifestProfileError> {
|
||||
validate_file_name_bytes(bytes)?;
|
||||
Ok(unsafe { String::from_utf8_unchecked(bytes.to_vec()) })
|
||||
}
|
||||
|
||||
fn der_count_elements(mut input: &[u8]) -> Result<usize, String> {
|
||||
let mut count: usize = 0;
|
||||
while !input.is_empty() {
|
||||
let (_tag, _value, rem) = der_take_tlv(input)?;
|
||||
input = rem;
|
||||
count += 1;
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
fn der_integer_to_u64(bytes: &[u8]) -> Result<u64, String> {
|
||||
if bytes.is_empty() {
|
||||
return Err("INTEGER empty".into());
|
||||
}
|
||||
// Reject negative (two's complement).
|
||||
if bytes[0] & 0x80 != 0 {
|
||||
return Err("INTEGER is negative".into());
|
||||
}
|
||||
if bytes.len() > 8 {
|
||||
return Err("INTEGER too large".into());
|
||||
}
|
||||
let mut v: u64 = 0;
|
||||
for &b in bytes {
|
||||
v = (v << 8) | (b as u64);
|
||||
}
|
||||
Ok(v)
|
||||
}
|
||||
|
||||
fn der_integer_to_bigunsigned(bytes: &[u8]) -> Result<BigUnsigned, ManifestProfileError> {
|
||||
if bytes.is_empty() {
|
||||
return Err(ManifestProfileError::InvalidManifestNumber);
|
||||
}
|
||||
// Two's complement: for non-negative values, a leading 0x00 may be present.
|
||||
if bytes[0] & 0x80 != 0 {
|
||||
return Err(ManifestProfileError::InvalidManifestNumber);
|
||||
}
|
||||
let mut start = 0usize;
|
||||
while start + 1 < bytes.len() && bytes[start] == 0 {
|
||||
start += 1;
|
||||
}
|
||||
let mut minimal = bytes[start..].to_vec();
|
||||
if minimal.is_empty() {
|
||||
minimal.push(0);
|
||||
}
|
||||
if minimal.len() > 20 {
|
||||
return Err(ManifestProfileError::ManifestNumberTooLong);
|
||||
}
|
||||
Ok(BigUnsigned { bytes_be: minimal })
|
||||
}
|
||||
|
||||
fn parse_generalized_time_bytes(bytes: &[u8]) -> Result<UtcTime, String> {
|
||||
// Accept "YYYYMMDDHHMMSSZ" and also allow optional fractional seconds (".fff...Z").
|
||||
if !bytes.is_ascii() {
|
||||
return Err("GeneralizedTime not ASCII".into());
|
||||
}
|
||||
let s = std::str::from_utf8(bytes).map_err(|e| e.to_string())?;
|
||||
if !s.ends_with('Z') {
|
||||
return Err("GeneralizedTime must end with 'Z'".into());
|
||||
}
|
||||
let core = &s[..s.len() - 1];
|
||||
let (main, frac) = core
|
||||
.split_once('.')
|
||||
.map_or((core, None), |(a, b)| (a, Some(b)));
|
||||
if main.len() != 14 || !main.bytes().all(|b| b.is_ascii_digit()) {
|
||||
return Err("GeneralizedTime must be YYYYMMDDHHMMSS[.fff]Z".into());
|
||||
}
|
||||
let year: i32 = main[0..4].parse().map_err(|_| "bad year")?;
|
||||
let month: u8 = main[4..6].parse().map_err(|_| "bad month")?;
|
||||
let day: u8 = main[6..8].parse().map_err(|_| "bad day")?;
|
||||
let hour: u8 = main[8..10].parse().map_err(|_| "bad hour")?;
|
||||
let minute: u8 = main[10..12].parse().map_err(|_| "bad minute")?;
|
||||
let second: u8 = main[12..14].parse().map_err(|_| "bad second")?;
|
||||
|
||||
let nanosecond: u32 = if let Some(frac) = frac {
|
||||
if frac.is_empty() || !frac.bytes().all(|b| b.is_ascii_digit()) {
|
||||
return Err("bad fractional seconds".into());
|
||||
}
|
||||
let mut ns: u32 = 0;
|
||||
let mut scale: u32 = 1_000_000_000;
|
||||
for (i, ch) in frac.bytes().enumerate() {
|
||||
if i >= 9 {
|
||||
break;
|
||||
}
|
||||
scale /= 10;
|
||||
ns += ((ch - b'0') as u32) * scale;
|
||||
}
|
||||
ns
|
||||
} else {
|
||||
0
|
||||
};
|
||||
|
||||
let date = time::Date::from_calendar_date(
|
||||
year,
|
||||
time::Month::try_from(month).map_err(|_| "bad month")?,
|
||||
day,
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
let t =
|
||||
time::Time::from_hms_nano(hour, minute, second, nanosecond).map_err(|e| e.to_string())?;
|
||||
Ok(date.with_time(t).assume_utc())
|
||||
}
|
||||
|
||||
fn oid_content_is_sha256(bytes: &[u8]) -> bool {
|
||||
// 2.16.840.1.101.3.4.2.1
|
||||
let mut arcs = oid_content_iter(bytes);
|
||||
const EXPECTED: &[u64] = &[2, 16, 840, 1, 101, 3, 4, 2, 1];
|
||||
for &e in EXPECTED {
|
||||
match arcs.next() {
|
||||
Some(v) if v == e => {}
|
||||
_ => return false,
|
||||
}
|
||||
}
|
||||
arcs.next().is_none()
|
||||
}
|
||||
|
||||
fn oid_content_to_string(bytes: &[u8]) -> String {
|
||||
let arcs: Vec<u64> = oid_content_iter(bytes).collect();
|
||||
if arcs.is_empty() {
|
||||
return "<invalid oid>".to_string();
|
||||
}
|
||||
let mut s = String::new();
|
||||
for (i, a) in arcs.iter().enumerate() {
|
||||
if i > 0 {
|
||||
s.push('.');
|
||||
}
|
||||
s.push_str(&a.to_string());
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
fn oid_content_iter(bytes: &[u8]) -> impl Iterator<Item = u64> + '_ {
|
||||
struct It<'a> {
|
||||
bytes: &'a [u8],
|
||||
pos: usize,
|
||||
first_done: bool,
|
||||
first_a0: u64,
|
||||
first_a1: u64,
|
||||
emit_first_idx: u8,
|
||||
}
|
||||
impl<'a> Iterator for It<'a> {
|
||||
type Item = u64;
|
||||
fn next(&mut self) -> Option<u64> {
|
||||
if !self.first_done {
|
||||
if self.bytes.is_empty() {
|
||||
self.first_done = true;
|
||||
return None;
|
||||
}
|
||||
let first = self.bytes[0] as u64;
|
||||
self.first_a0 = first / 40;
|
||||
self.first_a1 = first % 40;
|
||||
self.pos = 1;
|
||||
self.first_done = true;
|
||||
self.emit_first_idx = 0;
|
||||
}
|
||||
if self.emit_first_idx == 0 {
|
||||
self.emit_first_idx = 1;
|
||||
return Some(self.first_a0);
|
||||
}
|
||||
if self.emit_first_idx == 1 {
|
||||
self.emit_first_idx = 2;
|
||||
return Some(self.first_a1);
|
||||
}
|
||||
if self.pos >= self.bytes.len() {
|
||||
return None;
|
||||
}
|
||||
let mut v: u64 = 0;
|
||||
while self.pos < self.bytes.len() {
|
||||
let b = self.bytes[self.pos];
|
||||
self.pos += 1;
|
||||
v = (v << 7) | ((b & 0x7F) as u64);
|
||||
if b & 0x80 == 0 {
|
||||
return Some(v);
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
It {
|
||||
bytes,
|
||||
pos: 0,
|
||||
first_done: false,
|
||||
first_a0: 0,
|
||||
first_a1: 0,
|
||||
emit_first_idx: 0,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn tlv(tag: u8, value: &[u8]) -> Vec<u8> {
|
||||
assert!(value.len() < 128);
|
||||
let mut out = Vec::with_capacity(2 + value.len());
|
||||
out.push(tag);
|
||||
out.push(value.len() as u8);
|
||||
out.extend_from_slice(value);
|
||||
out
|
||||
}
|
||||
|
||||
fn tlv_long_len(tag: u8, len_bytes: &[u8], value: &[u8]) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(2 + len_bytes.len() + value.len());
|
||||
out.push(tag);
|
||||
out.push(0x80 | (len_bytes.len() as u8));
|
||||
out.extend_from_slice(len_bytes);
|
||||
out.extend_from_slice(value);
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn der_take_tlv_supports_short_and_long_form_lengths_and_errors() {
|
||||
let v = b"abc";
|
||||
let der = tlv(0x04, v);
|
||||
let (tag, val, rem) = der_take_tlv(&der).expect("short len");
|
||||
assert_eq!(tag, 0x04);
|
||||
assert_eq!(val, v);
|
||||
assert!(rem.is_empty());
|
||||
|
||||
// Long-form length with 1 length byte (130).
|
||||
let v = vec![b'x'; 130];
|
||||
let der = tlv_long_len(0x04, &[0x82], &v);
|
||||
let (tag, val, rem) = der_take_tlv(&der).expect("long len 1");
|
||||
assert_eq!(tag, 0x04);
|
||||
assert_eq!(val.len(), 130);
|
||||
assert!(rem.is_empty());
|
||||
|
||||
// Long-form length with 2 length bytes (256).
|
||||
let v = vec![b'y'; 256];
|
||||
let der = tlv_long_len(0x04, &[0x01, 0x00], &v);
|
||||
let (tag, val, rem) = der_take_tlv(&der).expect("long len 2");
|
||||
assert_eq!(tag, 0x04);
|
||||
assert_eq!(val.len(), 256);
|
||||
assert!(rem.is_empty());
|
||||
|
||||
assert!(der_take_tlv(&[]).is_err());
|
||||
assert!(der_take_tlv(&[0x04]).is_err());
|
||||
|
||||
// High-tag-number form not supported.
|
||||
assert!(der_take_tlv(&[0x1F, 0x01, 0x00]).is_err());
|
||||
|
||||
// Indefinite length is not allowed in DER.
|
||||
assert!(der_take_tlv(&[0x04, 0x80]).is_err());
|
||||
|
||||
// Invalid long-form length encoding.
|
||||
assert!(der_take_tlv(&[0x04, 0x81]).is_err());
|
||||
assert!(der_take_tlv(&[0x04, 0x89]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalized_time_bytes_accepts_fraction_and_rejects_invalid() {
|
||||
let t = parse_generalized_time_bytes(b"20260101000000Z").expect("basic time");
|
||||
assert_eq!(t.year(), 2026);
|
||||
|
||||
let t = parse_generalized_time_bytes(b"20260101000000.1Z").expect("fractional");
|
||||
assert_eq!(t.nanosecond(), 100_000_000);
|
||||
|
||||
assert!(parse_generalized_time_bytes(b"20260101000000").is_err());
|
||||
assert!(parse_generalized_time_bytes(b"20260101000000+00").is_err());
|
||||
assert!(parse_generalized_time_bytes(b"2026010100000Z").is_err());
|
||||
assert!(parse_generalized_time_bytes(b"20261301000000Z").is_err());
|
||||
assert!(parse_generalized_time_bytes(b"20260132000000Z").is_err());
|
||||
assert!(parse_generalized_time_bytes(&[0xFF]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn oid_helpers_accept_sha256_and_format_invalid() {
|
||||
// 2.16.840.1.101.3.4.2.1
|
||||
let sha256_oid_content = [0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01];
|
||||
assert!(oid_content_is_sha256(&sha256_oid_content));
|
||||
assert!(!oid_content_is_sha256(&[0x55, 0x04, 0x03])); // 2.5.4.3
|
||||
|
||||
assert_eq!(oid_content_to_string(&[]), "<invalid oid>".to_string());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_file_list_sha256_fast_counts_and_rejects_bad_hash() {
|
||||
fn file_and_hash(file: &str, digest: u8) -> Vec<u8> {
|
||||
let mut hash = vec![0u8; 33];
|
||||
hash[0] = 0; // unused bits
|
||||
for b in &mut hash[1..] {
|
||||
*b = digest;
|
||||
}
|
||||
let ia5 = tlv(0x16, file.as_bytes());
|
||||
let bit = tlv(0x03, &hash);
|
||||
let mut entry = Vec::new();
|
||||
entry.extend_from_slice(&ia5);
|
||||
entry.extend_from_slice(&bit);
|
||||
tlv(0x30, &entry)
|
||||
}
|
||||
|
||||
let mut list = Vec::new();
|
||||
list.extend_from_slice(&file_and_hash("A.cer", 0xAA));
|
||||
list.extend_from_slice(&file_and_hash("B.roa", 0xBB));
|
||||
assert_eq!(validate_file_list_sha256_fast(&list).expect("count"), 2);
|
||||
|
||||
// Wrong hash length.
|
||||
let mut bad = Vec::new();
|
||||
let ia5 = tlv(0x16, b"A.cer");
|
||||
let bit = tlv(0x03, &[0u8; 2]); // too short
|
||||
let mut entry = Vec::new();
|
||||
entry.extend_from_slice(&ia5);
|
||||
entry.extend_from_slice(&bit);
|
||||
bad.extend_from_slice(&tlv(0x30, &entry));
|
||||
assert!(matches!(
|
||||
validate_file_list_sha256_fast(&bad),
|
||||
Err(ManifestProfileError::InvalidHashLength(_))
|
||||
));
|
||||
}
|
||||
}
|
||||
13
crates/panda-rpki-validator/src/data_model/mod.rs
Normal file
13
crates/panda-rpki-validator/src/data_model/mod.rs
Normal file
@ -0,0 +1,13 @@
|
||||
pub mod aspa;
|
||||
pub mod common;
|
||||
pub mod crl;
|
||||
pub mod manifest;
|
||||
pub mod oid;
|
||||
pub mod rc;
|
||||
pub mod roa;
|
||||
pub mod signed_object;
|
||||
pub mod ta;
|
||||
pub mod tal;
|
||||
|
||||
#[cfg(feature = "full")]
|
||||
pub mod router_cert;
|
||||
78
crates/panda-rpki-validator/src/data_model/oid.rs
Normal file
78
crates/panda-rpki-validator/src/data_model/oid.rs
Normal file
@ -0,0 +1,78 @@
|
||||
pub const OID_SHA256: &str = "2.16.840.1.101.3.4.2.1";
|
||||
pub const OID_SHA256_RAW: &[u8] = &asn1_rs::oid!(raw 2.16.840.1.101.3.4.2.1);
|
||||
|
||||
pub const OID_SIGNED_DATA: &str = "1.2.840.113549.1.7.2";
|
||||
pub const OID_SIGNED_DATA_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.7.2);
|
||||
|
||||
pub const OID_CMS_ATTR_CONTENT_TYPE: &str = "1.2.840.113549.1.9.3";
|
||||
pub const OID_CMS_ATTR_CONTENT_TYPE_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.3);
|
||||
pub const OID_CMS_ATTR_MESSAGE_DIGEST: &str = "1.2.840.113549.1.9.4";
|
||||
pub const OID_CMS_ATTR_MESSAGE_DIGEST_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.4);
|
||||
pub const OID_CMS_ATTR_SIGNING_TIME: &str = "1.2.840.113549.1.9.5";
|
||||
pub const OID_CMS_ATTR_SIGNING_TIME_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.5);
|
||||
|
||||
pub const OID_RSA_ENCRYPTION: &str = "1.2.840.113549.1.1.1";
|
||||
pub const OID_RSA_ENCRYPTION_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.1.1);
|
||||
pub const OID_SHA256_WITH_RSA_ENCRYPTION: &str = "1.2.840.113549.1.1.11";
|
||||
pub const OID_SHA256_WITH_RSA_ENCRYPTION_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.1.11);
|
||||
|
||||
// X.509 extensions (RFC 5280 / RFC 6487)
|
||||
pub const OID_BASIC_CONSTRAINTS: &str = "2.5.29.19";
|
||||
pub const OID_BASIC_CONSTRAINTS_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.19);
|
||||
pub const OID_KEY_USAGE: &str = "2.5.29.15";
|
||||
pub const OID_KEY_USAGE_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.15);
|
||||
pub const OID_EXTENDED_KEY_USAGE: &str = "2.5.29.37";
|
||||
pub const OID_EXTENDED_KEY_USAGE_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.37);
|
||||
pub const OID_CRL_DISTRIBUTION_POINTS: &str = "2.5.29.31";
|
||||
pub const OID_CRL_DISTRIBUTION_POINTS_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.31);
|
||||
pub const OID_AUTHORITY_INFO_ACCESS: &str = "1.3.6.1.5.5.7.1.1";
|
||||
pub const OID_AUTHORITY_INFO_ACCESS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.1);
|
||||
pub const OID_CERTIFICATE_POLICIES: &str = "2.5.29.32";
|
||||
pub const OID_CERTIFICATE_POLICIES_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.32);
|
||||
pub const OID_QT_CPS: &str = "1.3.6.1.5.5.7.2.1";
|
||||
|
||||
pub const OID_AUTHORITY_KEY_IDENTIFIER: &str = "2.5.29.35";
|
||||
pub const OID_AUTHORITY_KEY_IDENTIFIER_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.35);
|
||||
pub const OID_CRL_NUMBER: &str = "2.5.29.20";
|
||||
pub const OID_CRL_NUMBER_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.20);
|
||||
pub const OID_SUBJECT_KEY_IDENTIFIER: &str = "2.5.29.14";
|
||||
pub const OID_SUBJECT_KEY_IDENTIFIER_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.14);
|
||||
|
||||
pub const OID_CT_RPKI_MANIFEST: &str = "1.2.840.113549.1.9.16.1.26";
|
||||
pub const OID_CT_RPKI_MANIFEST_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.26);
|
||||
pub const OID_CT_ROUTE_ORIGIN_AUTHZ: &str = "1.2.840.113549.1.9.16.1.24";
|
||||
pub const OID_CT_ROUTE_ORIGIN_AUTHZ_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.24);
|
||||
pub const OID_CT_ASPA: &str = "1.2.840.113549.1.9.16.1.49";
|
||||
pub const OID_CT_ASPA_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.49);
|
||||
|
||||
// X.509 extensions / access methods (RFC 5280 / RFC 6487)
|
||||
pub const OID_SUBJECT_INFO_ACCESS: &str = "1.3.6.1.5.5.7.1.11";
|
||||
pub const OID_SUBJECT_INFO_ACCESS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.11);
|
||||
pub const OID_AD_SIGNED_OBJECT: &str = "1.3.6.1.5.5.7.48.11";
|
||||
pub const OID_AD_SIGNED_OBJECT_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.11);
|
||||
|
||||
pub const OID_AD_CA_ISSUERS: &str = "1.3.6.1.5.5.7.48.2";
|
||||
pub const OID_AD_CA_ISSUERS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.2);
|
||||
pub const OID_AD_CA_REPOSITORY: &str = "1.3.6.1.5.5.7.48.5";
|
||||
pub const OID_AD_CA_REPOSITORY_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.5);
|
||||
pub const OID_AD_RPKI_MANIFEST: &str = "1.3.6.1.5.5.7.48.10";
|
||||
pub const OID_AD_RPKI_MANIFEST_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.10);
|
||||
pub const OID_AD_RPKI_NOTIFY: &str = "1.3.6.1.5.5.7.48.13";
|
||||
pub const OID_AD_RPKI_NOTIFY_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.13);
|
||||
|
||||
// RFC 3779 resource extensions (RFC 6487 profile)
|
||||
pub const OID_IP_ADDR_BLOCKS: &str = "1.3.6.1.5.5.7.1.7";
|
||||
pub const OID_IP_ADDR_BLOCKS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.7);
|
||||
pub const OID_AUTONOMOUS_SYS_IDS: &str = "1.3.6.1.5.5.7.1.8";
|
||||
pub const OID_AUTONOMOUS_SYS_IDS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.8);
|
||||
|
||||
// RPKI CP (RFC 6484 / RFC 6487)
|
||||
pub const OID_CP_IPADDR_ASNUMBER: &str = "1.3.6.1.5.5.7.14.2";
|
||||
pub const OID_CP_IPADDR_ASNUMBER_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.14.2);
|
||||
|
||||
pub const OID_CT_RPKI_CCR: &str = "1.2.840.113549.1.9.16.1.54";
|
||||
pub const OID_CT_RPKI_CCR_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.54);
|
||||
|
||||
pub const OID_KP_BGPSEC_ROUTER: &str = "1.3.6.1.5.5.7.3.30";
|
||||
pub const OID_EC_PUBLIC_KEY: &str = "1.2.840.10045.2.1";
|
||||
pub const OID_SECP256R1: &str = "1.2.840.10045.3.1.7";
|
||||
1868
crates/panda-rpki-validator/src/data_model/rc.rs
Normal file
1868
crates/panda-rpki-validator/src/data_model/rc.rs
Normal file
File diff suppressed because it is too large
Load Diff
633
crates/panda-rpki-validator/src/data_model/roa.rs
Normal file
633
crates/panda-rpki-validator/src/data_model/roa.rs
Normal file
@ -0,0 +1,633 @@
|
||||
use crate::data_model::common::{DerReader, der_take_tlv};
|
||||
use crate::data_model::oid::OID_CT_ROUTE_ORIGIN_AUTHZ;
|
||||
use crate::data_model::rc::{Afi as RcAfi, IpPrefix as RcIpPrefix, ResourceCertificate};
|
||||
use crate::data_model::signed_object::{
|
||||
RpkiSignedObject, RpkiSignedObjectParsed, SignedObjectDecodeError, SignedObjectParseError,
|
||||
SignedObjectValidateError,
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaObject {
|
||||
pub signed_object: RpkiSignedObject,
|
||||
pub econtent_type: String,
|
||||
pub roa: RoaEContent,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaObjectParsed {
|
||||
pub signed_object: RpkiSignedObjectParsed,
|
||||
pub econtent_type: String,
|
||||
pub roa: Option<RoaEContentParsed>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaEContent {
|
||||
pub version: u32,
|
||||
pub as_id: u32,
|
||||
pub ip_addr_blocks: Vec<RoaIpAddressFamily>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaEContentParsed {
|
||||
der: Vec<u8>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum RoaParseError {
|
||||
#[error("signed object parse error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
|
||||
SignedObject(#[from] SignedObjectParseError),
|
||||
#[error("ROA parse error: {0} (RFC 9582 §4; DER)")]
|
||||
Parse(String),
|
||||
|
||||
#[error("ROA trailing bytes: {0} bytes (RFC 9582 §4; DER)")]
|
||||
TrailingBytes(usize),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum RoaProfileError {
|
||||
#[error("signed object profile error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
|
||||
SignedObject(#[from] SignedObjectValidateError),
|
||||
|
||||
#[error("ROA eContentType must be {OID_CT_ROUTE_ORIGIN_AUTHZ}, got {0} (RFC 9582 §3)")]
|
||||
InvalidEContentType(String),
|
||||
|
||||
#[error("ROA profile decode error: {0} (RFC 9582 §4; DER)")]
|
||||
ProfileDecode(String),
|
||||
|
||||
#[error("RouteOriginAttestation must be a SEQUENCE of 2 or 3 elements, got {0} (RFC 9582 §4)")]
|
||||
InvalidAttestationSequenceLen(usize),
|
||||
|
||||
#[error("ROA version must be 0, got {0} (RFC 9582 §4.1)")]
|
||||
InvalidVersion(u64),
|
||||
|
||||
#[error("ROA asID out of range (0..=4294967295), got {0} (RFC 9582 §4.2)")]
|
||||
AsIdOutOfRange(u64),
|
||||
|
||||
#[error("ROA ipAddrBlocks must have length 1..2, got {0} (RFC 9582 §4; RFC 9582 §4.3.1)")]
|
||||
InvalidIpAddrBlocksLen(usize),
|
||||
|
||||
#[error("ROAIPAddressFamily must be a SEQUENCE of 2 elements (RFC 9582 §4.3.1)")]
|
||||
InvalidIpAddressFamily,
|
||||
|
||||
#[error("ROA addressFamily must be an OCTET STRING of 2 bytes (RFC 9582 §4.3.1)")]
|
||||
InvalidAddressFamily,
|
||||
|
||||
#[error("ROA addressFamily AFI not supported: {0:02X?} (RFC 9582 §4.3.1)")]
|
||||
UnsupportedAfi(Vec<u8>),
|
||||
|
||||
#[error("ROA contains duplicate AFI {0:?} (RFC 9582 §4.3.1)")]
|
||||
DuplicateAfi(RoaAfi),
|
||||
|
||||
#[error("ROAAddresses must have at least one entry (RFC 9582 §4.3.2)")]
|
||||
EmptyAddressList,
|
||||
|
||||
#[error("ROAIPAddress must be a SEQUENCE of 1..2 elements (RFC 9582 §4.3.2)")]
|
||||
InvalidRoaIpAddress,
|
||||
|
||||
#[error("ROAIPAddress.address must be a BIT STRING (RFC 9582 §4.3.2.1; RFC 3779 §2.2.3.8)")]
|
||||
InvalidPrefixBitString,
|
||||
|
||||
#[error(
|
||||
"ROAIPAddress.address has invalid unused bits encoding (RFC 9582 §4.3.2.1; RFC 3779 §2.2.3.8)"
|
||||
)]
|
||||
InvalidPrefixUnusedBits,
|
||||
|
||||
#[error(
|
||||
"ROAIPAddress.address prefix length {prefix_len} out of range for {afi:?} (RFC 9582 §4.3.2.1)"
|
||||
)]
|
||||
PrefixLenOutOfRange { afi: RoaAfi, prefix_len: u16 },
|
||||
|
||||
#[error(
|
||||
"ROAIPAddress.maxLength out of range for {afi:?}: prefix_len={prefix_len}, max_len={max_len} (RFC 9582 §4.3.2.2)"
|
||||
)]
|
||||
InvalidMaxLength {
|
||||
afi: RoaAfi,
|
||||
prefix_len: u16,
|
||||
max_len: u16,
|
||||
},
|
||||
}
|
||||
|
||||
impl From<SignedObjectDecodeError> for RoaProfileError {
|
||||
fn from(value: SignedObjectDecodeError) -> Self {
|
||||
match value {
|
||||
SignedObjectDecodeError::Parse(e) => RoaProfileError::ProfileDecode(e.to_string()),
|
||||
SignedObjectDecodeError::Validate(e) => RoaProfileError::SignedObject(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum RoaDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] RoaParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] RoaProfileError),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum RoaValidateError {
|
||||
#[error("ROA EE certificate must not contain AS resources extension (RFC 9582 §5)")]
|
||||
EeAsResourcesPresent,
|
||||
|
||||
#[error("ROA EE certificate must contain IP resources extension (RFC 9582 §5)")]
|
||||
EeIpResourcesMissing,
|
||||
|
||||
#[error("ROA EE certificate IP resources must not use inherit (RFC 9582 §5)")]
|
||||
EeIpResourcesInherit,
|
||||
|
||||
#[error(
|
||||
"ROA prefix not covered by EE certificate IP resources: {afi:?} {addr:?}/{prefix_len} (RFC 9582 §5; RFC 3779 §2.3)"
|
||||
)]
|
||||
PrefixNotInEeResources {
|
||||
afi: RoaAfi,
|
||||
addr: Vec<u8>,
|
||||
prefix_len: u16,
|
||||
},
|
||||
}
|
||||
|
||||
impl RoaObject {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<RoaObjectParsed, RoaParseError> {
|
||||
let signed_object = RpkiSignedObject::parse_der(der)?;
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
let roa = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent
|
||||
.as_deref()
|
||||
.map(RoaEContent::parse_der)
|
||||
.transpose()?;
|
||||
Ok(RoaObjectParsed {
|
||||
signed_object,
|
||||
econtent_type,
|
||||
roa,
|
||||
})
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `RoaObject` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `RoaObjectParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), RoaProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, RoaDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
pub fn decode_der_with_strict_options(
|
||||
der: &[u8],
|
||||
strict_cms_der: bool,
|
||||
strict_name: bool,
|
||||
) -> Result<Self, RoaDecodeError> {
|
||||
let signed_object =
|
||||
RpkiSignedObject::decode_der_with_strict_options(der, strict_cms_der, strict_name)
|
||||
.map_err(RoaProfileError::from)?;
|
||||
Self::from_signed_object(signed_object)
|
||||
}
|
||||
|
||||
pub fn from_signed_object(signed_object: RpkiSignedObject) -> Result<Self, RoaDecodeError> {
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
if econtent_type != OID_CT_ROUTE_ORIGIN_AUTHZ {
|
||||
return Err(RoaProfileError::InvalidEContentType(econtent_type).into());
|
||||
}
|
||||
|
||||
let roa = RoaEContent::decode_der(&signed_object.signed_data.encap_content_info.econtent)?;
|
||||
Ok(Self {
|
||||
roa,
|
||||
signed_object,
|
||||
econtent_type: OID_CT_ROUTE_ORIGIN_AUTHZ.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Validate this ROA's embedded EE certificate resources.
|
||||
pub fn validate_embedded_ee_cert(&self) -> Result<(), RoaValidateError> {
|
||||
let ee = &self.signed_object.signed_data.certificates[0].resource_cert;
|
||||
self.roa.validate_against_ee_cert(ee)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(
|
||||
Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, serde::Serialize, serde::Deserialize,
|
||||
)]
|
||||
pub enum RoaAfi {
|
||||
Ipv4,
|
||||
Ipv6,
|
||||
}
|
||||
|
||||
impl RoaAfi {
|
||||
fn ub(self) -> u16 {
|
||||
match self {
|
||||
RoaAfi::Ipv4 => 32,
|
||||
RoaAfi::Ipv6 => 128,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RoaIpAddressFamily {
|
||||
pub afi: RoaAfi,
|
||||
pub addresses: Vec<RoaIpAddress>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub struct RoaIpAddress {
|
||||
pub prefix: IpPrefix,
|
||||
pub max_length: Option<u16>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub struct IpPrefix {
|
||||
pub afi: RoaAfi,
|
||||
/// Prefix length in bits.
|
||||
pub prefix_len: u16,
|
||||
/// Network order address bytes (always 16 bytes), with host bits cleared.
|
||||
///
|
||||
/// For IPv4 prefixes, only the first 4 bytes are used and the remaining 12 bytes are zero.
|
||||
pub addr: [u8; 16],
|
||||
}
|
||||
|
||||
impl IpPrefix {
|
||||
pub fn addr_bytes(&self) -> &[u8] {
|
||||
match self.afi {
|
||||
RoaAfi::Ipv4 => &self.addr[..4],
|
||||
RoaAfi::Ipv6 => &self.addr[..16],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl RoaEContent {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<RoaEContentParsed, RoaParseError> {
|
||||
let (_tag, _value, rem) = der_take_tlv(der).map_err(RoaParseError::Parse)?;
|
||||
if !rem.is_empty() {
|
||||
return Err(RoaParseError::TrailingBytes(rem.len()));
|
||||
}
|
||||
Ok(RoaEContentParsed { der: der.to_vec() })
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `RoaEContent` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `RoaEContentParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), RoaProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Decode the DER-encoded RouteOriginAttestation defined in RFC 9582 §4 (`parse + validate`).
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, RoaDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
pub fn canonicalize(&mut self) {
|
||||
self.ip_addr_blocks.sort_by_key(|f| f.afi);
|
||||
for fam in &mut self.ip_addr_blocks {
|
||||
fam.addresses.sort();
|
||||
fam.addresses.dedup();
|
||||
}
|
||||
}
|
||||
|
||||
/// Validate ROA payload against the embedded EE resource certificate (RFC 9582 §5).
|
||||
///
|
||||
/// This performs the EE/payload semantic checks that do not require certificate path
|
||||
/// validation.
|
||||
pub fn validate_against_ee_cert(
|
||||
&self,
|
||||
ee: &ResourceCertificate,
|
||||
) -> Result<(), RoaValidateError> {
|
||||
if ee.tbs.extensions.as_resources.is_some() {
|
||||
return Err(RoaValidateError::EeAsResourcesPresent);
|
||||
}
|
||||
|
||||
let ip = ee
|
||||
.tbs
|
||||
.extensions
|
||||
.ip_resources
|
||||
.as_ref()
|
||||
.ok_or(RoaValidateError::EeIpResourcesMissing)?;
|
||||
|
||||
if ip.has_any_inherit() {
|
||||
return Err(RoaValidateError::EeIpResourcesInherit);
|
||||
}
|
||||
|
||||
for fam in &self.ip_addr_blocks {
|
||||
for entry in &fam.addresses {
|
||||
let rc_prefix = roa_prefix_to_rc(&entry.prefix);
|
||||
if !ip.contains_prefix(&rc_prefix) {
|
||||
return Err(RoaValidateError::PrefixNotInEeResources {
|
||||
afi: entry.prefix.afi,
|
||||
addr: entry.prefix.addr_bytes().to_vec(),
|
||||
prefix_len: entry.prefix.prefix_len,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl RoaObjectParsed {
|
||||
pub fn validate_profile(self) -> Result<RoaObject, RoaProfileError> {
|
||||
let signed_object = self.signed_object.validate_profile()?;
|
||||
let econtent_type = signed_object
|
||||
.signed_data
|
||||
.encap_content_info
|
||||
.econtent_type
|
||||
.clone();
|
||||
if econtent_type != OID_CT_ROUTE_ORIGIN_AUTHZ {
|
||||
return Err(RoaProfileError::InvalidEContentType(econtent_type));
|
||||
}
|
||||
let roa = self
|
||||
.roa
|
||||
.ok_or_else(|| RoaProfileError::ProfileDecode("ROA.eContent missing".into()))?
|
||||
.validate_profile()?;
|
||||
Ok(RoaObject {
|
||||
signed_object,
|
||||
econtent_type: OID_CT_ROUTE_ORIGIN_AUTHZ.to_string(),
|
||||
roa,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl RoaEContentParsed {
|
||||
pub fn validate_profile(self) -> Result<RoaEContent, RoaProfileError> {
|
||||
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
|
||||
let mut n = 0usize;
|
||||
while !r.is_empty() {
|
||||
r.skip_any()?;
|
||||
n += 1;
|
||||
}
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
let mut r = DerReader::new(&self.der);
|
||||
let mut seq = r
|
||||
.take_sequence()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e))?;
|
||||
if !r.is_empty() {
|
||||
return Err(RoaProfileError::ProfileDecode(
|
||||
"trailing bytes after RouteOriginAttestation".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let elem_count =
|
||||
count_elements(seq).map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if elem_count != 2 && elem_count != 3 {
|
||||
return Err(RoaProfileError::InvalidAttestationSequenceLen(elem_count));
|
||||
}
|
||||
|
||||
let mut version: u32 = 0;
|
||||
if elem_count == 3 {
|
||||
if seq
|
||||
.peek_tag()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?
|
||||
!= 0xA0
|
||||
{
|
||||
return Err(RoaProfileError::ProfileDecode(
|
||||
"RouteOriginAttestation.version must be [0] EXPLICIT INTEGER".into(),
|
||||
));
|
||||
}
|
||||
let (inner_tag, inner_val) = seq
|
||||
.take_explicit(0xA0)
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if inner_tag != 0x02 {
|
||||
return Err(RoaProfileError::ProfileDecode(
|
||||
"RouteOriginAttestation.version must be [0] EXPLICIT INTEGER".into(),
|
||||
));
|
||||
}
|
||||
let v = crate::data_model::common::der_uint_from_bytes(inner_val)
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if v != 0 {
|
||||
return Err(RoaProfileError::InvalidVersion(v));
|
||||
}
|
||||
version = 0;
|
||||
}
|
||||
|
||||
let as_id_u64 = seq
|
||||
.take_uint_u64()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if as_id_u64 > u32::MAX as u64 {
|
||||
return Err(RoaProfileError::AsIdOutOfRange(as_id_u64));
|
||||
}
|
||||
let as_id = as_id_u64 as u32;
|
||||
let ip_addr_blocks = parse_ip_addr_blocks_cursor(
|
||||
seq.take_sequence()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(format!("ipAddrBlocks: {e}")))?,
|
||||
)?;
|
||||
|
||||
if !seq.is_empty() {
|
||||
// Extra elements beyond the expected 2..3.
|
||||
let extra =
|
||||
count_elements(seq).map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
return Err(RoaProfileError::InvalidAttestationSequenceLen(
|
||||
elem_count + extra,
|
||||
));
|
||||
}
|
||||
|
||||
let mut out = RoaEContent {
|
||||
version,
|
||||
as_id,
|
||||
ip_addr_blocks,
|
||||
};
|
||||
out.canonicalize();
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
|
||||
fn roa_prefix_to_rc(p: &IpPrefix) -> RcIpPrefix {
|
||||
let afi = match p.afi {
|
||||
RoaAfi::Ipv4 => RcAfi::Ipv4,
|
||||
RoaAfi::Ipv6 => RcAfi::Ipv6,
|
||||
};
|
||||
RcIpPrefix {
|
||||
afi,
|
||||
prefix_len: p.prefix_len,
|
||||
addr: p.addr_bytes().to_vec(),
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_ip_addr_blocks_cursor(
|
||||
mut seq: DerReader<'_>,
|
||||
) -> Result<Vec<RoaIpAddressFamily>, RoaProfileError> {
|
||||
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
|
||||
let mut n = 0usize;
|
||||
while !r.is_empty() {
|
||||
r.skip_any()?;
|
||||
n += 1;
|
||||
}
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
let fam_count =
|
||||
count_elements(seq).map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
if fam_count == 0 || fam_count > 2 {
|
||||
return Err(RoaProfileError::InvalidIpAddrBlocksLen(fam_count));
|
||||
}
|
||||
|
||||
let mut out: Vec<RoaIpAddressFamily> = Vec::with_capacity(fam_count);
|
||||
while !seq.is_empty() {
|
||||
let family = parse_ip_address_family_cursor(
|
||||
seq.take_sequence()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?,
|
||||
)?;
|
||||
if out.iter().any(|f| f.afi == family.afi) {
|
||||
return Err(RoaProfileError::DuplicateAfi(family.afi));
|
||||
}
|
||||
out.push(family);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn parse_ip_address_family_cursor(
|
||||
mut fam: DerReader<'_>,
|
||||
) -> Result<RoaIpAddressFamily, RoaProfileError> {
|
||||
let afi = {
|
||||
let bytes = fam
|
||||
.take_octet_string()
|
||||
.map_err(|_e| RoaProfileError::InvalidAddressFamily)?;
|
||||
if bytes.len() != 2 {
|
||||
return Err(RoaProfileError::InvalidAddressFamily);
|
||||
}
|
||||
match bytes {
|
||||
[0x00, 0x01] => RoaAfi::Ipv4,
|
||||
[0x00, 0x02] => RoaAfi::Ipv6,
|
||||
_ => return Err(RoaProfileError::UnsupportedAfi(bytes.to_vec())),
|
||||
}
|
||||
};
|
||||
|
||||
let mut addrs = fam
|
||||
.take_sequence()
|
||||
.map_err(|_e| RoaProfileError::InvalidIpAddressFamily)?;
|
||||
if !fam.is_empty() {
|
||||
return Err(RoaProfileError::InvalidIpAddressFamily);
|
||||
}
|
||||
|
||||
if addrs.is_empty() {
|
||||
return Err(RoaProfileError::EmptyAddressList);
|
||||
}
|
||||
let mut addresses: Vec<RoaIpAddress> = Vec::new();
|
||||
while !addrs.is_empty() {
|
||||
let entry = addrs
|
||||
.take_sequence()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
addresses.push(parse_roa_ip_address_cursor(afi, entry)?);
|
||||
}
|
||||
|
||||
Ok(RoaIpAddressFamily { afi, addresses })
|
||||
}
|
||||
|
||||
fn parse_roa_ip_address_cursor(
|
||||
afi: RoaAfi,
|
||||
mut seq: DerReader<'_>,
|
||||
) -> Result<RoaIpAddress, RoaProfileError> {
|
||||
if seq.is_empty() {
|
||||
return Err(RoaProfileError::InvalidRoaIpAddress);
|
||||
}
|
||||
|
||||
let (unused_bits, bytes) = seq
|
||||
.take_bit_string()
|
||||
.map_err(|_e| RoaProfileError::InvalidPrefixBitString)?;
|
||||
let prefix = parse_prefix_bits_bytes(afi, unused_bits, bytes)?;
|
||||
|
||||
let max_length = if !seq.is_empty() {
|
||||
let v = seq
|
||||
.take_uint_u64()
|
||||
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
|
||||
let max_len: u16 = v
|
||||
.try_into()
|
||||
.map_err(|_e| RoaProfileError::InvalidMaxLength {
|
||||
afi,
|
||||
prefix_len: prefix.prefix_len,
|
||||
max_len: u16::MAX,
|
||||
})?;
|
||||
Some(max_len)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
if !seq.is_empty() {
|
||||
return Err(RoaProfileError::InvalidRoaIpAddress);
|
||||
}
|
||||
|
||||
if let Some(max_len) = max_length {
|
||||
let ub = afi.ub();
|
||||
if max_len > ub || max_len < prefix.prefix_len {
|
||||
return Err(RoaProfileError::InvalidMaxLength {
|
||||
afi,
|
||||
prefix_len: prefix.prefix_len,
|
||||
max_len,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Ok(RoaIpAddress { prefix, max_length })
|
||||
}
|
||||
|
||||
fn parse_prefix_bits_bytes(
|
||||
afi: RoaAfi,
|
||||
unused_bits: u8,
|
||||
bytes: &[u8],
|
||||
) -> Result<IpPrefix, RoaProfileError> {
|
||||
if unused_bits > 7 {
|
||||
return Err(RoaProfileError::InvalidPrefixUnusedBits);
|
||||
}
|
||||
if bytes.is_empty() {
|
||||
if unused_bits != 0 {
|
||||
return Err(RoaProfileError::InvalidPrefixUnusedBits);
|
||||
}
|
||||
} else if unused_bits != 0 {
|
||||
let mask = (1u8 << unused_bits) - 1;
|
||||
if (bytes[bytes.len() - 1] & mask) != 0 {
|
||||
return Err(RoaProfileError::InvalidPrefixUnusedBits);
|
||||
}
|
||||
}
|
||||
|
||||
let prefix_len = (bytes.len() * 8)
|
||||
.checked_sub(unused_bits as usize)
|
||||
.ok_or(RoaProfileError::InvalidPrefixUnusedBits)? as u16;
|
||||
if prefix_len > afi.ub() {
|
||||
return Err(RoaProfileError::PrefixLenOutOfRange { afi, prefix_len });
|
||||
}
|
||||
|
||||
let addr = canonicalize_prefix_addr(afi, prefix_len, bytes);
|
||||
Ok(IpPrefix {
|
||||
afi,
|
||||
prefix_len,
|
||||
addr,
|
||||
})
|
||||
}
|
||||
|
||||
fn canonicalize_prefix_addr(afi: RoaAfi, prefix_len: u16, bytes: &[u8]) -> [u8; 16] {
|
||||
let full_len = match afi {
|
||||
RoaAfi::Ipv4 => 4,
|
||||
RoaAfi::Ipv6 => 16,
|
||||
};
|
||||
let mut addr = [0u8; 16];
|
||||
let copy_len = bytes.len().min(full_len);
|
||||
addr[..copy_len].copy_from_slice(&bytes[..copy_len]);
|
||||
|
||||
if prefix_len == 0 {
|
||||
return addr;
|
||||
}
|
||||
|
||||
let last_prefix_bit = (prefix_len - 1) as usize;
|
||||
let last_prefix_byte = last_prefix_bit / 8;
|
||||
let rem = (prefix_len % 8) as u8;
|
||||
if rem != 0 {
|
||||
let mask: u8 = 0xFF << (8 - rem);
|
||||
if last_prefix_byte < full_len {
|
||||
addr[last_prefix_byte] &= mask;
|
||||
}
|
||||
}
|
||||
addr
|
||||
}
|
||||
363
crates/panda-rpki-validator/src/data_model/router_cert.rs
Normal file
363
crates/panda-rpki-validator/src/data_model/router_cert.rs
Normal file
@ -0,0 +1,363 @@
|
||||
use crate::data_model::oid::{
|
||||
OID_EC_PUBLIC_KEY, OID_EXTENDED_KEY_USAGE_RAW, OID_KP_BGPSEC_ROUTER, OID_SECP256R1,
|
||||
};
|
||||
use crate::data_model::rc::{
|
||||
AsIdOrRange, AsIdentifierChoice, ResourceCertKind, ResourceCertificate,
|
||||
ResourceCertificateParseError, ResourceCertificateParsed, ResourceCertificateProfileError,
|
||||
ResourceCertificateRole,
|
||||
};
|
||||
use crate::validation::cert_path::{CertPathError, validate_ee_cert_path_with_predecoded_ee};
|
||||
use x509_parser::extensions::ParsedExtension;
|
||||
use x509_parser::prelude::{FromDer, X509Certificate};
|
||||
use x509_parser::public_key::PublicKey;
|
||||
use x509_parser::x509::SubjectPublicKeyInfo;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct BgpsecRouterCertificateParsed {
|
||||
pub rc_parsed: ResourceCertificateParsed,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct BgpsecRouterCertificate {
|
||||
pub raw_der: Vec<u8>,
|
||||
pub resource_cert: ResourceCertificate,
|
||||
pub subject_key_identifier: Vec<u8>,
|
||||
pub spki_der: Vec<u8>,
|
||||
pub asns: Vec<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum BgpsecRouterCertificateParseError {
|
||||
#[error("resource certificate parse error: {0} (RFC 5280 §4.1; RFC 6487 §4; RFC 8209 §3.1)")]
|
||||
ResourceCertificate(#[from] ResourceCertificateParseError),
|
||||
|
||||
#[error("X.509 parse error: {0} (RFC 5280 §4.1; RFC 8209 §3.1)")]
|
||||
X509(String),
|
||||
|
||||
#[error("trailing bytes after router certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
|
||||
TrailingBytes(usize),
|
||||
|
||||
#[error("router SubjectPublicKeyInfo parse error: {0} (RFC 5280 §4.1.2.7; RFC 8208 §3.1)")]
|
||||
SpkiParse(String),
|
||||
|
||||
#[error("trailing bytes after router SubjectPublicKeyInfo DER: {0} bytes (DER; RFC 8208 §3.1)")]
|
||||
SpkiTrailingBytes(usize),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum BgpsecRouterCertificateProfileError {
|
||||
#[error("resource certificate profile error: {0} (RFC 6487 §4; RFC 8209 §3.1)")]
|
||||
ResourceCertificate(#[from] ResourceCertificateProfileError),
|
||||
|
||||
#[error("BGPsec router certificate must be an EE certificate (RFC 8209 §3.1)")]
|
||||
NotEe,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate must contain SubjectKeyIdentifier (RFC 6487 §4.8.2; RFC 8209 §3.3)"
|
||||
)]
|
||||
MissingSki,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate must include ExtendedKeyUsage (RFC 8209 §3.1.3.2; RFC 8209 §3.3)"
|
||||
)]
|
||||
MissingExtendedKeyUsage,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate ExtendedKeyUsage must be non-critical (RFC 6487 §4.8.4; RFC 8209 §3.1.3.2)"
|
||||
)]
|
||||
ExtendedKeyUsageCriticality,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate ExtendedKeyUsage must contain id-kp-bgpsec-router ({OID_KP_BGPSEC_ROUTER}) (RFC 8209 §3.1.3.2; RFC 8209 §3.3)"
|
||||
)]
|
||||
MissingBgpsecRouterEku,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate MUST NOT include Subject Information Access (RFC 8209 §3.1.3.3; RFC 8209 §3.3)"
|
||||
)]
|
||||
SubjectInfoAccessPresent,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate MUST NOT include IP resources extension (RFC 8209 §3.1.3.4; RFC 8209 §3.3)"
|
||||
)]
|
||||
IpResourcesPresent,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate MUST include AS resources extension (RFC 8209 §3.1.3.5; RFC 8209 §3.3)"
|
||||
)]
|
||||
AsResourcesMissing,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate AS resources MUST include one or more ASNs (RFC 8209 §3.1.3.5)"
|
||||
)]
|
||||
AsResourcesAsnumMissing,
|
||||
|
||||
#[error("BGPsec router certificate AS resources MUST NOT use inherit (RFC 8209 §3.1.3.5)")]
|
||||
AsResourcesInherit,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate AS resources MUST contain explicit ASNs, not ranges (RFC 8209 §3.1.3.5)"
|
||||
)]
|
||||
AsResourcesRangeNotAllowed,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate subjectPublicKeyInfo.algorithm must be id-ecPublicKey ({OID_EC_PUBLIC_KEY}) (RFC 8208 §3.1)"
|
||||
)]
|
||||
SpkiAlgorithmNotEcPublicKey,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate subjectPublicKeyInfo.parameters must be secp256r1 ({OID_SECP256R1}) (RFC 8208 §3.1)"
|
||||
)]
|
||||
SpkiWrongCurve,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate subjectPublicKeyInfo.parameters missing or invalid (RFC 8208 §3.1)"
|
||||
)]
|
||||
SpkiParametersMissingOrInvalid,
|
||||
|
||||
#[error(
|
||||
"BGPsec router certificate subjectPublicKey MUST be uncompressed P-256 ECPoint (RFC 8208 §3.1)"
|
||||
)]
|
||||
SpkiEcPointNotUncompressedP256,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum BgpsecRouterCertificateDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] BgpsecRouterCertificateParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] BgpsecRouterCertificateProfileError),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum BgpsecRouterCertificatePathError {
|
||||
#[error("{0}")]
|
||||
Decode(#[from] BgpsecRouterCertificateDecodeError),
|
||||
|
||||
#[error("{0}")]
|
||||
CertPath(#[from] CertPathError),
|
||||
}
|
||||
|
||||
impl BgpsecRouterCertificate {
|
||||
pub fn parse_der(
|
||||
der: &[u8],
|
||||
) -> Result<BgpsecRouterCertificateParsed, BgpsecRouterCertificateParseError> {
|
||||
let (rem, cert) = X509Certificate::from_der(der)
|
||||
.map_err(|e| BgpsecRouterCertificateParseError::X509(e.to_string()))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(BgpsecRouterCertificateParseError::TrailingBytes(rem.len()));
|
||||
}
|
||||
let (spki_rem, _spki) =
|
||||
SubjectPublicKeyInfo::from_der(cert.tbs_certificate.subject_pki.raw)
|
||||
.map_err(|e| BgpsecRouterCertificateParseError::SpkiParse(e.to_string()))?;
|
||||
if !spki_rem.is_empty() {
|
||||
return Err(BgpsecRouterCertificateParseError::SpkiTrailingBytes(
|
||||
spki_rem.len(),
|
||||
));
|
||||
}
|
||||
let rc_parsed = ResourceCertificate::parse_der(der)?;
|
||||
Ok(BgpsecRouterCertificateParsed { rc_parsed })
|
||||
}
|
||||
|
||||
pub fn validate_profile(&self) -> Result<(), BgpsecRouterCertificateProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, BgpsecRouterCertificateDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
pub fn from_der(der: &[u8]) -> Result<Self, BgpsecRouterCertificateDecodeError> {
|
||||
Self::decode_der(der)
|
||||
}
|
||||
|
||||
pub fn validate_path_with_prevalidated_issuer(
|
||||
der: &[u8],
|
||||
issuer_ca: &ResourceCertificate,
|
||||
issuer_spki: &SubjectPublicKeyInfo<'_>,
|
||||
issuer_crl: &crate::data_model::crl::RpkixCrl,
|
||||
issuer_crl_revoked_serials: &std::collections::HashSet<Vec<u8>>,
|
||||
issuer_ca_rsync_uri: Option<&str>,
|
||||
issuer_crl_rsync_uri: Option<&str>,
|
||||
validation_time: time::OffsetDateTime,
|
||||
) -> Result<Self, BgpsecRouterCertificatePathError> {
|
||||
let cert = Self::decode_der(der)?;
|
||||
validate_ee_cert_path_with_predecoded_ee(
|
||||
&cert.resource_cert,
|
||||
der,
|
||||
issuer_ca,
|
||||
issuer_spki,
|
||||
issuer_crl,
|
||||
issuer_crl_revoked_serials,
|
||||
issuer_ca_rsync_uri,
|
||||
issuer_crl_rsync_uri,
|
||||
validation_time,
|
||||
)?;
|
||||
Ok(cert)
|
||||
}
|
||||
}
|
||||
|
||||
impl BgpsecRouterCertificateParsed {
|
||||
pub fn validate_profile(
|
||||
self,
|
||||
) -> Result<BgpsecRouterCertificate, BgpsecRouterCertificateProfileError> {
|
||||
let rc = self.rc_parsed.validate_profile()?;
|
||||
rc.validate_rfc6487_profile(ResourceCertificateRole::RouterEe)?;
|
||||
if rc.kind != ResourceCertKind::Ee {
|
||||
return Err(BgpsecRouterCertificateProfileError::NotEe);
|
||||
}
|
||||
let ski = rc
|
||||
.tbs
|
||||
.extensions
|
||||
.subject_key_identifier
|
||||
.clone()
|
||||
.ok_or(BgpsecRouterCertificateProfileError::MissingSki)?;
|
||||
|
||||
if rc.tbs.extensions.subject_info_access.is_some() {
|
||||
return Err(BgpsecRouterCertificateProfileError::SubjectInfoAccessPresent);
|
||||
}
|
||||
if rc.tbs.extensions.ip_resources.is_some() {
|
||||
return Err(BgpsecRouterCertificateProfileError::IpResourcesPresent);
|
||||
}
|
||||
let as_resources = rc
|
||||
.tbs
|
||||
.extensions
|
||||
.as_resources
|
||||
.as_ref()
|
||||
.ok_or(BgpsecRouterCertificateProfileError::AsResourcesMissing)?;
|
||||
let asns = extract_router_asns(as_resources)?;
|
||||
|
||||
let (rem, cert) = X509Certificate::from_der(&rc.raw_der).map_err(|e| {
|
||||
BgpsecRouterCertificateProfileError::ResourceCertificate(
|
||||
ResourceCertificateProfileError::InvalidCertificatePolicy(e.to_string()),
|
||||
)
|
||||
})?;
|
||||
if !rem.is_empty() {
|
||||
return Err(BgpsecRouterCertificateProfileError::ResourceCertificate(
|
||||
ResourceCertificateProfileError::InvalidCertificatePolicy(format!(
|
||||
"trailing bytes after router certificate DER: {}",
|
||||
rem.len()
|
||||
)),
|
||||
));
|
||||
}
|
||||
validate_router_eku(&cert)?;
|
||||
validate_router_spki(&rc.tbs.subject_public_key_info)?;
|
||||
|
||||
Ok(BgpsecRouterCertificate {
|
||||
raw_der: rc.raw_der.clone(),
|
||||
resource_cert: rc.clone(),
|
||||
subject_key_identifier: ski,
|
||||
spki_der: rc.tbs.subject_public_key_info.clone(),
|
||||
asns,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn extract_router_asns(
|
||||
as_resources: &crate::data_model::rc::AsResourceSet,
|
||||
) -> Result<Vec<u32>, BgpsecRouterCertificateProfileError> {
|
||||
let asnum = as_resources
|
||||
.asnum
|
||||
.as_ref()
|
||||
.ok_or(BgpsecRouterCertificateProfileError::AsResourcesAsnumMissing)?;
|
||||
if matches!(asnum, AsIdentifierChoice::Inherit)
|
||||
|| matches!(as_resources.rdi.as_ref(), Some(AsIdentifierChoice::Inherit))
|
||||
{
|
||||
return Err(BgpsecRouterCertificateProfileError::AsResourcesInherit);
|
||||
}
|
||||
let AsIdentifierChoice::AsIdsOrRanges(items) = asnum else {
|
||||
return Err(BgpsecRouterCertificateProfileError::AsResourcesInherit);
|
||||
};
|
||||
if items.is_empty() {
|
||||
return Err(BgpsecRouterCertificateProfileError::AsResourcesAsnumMissing);
|
||||
}
|
||||
let mut asns = Vec::with_capacity(items.len());
|
||||
for item in items {
|
||||
match item {
|
||||
AsIdOrRange::Id(v) => asns.push(*v),
|
||||
AsIdOrRange::Range { .. } => {
|
||||
return Err(BgpsecRouterCertificateProfileError::AsResourcesRangeNotAllowed);
|
||||
}
|
||||
}
|
||||
}
|
||||
asns.sort_unstable();
|
||||
asns.dedup();
|
||||
Ok(asns)
|
||||
}
|
||||
|
||||
fn validate_router_eku(
|
||||
cert: &X509Certificate<'_>,
|
||||
) -> Result<(), BgpsecRouterCertificateProfileError> {
|
||||
let mut matches = cert
|
||||
.tbs_certificate
|
||||
.extensions()
|
||||
.iter()
|
||||
.filter(|ext| ext.oid.as_bytes() == OID_EXTENDED_KEY_USAGE_RAW);
|
||||
let Some(ext) = matches.next() else {
|
||||
return Err(BgpsecRouterCertificateProfileError::MissingExtendedKeyUsage);
|
||||
};
|
||||
if matches.next().is_some() {
|
||||
return Err(BgpsecRouterCertificateProfileError::MissingExtendedKeyUsage);
|
||||
}
|
||||
if ext.critical {
|
||||
return Err(BgpsecRouterCertificateProfileError::ExtendedKeyUsageCriticality);
|
||||
}
|
||||
let ParsedExtension::ExtendedKeyUsage(eku) = ext.parsed_extension() else {
|
||||
return Err(BgpsecRouterCertificateProfileError::MissingExtendedKeyUsage);
|
||||
};
|
||||
let found = eku
|
||||
.other
|
||||
.iter()
|
||||
.any(|oid| oid.to_id_string() == OID_KP_BGPSEC_ROUTER);
|
||||
if !found {
|
||||
return Err(BgpsecRouterCertificateProfileError::MissingBgpsecRouterEku);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_router_spki(spki_der: &[u8]) -> Result<(), BgpsecRouterCertificateProfileError> {
|
||||
let (rem, spki) = SubjectPublicKeyInfo::from_der(spki_der)
|
||||
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid)?;
|
||||
if !rem.is_empty() {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
|
||||
}
|
||||
if spki.algorithm.algorithm.to_id_string() != OID_EC_PUBLIC_KEY {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiAlgorithmNotEcPublicKey);
|
||||
}
|
||||
let Some(params) = spki.algorithm.parameters.as_ref() else {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
|
||||
};
|
||||
if params.header.tag().0 != 0x06 {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
|
||||
}
|
||||
let mut der = Vec::with_capacity(params.data.len() + 2);
|
||||
der.push(0x06);
|
||||
if params.data.len() >= 0x80 {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
|
||||
}
|
||||
der.push(params.data.len() as u8);
|
||||
der.extend_from_slice(params.data);
|
||||
let (prem, oid) = der_parser::der::parse_der_oid(&der)
|
||||
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid)?;
|
||||
if !prem.is_empty() {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
|
||||
}
|
||||
let curve = oid
|
||||
.as_oid_val()
|
||||
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid)?
|
||||
.to_string();
|
||||
if curve != OID_SECP256R1 {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiWrongCurve);
|
||||
}
|
||||
let parsed = spki
|
||||
.parsed()
|
||||
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiEcPointNotUncompressedP256)?;
|
||||
let PublicKey::EC(ec) = parsed else {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiAlgorithmNotEcPublicKey);
|
||||
};
|
||||
if ec.data().len() != 65 || ec.data().first() != Some(&0x04) {
|
||||
return Err(BgpsecRouterCertificateProfileError::SpkiEcPointNotUncompressedP256);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
1629
crates/panda-rpki-validator/src/data_model/signed_object.rs
Normal file
1629
crates/panda-rpki-validator/src/data_model/signed_object.rs
Normal file
File diff suppressed because it is too large
Load Diff
312
crates/panda-rpki-validator/src/data_model/ta.rs
Normal file
312
crates/panda-rpki-validator/src/data_model/ta.rs
Normal file
@ -0,0 +1,312 @@
|
||||
use url::Url;
|
||||
use x509_parser::prelude::{FromDer, X509Certificate};
|
||||
|
||||
use crate::data_model::oid::OID_CP_IPADDR_ASNUMBER;
|
||||
use crate::data_model::rc::{
|
||||
AsIdentifierChoice, IpAddressChoice, ResourceCertKind, ResourceCertificate,
|
||||
ResourceCertificateParseError, ResourceCertificateParsed, ResourceCertificateProfileError,
|
||||
ResourceCertificateRole,
|
||||
};
|
||||
use crate::data_model::tal::Tal;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct TaCertificate {
|
||||
pub raw_der: Vec<u8>,
|
||||
pub rc_ca: ResourceCertificate,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TaCertificateParseError {
|
||||
#[error("TA certificate parse error: {0} (RFC 5280 §4.1; RFC 6487 §4; RFC 8630 §2.3)")]
|
||||
ResourceCertificate(#[from] ResourceCertificateParseError),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct TaCertificateParsed {
|
||||
pub rc_parsed: ResourceCertificateParsed,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TaCertificateProfileError {
|
||||
#[error("resource certificate profile error: {0} (RFC 5280 §4; RFC 6487 §4)")]
|
||||
ResourceCertificate(#[from] ResourceCertificateProfileError),
|
||||
|
||||
#[error("TA certificate must be a CA certificate (RFC 8630 §2.3; RFC 6487 §4.8.1)")]
|
||||
NotCa,
|
||||
|
||||
#[error(
|
||||
"TA certificate must be self-signed (issuer DN must equal subject DN) (RFC 8630 §2.3; RFC 5280 §4.1.2.4)"
|
||||
)]
|
||||
NotSelfSignedIssuerSubject,
|
||||
|
||||
#[error(
|
||||
"TA certificate must contain certificatePolicies ipAddr-asNumber ({OID_CP_IPADDR_ASNUMBER}) (RFC 6487 §4.8.9; RFC 8630 §2.3)"
|
||||
)]
|
||||
MissingOrInvalidCertificatePolicies,
|
||||
|
||||
#[error("TA certificate must contain SubjectKeyIdentifier (RFC 6487 §4.8.2; RFC 8630 §2.3)")]
|
||||
MissingSubjectKeyIdentifier,
|
||||
|
||||
#[error(
|
||||
"TA certificate must contain at least one RFC 3779 resource extension (IP or AS) (RFC 6487 §4.8.10-§4.8.11; RFC 8630 §2.3)"
|
||||
)]
|
||||
ResourcesMissing,
|
||||
|
||||
#[error("TA certificate resources must be non-empty (RFC 8630 §2.3)")]
|
||||
ResourcesEmpty,
|
||||
|
||||
#[error(
|
||||
"TA certificate MUST NOT use inherit in IP resources (RFC 8630 §2.3; RFC 3779 §2.2.3.5)"
|
||||
)]
|
||||
IpResourcesInherit,
|
||||
|
||||
#[error(
|
||||
"TA certificate MUST NOT use inherit in AS resources (RFC 8630 §2.3; RFC 3779 §3.2.3.3)"
|
||||
)]
|
||||
AsResourcesInherit,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TaCertificateDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] TaCertificateParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] TaCertificateProfileError),
|
||||
}
|
||||
|
||||
/// Backwards-compatible name: TA certificate errors from parse+validate.
|
||||
pub type TaCertificateError = TaCertificateDecodeError;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TaCertificateVerifyError {
|
||||
#[error("TA certificate parse error: {0} (RFC 5280 §4.1; RFC 8630 §2.3)")]
|
||||
Parse(String),
|
||||
|
||||
#[error("trailing bytes after TA certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
|
||||
TrailingBytes(usize),
|
||||
|
||||
#[error(
|
||||
"TA certificate self-signature verification failed: {0} (RFC 8630 §2.3; RFC 5280 §6.1)"
|
||||
)]
|
||||
InvalidSelfSignature(String),
|
||||
}
|
||||
|
||||
impl TaCertificate {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_der(der: &[u8]) -> Result<TaCertificateParsed, TaCertificateParseError> {
|
||||
Ok(TaCertificateParsed {
|
||||
rc_parsed: ResourceCertificate::parse_der(der)?,
|
||||
})
|
||||
}
|
||||
|
||||
/// Profile validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `TaCertificate` is already profile-validated when constructed via `decode_der()` /
|
||||
/// `TaCertificateParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), TaCertificateProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Decode a TA certificate (`parse + validate`).
|
||||
pub fn decode_der(der: &[u8]) -> Result<Self, TaCertificateDecodeError> {
|
||||
Ok(Self::parse_der(der)?.validate_profile()?)
|
||||
}
|
||||
|
||||
pub fn decode_der_with_strict_name(der: &[u8]) -> Result<Self, TaCertificateDecodeError> {
|
||||
let ta = Self::decode_der(der)?;
|
||||
ta.rc_ca
|
||||
.validate_strict_name_profile()
|
||||
.map_err(TaCertificateProfileError::from)?;
|
||||
Ok(ta)
|
||||
}
|
||||
|
||||
/// Backwards-compatible helper (historical name).
|
||||
pub fn from_der(der: &[u8]) -> Result<Self, TaCertificateError> {
|
||||
Self::decode_der(der)
|
||||
}
|
||||
|
||||
pub fn spki_der(&self) -> &[u8] {
|
||||
&self.rc_ca.tbs.subject_public_key_info
|
||||
}
|
||||
|
||||
/// Verify step of scheme A (`parse → validate → verify`).
|
||||
pub fn verify_self_signature(&self) -> Result<(), TaCertificateVerifyError> {
|
||||
let (rem, cert) = X509Certificate::from_der(&self.raw_der)
|
||||
.map_err(|e| TaCertificateVerifyError::Parse(e.to_string()))?;
|
||||
if !rem.is_empty() {
|
||||
return Err(TaCertificateVerifyError::TrailingBytes(rem.len()));
|
||||
}
|
||||
cert.verify_signature(None)
|
||||
.map_err(|e| TaCertificateVerifyError::InvalidSelfSignature(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate TA-specific semantic constraints on a parsed Resource Certificate.
|
||||
///
|
||||
/// Note: this does not verify the X.509 signature; it is intended for higher-level logic and
|
||||
/// for unit tests that exercise individual constraint branches.
|
||||
pub fn validate_rc_constraints(
|
||||
rc_ca: &ResourceCertificate,
|
||||
) -> Result<(), TaCertificateProfileError> {
|
||||
if rc_ca.kind != ResourceCertKind::Ca {
|
||||
return Err(TaCertificateProfileError::NotCa);
|
||||
}
|
||||
|
||||
if rc_ca.tbs.extensions.certificate_policies_oid.as_deref() != Some(OID_CP_IPADDR_ASNUMBER)
|
||||
{
|
||||
return Err(TaCertificateProfileError::MissingOrInvalidCertificatePolicies);
|
||||
}
|
||||
if rc_ca.tbs.extensions.subject_key_identifier.is_none() {
|
||||
return Err(TaCertificateProfileError::MissingSubjectKeyIdentifier);
|
||||
}
|
||||
|
||||
let ip = rc_ca.tbs.extensions.ip_resources.as_ref();
|
||||
let asn = rc_ca.tbs.extensions.as_resources.as_ref();
|
||||
if ip.is_none() && asn.is_none() {
|
||||
return Err(TaCertificateProfileError::ResourcesMissing);
|
||||
}
|
||||
|
||||
let mut has_any_resource = false;
|
||||
|
||||
if let Some(ip) = ip {
|
||||
if ip.has_any_inherit() {
|
||||
return Err(TaCertificateProfileError::IpResourcesInherit);
|
||||
}
|
||||
for fam in &ip.families {
|
||||
match &fam.choice {
|
||||
IpAddressChoice::Inherit => {
|
||||
return Err(TaCertificateProfileError::IpResourcesInherit);
|
||||
}
|
||||
IpAddressChoice::AddressesOrRanges(items) => {
|
||||
if !items.is_empty() {
|
||||
has_any_resource = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(asn) = asn {
|
||||
if matches!(asn.asnum, Some(AsIdentifierChoice::Inherit))
|
||||
|| matches!(asn.rdi, Some(AsIdentifierChoice::Inherit))
|
||||
{
|
||||
return Err(TaCertificateProfileError::AsResourcesInherit);
|
||||
}
|
||||
if let Some(AsIdentifierChoice::AsIdsOrRanges(items)) = asn.asnum.as_ref() {
|
||||
if !items.is_empty() {
|
||||
has_any_resource = true;
|
||||
}
|
||||
}
|
||||
if let Some(AsIdentifierChoice::AsIdsOrRanges(items)) = asn.rdi.as_ref() {
|
||||
if !items.is_empty() {
|
||||
has_any_resource = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !has_any_resource {
|
||||
return Err(TaCertificateProfileError::ResourcesEmpty);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl TaCertificateParsed {
|
||||
pub fn validate_profile(self) -> Result<TaCertificate, TaCertificateProfileError> {
|
||||
let rc_ca = self.rc_parsed.validate_profile()?;
|
||||
if rc_ca.kind != ResourceCertKind::Ca {
|
||||
return Err(TaCertificateProfileError::NotCa);
|
||||
}
|
||||
rc_ca.validate_rfc6487_profile(ResourceCertificateRole::TrustAnchor)?;
|
||||
|
||||
if rc_ca.tbs.issuer_name != rc_ca.tbs.subject_name {
|
||||
return Err(TaCertificateProfileError::NotSelfSignedIssuerSubject);
|
||||
}
|
||||
|
||||
TaCertificate::validate_rc_constraints(&rc_ca)?;
|
||||
|
||||
Ok(TaCertificate {
|
||||
raw_der: rc_ca.raw_der.clone(),
|
||||
rc_ca,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct TrustAnchor {
|
||||
pub tal: Tal,
|
||||
pub ta_certificate: TaCertificate,
|
||||
pub resolved_ta_uri: Option<Url>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TrustAnchorError {
|
||||
#[error("TA certificate error: {0} (RFC 8630 §2.3)")]
|
||||
TaCertificate(#[from] TaCertificateDecodeError),
|
||||
|
||||
#[error("TA certificate self-signature error: {0} (RFC 8630 §2.3)")]
|
||||
TaSelfSignature(#[from] TaCertificateVerifyError),
|
||||
|
||||
#[error("{0}")]
|
||||
Bind(#[from] TrustAnchorBindError),
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TrustAnchorBindError {
|
||||
#[error("resolved TA URI not listed in TAL: {0} (RFC 8630 §2.2-§2.3)")]
|
||||
ResolvedUriNotInTal(String),
|
||||
|
||||
#[error(
|
||||
"TAL SPKI does not match TA certificate SubjectPublicKeyInfo (RFC 8630 §2.3; RFC 5280 §4.1.2.7)"
|
||||
)]
|
||||
TalSpkiMismatch,
|
||||
}
|
||||
|
||||
impl TrustAnchor {
|
||||
/// Bind a TAL and a downloaded TA certificate.
|
||||
///
|
||||
/// This does not download anything; it only validates the binding rules from RFC 8630 §2.3.
|
||||
pub fn bind_der(
|
||||
tal: Tal,
|
||||
ta_der: &[u8],
|
||||
resolved_uri: Option<&Url>,
|
||||
) -> Result<Self, TrustAnchorError> {
|
||||
let ta_certificate = TaCertificate::decode_der(ta_der)?;
|
||||
ta_certificate.verify_self_signature()?;
|
||||
Ok(Self::bind(tal, ta_certificate, resolved_uri)?)
|
||||
}
|
||||
|
||||
pub fn bind_der_with_strict_name(
|
||||
tal: Tal,
|
||||
ta_der: &[u8],
|
||||
resolved_uri: Option<&Url>,
|
||||
) -> Result<Self, TrustAnchorError> {
|
||||
let ta_certificate = TaCertificate::decode_der_with_strict_name(ta_der)?;
|
||||
ta_certificate.verify_self_signature()?;
|
||||
Ok(Self::bind(tal, ta_certificate, resolved_uri)?)
|
||||
}
|
||||
|
||||
pub fn bind(
|
||||
tal: Tal,
|
||||
ta_certificate: TaCertificate,
|
||||
resolved_uri: Option<&Url>,
|
||||
) -> Result<Self, TrustAnchorBindError> {
|
||||
if let Some(u) = resolved_uri {
|
||||
if !tal.ta_uris.iter().any(|x| x == u) {
|
||||
return Err(TrustAnchorBindError::ResolvedUriNotInTal(u.to_string()));
|
||||
}
|
||||
}
|
||||
|
||||
if tal.subject_public_key_info_der != ta_certificate.spki_der() {
|
||||
return Err(TrustAnchorBindError::TalSpkiMismatch);
|
||||
}
|
||||
|
||||
Ok(Self {
|
||||
tal,
|
||||
ta_certificate,
|
||||
resolved_ta_uri: resolved_uri.cloned(),
|
||||
})
|
||||
}
|
||||
}
|
||||
183
crates/panda-rpki-validator/src/data_model/tal.rs
Normal file
183
crates/panda-rpki-validator/src/data_model/tal.rs
Normal file
@ -0,0 +1,183 @@
|
||||
use base64::Engine;
|
||||
use url::Url;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct TalParsed {
|
||||
pub raw: Vec<u8>,
|
||||
/// Lines split by '\n' and normalized by stripping a trailing '\r' per line.
|
||||
pub lines: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct Tal {
|
||||
pub raw: Vec<u8>,
|
||||
pub comments: Vec<String>,
|
||||
pub ta_uris: Vec<Url>,
|
||||
pub subject_public_key_info_der: Vec<u8>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TalParseError {
|
||||
#[error("TAL must be valid UTF-8 (RFC 8630 §2.2)")]
|
||||
InvalidUtf8,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TalProfileError {
|
||||
#[error("TAL comments must appear only at the beginning (RFC 8630 §2.2)")]
|
||||
CommentAfterHeader,
|
||||
|
||||
#[error("TAL must contain at least one TA URI line (RFC 8630 §2.2)")]
|
||||
MissingTaUris,
|
||||
|
||||
#[error(
|
||||
"TAL must contain an empty line separator between URI list and SPKI base64 (RFC 8630 §2.2)"
|
||||
)]
|
||||
MissingSeparatorEmptyLine,
|
||||
|
||||
#[error("TAL TA URI invalid: {0} (RFC 8630 §2.2)")]
|
||||
InvalidUri(String),
|
||||
|
||||
#[error("TAL TA URI scheme must be rsync or https, got {0} (RFC 8630 §2.2)")]
|
||||
UnsupportedUriScheme(String),
|
||||
|
||||
#[error(
|
||||
"TAL TA URI must reference a single object (must not end with '/'): {0} (RFC 8630 §2.3)"
|
||||
)]
|
||||
UriIsDirectory(String),
|
||||
|
||||
#[error(
|
||||
"TAL must contain base64-encoded SubjectPublicKeyInfo after the separator (RFC 8630 §2.2)"
|
||||
)]
|
||||
MissingSpki,
|
||||
|
||||
#[error("TAL SPKI base64 decode failed (RFC 8630 §2.2)")]
|
||||
SpkiBase64Decode,
|
||||
|
||||
#[error("TAL SPKI DER is empty (RFC 8630 §2.2)")]
|
||||
SpkiDerEmpty,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TalDecodeError {
|
||||
#[error("{0}")]
|
||||
Parse(#[from] TalParseError),
|
||||
|
||||
#[error("{0}")]
|
||||
Validate(#[from] TalProfileError),
|
||||
}
|
||||
|
||||
impl Tal {
|
||||
/// Parse step of scheme A (`parse → validate → verify`).
|
||||
pub fn parse_bytes(input: &[u8]) -> Result<TalParsed, TalParseError> {
|
||||
let raw = input.to_vec();
|
||||
let text = std::str::from_utf8(input).map_err(|_| TalParseError::InvalidUtf8)?;
|
||||
|
||||
let lines: Vec<String> = text
|
||||
.split('\n')
|
||||
.map(|l| l.strip_suffix('\r').unwrap_or(l).to_string())
|
||||
.collect();
|
||||
|
||||
Ok(TalParsed { raw, lines })
|
||||
}
|
||||
|
||||
/// Validate step of scheme A (`parse → validate → verify`).
|
||||
///
|
||||
/// `Tal` is already profile-validated when constructed via `decode_bytes()` /
|
||||
/// `TalParsed::validate_profile()`.
|
||||
pub fn validate_profile(&self) -> Result<(), TalProfileError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn decode_bytes(input: &[u8]) -> Result<Self, TalDecodeError> {
|
||||
Ok(Self::parse_bytes(input)?.validate_profile()?)
|
||||
}
|
||||
}
|
||||
|
||||
impl TalParsed {
|
||||
pub fn validate_profile(self) -> Result<Tal, TalProfileError> {
|
||||
let mut idx = 0usize;
|
||||
|
||||
// 1) Leading comments.
|
||||
let mut comments: Vec<String> = Vec::new();
|
||||
while idx < self.lines.len() && self.lines[idx].starts_with('#') {
|
||||
comments.push(self.lines[idx][1..].to_string());
|
||||
idx += 1;
|
||||
}
|
||||
|
||||
// 2) URI list (one or more non-empty lines).
|
||||
let mut ta_uris: Vec<Url> = Vec::new();
|
||||
while idx < self.lines.len() {
|
||||
let line = self.lines[idx].trim();
|
||||
if line.is_empty() {
|
||||
break;
|
||||
}
|
||||
if line.starts_with('#') {
|
||||
return Err(TalProfileError::CommentAfterHeader);
|
||||
}
|
||||
let url = match Url::parse(line) {
|
||||
Ok(u) => u,
|
||||
Err(_) => {
|
||||
if !ta_uris.is_empty() {
|
||||
return Err(TalProfileError::MissingSeparatorEmptyLine);
|
||||
}
|
||||
return Err(TalProfileError::InvalidUri(line.to_string()));
|
||||
}
|
||||
};
|
||||
match url.scheme() {
|
||||
"rsync" | "https" => {}
|
||||
s => return Err(TalProfileError::UnsupportedUriScheme(s.to_string())),
|
||||
}
|
||||
if url.path().ends_with('/') {
|
||||
return Err(TalProfileError::UriIsDirectory(line.to_string()));
|
||||
}
|
||||
if url
|
||||
.path_segments()
|
||||
.and_then(|mut s| s.next_back())
|
||||
.unwrap_or("")
|
||||
.is_empty()
|
||||
{
|
||||
return Err(TalProfileError::UriIsDirectory(line.to_string()));
|
||||
}
|
||||
ta_uris.push(url);
|
||||
idx += 1;
|
||||
}
|
||||
|
||||
if ta_uris.is_empty() {
|
||||
return Err(TalProfileError::MissingTaUris);
|
||||
}
|
||||
|
||||
// 3) Empty line separator (must exist).
|
||||
if idx >= self.lines.len() || !self.lines[idx].trim().is_empty() {
|
||||
return Err(TalProfileError::MissingSeparatorEmptyLine);
|
||||
}
|
||||
idx += 1;
|
||||
|
||||
// 4) Base64(SPKI DER) remainder; allow line wrapping.
|
||||
let mut b64 = String::new();
|
||||
while idx < self.lines.len() {
|
||||
let line = self.lines[idx].trim();
|
||||
if !line.is_empty() {
|
||||
b64.push_str(line);
|
||||
}
|
||||
idx += 1;
|
||||
}
|
||||
if b64.is_empty() {
|
||||
return Err(TalProfileError::MissingSpki);
|
||||
}
|
||||
|
||||
let spki_der = base64::engine::general_purpose::STANDARD
|
||||
.decode(b64.as_bytes())
|
||||
.map_err(|_| TalProfileError::SpkiBase64Decode)?;
|
||||
if spki_der.is_empty() {
|
||||
return Err(TalProfileError::SpkiDerEmpty);
|
||||
}
|
||||
|
||||
Ok(Tal {
|
||||
raw: self.raw,
|
||||
comments,
|
||||
ta_uris,
|
||||
subject_public_key_info_der: spki_der,
|
||||
})
|
||||
}
|
||||
}
|
||||
234
crates/panda-rpki-validator/src/fetch/current_repository.rs
Normal file
234
crates/panda-rpki-validator/src/fetch/current_repository.rs
Normal file
@ -0,0 +1,234 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::fetch::rsync::{RsyncFetchError, RsyncFetchResult, RsyncFetcher};
|
||||
use crate::fetch::rsync_system::{
|
||||
RsyncScopePolicy, scoped_rsync_failure_dedup_key, scoped_rsync_fetch_uri,
|
||||
};
|
||||
use crate::storage::{RepositoryViewState, RocksStore};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct CurrentRepositoryViewRsyncFetcher {
|
||||
store: Arc<RocksStore>,
|
||||
scope_policy: RsyncScopePolicy,
|
||||
}
|
||||
|
||||
impl CurrentRepositoryViewRsyncFetcher {
|
||||
pub fn new(store: Arc<RocksStore>, scope_policy: RsyncScopePolicy) -> Self {
|
||||
Self {
|
||||
store,
|
||||
scope_policy,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncFetcher for CurrentRepositoryViewRsyncFetcher {
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
|
||||
// The source run's live fetcher may have fetched a module root rather
|
||||
// than the publication-point URI that triggered it. Read that same
|
||||
// frozen-view prefix here so transport prefetch request identities and
|
||||
// their materialized object sets stay equivalent.
|
||||
let base = scoped_rsync_fetch_uri(self.scope_policy, rsync_base_uri);
|
||||
let entries = self
|
||||
.store
|
||||
.list_repository_view_entries_with_prefix(&base)
|
||||
.map_err(|error| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"list frozen repository view failed for {base}: {error}"
|
||||
))
|
||||
})?;
|
||||
let mut objects = Vec::with_capacity(entries.len());
|
||||
for entry in entries {
|
||||
if !matches!(
|
||||
entry.state,
|
||||
RepositoryViewState::Present | RepositoryViewState::Replaced
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
let bytes = self
|
||||
.store
|
||||
.load_current_object_bytes_by_uri(&entry.rsync_uri)
|
||||
.map_err(|error| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"load frozen repository object failed for {}: {error}",
|
||||
entry.rsync_uri
|
||||
))
|
||||
})?
|
||||
.ok_or_else(|| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"frozen repository object missing for {}",
|
||||
entry.rsync_uri
|
||||
))
|
||||
})?;
|
||||
objects.push((entry.rsync_uri, bytes));
|
||||
}
|
||||
objects.sort_by(|left, right| left.0.cmp(&right.0));
|
||||
if objects.is_empty() {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"frozen repository view contains no current objects under {base}"
|
||||
)));
|
||||
}
|
||||
Ok(objects)
|
||||
}
|
||||
|
||||
fn fetch_object(&self, rsync_uri: &str) -> RsyncFetchResult<Vec<u8>> {
|
||||
self.store
|
||||
.load_current_object_bytes_by_uri(rsync_uri)
|
||||
.map_err(|error| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"load frozen repository object failed for {rsync_uri}: {error}"
|
||||
))
|
||||
})?
|
||||
.ok_or_else(|| {
|
||||
RsyncFetchError::Fetch(format!("frozen repository object not found: {rsync_uri}"))
|
||||
})
|
||||
}
|
||||
|
||||
fn dedup_key(&self, rsync_base_uri: &str) -> String {
|
||||
scoped_rsync_fetch_uri(self.scope_policy, rsync_base_uri)
|
||||
}
|
||||
|
||||
fn failure_dedup_key(&self, rsync_base_uri: &str) -> Option<String> {
|
||||
scoped_rsync_failure_dedup_key(self.scope_policy, rsync_base_uri)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::storage::{RepositoryViewEntry, RocksStore};
|
||||
|
||||
#[test]
|
||||
fn current_repository_fetcher_returns_sorted_present_objects() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let store = Arc::new(RocksStore::open(dir.path()).expect("open store"));
|
||||
let entries = [
|
||||
("rsync://example.test/repo/b.roa", b"b".as_slice()),
|
||||
("rsync://example.test/repo/a.mft", b"a".as_slice()),
|
||||
];
|
||||
for (uri, bytes) in entries {
|
||||
let hash = hex::encode(crate::cir::sha256(bytes));
|
||||
store
|
||||
.put_blob_bytes_batch(&[(hash.clone(), bytes.to_vec())])
|
||||
.expect("put blob");
|
||||
store
|
||||
.put_repository_view_entry(&RepositoryViewEntry {
|
||||
rsync_uri: uri.to_string(),
|
||||
repository_source: Some("fixture".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
current_hash: Some(hash),
|
||||
})
|
||||
.expect("put view");
|
||||
}
|
||||
let fetcher = CurrentRepositoryViewRsyncFetcher::new(store, RsyncScopePolicy::default());
|
||||
let objects = fetcher
|
||||
.fetch_objects("rsync://example.test/repo/")
|
||||
.expect("fetch objects");
|
||||
assert_eq!(
|
||||
objects
|
||||
.iter()
|
||||
.map(|(uri, _)| uri.as_str())
|
||||
.collect::<Vec<_>>(),
|
||||
vec![
|
||||
"rsync://example.test/repo/a.mft",
|
||||
"rsync://example.test/repo/b.roa"
|
||||
]
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher
|
||||
.fetch_object("rsync://example.test/repo/a.mft")
|
||||
.expect("fetch one"),
|
||||
b"a"
|
||||
);
|
||||
assert!(
|
||||
fetcher
|
||||
.fetch_object("rsync://example.test/repo/missing.roa")
|
||||
.is_err()
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher.dedup_key("rsync://example.test/repo"),
|
||||
"rsync://example.test/repo/"
|
||||
);
|
||||
assert!(
|
||||
fetcher
|
||||
.fetch_objects("rsync://empty.example/repo/")
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("no current objects")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repository_fetcher_ignores_withdrawn_and_reports_missing_blob() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let store = Arc::new(RocksStore::open(dir.path()).expect("open store"));
|
||||
store
|
||||
.put_repository_view_entry(&RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/withdrawn.roa".to_string(),
|
||||
repository_source: Some("fixture".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Withdrawn,
|
||||
current_hash: None,
|
||||
})
|
||||
.expect("put withdrawn view");
|
||||
let missing_hash = "ab".repeat(32);
|
||||
store
|
||||
.put_repository_view_entry(&RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/missing.roa".to_string(),
|
||||
repository_source: Some("fixture".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
current_hash: Some(missing_hash),
|
||||
})
|
||||
.expect("put missing view");
|
||||
let fetcher = CurrentRepositoryViewRsyncFetcher::new(store, RsyncScopePolicy::default());
|
||||
let error = fetcher
|
||||
.fetch_objects("rsync://example.test/repo/")
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(error.contains("blob bytes missing"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_repository_fetcher_replays_module_scope_and_failure_dedup() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let store = Arc::new(RocksStore::open(dir.path()).expect("open store"));
|
||||
for (uri, bytes) in [
|
||||
("rsync://example.test/repo/ca/a.mft", b"a".as_slice()),
|
||||
("rsync://example.test/repo/other/b.roa", b"b".as_slice()),
|
||||
] {
|
||||
let hash = hex::encode(crate::cir::sha256(bytes));
|
||||
store
|
||||
.put_blob_bytes_batch(&[(hash.clone(), bytes.to_vec())])
|
||||
.expect("put blob");
|
||||
store
|
||||
.put_repository_view_entry(&RepositoryViewEntry {
|
||||
rsync_uri: uri.to_string(),
|
||||
repository_source: Some("fixture".to_string()),
|
||||
object_type: Some("fixture".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
current_hash: Some(hash),
|
||||
})
|
||||
.expect("put view");
|
||||
}
|
||||
|
||||
let module_fetcher = CurrentRepositoryViewRsyncFetcher::new(
|
||||
Arc::clone(&store),
|
||||
RsyncScopePolicy::ModuleRoot,
|
||||
);
|
||||
let objects = module_fetcher
|
||||
.fetch_objects("rsync://example.test/repo/ca/")
|
||||
.expect("fetch module scope");
|
||||
assert_eq!(objects.len(), 2);
|
||||
assert_eq!(
|
||||
module_fetcher.dedup_key("rsync://example.test/repo/ca/"),
|
||||
"rsync://example.test/repo/"
|
||||
);
|
||||
|
||||
let host_fetcher = CurrentRepositoryViewRsyncFetcher::new(store, RsyncScopePolicy::Host);
|
||||
assert_eq!(
|
||||
host_fetcher.failure_dedup_key("rsync://example.test/repo/ca/"),
|
||||
Some("rsync://example.test/".to_string())
|
||||
);
|
||||
}
|
||||
}
|
||||
652
crates/panda-rpki-validator/src/fetch/http.rs
Normal file
652
crates/panda-rpki-validator/src/fetch/http.rs
Normal file
@ -0,0 +1,652 @@
|
||||
use std::cell::RefCell;
|
||||
use std::io::Write;
|
||||
use std::time::Duration;
|
||||
|
||||
use reqwest::blocking::Client;
|
||||
use reqwest::header::HeaderMap;
|
||||
|
||||
use crate::sync::rrdp::Fetcher;
|
||||
|
||||
thread_local! {
|
||||
static HTTP_TIMEOUT_OVERRIDE: RefCell<Option<Duration>> = const { RefCell::new(None) };
|
||||
}
|
||||
|
||||
pub fn with_scoped_http_timeout_override<R>(timeout: Duration, f: impl FnOnce() -> R) -> R {
|
||||
HTTP_TIMEOUT_OVERRIDE.with(|cell| {
|
||||
let previous = cell.replace(Some(timeout));
|
||||
let result = f();
|
||||
let _ = cell.replace(previous);
|
||||
result
|
||||
})
|
||||
}
|
||||
|
||||
/// Default User-Agent sent with every outgoing HTTP request (RRDP
|
||||
/// notification/snapshot/delta, TAL/TA downloads, daemon probes).
|
||||
pub const DEFAULT_HTTP_USER_AGENT: &str = "panda-rpki/0.2";
|
||||
|
||||
/// Environment variable that overrides the HTTP User-Agent. Unset, empty, or
|
||||
/// values containing characters that are illegal in a header value fall back
|
||||
/// to [`DEFAULT_HTTP_USER_AGENT`].
|
||||
pub const HTTP_USER_AGENT_ENV: &str = "RPKI_HTTP_USER_AGENT";
|
||||
|
||||
fn resolve_http_user_agent(env_value: Option<String>) -> String {
|
||||
let Some(value) = env_value else {
|
||||
return DEFAULT_HTTP_USER_AGENT.to_string();
|
||||
};
|
||||
let trimmed = value.trim();
|
||||
let valid = !trimmed.is_empty() && trimmed.bytes().all(|b| (0x20..=0x7e).contains(&b));
|
||||
if valid {
|
||||
trimmed.to_string()
|
||||
} else {
|
||||
DEFAULT_HTTP_USER_AGENT.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct HttpFetcherConfig {
|
||||
/// Connection-establishment timeout for HTTP requests.
|
||||
pub connect_timeout: Duration,
|
||||
/// Short timeout used for connection establishment and small metadata objects.
|
||||
pub timeout: Duration,
|
||||
/// Larger timeout used for RRDP snapshot / delta bodies.
|
||||
pub large_body_timeout: Duration,
|
||||
pub user_agent: String,
|
||||
/// Extra PEM trust anchors for HTTPS transport tests or private RRDP endpoints.
|
||||
pub extra_root_certificates_pem: Vec<Vec<u8>>,
|
||||
}
|
||||
|
||||
impl Default for HttpFetcherConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
connect_timeout: Duration::from_secs(15),
|
||||
timeout: Duration::from_secs(30),
|
||||
large_body_timeout: Duration::from_secs(180),
|
||||
user_agent: resolve_http_user_agent(std::env::var(HTTP_USER_AGENT_ENV).ok()),
|
||||
extra_root_certificates_pem: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Minimal blocking HTTP(S) fetcher for stage2.
|
||||
///
|
||||
/// This is used for:
|
||||
/// - downloading TAL / TA certificates (RFC 8630 §2)
|
||||
/// - fetching RRDP notification/snapshot files (RFC 8182 §3.4)
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct BlockingHttpFetcher {
|
||||
short_client: Client,
|
||||
large_body_client: Client,
|
||||
retry_short_client: Client,
|
||||
short_timeout: Duration,
|
||||
large_body_timeout: Duration,
|
||||
}
|
||||
|
||||
impl BlockingHttpFetcher {
|
||||
pub fn new(config: HttpFetcherConfig) -> Result<Self, String> {
|
||||
let short_timeout = config.timeout;
|
||||
let large_body_timeout = std::cmp::max(config.large_body_timeout, config.timeout);
|
||||
let connect_timeout = std::cmp::min(config.connect_timeout, config.timeout);
|
||||
let short_client = Self::client_builder(
|
||||
&config,
|
||||
connect_timeout,
|
||||
config.timeout,
|
||||
config.user_agent.clone(),
|
||||
)?
|
||||
.build()
|
||||
.map_err(|e| e.to_string())?;
|
||||
let large_body_client = Self::client_builder(
|
||||
&config,
|
||||
connect_timeout,
|
||||
large_body_timeout,
|
||||
config.user_agent.clone(),
|
||||
)?
|
||||
.build()
|
||||
.map_err(|e| e.to_string())?;
|
||||
let retry_short_client = Self::client_builder(
|
||||
&config,
|
||||
Duration::from_secs(1),
|
||||
Duration::from_secs(1),
|
||||
config.user_agent.clone(),
|
||||
)?
|
||||
.build()
|
||||
.map_err(|e| e.to_string())?;
|
||||
Ok(Self {
|
||||
short_client,
|
||||
large_body_client,
|
||||
retry_short_client,
|
||||
short_timeout,
|
||||
large_body_timeout,
|
||||
})
|
||||
}
|
||||
|
||||
fn client_builder(
|
||||
config: &HttpFetcherConfig,
|
||||
connect_timeout: Duration,
|
||||
timeout: Duration,
|
||||
user_agent: String,
|
||||
) -> Result<reqwest::blocking::ClientBuilder, String> {
|
||||
let mut builder = Client::builder()
|
||||
.connect_timeout(connect_timeout)
|
||||
.timeout(timeout)
|
||||
.user_agent(user_agent);
|
||||
for (idx, pem) in config.extra_root_certificates_pem.iter().enumerate() {
|
||||
let certificate = reqwest::Certificate::from_pem(pem)
|
||||
.map_err(|e| format!("parse HTTP root certificate #{idx} failed: {e}"))?;
|
||||
builder = builder.add_root_certificate(certificate);
|
||||
}
|
||||
Ok(builder)
|
||||
}
|
||||
|
||||
pub fn fetch_bytes(&self, uri: &str) -> Result<Vec<u8>, String> {
|
||||
let started = std::time::Instant::now();
|
||||
let (client, timeout_profile, timeout_value) = self.client_for_uri(uri);
|
||||
let resp = client.get(uri).send().map_err(|e| {
|
||||
let msg = format!("http request failed: {e:?}");
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_failed",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"stage": "request",
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
"error": msg,
|
||||
}),
|
||||
);
|
||||
msg
|
||||
})?;
|
||||
|
||||
let status = resp.status();
|
||||
let headers = resp.headers().clone();
|
||||
if !status.is_success() {
|
||||
let body_preview = resp
|
||||
.text()
|
||||
.ok()
|
||||
.map(|text| text.chars().take(160).collect::<String>());
|
||||
let body_prefix = body_preview
|
||||
.clone()
|
||||
.unwrap_or_else(|| "<unavailable>".to_string());
|
||||
let msg = format!(
|
||||
"http status {status}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}; body_prefix={}",
|
||||
header_value(&headers, "content-type"),
|
||||
header_value(&headers, "content-encoding"),
|
||||
header_value(&headers, "content-length"),
|
||||
header_value(&headers, "transfer-encoding"),
|
||||
body_prefix,
|
||||
);
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_failed",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"stage": "status",
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
"status": status.as_u16(),
|
||||
"content_type": header_value_opt(&headers, "content-type"),
|
||||
"content_encoding": header_value_opt(&headers, "content-encoding"),
|
||||
"content_length": header_value_opt(&headers, "content-length"),
|
||||
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
|
||||
"body_prefix": body_preview,
|
||||
"error": msg,
|
||||
}),
|
||||
);
|
||||
return Err(msg);
|
||||
}
|
||||
|
||||
match resp.bytes() {
|
||||
Ok(bytes) => {
|
||||
let duration_ms = started.elapsed().as_millis() as u64;
|
||||
if (duration_ms as f64) / 1000.0 >= crate::progress_log::slow_threshold_secs() {
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_slow",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"status": status.as_u16(),
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": duration_ms,
|
||||
"bytes": bytes.len(),
|
||||
"content_type": header_value_opt(&headers, "content-type"),
|
||||
"content_encoding": header_value_opt(&headers, "content-encoding"),
|
||||
"content_length": header_value_opt(&headers, "content-length"),
|
||||
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
|
||||
}),
|
||||
);
|
||||
}
|
||||
Ok(bytes.to_vec())
|
||||
}
|
||||
Err(e) => {
|
||||
let msg = format!(
|
||||
"http read body failed: {e}; status={}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}",
|
||||
status,
|
||||
header_value(&headers, "content-type"),
|
||||
header_value(&headers, "content-encoding"),
|
||||
header_value(&headers, "content-length"),
|
||||
header_value(&headers, "transfer-encoding"),
|
||||
);
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_failed",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"stage": "read_body",
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
"status": status.as_u16(),
|
||||
"content_type": header_value_opt(&headers, "content-type"),
|
||||
"content_encoding": header_value_opt(&headers, "content-encoding"),
|
||||
"content_length": header_value_opt(&headers, "content-length"),
|
||||
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
|
||||
"error": msg,
|
||||
}),
|
||||
);
|
||||
Err(msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn client_for_uri(&self, uri: &str) -> (&Client, &'static str, Duration) {
|
||||
if let Some(timeout) = HTTP_TIMEOUT_OVERRIDE.with(|cell| *cell.borrow()) {
|
||||
return (&self.retry_short_client, "retry_short", timeout);
|
||||
}
|
||||
if uses_large_body_timeout(uri) {
|
||||
(
|
||||
&self.large_body_client,
|
||||
"large_body",
|
||||
self.large_body_timeout,
|
||||
)
|
||||
} else {
|
||||
(&self.short_client, "short", self.short_timeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Fetcher for BlockingHttpFetcher {
|
||||
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
|
||||
self.fetch_bytes(uri)
|
||||
}
|
||||
|
||||
fn fetch_to_writer(&self, uri: &str, out: &mut dyn Write) -> Result<u64, String> {
|
||||
let started = std::time::Instant::now();
|
||||
let (client, timeout_profile, timeout_value) = self.client_for_uri(uri);
|
||||
let resp = client.get(uri).send().map_err(|e| {
|
||||
let msg = format!("http request failed: {e:?}");
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_failed",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"stage": "request",
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
"error": msg,
|
||||
}),
|
||||
);
|
||||
msg
|
||||
})?;
|
||||
|
||||
let status = resp.status();
|
||||
let headers = resp.headers().clone();
|
||||
if !status.is_success() {
|
||||
let body_preview = resp
|
||||
.text()
|
||||
.ok()
|
||||
.map(|text| text.chars().take(160).collect::<String>());
|
||||
let body_prefix = body_preview
|
||||
.clone()
|
||||
.unwrap_or_else(|| "<unavailable>".to_string());
|
||||
let msg = format!(
|
||||
"http status {status}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}; body_prefix={}",
|
||||
header_value(&headers, "content-type"),
|
||||
header_value(&headers, "content-encoding"),
|
||||
header_value(&headers, "content-length"),
|
||||
header_value(&headers, "transfer-encoding"),
|
||||
body_prefix,
|
||||
);
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_failed",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"stage": "status",
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
"status": status.as_u16(),
|
||||
"content_type": header_value_opt(&headers, "content-type"),
|
||||
"content_encoding": header_value_opt(&headers, "content-encoding"),
|
||||
"content_length": header_value_opt(&headers, "content-length"),
|
||||
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
|
||||
"body_prefix": body_preview,
|
||||
"error": msg,
|
||||
}),
|
||||
);
|
||||
return Err(msg);
|
||||
}
|
||||
|
||||
let mut resp = resp;
|
||||
match resp.copy_to(out) {
|
||||
Ok(bytes) => {
|
||||
let duration_ms = started.elapsed().as_millis() as u64;
|
||||
if (duration_ms as f64) / 1000.0 >= crate::progress_log::slow_threshold_secs() {
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_slow",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"status": status.as_u16(),
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": duration_ms,
|
||||
"bytes": bytes,
|
||||
"content_type": header_value_opt(&headers, "content-type"),
|
||||
"content_encoding": header_value_opt(&headers, "content-encoding"),
|
||||
"content_length": header_value_opt(&headers, "content-length"),
|
||||
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
|
||||
}),
|
||||
);
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
Err(e) => {
|
||||
let msg = format!(
|
||||
"http stream body failed: {e}; status={}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}",
|
||||
status,
|
||||
header_value(&headers, "content-type"),
|
||||
header_value(&headers, "content-encoding"),
|
||||
header_value(&headers, "content-length"),
|
||||
header_value(&headers, "transfer-encoding"),
|
||||
);
|
||||
crate::progress_log::emit(
|
||||
"http_fetch_failed",
|
||||
serde_json::json!({
|
||||
"uri": uri,
|
||||
"stage": "stream_body",
|
||||
"timeout_profile": timeout_profile,
|
||||
"request_timeout_ms": timeout_value.as_millis() as u64,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
"status": status.as_u16(),
|
||||
"content_type": header_value_opt(&headers, "content-type"),
|
||||
"content_encoding": header_value_opt(&headers, "content-encoding"),
|
||||
"content_length": header_value_opt(&headers, "content-length"),
|
||||
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
|
||||
"error": msg,
|
||||
}),
|
||||
);
|
||||
Err(msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn header_value(headers: &HeaderMap, name: &str) -> String {
|
||||
header_value_opt(headers, name).unwrap_or_else(|| "<none>".to_string())
|
||||
}
|
||||
|
||||
fn header_value_opt(headers: &HeaderMap, name: &str) -> Option<String> {
|
||||
headers
|
||||
.get(name)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.map(|v| v.to_string())
|
||||
}
|
||||
|
||||
fn uses_large_body_timeout(uri: &str) -> bool {
|
||||
uri.starts_with("https://") && uri.ends_with(".xml") && !uri.ends_with("notification.xml")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::{Read, Write};
|
||||
use std::net::TcpListener;
|
||||
use std::thread;
|
||||
use std::time::Duration as StdDuration;
|
||||
|
||||
fn spawn_one_shot_http_server(status_line: &'static str, body: &'static [u8]) -> String {
|
||||
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
|
||||
let addr = listener.local_addr().expect("addr");
|
||||
thread::spawn(move || {
|
||||
let (mut stream, _) = listener.accept().expect("accept");
|
||||
let mut buf = [0u8; 1024];
|
||||
let _ = stream.read(&mut buf);
|
||||
let hdr = format!(
|
||||
"{status_line}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
|
||||
body.len()
|
||||
);
|
||||
stream.write_all(hdr.as_bytes()).expect("write hdr");
|
||||
stream.write_all(body).expect("write body");
|
||||
});
|
||||
format!("http://{}/", addr)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_bytes_returns_body_on_success() {
|
||||
let url = spawn_one_shot_http_server("HTTP/1.1 200 OK", b"hello");
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(2),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
let got = http.fetch_bytes(&url).expect("fetch");
|
||||
assert_eq!(got, b"hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_bytes_rejects_non_success_status() {
|
||||
let url = spawn_one_shot_http_server("HTTP/1.1 404 Not Found", b"");
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(2),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
let err = http.fetch_bytes(&url).unwrap_err();
|
||||
assert!(err.contains("http status"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_bytes_times_out_on_idle_body_read() {
|
||||
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
|
||||
let addr = listener.local_addr().expect("addr");
|
||||
thread::spawn(move || {
|
||||
let (mut stream, _) = listener.accept().expect("accept");
|
||||
let mut buf = [0u8; 1024];
|
||||
let _ = stream.read(&mut buf);
|
||||
stream
|
||||
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\nConnection: close\r\n\r\nh")
|
||||
.expect("write partial body");
|
||||
std::thread::sleep(StdDuration::from_secs(2));
|
||||
let _ = stream.write_all(b"ello");
|
||||
});
|
||||
let url = format!("http://{}/", addr);
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(1),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
let err = http.fetch_bytes(&url).unwrap_err();
|
||||
assert!(err.contains("http read body failed"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_to_writer_streams_body_on_success() {
|
||||
let url = spawn_one_shot_http_server("HTTP/1.1 200 OK", b"writer-body");
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(2),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
let mut out = Vec::new();
|
||||
let bytes = http.fetch_to_writer(&url, &mut out).expect("stream");
|
||||
assert_eq!(bytes, 11);
|
||||
assert_eq!(out, b"writer-body");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_to_writer_rejects_non_success_status() {
|
||||
let url = spawn_one_shot_http_server("HTTP/1.1 500 Internal Server Error", b"boom");
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(2),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
let mut out = Vec::new();
|
||||
let err = http.fetch_to_writer(&url, &mut out).unwrap_err();
|
||||
assert!(err.contains("http status"), "{err}");
|
||||
assert!(out.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_to_writer_times_out_on_idle_stream_read() {
|
||||
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
|
||||
let addr = listener.local_addr().expect("addr");
|
||||
thread::spawn(move || {
|
||||
let (mut stream, _) = listener.accept().expect("accept");
|
||||
let mut buf = [0u8; 1024];
|
||||
let _ = stream.read(&mut buf);
|
||||
stream
|
||||
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\nConnection: close\r\n\r\nh")
|
||||
.expect("write partial body");
|
||||
std::thread::sleep(StdDuration::from_secs(2));
|
||||
let _ = stream.write_all(b"ello");
|
||||
});
|
||||
let url = format!("http://{}/", addr);
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(1),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
let mut out = Vec::new();
|
||||
let err = http.fetch_to_writer(&url, &mut out).unwrap_err();
|
||||
assert!(err.contains("http stream body failed"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn uses_large_body_timeout_selects_rrdp_snapshot_and_delta_not_notification() {
|
||||
assert!(!uses_large_body_timeout(
|
||||
"https://rrdp.example.test/notification.xml"
|
||||
));
|
||||
assert!(uses_large_body_timeout(
|
||||
"https://rrdp.example.test/session/123/snapshot.xml"
|
||||
));
|
||||
assert!(uses_large_body_timeout(
|
||||
"https://rrdp.example.test/session/123/delta-42.xml"
|
||||
));
|
||||
assert!(!uses_large_body_timeout(
|
||||
"https://tal.example.test/example.tal"
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn client_for_uri_selects_expected_timeout_profile() {
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(3),
|
||||
large_body_timeout: Duration::from_secs(9),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
|
||||
let (_, profile_short, timeout_short) =
|
||||
http.client_for_uri("https://example.test/root.tal");
|
||||
assert_eq!(profile_short, "short");
|
||||
assert_eq!(timeout_short, Duration::from_secs(3));
|
||||
|
||||
let (_, profile_large, timeout_large) =
|
||||
http.client_for_uri("https://rrdp.example.test/session/1/snapshot.xml");
|
||||
assert_eq!(profile_large, "large_body");
|
||||
assert_eq!(timeout_large, Duration::from_secs(9));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_http_user_agent_falls_back_to_default() {
|
||||
assert_eq!(resolve_http_user_agent(None), DEFAULT_HTTP_USER_AGENT);
|
||||
assert_eq!(
|
||||
resolve_http_user_agent(Some(String::new())),
|
||||
DEFAULT_HTTP_USER_AGENT
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_http_user_agent(Some(" ".to_string())),
|
||||
DEFAULT_HTTP_USER_AGENT
|
||||
);
|
||||
// Control characters are illegal in header values; fall back instead
|
||||
// of letting reqwest fail to build the client.
|
||||
assert_eq!(
|
||||
resolve_http_user_agent(Some("bad\u{1}ua".to_string())),
|
||||
DEFAULT_HTTP_USER_AGENT
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_http_user_agent(Some("non-ascii-ua-中".to_string())),
|
||||
DEFAULT_HTTP_USER_AGENT
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_http_user_agent_accepts_custom_value() {
|
||||
assert_eq!(
|
||||
resolve_http_user_agent(Some("panda-rpki/9.9 (test)".to_string())),
|
||||
"panda-rpki/9.9 (test)"
|
||||
);
|
||||
// Surrounding whitespace is trimmed.
|
||||
assert_eq!(
|
||||
resolve_http_user_agent(Some(" custom/1.0 ".to_string())),
|
||||
"custom/1.0"
|
||||
);
|
||||
}
|
||||
|
||||
fn spawn_user_agent_capture_server() -> (String, std::sync::mpsc::Receiver<String>) {
|
||||
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
|
||||
let addr = listener.local_addr().expect("addr");
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
thread::spawn(move || {
|
||||
let (mut stream, _) = listener.accept().expect("accept");
|
||||
let mut buf = [0u8; 4096];
|
||||
let read = stream.read(&mut buf).expect("read request");
|
||||
let request = String::from_utf8_lossy(&buf[..read]).to_string();
|
||||
let body = b"ok";
|
||||
let hdr = format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
|
||||
body.len()
|
||||
);
|
||||
stream.write_all(hdr.as_bytes()).expect("write hdr");
|
||||
stream.write_all(body).expect("write body");
|
||||
let _ = tx.send(request);
|
||||
});
|
||||
(format!("http://{}/", addr), rx)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_bytes_sends_configured_user_agent_header() {
|
||||
let (url, rx) = spawn_user_agent_capture_server();
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(2),
|
||||
user_agent: "panda-rpki/0.2".to_string(),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
http.fetch_bytes(&url).expect("fetch");
|
||||
let request = rx.recv().expect("request captured");
|
||||
assert!(
|
||||
request
|
||||
.to_ascii_lowercase()
|
||||
.contains("user-agent: panda-rpki/0.2"),
|
||||
"{request}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fetch_bytes_sends_custom_user_agent_header() {
|
||||
let (url, rx) = spawn_user_agent_capture_server();
|
||||
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
|
||||
timeout: Duration::from_secs(2),
|
||||
user_agent: "custom-marker/7.7".to_string(),
|
||||
..HttpFetcherConfig::default()
|
||||
})
|
||||
.expect("http");
|
||||
http.fetch_bytes(&url).expect("fetch");
|
||||
let request = rx.recv().expect("request captured");
|
||||
assert!(
|
||||
request
|
||||
.to_ascii_lowercase()
|
||||
.contains("user-agent: custom-marker/7.7"),
|
||||
"{request}"
|
||||
);
|
||||
}
|
||||
}
|
||||
4
crates/panda-rpki-validator/src/fetch/mod.rs
Normal file
4
crates/panda-rpki-validator/src/fetch/mod.rs
Normal file
@ -0,0 +1,4 @@
|
||||
pub mod current_repository;
|
||||
pub mod http;
|
||||
pub mod rsync;
|
||||
pub mod rsync_system;
|
||||
206
crates/panda-rpki-validator/src/fetch/rsync.rs
Normal file
206
crates/panda-rpki-validator/src/fetch/rsync.rs
Normal file
@ -0,0 +1,206 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum RsyncFetchError {
|
||||
#[error("rsync fetch error: {0}")]
|
||||
Fetch(String),
|
||||
}
|
||||
|
||||
pub type RsyncFetchResult<T> = Result<T, RsyncFetchError>;
|
||||
|
||||
pub fn normalize_rsync_base_uri(s: &str) -> String {
|
||||
if s.ends_with('/') {
|
||||
s.to_string()
|
||||
} else {
|
||||
format!("{s}/")
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetch repository objects from a publication point.
|
||||
///
|
||||
/// v1: this is intentionally abstract so unit tests can use a mock, and later we can
|
||||
/// back it by calling the system `rsync` binary (RFC 6481 §5; RFC 8182 §3.4.5).
|
||||
pub trait RsyncFetcher: Send + Sync {
|
||||
/// Return a list of objects as `(rsync_uri, bytes)` pairs.
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>>;
|
||||
|
||||
/// Fetch one object by exact rsync URI.
|
||||
///
|
||||
/// The default implementation fetches the parent directory and filters the exact
|
||||
/// object. Live fetchers should override this to avoid widening one-object TAL
|
||||
/// bootstrap fetches into whole publication point or module synchronizations.
|
||||
fn fetch_object(&self, rsync_uri: &str) -> RsyncFetchResult<Vec<u8>> {
|
||||
let base = parent_rsync_uri(rsync_uri).map_err(RsyncFetchError::Fetch)?;
|
||||
self.fetch_objects(&base)?
|
||||
.into_iter()
|
||||
.find(|(uri, _)| uri == rsync_uri)
|
||||
.map(|(_, bytes)| bytes)
|
||||
.ok_or_else(|| RsyncFetchError::Fetch(format!("rsync object not found: {rsync_uri}")))
|
||||
}
|
||||
|
||||
/// Stream fetched objects to a visitor without requiring callers to materialize the
|
||||
/// full result vector in memory.
|
||||
fn visit_objects(
|
||||
&self,
|
||||
rsync_base_uri: &str,
|
||||
visitor: &mut dyn FnMut(String, Vec<u8>) -> Result<(), String>,
|
||||
) -> RsyncFetchResult<(usize, u64)> {
|
||||
let objects = self.fetch_objects(rsync_base_uri)?;
|
||||
let mut count = 0usize;
|
||||
let mut bytes_total = 0u64;
|
||||
for (uri, bytes) in objects {
|
||||
bytes_total += bytes.len() as u64;
|
||||
count += 1;
|
||||
visitor(uri, bytes).map_err(RsyncFetchError::Fetch)?;
|
||||
}
|
||||
Ok((count, bytes_total))
|
||||
}
|
||||
|
||||
/// Return the deduplication key used by orchestration layers.
|
||||
///
|
||||
/// By default this is the normalized publication point base URI. Fetchers that
|
||||
/// intentionally widen their fetch scope (for example to a full rsync module)
|
||||
/// should override this so callers can safely deduplicate at the same scope.
|
||||
fn dedup_key(&self, rsync_base_uri: &str) -> String {
|
||||
normalize_rsync_base_uri(rsync_base_uri)
|
||||
}
|
||||
|
||||
/// Return an optional failure-only deduplication key.
|
||||
///
|
||||
/// This key is only used to short-circuit repeated failed rsync fallbacks. It must
|
||||
/// not be used to reuse successful fetch results unless it is identical to
|
||||
/// `dedup_key`, because a successful fetch for one publication point does not imply
|
||||
/// that another publication point under the same host has been fetched.
|
||||
fn failure_dedup_key(&self, _rsync_base_uri: &str) -> Option<String> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn parent_rsync_uri(rsync_uri: &str) -> Result<String, String> {
|
||||
let parsed = url::Url::parse(rsync_uri).map_err(|e| e.to_string())?;
|
||||
if parsed.scheme() != "rsync" {
|
||||
return Err(format!("not an rsync URI: {rsync_uri}"));
|
||||
}
|
||||
let host = parsed
|
||||
.host_str()
|
||||
.ok_or_else(|| format!("missing host in rsync URI: {rsync_uri}"))?;
|
||||
let segments = parsed
|
||||
.path_segments()
|
||||
.ok_or_else(|| format!("missing path in rsync URI: {rsync_uri}"))?
|
||||
.collect::<Vec<_>>();
|
||||
if segments.is_empty() || segments.last().copied().unwrap_or_default().is_empty() {
|
||||
return Err(format!(
|
||||
"rsync URI must reference a file object: {rsync_uri}"
|
||||
));
|
||||
}
|
||||
let parent_segments = &segments[..segments.len() - 1];
|
||||
let mut parent = format!("rsync://{host}/");
|
||||
if !parent_segments.is_empty() {
|
||||
parent.push_str(&parent_segments.join("/"));
|
||||
parent.push('/');
|
||||
}
|
||||
Ok(parent)
|
||||
}
|
||||
|
||||
/// A simple "rsync" implementation backed by a local directory.
|
||||
///
|
||||
/// This is primarily meant for offline tests and fixtures. The key generation mimics rsync URIs:
|
||||
/// `rsync_base_uri` + relative path (with `/` separators).
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct LocalDirRsyncFetcher {
|
||||
pub root_dir: PathBuf,
|
||||
}
|
||||
|
||||
impl LocalDirRsyncFetcher {
|
||||
pub fn new(root_dir: impl Into<PathBuf>) -> Self {
|
||||
Self {
|
||||
root_dir: root_dir.into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncFetcher for LocalDirRsyncFetcher {
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
|
||||
let base = normalize_rsync_base_uri(rsync_base_uri);
|
||||
let mut out = Vec::new();
|
||||
walk_dir_collect(&self.root_dir, &self.root_dir, &base, &mut out)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e))?;
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
|
||||
fn walk_dir_collect(
|
||||
root: &Path,
|
||||
current: &Path,
|
||||
rsync_base_uri: &str,
|
||||
out: &mut Vec<(String, Vec<u8>)>,
|
||||
) -> Result<(), String> {
|
||||
let rd = std::fs::read_dir(current).map_err(|e| e.to_string())?;
|
||||
for entry in rd {
|
||||
let entry = entry.map_err(|e| e.to_string())?;
|
||||
let path = entry.path();
|
||||
let meta = entry.metadata().map_err(|e| e.to_string())?;
|
||||
if meta.is_dir() {
|
||||
walk_dir_collect(root, &path, rsync_base_uri, out)?;
|
||||
continue;
|
||||
}
|
||||
if !meta.is_file() {
|
||||
continue;
|
||||
}
|
||||
let rel = path
|
||||
.strip_prefix(root)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
let uri = format!("{rsync_base_uri}{rel}");
|
||||
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
|
||||
out.push((uri, bytes));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn local_dir_rsync_fetcher_collects_files_and_normalizes_base_uri() {
|
||||
let tmp = tempfile::tempdir().expect("tempdir");
|
||||
std::fs::create_dir_all(tmp.path().join("nested")).expect("mkdir");
|
||||
std::fs::write(tmp.path().join("a.mft"), b"a").expect("write");
|
||||
std::fs::write(tmp.path().join("nested").join("b.roa"), b"b").expect("write");
|
||||
|
||||
let f = LocalDirRsyncFetcher::new(tmp.path());
|
||||
let mut objects = f
|
||||
.fetch_objects("rsync://example.net/repo")
|
||||
.expect("fetch_objects");
|
||||
objects.sort_by(|(a, _), (b, _)| a.cmp(b));
|
||||
|
||||
assert_eq!(objects.len(), 2);
|
||||
assert_eq!(objects[0].0, "rsync://example.net/repo/a.mft");
|
||||
assert_eq!(objects[0].1, b"a");
|
||||
assert_eq!(objects[1].0, "rsync://example.net/repo/nested/b.roa");
|
||||
assert_eq!(objects[1].1, b"b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn local_dir_rsync_fetcher_reports_read_dir_errors() {
|
||||
let tmp = tempfile::tempdir().expect("tempdir");
|
||||
let missing = tmp.path().join("missing");
|
||||
let f = LocalDirRsyncFetcher::new(missing);
|
||||
let err = f.fetch_objects("rsync://example.net/repo").unwrap_err();
|
||||
match err {
|
||||
RsyncFetchError::Fetch(msg) => assert!(!msg.is_empty()),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_dedup_key_is_normalized_base_uri() {
|
||||
let tmp = tempfile::tempdir().expect("tempdir");
|
||||
let fetcher = LocalDirRsyncFetcher::new(tmp.path());
|
||||
assert_eq!(
|
||||
fetcher.dedup_key("rsync://example.net/repo"),
|
||||
"rsync://example.net/repo/"
|
||||
);
|
||||
}
|
||||
}
|
||||
944
crates/panda-rpki-validator/src/fetch/rsync_system.rs
Normal file
944
crates/panda-rpki-validator/src/fetch/rsync_system.rs
Normal file
@ -0,0 +1,944 @@
|
||||
use std::cell::RefCell;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
use std::process::Stdio;
|
||||
use std::thread;
|
||||
use std::time::Duration;
|
||||
use std::time::Instant;
|
||||
|
||||
use sha2::Digest;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::fetch::rsync::{
|
||||
RsyncFetchError, RsyncFetchResult, RsyncFetcher, normalize_rsync_base_uri,
|
||||
};
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "kebab-case")]
|
||||
pub enum RsyncScopePolicy {
|
||||
Host,
|
||||
PublicationPoint,
|
||||
ModuleRoot,
|
||||
}
|
||||
|
||||
impl Default for RsyncScopePolicy {
|
||||
fn default() -> Self {
|
||||
Self::ModuleRoot
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncScopePolicy {
|
||||
pub fn parse_cli_value(value: &str) -> Result<Self, String> {
|
||||
match value {
|
||||
"host" => Ok(Self::Host),
|
||||
"publication-point" => Ok(Self::PublicationPoint),
|
||||
"module-root" => Ok(Self::ModuleRoot),
|
||||
_ => Err(format!(
|
||||
"invalid --rsync-scope: {value}; expected host, publication-point, or module-root"
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct SystemRsyncConfig {
|
||||
pub rsync_bin: PathBuf,
|
||||
pub connect_timeout: Duration,
|
||||
pub timeout: Duration,
|
||||
pub extra_args: Vec<String>,
|
||||
/// Optional root directory for persistent rsync mirrors.
|
||||
///
|
||||
/// When set, callers may choose to sync into stable subdirectories under this
|
||||
/// root (instead of a temporary directory) to benefit from rsync's incremental
|
||||
/// behavior across runs.
|
||||
///
|
||||
/// Note: actual mirror behavior is implemented separately from config wiring.
|
||||
pub mirror_root: Option<PathBuf>,
|
||||
pub scope_policy: RsyncScopePolicy,
|
||||
}
|
||||
|
||||
impl Default for SystemRsyncConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
rsync_bin: PathBuf::from("rsync"),
|
||||
connect_timeout: Duration::from_secs(15),
|
||||
timeout: Duration::from_secs(30),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A `RsyncFetcher` implementation backed by the system `rsync` binary.
|
||||
///
|
||||
/// This is intended for live stage2 runs. For unit tests and offline fixtures,
|
||||
/// prefer `LocalDirRsyncFetcher`.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct SystemRsyncFetcher {
|
||||
config: SystemRsyncConfig,
|
||||
}
|
||||
|
||||
thread_local! {
|
||||
static RSYNC_TIMEOUT_OVERRIDE: RefCell<Option<Duration>> = const { RefCell::new(None) };
|
||||
static RSYNC_FAIL_FAST_PROFILE: RefCell<Option<RsyncFailFastProfile>> = const { RefCell::new(None) };
|
||||
}
|
||||
|
||||
pub fn with_scoped_rsync_timeout_override<R>(timeout: Duration, f: impl FnOnce() -> R) -> R {
|
||||
RSYNC_TIMEOUT_OVERRIDE.with(|cell| {
|
||||
let previous = cell.replace(Some(timeout));
|
||||
let result = f();
|
||||
let _ = cell.replace(previous);
|
||||
result
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct RsyncFailFastProfile {
|
||||
pub initial_wall_clock_timeout: Duration,
|
||||
pub max_wall_clock_timeout: Duration,
|
||||
pub max_attempts: usize,
|
||||
}
|
||||
|
||||
pub fn with_scoped_rsync_fail_fast_profile<R>(
|
||||
profile: RsyncFailFastProfile,
|
||||
f: impl FnOnce() -> R,
|
||||
) -> R {
|
||||
RSYNC_FAIL_FAST_PROFILE.with(|cell| {
|
||||
let previous = cell.replace(Some(profile));
|
||||
let result = f();
|
||||
let _ = cell.replace(previous);
|
||||
result
|
||||
})
|
||||
}
|
||||
|
||||
impl SystemRsyncFetcher {
|
||||
pub fn new(config: SystemRsyncConfig) -> Self {
|
||||
Self { config }
|
||||
}
|
||||
|
||||
fn mirror_dst_dir(&self, normalized_rsync_base_uri: &str) -> Result<Option<PathBuf>, String> {
|
||||
let Some(root) = self.config.mirror_root.as_ref() else {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
std::fs::create_dir_all(root)
|
||||
.map_err(|e| format!("create rsync mirror root failed: {}: {e}", root.display()))?;
|
||||
|
||||
let hash = hex::encode(sha2::Sha256::digest(normalized_rsync_base_uri.as_bytes()));
|
||||
let dir = root.join(hash);
|
||||
std::fs::create_dir_all(&dir).map_err(|e| {
|
||||
format!(
|
||||
"create rsync mirror directory failed: {}: {e}",
|
||||
dir.display()
|
||||
)
|
||||
})?;
|
||||
Ok(Some(dir))
|
||||
}
|
||||
|
||||
fn run_rsync(&self, src: &str, dst: &Path) -> Result<(), String> {
|
||||
let fail_fast = RSYNC_FAIL_FAST_PROFILE.with(|cell| *cell.borrow());
|
||||
if let Some(profile) = fail_fast {
|
||||
return self.run_rsync_fail_fast(src, dst, profile);
|
||||
}
|
||||
self.run_rsync_once(src, dst, None, false)
|
||||
}
|
||||
|
||||
fn run_rsync_once(
|
||||
&self,
|
||||
src: &str,
|
||||
dst: &Path,
|
||||
wall_clock_timeout: Option<Duration>,
|
||||
keep_partial: bool,
|
||||
) -> Result<(), String> {
|
||||
// `--timeout` is I/O timeout in seconds (applies to network reads/writes).
|
||||
let timeout =
|
||||
RSYNC_TIMEOUT_OVERRIDE.with(|cell| cell.borrow().unwrap_or(self.config.timeout));
|
||||
let connect_timeout_secs = self.config.connect_timeout.as_secs().max(1).to_string();
|
||||
let timeout_secs = timeout.as_secs().max(1).to_string();
|
||||
let is_remote_rsync = src.starts_with("rsync://");
|
||||
|
||||
let mut cmd = Command::new(&self.config.rsync_bin);
|
||||
cmd.arg("-rt")
|
||||
.arg("--delete")
|
||||
.arg("--timeout")
|
||||
.arg(timeout_secs)
|
||||
.args(&self.config.extra_args);
|
||||
if is_remote_rsync {
|
||||
cmd.arg("--contimeout").arg(connect_timeout_secs);
|
||||
}
|
||||
if keep_partial {
|
||||
cmd.arg("--partial");
|
||||
}
|
||||
cmd.arg(src)
|
||||
.arg(dst)
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped());
|
||||
|
||||
let mut child = cmd
|
||||
.spawn()
|
||||
.map_err(|e| format!("rsync spawn failed: {e}"))?;
|
||||
if let Some(limit) = wall_clock_timeout {
|
||||
let started = Instant::now();
|
||||
loop {
|
||||
match child
|
||||
.try_wait()
|
||||
.map_err(|e| format!("rsync wait failed: {e}"))?
|
||||
{
|
||||
Some(_status) => {
|
||||
let out = child
|
||||
.wait_with_output()
|
||||
.map_err(|e| format!("rsync wait_with_output failed: {e}"))?;
|
||||
if out.status.success() {
|
||||
return Ok(());
|
||||
}
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
let stdout = String::from_utf8_lossy(&out.stdout);
|
||||
return Err(format!(
|
||||
"rsync failed: status={} stdout={} stderr={}",
|
||||
out.status,
|
||||
stdout.trim(),
|
||||
stderr.trim()
|
||||
));
|
||||
}
|
||||
None => {
|
||||
if started.elapsed() >= limit {
|
||||
let _ = child.kill();
|
||||
let out = child
|
||||
.wait_with_output()
|
||||
.map_err(|e| format!("rsync wait_with_output failed: {e}"))?;
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
let stdout = String::from_utf8_lossy(&out.stdout);
|
||||
return Err(format!(
|
||||
"rsync wall-clock timeout after {}s: stdout={} stderr={}",
|
||||
limit.as_secs(),
|
||||
stdout.trim(),
|
||||
stderr.trim()
|
||||
));
|
||||
}
|
||||
thread::sleep(Duration::from_millis(100));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let out = child
|
||||
.wait_with_output()
|
||||
.map_err(|e| format!("rsync wait_with_output failed: {e}"))?;
|
||||
if out.status.success() {
|
||||
return Ok(());
|
||||
}
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
let stdout = String::from_utf8_lossy(&out.stdout);
|
||||
Err(format!(
|
||||
"rsync failed: status={} stdout={} stderr={}",
|
||||
out.status,
|
||||
stdout.trim(),
|
||||
stderr.trim()
|
||||
))
|
||||
}
|
||||
|
||||
fn run_rsync_fail_fast(
|
||||
&self,
|
||||
src: &str,
|
||||
dst: &Path,
|
||||
profile: RsyncFailFastProfile,
|
||||
) -> Result<(), String> {
|
||||
let mut attempt = 0usize;
|
||||
let mut timeout = profile.initial_wall_clock_timeout;
|
||||
let mut previous_progress = (0usize, 0u64);
|
||||
let mut zero_progress_attempts = 0usize;
|
||||
let max_timeout = std::cmp::max(
|
||||
profile.max_wall_clock_timeout,
|
||||
profile.initial_wall_clock_timeout,
|
||||
);
|
||||
|
||||
loop {
|
||||
attempt += 1;
|
||||
match self.run_rsync_once(src, dst, Some(timeout), true) {
|
||||
Ok(()) => return Ok(()),
|
||||
Err(err) => {
|
||||
if is_hard_fail_rsync_error(&err) {
|
||||
return Err(format!(
|
||||
"rsync fail-fast hard-fail on attempt {}: {}",
|
||||
attempt, err
|
||||
));
|
||||
}
|
||||
if !err.contains("wall-clock timeout") {
|
||||
return Err(err);
|
||||
}
|
||||
let progress = dir_progress(dst)
|
||||
.map_err(|e| format!("rsync fail-fast progress stat failed: {e}"))?;
|
||||
if progress == (0, 0) {
|
||||
zero_progress_attempts += 1;
|
||||
if zero_progress_attempts >= 2 || attempt >= profile.max_attempts {
|
||||
return Err(format!(
|
||||
"rsync fail-fast gave up after {} attempts with no progress: {}",
|
||||
attempt, err
|
||||
));
|
||||
}
|
||||
} else if progress == previous_progress {
|
||||
return Err(format!(
|
||||
"rsync fail-fast gave up after {} attempts with no additional progress: {}",
|
||||
attempt, err
|
||||
));
|
||||
} else {
|
||||
previous_progress = progress;
|
||||
}
|
||||
|
||||
if attempt >= profile.max_attempts {
|
||||
return Err(format!(
|
||||
"rsync fail-fast exhausted {} attempts: {}",
|
||||
profile.max_attempts, err
|
||||
));
|
||||
}
|
||||
timeout = std::cmp::min(timeout.saturating_mul(2), max_timeout);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn scope_fetch_uri(&self, rsync_base_uri: &str) -> String {
|
||||
scoped_rsync_fetch_uri(self.config.scope_policy, rsync_base_uri)
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncFetcher for SystemRsyncFetcher {
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
|
||||
let mut out = Vec::new();
|
||||
self.visit_objects(rsync_base_uri, &mut |uri, bytes| {
|
||||
out.push((uri, bytes));
|
||||
Ok(())
|
||||
})?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn fetch_object(&self, rsync_uri: &str) -> RsyncFetchResult<Vec<u8>> {
|
||||
let parsed =
|
||||
url::Url::parse(rsync_uri).map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
if parsed.scheme() != "rsync" {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"not an rsync URI: {rsync_uri}"
|
||||
)));
|
||||
}
|
||||
let file_name = parsed
|
||||
.path_segments()
|
||||
.and_then(|segments| segments.filter(|segment| !segment.is_empty()).next_back())
|
||||
.ok_or_else(|| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"rsync URI must reference a file object: {rsync_uri}"
|
||||
))
|
||||
})?;
|
||||
let tmp = TempDir::new().map_err(RsyncFetchError::Fetch)?;
|
||||
self.run_rsync(rsync_uri, tmp.path())
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
let object_path = tmp.path().join(file_name);
|
||||
std::fs::read(&object_path).map_err(|e| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"read fetched rsync object failed: {}: {e}",
|
||||
object_path.display()
|
||||
))
|
||||
})
|
||||
}
|
||||
|
||||
fn visit_objects(
|
||||
&self,
|
||||
rsync_base_uri: &str,
|
||||
visitor: &mut dyn FnMut(String, Vec<u8>) -> Result<(), String>,
|
||||
) -> RsyncFetchResult<(usize, u64)> {
|
||||
let base = self.scope_fetch_uri(rsync_base_uri);
|
||||
let mut count = 0usize;
|
||||
let mut bytes_total = 0u64;
|
||||
let mut wrapped = |uri: String, bytes: Vec<u8>| -> Result<(), String> {
|
||||
bytes_total += bytes.len() as u64;
|
||||
count += 1;
|
||||
visitor(uri, bytes)
|
||||
};
|
||||
|
||||
if let Some(dst) = self
|
||||
.mirror_dst_dir(&base)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?
|
||||
{
|
||||
self.run_rsync(&base, &dst)
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
walk_dir_visit(&dst, &dst, &base, &mut wrapped).map_err(RsyncFetchError::Fetch)?;
|
||||
return Ok((count, bytes_total));
|
||||
}
|
||||
|
||||
let tmp = TempDir::new().map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
self.run_rsync(&base, tmp.path())
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
walk_dir_visit(tmp.path(), tmp.path(), &base, &mut wrapped)
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
Ok((count, bytes_total))
|
||||
}
|
||||
|
||||
fn dedup_key(&self, rsync_base_uri: &str) -> String {
|
||||
self.scope_fetch_uri(rsync_base_uri)
|
||||
}
|
||||
|
||||
fn failure_dedup_key(&self, rsync_base_uri: &str) -> Option<String> {
|
||||
scoped_rsync_failure_dedup_key(self.config.scope_policy, rsync_base_uri)
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the exact successful-fetch scope used by the live rsync fetcher.
|
||||
///
|
||||
/// The request scope is shared by live and replay transport paths so their
|
||||
/// request keys and object projection scope remain equivalent.
|
||||
pub fn scoped_rsync_fetch_uri(scope_policy: RsyncScopePolicy, rsync_base_uri: &str) -> String {
|
||||
match scope_policy {
|
||||
RsyncScopePolicy::Host | RsyncScopePolicy::PublicationPoint => {
|
||||
normalize_rsync_base_uri(rsync_base_uri)
|
||||
}
|
||||
RsyncScopePolicy::ModuleRoot => rsync_module_root_uri(rsync_base_uri)
|
||||
.unwrap_or_else(|| normalize_rsync_base_uri(rsync_base_uri)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the live fetcher's failure-deduplication key for the configured
|
||||
/// scope. This is part of the transport request identity, even though host
|
||||
/// scope deliberately does not widen the successful fetch URI.
|
||||
pub fn scoped_rsync_failure_dedup_key(
|
||||
scope_policy: RsyncScopePolicy,
|
||||
rsync_base_uri: &str,
|
||||
) -> Option<String> {
|
||||
match scope_policy {
|
||||
RsyncScopePolicy::Host => rsync_host_scope_uri(rsync_base_uri),
|
||||
RsyncScopePolicy::PublicationPoint | RsyncScopePolicy::ModuleRoot => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn rsync_host_scope_uri(rsync_base_uri: &str) -> Option<String> {
|
||||
let parsed = url::Url::parse(rsync_base_uri).ok()?;
|
||||
if parsed.scheme() != "rsync" {
|
||||
return None;
|
||||
}
|
||||
Some(format!("rsync://{}/", parsed.host_str()?))
|
||||
}
|
||||
|
||||
struct TempDir {
|
||||
path: PathBuf,
|
||||
}
|
||||
|
||||
impl TempDir {
|
||||
fn new() -> Result<Self, String> {
|
||||
let mut p = std::env::temp_dir();
|
||||
p.push(format!("rpki-system-rsync-{}", Uuid::new_v4()));
|
||||
std::fs::create_dir_all(&p).map_err(|e| e.to_string())?;
|
||||
Ok(Self { path: p })
|
||||
}
|
||||
|
||||
fn path(&self) -> &Path {
|
||||
&self.path
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TempDir {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.path);
|
||||
}
|
||||
}
|
||||
|
||||
fn rsync_module_root_uri(s: &str) -> Option<String> {
|
||||
let normalized = normalize_rsync_base_uri(s);
|
||||
let rest = normalized.strip_prefix("rsync://")?;
|
||||
let mut host_and_path = rest.splitn(2, '/');
|
||||
let authority = host_and_path.next()?;
|
||||
let path = host_and_path.next()?;
|
||||
let mut segments: Vec<&str> = path
|
||||
.split('/')
|
||||
.filter(|segment| !segment.is_empty())
|
||||
.collect();
|
||||
if segments.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let module = segments.remove(0);
|
||||
Some(format!("rsync://{authority}/{module}/"))
|
||||
}
|
||||
|
||||
fn walk_dir_collect(
|
||||
root: &Path,
|
||||
current: &Path,
|
||||
rsync_base_uri: &str,
|
||||
out: &mut Vec<(String, Vec<u8>)>,
|
||||
) -> Result<(), String> {
|
||||
let rd = std::fs::read_dir(current).map_err(|e| e.to_string())?;
|
||||
for entry in rd {
|
||||
let entry = entry.map_err(|e| e.to_string())?;
|
||||
let path = entry.path();
|
||||
let meta = entry.metadata().map_err(|e| e.to_string())?;
|
||||
if meta.is_dir() {
|
||||
walk_dir_collect(root, &path, rsync_base_uri, out)?;
|
||||
continue;
|
||||
}
|
||||
if !meta.is_file() {
|
||||
continue;
|
||||
}
|
||||
let rel = path
|
||||
.strip_prefix(root)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
let uri = format!("{rsync_base_uri}{rel}");
|
||||
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
|
||||
out.push((uri, bytes));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn dir_progress(root: &Path) -> Result<(usize, u64), String> {
|
||||
if !root.exists() {
|
||||
return Ok((0, 0));
|
||||
}
|
||||
let mut files = 0usize;
|
||||
let mut bytes = 0u64;
|
||||
let mut stack = vec![root.to_path_buf()];
|
||||
while let Some(path) = stack.pop() {
|
||||
let rd = std::fs::read_dir(&path).map_err(|e| e.to_string())?;
|
||||
for entry in rd {
|
||||
let entry = entry.map_err(|e| e.to_string())?;
|
||||
let path = entry.path();
|
||||
let meta = entry.metadata().map_err(|e| e.to_string())?;
|
||||
if meta.is_dir() {
|
||||
stack.push(path);
|
||||
} else if meta.is_file() {
|
||||
files += 1;
|
||||
bytes += meta.len();
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok((files, bytes))
|
||||
}
|
||||
|
||||
fn is_hard_fail_rsync_error(msg: &str) -> bool {
|
||||
let lower = msg.to_ascii_lowercase();
|
||||
lower.contains("no route to host")
|
||||
|| lower.contains("network is unreachable")
|
||||
|| lower.contains("connection refused")
|
||||
|| lower.contains("name or service not known")
|
||||
}
|
||||
|
||||
fn walk_dir_visit(
|
||||
root: &Path,
|
||||
current: &Path,
|
||||
rsync_base_uri: &str,
|
||||
visitor: &mut dyn FnMut(String, Vec<u8>) -> Result<(), String>,
|
||||
) -> Result<(), String> {
|
||||
let rd = std::fs::read_dir(current).map_err(|e| e.to_string())?;
|
||||
for entry in rd {
|
||||
let entry = entry.map_err(|e| e.to_string())?;
|
||||
let path = entry.path();
|
||||
let meta = entry.metadata().map_err(|e| e.to_string())?;
|
||||
if meta.is_dir() {
|
||||
walk_dir_visit(root, &path, rsync_base_uri, visitor)?;
|
||||
continue;
|
||||
}
|
||||
if !meta.is_file() {
|
||||
continue;
|
||||
}
|
||||
let rel = path
|
||||
.strip_prefix(root)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
let uri = format!("{rsync_base_uri}{rel}");
|
||||
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
|
||||
visitor(uri, bytes)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn normalize_rsync_base_uri_appends_slash_when_missing() {
|
||||
assert_eq!(
|
||||
normalize_rsync_base_uri("rsync://example.net/repo"),
|
||||
"rsync://example.net/repo/".to_string()
|
||||
);
|
||||
assert_eq!(
|
||||
normalize_rsync_base_uri("rsync://example.net/repo/"),
|
||||
"rsync://example.net/repo/".to_string()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn walk_dir_collect_collects_files_and_normalizes_backslashes_in_uri() {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let root = temp.path();
|
||||
std::fs::create_dir_all(root.join("sub")).expect("mkdir");
|
||||
std::fs::write(root.join("sub").join("a.cer"), b"x").expect("write");
|
||||
std::fs::write(root.join("b\\c.mft"), b"y").expect("write backslash file");
|
||||
|
||||
let mut out: Vec<(String, Vec<u8>)> = Vec::new();
|
||||
walk_dir_collect(root, root, "rsync://example.net/repo/", &mut out).expect("walk");
|
||||
out.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
|
||||
assert_eq!(out.len(), 2);
|
||||
assert_eq!(out[0].0, "rsync://example.net/repo/b/c.mft");
|
||||
assert_eq!(out[0].1, b"y");
|
||||
assert_eq!(out[1].0, "rsync://example.net/repo/sub/a.cer");
|
||||
assert_eq!(out[1].1, b"x");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rsync_module_root_uri_returns_host_and_module_only() {
|
||||
assert_eq!(
|
||||
rsync_module_root_uri("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
Some("rsync://example.net/repo/".to_string())
|
||||
);
|
||||
assert_eq!(
|
||||
rsync_module_root_uri("rsync://example.net/repo/ta/"),
|
||||
Some("rsync://example.net/repo/".to_string())
|
||||
);
|
||||
assert_eq!(
|
||||
rsync_module_root_uri("rsync://example.net/repo/"),
|
||||
Some("rsync://example.net/repo/".to_string())
|
||||
);
|
||||
assert_eq!(rsync_module_root_uri("https://example.net/repo"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rsync_host_scope_uri_returns_host_only() {
|
||||
assert_eq!(
|
||||
rsync_host_scope_uri("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
Some("rsync://example.net/".to_string())
|
||||
);
|
||||
assert_eq!(rsync_host_scope_uri("https://example.net/repo"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_rsync_dedup_key_uses_module_root_by_default() {
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig::default());
|
||||
assert_eq!(
|
||||
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
"rsync://example.net/repo/"
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher.failure_dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_rsync_host_scope_does_not_widen_success_fetch_scope() {
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
scope_policy: RsyncScopePolicy::Host,
|
||||
..SystemRsyncConfig::default()
|
||||
});
|
||||
assert_eq!(
|
||||
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
"rsync://example.net/repo/ta/ca/publication-point/"
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher.scope_fetch_uri("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
"rsync://example.net/repo/ta/ca/publication-point/"
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher.failure_dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
Some("rsync://example.net/".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_rsync_dedup_key_uses_publication_point_scope_when_configured() {
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
scope_policy: RsyncScopePolicy::PublicationPoint,
|
||||
..SystemRsyncConfig::default()
|
||||
});
|
||||
assert_eq!(
|
||||
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
"rsync://example.net/repo/ta/ca/publication-point/"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_rsync_dedup_key_uses_module_root_when_configured() {
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
scope_policy: RsyncScopePolicy::ModuleRoot,
|
||||
..SystemRsyncConfig::default()
|
||||
});
|
||||
assert_eq!(
|
||||
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
|
||||
"rsync://example.net/repo/"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_rsync_fetcher_reports_spawn_and_exit_errors() {
|
||||
let dst = tempfile::tempdir().expect("tempdir");
|
||||
|
||||
// 1) Spawn error.
|
||||
let f = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: PathBuf::from("/this/does/not/exist/rsync"),
|
||||
connect_timeout: Duration::from_secs(1),
|
||||
timeout: Duration::from_secs(1),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
let e = f
|
||||
.run_rsync("rsync://example.net/repo/", dst.path())
|
||||
.expect_err("spawn must fail");
|
||||
assert!(e.contains("rsync spawn failed:"), "{e}");
|
||||
|
||||
// 2) Non-zero exit status.
|
||||
let f = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: PathBuf::from("false"),
|
||||
connect_timeout: Duration::from_secs(1),
|
||||
timeout: Duration::from_secs(1),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
let e = f
|
||||
.run_rsync("rsync://example.net/repo/", dst.path())
|
||||
.expect_err("false must fail");
|
||||
assert!(e.contains("rsync failed:"), "{e}");
|
||||
assert!(e.contains("status="), "{e}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mirror_dst_dir_reports_root_creation_error() {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let root_file = temp.path().join("mirror-root-file");
|
||||
std::fs::write(&root_file, b"not a directory").expect("write root file");
|
||||
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: PathBuf::from("rsync"),
|
||||
connect_timeout: Duration::from_secs(1),
|
||||
timeout: Duration::from_secs(1),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: Some(root_file.clone()),
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
|
||||
let err = fetcher
|
||||
.mirror_dst_dir("rsync://example.net/repo/")
|
||||
.expect_err("file mirror root must fail");
|
||||
assert!(err.contains("create rsync mirror root failed"), "{err}");
|
||||
assert!(err.contains(&root_file.display().to_string()), "{err}");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn mirror_dst_dir_reports_directory_creation_error_inside_root() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let root = temp.path().join("mirror");
|
||||
std::fs::create_dir_all(&root).expect("mkdir root");
|
||||
let mut perms = std::fs::metadata(&root).expect("metadata").permissions();
|
||||
perms.set_mode(0o555);
|
||||
std::fs::set_permissions(&root, perms).expect("chmod root readonly");
|
||||
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: PathBuf::from("rsync"),
|
||||
connect_timeout: Duration::from_secs(1),
|
||||
timeout: Duration::from_secs(1),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: Some(root.clone()),
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
|
||||
let err = fetcher
|
||||
.mirror_dst_dir("rsync://example.net/repo/")
|
||||
.expect_err("readonly mirror root must fail");
|
||||
assert!(
|
||||
err.contains("create rsync mirror directory failed"),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let mut perms = std::fs::metadata(&root).expect("metadata").permissions();
|
||||
perms.set_mode(0o755);
|
||||
std::fs::set_permissions(&root, perms).expect("restore perms");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn walk_dir_collect_ignores_non_file_entries() {
|
||||
use std::os::unix::net::UnixListener;
|
||||
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let root = temp.path();
|
||||
std::fs::write(root.join("a.cer"), b"x").expect("write file");
|
||||
let socket_path = root.join("skip.sock");
|
||||
let _listener = UnixListener::bind(&socket_path).expect("bind socket");
|
||||
|
||||
let mut out: Vec<(String, Vec<u8>)> = Vec::new();
|
||||
walk_dir_collect(root, root, "rsync://example.net/repo/", &mut out).expect("walk");
|
||||
|
||||
assert_eq!(out.len(), 1);
|
||||
assert_eq!(out[0].0, "rsync://example.net/repo/a.cer");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn rsync_fail_fast_retries_when_progress_is_made() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let script = temp.path().join("fake-rsync.sh");
|
||||
let state = temp.path().join("state.txt");
|
||||
std::fs::write(
|
||||
&script,
|
||||
format!(
|
||||
"#!/usr/bin/env bash\nset -euo pipefail\nSTATE=\"{}\"\nDST=\"${{@: -1}}\"\nCOUNT=0\nif [[ -f \"$STATE\" ]]; then COUNT=$(cat \"$STATE\"); fi\nCOUNT=$((COUNT+1))\necho \"$COUNT\" > \"$STATE\"\nmkdir -p \"$DST\"\nif [[ \"$COUNT\" -eq 1 ]]; then\n echo first > \"$DST/part1\"\n sleep 2\nelse\n echo second > \"$DST/part2\"\nfi\n",
|
||||
state.display()
|
||||
),
|
||||
)
|
||||
.expect("write script");
|
||||
let mut perms = std::fs::metadata(&script).unwrap().permissions();
|
||||
perms.set_mode(0o755);
|
||||
std::fs::set_permissions(&script, perms).unwrap();
|
||||
|
||||
let dst = temp.path().join("dst");
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: script,
|
||||
connect_timeout: Duration::from_secs(15),
|
||||
timeout: Duration::from_secs(60),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
|
||||
fetcher
|
||||
.run_rsync_fail_fast(
|
||||
"rsync://example.net/repo/",
|
||||
&dst,
|
||||
RsyncFailFastProfile {
|
||||
initial_wall_clock_timeout: Duration::from_secs(1),
|
||||
max_wall_clock_timeout: Duration::from_secs(4),
|
||||
max_attempts: 3,
|
||||
},
|
||||
)
|
||||
.expect("eventual success");
|
||||
|
||||
assert!(dst.join("part1").exists());
|
||||
assert!(dst.join("part2").exists());
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn rsync_fail_fast_gives_up_after_two_zero_progress_timeouts() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let script = temp.path().join("fake-rsync.sh");
|
||||
std::fs::write(&script, "#!/usr/bin/env bash\nset -euo pipefail\nsleep 5\n")
|
||||
.expect("write script");
|
||||
let mut perms = std::fs::metadata(&script).unwrap().permissions();
|
||||
perms.set_mode(0o755);
|
||||
std::fs::set_permissions(&script, perms).unwrap();
|
||||
|
||||
let dst = temp.path().join("dst");
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: script,
|
||||
connect_timeout: Duration::from_secs(15),
|
||||
timeout: Duration::from_secs(60),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
|
||||
let err = fetcher
|
||||
.run_rsync_fail_fast(
|
||||
"rsync://example.net/repo/",
|
||||
&dst,
|
||||
RsyncFailFastProfile {
|
||||
initial_wall_clock_timeout: Duration::from_secs(1),
|
||||
max_wall_clock_timeout: Duration::from_secs(2),
|
||||
max_attempts: 4,
|
||||
},
|
||||
)
|
||||
.expect_err("must fail");
|
||||
assert!(err.contains("no progress"), "{err}");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn rsync_fail_fast_hard_fail_stops_after_first_attempt() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let script = temp.path().join("fake-rsync.sh");
|
||||
let state = temp.path().join("state.txt");
|
||||
std::fs::write(
|
||||
&script,
|
||||
format!(
|
||||
"#!/usr/bin/env bash\nset -euo pipefail\nSTATE=\"{}\"\nCOUNT=0\nif [[ -f \"$STATE\" ]]; then COUNT=$(cat \"$STATE\"); fi\nCOUNT=$((COUNT+1))\necho \"$COUNT\" > \"$STATE\"\necho 'rsync: [Receiver] failed to connect to host (1.2.3.4): Connection refused (111)' >&2\nexit 10\n",
|
||||
state.display()
|
||||
),
|
||||
)
|
||||
.expect("write script");
|
||||
let mut perms = std::fs::metadata(&script).unwrap().permissions();
|
||||
perms.set_mode(0o755);
|
||||
std::fs::set_permissions(&script, perms).unwrap();
|
||||
|
||||
let dst = temp.path().join("dst");
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: script,
|
||||
connect_timeout: Duration::from_secs(15),
|
||||
timeout: Duration::from_secs(60),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
|
||||
let err = fetcher
|
||||
.run_rsync_fail_fast(
|
||||
"rsync://example.net/repo/",
|
||||
&dst,
|
||||
RsyncFailFastProfile {
|
||||
initial_wall_clock_timeout: Duration::from_secs(10),
|
||||
max_wall_clock_timeout: Duration::from_secs(80),
|
||||
max_attempts: 4,
|
||||
},
|
||||
)
|
||||
.expect_err("must hard fail");
|
||||
assert!(err.contains("hard-fail"), "{err}");
|
||||
let count = std::fs::read_to_string(&state).unwrap();
|
||||
assert_eq!(count.trim(), "1");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn run_rsync_once_passes_contimeout_and_timeout_args() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let script = temp.path().join("capture-rsync.sh");
|
||||
let args_file = temp.path().join("args.txt");
|
||||
std::fs::write(
|
||||
&script,
|
||||
format!(
|
||||
"#!/usr/bin/env bash\nset -euo pipefail\nprintf '%s\\n' \"$@\" > \"{}\"\nDST=\"${{@: -1}}\"\nmkdir -p \"$DST\"\n",
|
||||
args_file.display()
|
||||
),
|
||||
)
|
||||
.expect("write script");
|
||||
let mut perms = std::fs::metadata(&script).unwrap().permissions();
|
||||
perms.set_mode(0o755);
|
||||
std::fs::set_permissions(&script, perms).unwrap();
|
||||
|
||||
let dst = temp.path().join("dst");
|
||||
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
|
||||
rsync_bin: script,
|
||||
connect_timeout: Duration::from_secs(15),
|
||||
timeout: Duration::from_secs(30),
|
||||
extra_args: Vec::new(),
|
||||
mirror_root: None,
|
||||
scope_policy: RsyncScopePolicy::default(),
|
||||
});
|
||||
|
||||
fetcher
|
||||
.run_rsync_once("rsync://example.net/repo/", &dst, None, false)
|
||||
.expect("rsync");
|
||||
|
||||
let args = std::fs::read_to_string(&args_file).expect("read args");
|
||||
assert!(args.contains("--contimeout\n15\n"), "{args}");
|
||||
assert!(args.contains("--timeout\n30\n"), "{args}");
|
||||
}
|
||||
}
|
||||
54
crates/panda-rpki-validator/src/lib.rs
Normal file
54
crates/panda-rpki-validator/src/lib.rs
Normal file
@ -0,0 +1,54 @@
|
||||
//! Public package boundary for the Panda RPKI synchronization validator.
|
||||
//!
|
||||
//! The package intentionally keeps the audited implementation in one crate
|
||||
//! while exposing only the normal synchronization/validation CLI. A separate
|
||||
//! offline revalidation workflow remains in the original private repository
|
||||
//! until a follow-up backlog item defines its public contract.
|
||||
//!
|
||||
//! The implementation is extracted from the private `rpki` tree. During the
|
||||
//! parity window we intentionally keep the upstream implementation's lint
|
||||
//! profile unchanged; functional and output-equivalence gates take precedence
|
||||
//! over behavior-changing lint cleanup.
|
||||
#![allow(clippy::all)]
|
||||
#![allow(clippy::pedantic, clippy::nursery, clippy::restriction)]
|
||||
#![allow(dead_code, deprecated, unused_mut)]
|
||||
|
||||
pub mod analysis;
|
||||
pub mod audit;
|
||||
pub mod audit_downloads;
|
||||
pub mod blob_store;
|
||||
pub mod ccr;
|
||||
pub mod cir;
|
||||
pub mod cli;
|
||||
pub mod contract;
|
||||
pub mod crypto_sig_cache;
|
||||
pub mod current_repo_index;
|
||||
pub mod daemon;
|
||||
pub mod data_model;
|
||||
pub mod fetch;
|
||||
pub mod memory_telemetry;
|
||||
pub mod object_projection;
|
||||
pub mod parallel;
|
||||
pub mod policy;
|
||||
pub mod progress_log;
|
||||
pub mod replay;
|
||||
pub mod report;
|
||||
pub mod storage;
|
||||
pub mod sync;
|
||||
pub mod ta_constraints;
|
||||
pub mod validation;
|
||||
|
||||
pub const COMPONENT_NAME: &str = "panda-rpki-validator";
|
||||
pub const COMPONENT_VERSION: &str = env!("CARGO_PKG_VERSION");
|
||||
|
||||
/// Runs the normal synchronization/validation CLI.
|
||||
pub fn run<I, S>(args: I) -> Result<(), String>
|
||||
where
|
||||
I: IntoIterator<Item = S>,
|
||||
S: Into<String>,
|
||||
{
|
||||
let mut argv = Vec::with_capacity(1);
|
||||
argv.push(COMPONENT_NAME.to_string());
|
||||
argv.extend(args.into_iter().map(Into::into));
|
||||
cli::run(&argv)
|
||||
}
|
||||
6
crates/panda-rpki-validator/src/main.rs
Normal file
6
crates/panda-rpki-validator/src/main.rs
Normal file
@ -0,0 +1,6 @@
|
||||
fn main() {
|
||||
if let Err(error) = panda_rpki_validator::run(std::env::args().skip(1)) {
|
||||
eprintln!("panda-rpki-validator: {error}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
348
crates/panda-rpki-validator/src/memory_telemetry.rs
Normal file
348
crates/panda-rpki-validator/src/memory_telemetry.rs
Normal file
@ -0,0 +1,348 @@
|
||||
use serde::Serialize;
|
||||
|
||||
use crate::storage::RocksDbMemorySnapshot;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct MallocTrimProbe {
|
||||
pub supported: bool,
|
||||
pub return_value: Option<i32>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct ProcessMemorySnapshot {
|
||||
pub label: String,
|
||||
pub vm_rss_kb: Option<u64>,
|
||||
pub vm_size_kb: Option<u64>,
|
||||
pub vm_data_kb: Option<u64>,
|
||||
pub vm_swap_kb: Option<u64>,
|
||||
pub rss_anon_kb: Option<u64>,
|
||||
pub rss_file_kb: Option<u64>,
|
||||
pub rss_shmem_kb: Option<u64>,
|
||||
pub threads: Option<u64>,
|
||||
pub fd_count: Option<u64>,
|
||||
pub smaps_rollup: Option<SmapsRollupSnapshot>,
|
||||
pub smaps_mapping_summary: Option<SmapsMappingSummary>,
|
||||
pub errors: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct SmapsRollupSnapshot {
|
||||
pub rss_kb: Option<u64>,
|
||||
pub pss_kb: Option<u64>,
|
||||
pub shared_clean_kb: Option<u64>,
|
||||
pub shared_dirty_kb: Option<u64>,
|
||||
pub private_clean_kb: Option<u64>,
|
||||
pub private_dirty_kb: Option<u64>,
|
||||
pub anonymous_kb: Option<u64>,
|
||||
pub swap_kb: Option<u64>,
|
||||
pub swap_pss_kb: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct SmapsMappingSummary {
|
||||
pub heap: SmapsMappingCategory,
|
||||
pub anonymous_mmap: SmapsMappingCategory,
|
||||
pub file_backed: SmapsMappingCategory,
|
||||
pub stack: SmapsMappingCategory,
|
||||
pub special: SmapsMappingCategory,
|
||||
pub total: SmapsMappingCategory,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct SmapsMappingCategory {
|
||||
pub mappings: u64,
|
||||
pub size_kb: u64,
|
||||
pub rss_kb: u64,
|
||||
pub pss_kb: u64,
|
||||
pub private_clean_kb: u64,
|
||||
pub private_dirty_kb: u64,
|
||||
pub anonymous_kb: u64,
|
||||
pub largest_mapping_rss_kb: u64,
|
||||
pub large_mapping_count_64m: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct MemoryTelemetryCheckpoint {
|
||||
pub label: String,
|
||||
pub elapsed_ms: u64,
|
||||
pub process: ProcessMemorySnapshot,
|
||||
pub rocksdb: RocksDbMemorySnapshot,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct MemoryTelemetrySummary {
|
||||
pub checkpoints: Vec<MemoryTelemetryCheckpoint>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub object_graph: Option<ObjectGraphMemorySummary>,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub malloc_trim_probes: Vec<MallocTrimProbe>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct ObjectGraphMemorySummary {
|
||||
pub captured_at_label: String,
|
||||
pub total_estimated_bytes: u64,
|
||||
pub sections: Vec<ObjectGraphMemorySection>,
|
||||
pub notes: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct ObjectGraphMemorySection {
|
||||
pub name: String,
|
||||
pub item_count: u64,
|
||||
pub shallow_bytes: u64,
|
||||
pub heap_bytes: u64,
|
||||
pub estimated_bytes: u64,
|
||||
pub string_count: u64,
|
||||
pub string_bytes: u64,
|
||||
pub string_capacity_bytes: u64,
|
||||
pub vec_count: u64,
|
||||
pub vec_heap_bytes: u64,
|
||||
pub vec_capacity_bytes: u64,
|
||||
pub details: Vec<ObjectGraphMemoryMetric>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||
pub struct ObjectGraphMemoryMetric {
|
||||
pub name: String,
|
||||
pub value: u64,
|
||||
}
|
||||
|
||||
pub fn process_memory_snapshot(label: impl Into<String>) -> ProcessMemorySnapshot {
|
||||
let label = label.into();
|
||||
let mut snapshot = ProcessMemorySnapshot {
|
||||
label,
|
||||
vm_rss_kb: None,
|
||||
vm_size_kb: None,
|
||||
vm_data_kb: None,
|
||||
vm_swap_kb: None,
|
||||
rss_anon_kb: None,
|
||||
rss_file_kb: None,
|
||||
rss_shmem_kb: None,
|
||||
threads: None,
|
||||
fd_count: current_fd_count(),
|
||||
smaps_rollup: None,
|
||||
smaps_mapping_summary: None,
|
||||
errors: Vec::new(),
|
||||
};
|
||||
|
||||
match std::fs::read_to_string("/proc/self/status") {
|
||||
Ok(status) => parse_status(&status, &mut snapshot),
|
||||
Err(err) => snapshot
|
||||
.errors
|
||||
.push(format!("read /proc/self/status failed: {err}")),
|
||||
}
|
||||
|
||||
match std::fs::read_to_string("/proc/self/smaps_rollup") {
|
||||
Ok(smaps) => snapshot.smaps_rollup = Some(parse_smaps_rollup(&smaps)),
|
||||
Err(err) => snapshot
|
||||
.errors
|
||||
.push(format!("read /proc/self/smaps_rollup failed: {err}")),
|
||||
}
|
||||
|
||||
match std::fs::read_to_string("/proc/self/smaps") {
|
||||
Ok(smaps) => snapshot.smaps_mapping_summary = Some(parse_smaps_mapping_summary(&smaps)),
|
||||
Err(err) => snapshot
|
||||
.errors
|
||||
.push(format!("read /proc/self/smaps failed: {err}")),
|
||||
}
|
||||
|
||||
snapshot
|
||||
}
|
||||
|
||||
pub fn malloc_trim_probe() -> MallocTrimProbe {
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
{
|
||||
MallocTrimProbe {
|
||||
supported: true,
|
||||
return_value: Some(unsafe { malloc_trim(0) }),
|
||||
}
|
||||
}
|
||||
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
|
||||
{
|
||||
MallocTrimProbe {
|
||||
supported: false,
|
||||
return_value: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
unsafe extern "C" {
|
||||
fn malloc_trim(pad: usize) -> i32;
|
||||
}
|
||||
|
||||
fn current_fd_count() -> Option<u64> {
|
||||
std::fs::read_dir("/proc/self/fd")
|
||||
.ok()
|
||||
.map(|entries| entries.filter_map(Result::ok).count() as u64)
|
||||
}
|
||||
|
||||
fn parse_status(status: &str, snapshot: &mut ProcessMemorySnapshot) {
|
||||
for line in status.lines() {
|
||||
let Some((key, value)) = line.split_once(':') else {
|
||||
continue;
|
||||
};
|
||||
let parsed = parse_kb_or_plain_u64(value);
|
||||
match key {
|
||||
"VmRSS" => snapshot.vm_rss_kb = parsed,
|
||||
"VmSize" => snapshot.vm_size_kb = parsed,
|
||||
"VmData" => snapshot.vm_data_kb = parsed,
|
||||
"VmSwap" => snapshot.vm_swap_kb = parsed,
|
||||
"RssAnon" => snapshot.rss_anon_kb = parsed,
|
||||
"RssFile" => snapshot.rss_file_kb = parsed,
|
||||
"RssShmem" => snapshot.rss_shmem_kb = parsed,
|
||||
"Threads" => snapshot.threads = parsed,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_smaps_rollup(smaps: &str) -> SmapsRollupSnapshot {
|
||||
let mut snapshot = SmapsRollupSnapshot::default();
|
||||
for line in smaps.lines() {
|
||||
let Some((key, value)) = line.split_once(':') else {
|
||||
continue;
|
||||
};
|
||||
let parsed = parse_kb_or_plain_u64(value);
|
||||
match key {
|
||||
"Rss" => snapshot.rss_kb = parsed,
|
||||
"Pss" => snapshot.pss_kb = parsed,
|
||||
"Shared_Clean" => snapshot.shared_clean_kb = parsed,
|
||||
"Shared_Dirty" => snapshot.shared_dirty_kb = parsed,
|
||||
"Private_Clean" => snapshot.private_clean_kb = parsed,
|
||||
"Private_Dirty" => snapshot.private_dirty_kb = parsed,
|
||||
"Anonymous" => snapshot.anonymous_kb = parsed,
|
||||
"Swap" => snapshot.swap_kb = parsed,
|
||||
"SwapPss" => snapshot.swap_pss_kb = parsed,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
snapshot
|
||||
}
|
||||
|
||||
fn parse_smaps_mapping_summary(smaps: &str) -> SmapsMappingSummary {
|
||||
let mut summary = SmapsMappingSummary::default();
|
||||
let mut current_path = String::new();
|
||||
let mut current = SmapsMappingCategory::default();
|
||||
let mut have_mapping = false;
|
||||
|
||||
for line in smaps.lines() {
|
||||
if is_smaps_mapping_header(line) {
|
||||
if have_mapping {
|
||||
add_mapping(&mut summary, ¤t_path, ¤t);
|
||||
}
|
||||
current_path = smaps_header_path(line);
|
||||
current = SmapsMappingCategory {
|
||||
mappings: 1,
|
||||
..SmapsMappingCategory::default()
|
||||
};
|
||||
have_mapping = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if !have_mapping {
|
||||
continue;
|
||||
}
|
||||
|
||||
let Some((key, value)) = line.split_once(':') else {
|
||||
continue;
|
||||
};
|
||||
let parsed = parse_kb_or_plain_u64(value).unwrap_or(0);
|
||||
match key {
|
||||
"Size" => current.size_kb = parsed,
|
||||
"Rss" => current.rss_kb = parsed,
|
||||
"Pss" => current.pss_kb = parsed,
|
||||
"Private_Clean" => current.private_clean_kb = parsed,
|
||||
"Private_Dirty" => current.private_dirty_kb = parsed,
|
||||
"Anonymous" => current.anonymous_kb = parsed,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
if have_mapping {
|
||||
add_mapping(&mut summary, ¤t_path, ¤t);
|
||||
}
|
||||
|
||||
summary
|
||||
}
|
||||
|
||||
fn is_smaps_mapping_header(line: &str) -> bool {
|
||||
let mut parts = line.split_whitespace();
|
||||
let Some(range) = parts.next() else {
|
||||
return false;
|
||||
};
|
||||
let Some(perms) = parts.next() else {
|
||||
return false;
|
||||
};
|
||||
let Some((start, end)) = range.split_once('-') else {
|
||||
return false;
|
||||
};
|
||||
!start.is_empty()
|
||||
&& !end.is_empty()
|
||||
&& start.as_bytes().iter().all(u8::is_ascii_hexdigit)
|
||||
&& end.as_bytes().iter().all(u8::is_ascii_hexdigit)
|
||||
&& perms.len() == 4
|
||||
&& perms
|
||||
.as_bytes()
|
||||
.iter()
|
||||
.all(|b| matches!(b, b'r' | b'w' | b'x' | b's' | b'p' | b'-'))
|
||||
}
|
||||
|
||||
fn smaps_header_path(line: &str) -> String {
|
||||
line.split_whitespace()
|
||||
.skip(5)
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ")
|
||||
}
|
||||
|
||||
fn add_mapping(summary: &mut SmapsMappingSummary, path: &str, mapping: &SmapsMappingCategory) {
|
||||
add_category(&mut summary.total, mapping);
|
||||
match mapping_category(path) {
|
||||
MappingCategory::Heap => add_category(&mut summary.heap, mapping),
|
||||
MappingCategory::AnonymousMmap => add_category(&mut summary.anonymous_mmap, mapping),
|
||||
MappingCategory::FileBacked => add_category(&mut summary.file_backed, mapping),
|
||||
MappingCategory::Stack => add_category(&mut summary.stack, mapping),
|
||||
MappingCategory::Special => add_category(&mut summary.special, mapping),
|
||||
}
|
||||
}
|
||||
|
||||
fn add_category(target: &mut SmapsMappingCategory, source: &SmapsMappingCategory) {
|
||||
target.mappings += source.mappings;
|
||||
target.size_kb += source.size_kb;
|
||||
target.rss_kb += source.rss_kb;
|
||||
target.pss_kb += source.pss_kb;
|
||||
target.private_clean_kb += source.private_clean_kb;
|
||||
target.private_dirty_kb += source.private_dirty_kb;
|
||||
target.anonymous_kb += source.anonymous_kb;
|
||||
target.largest_mapping_rss_kb = target.largest_mapping_rss_kb.max(source.rss_kb);
|
||||
if source.rss_kb >= 64 * 1024 {
|
||||
target.large_mapping_count_64m += source.mappings;
|
||||
}
|
||||
}
|
||||
|
||||
enum MappingCategory {
|
||||
Heap,
|
||||
AnonymousMmap,
|
||||
FileBacked,
|
||||
Stack,
|
||||
Special,
|
||||
}
|
||||
|
||||
fn mapping_category(path: &str) -> MappingCategory {
|
||||
if path == "[heap]" {
|
||||
MappingCategory::Heap
|
||||
} else if path.starts_with("[stack") {
|
||||
MappingCategory::Stack
|
||||
} else if path.is_empty() {
|
||||
MappingCategory::AnonymousMmap
|
||||
} else if path.starts_with('/') {
|
||||
MappingCategory::FileBacked
|
||||
} else {
|
||||
MappingCategory::Special
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_kb_or_plain_u64(value: &str) -> Option<u64> {
|
||||
value.split_whitespace().next()?.parse::<u64>().ok()
|
||||
}
|
||||
668
crates/panda-rpki-validator/src/object_projection.rs
Normal file
668
crates/panda-rpki-validator/src/object_projection.rs
Normal file
@ -0,0 +1,668 @@
|
||||
use std::net::{Ipv4Addr, Ipv6Addr};
|
||||
use std::path::Path;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::{Value, json};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::data_model::aspa::AspaObject;
|
||||
use crate::data_model::crl::RpkixCrl;
|
||||
use crate::data_model::manifest::ManifestObject;
|
||||
use crate::data_model::rc::{
|
||||
AccessDescription, RcExtensions, ResourceCertificate, SubjectInfoAccess,
|
||||
};
|
||||
use crate::data_model::roa::{IpPrefix as RoaIpPrefix, RoaAfi, RoaObject};
|
||||
use crate::data_model::signed_object::{
|
||||
ResourceEeCertificate, RpkiSignedObject, SignedAttrsProfiled, SignerInfoProfiled,
|
||||
};
|
||||
use crate::data_model::ta::TaCertificate;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ObjectType {
|
||||
Auto,
|
||||
Cer,
|
||||
Mft,
|
||||
Crl,
|
||||
Roa,
|
||||
Aspa,
|
||||
}
|
||||
|
||||
impl ObjectType {
|
||||
pub fn parse(value: &str) -> Result<Self, String> {
|
||||
match value.to_ascii_lowercase().as_str() {
|
||||
"auto" => Ok(Self::Auto),
|
||||
"cer" | ".cer" | "cert" | "certificate" => Ok(Self::Cer),
|
||||
"mft" | ".mft" | "manifest" => Ok(Self::Mft),
|
||||
"crl" | ".crl" => Ok(Self::Crl),
|
||||
"roa" | ".roa" => Ok(Self::Roa),
|
||||
"asa" | ".asa" | "aspa" => Ok(Self::Aspa),
|
||||
_ => Err(format!("unsupported object type: {value}")),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn label(self) -> &'static str {
|
||||
object_type_label(self)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn resolve_object_type(object_type: ObjectType, path: &Path) -> Result<ObjectType, String> {
|
||||
if object_type != ObjectType::Auto {
|
||||
return Ok(object_type);
|
||||
}
|
||||
match path
|
||||
.extension()
|
||||
.and_then(|v| v.to_str())
|
||||
.map(|v| v.to_ascii_lowercase())
|
||||
.as_deref()
|
||||
{
|
||||
Some("cer") => Ok(ObjectType::Cer),
|
||||
Some("mft") => Ok(ObjectType::Mft),
|
||||
Some("crl") => Ok(ObjectType::Crl),
|
||||
Some("roa") => Ok(ObjectType::Roa),
|
||||
Some("asa") | Some("aspa") => Ok(ObjectType::Aspa),
|
||||
_ => Err(format!(
|
||||
"cannot infer object type from path: {}",
|
||||
path.display()
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ObjectProjectionRecord {
|
||||
pub schema_version: u32,
|
||||
pub sha256: String,
|
||||
pub object_type: String,
|
||||
pub parse_status: String,
|
||||
pub error_summary: Option<String>,
|
||||
pub projection: Value,
|
||||
}
|
||||
|
||||
pub fn build_object_projection(
|
||||
object_type: ObjectType,
|
||||
input_path: &Path,
|
||||
bytes: &[u8],
|
||||
entry_limit: usize,
|
||||
) -> ObjectProjectionRecord {
|
||||
let resolved = match resolve_object_type(object_type, input_path) {
|
||||
Ok(value) => value,
|
||||
Err(err) => {
|
||||
return ObjectProjectionRecord {
|
||||
schema_version: 1,
|
||||
sha256: sha256_hex(bytes),
|
||||
object_type: "unknown".to_string(),
|
||||
parse_status: "error".to_string(),
|
||||
error_summary: Some(err),
|
||||
projection: json!({"decode": {"profileValid": false}}),
|
||||
};
|
||||
}
|
||||
};
|
||||
let projection = parse_object_json(resolved, input_path, bytes, entry_limit);
|
||||
let parse_status = if projection
|
||||
.get("object")
|
||||
.and_then(|v| v.get("decode"))
|
||||
.and_then(|v| v.get("profileValid"))
|
||||
.and_then(Value::as_bool)
|
||||
.unwrap_or(false)
|
||||
{
|
||||
"ok"
|
||||
} else {
|
||||
"error"
|
||||
};
|
||||
let error_summary = projection
|
||||
.get("object")
|
||||
.and_then(|v| v.get("decode"))
|
||||
.and_then(|v| v.get("error"))
|
||||
.and_then(Value::as_str)
|
||||
.map(str::to_string);
|
||||
ObjectProjectionRecord {
|
||||
schema_version: 1,
|
||||
sha256: sha256_hex(bytes),
|
||||
object_type: resolved.label().to_string(),
|
||||
parse_status: parse_status.to_string(),
|
||||
error_summary,
|
||||
projection,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_object_json(
|
||||
object_type: ObjectType,
|
||||
input_path: &Path,
|
||||
bytes: &[u8],
|
||||
entry_limit: usize,
|
||||
) -> Value {
|
||||
let object = match object_type {
|
||||
ObjectType::Auto => unreachable!("auto must be resolved"),
|
||||
ObjectType::Cer => parse_cer_json(bytes),
|
||||
ObjectType::Mft => parse_mft_json(bytes, entry_limit),
|
||||
ObjectType::Crl => parse_crl_json(bytes, entry_limit),
|
||||
ObjectType::Roa => parse_roa_json(bytes, entry_limit),
|
||||
ObjectType::Aspa => parse_aspa_json(bytes, entry_limit),
|
||||
};
|
||||
json!({
|
||||
"tool": "rpki_object_parse",
|
||||
"schemaVersion": 1,
|
||||
"input": {
|
||||
"path": input_path.display().to_string(),
|
||||
"type": object_type_label(object_type),
|
||||
"bytes": bytes_summary(bytes),
|
||||
},
|
||||
"object": object,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn parse_cer_json(bytes: &[u8]) -> Value {
|
||||
match ResourceCertificate::decode_der(bytes) {
|
||||
Ok(cert) => {
|
||||
let ta_profile = match TaCertificate::decode_der(bytes) {
|
||||
Ok(ta) => json!({
|
||||
"valid": true,
|
||||
"selfSignature": result_json(ta.verify_self_signature().map_err(|e| e.to_string())),
|
||||
}),
|
||||
Err(err) => json!({
|
||||
"valid": false,
|
||||
"error": err.to_string(),
|
||||
}),
|
||||
};
|
||||
json!({
|
||||
"type": "cer",
|
||||
"decode": {"profileValid": true},
|
||||
"resourceCertificate": resource_certificate_json(&cert),
|
||||
"trustAnchorProfile": ta_profile,
|
||||
})
|
||||
}
|
||||
Err(err) => json!({
|
||||
"type": "cer",
|
||||
"decode": {"profileValid": false, "error": err.to_string()},
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_mft_json(bytes: &[u8], entry_limit: usize) -> Value {
|
||||
match ManifestObject::decode_der(bytes) {
|
||||
Ok(mft) => {
|
||||
let files = mft.manifest.parse_files();
|
||||
let (file_sample, file_list_error) = match files {
|
||||
Ok(entries) => (
|
||||
json!({
|
||||
"count": entries.len(),
|
||||
"truncated": entries.len() > entry_limit,
|
||||
"entries": entries.iter().take(entry_limit).map(|item| {
|
||||
json!({"fileName": item.file_name, "hashHex": hex::encode(item.hash_bytes)})
|
||||
}).collect::<Vec<_>>(),
|
||||
}),
|
||||
Value::Null,
|
||||
),
|
||||
Err(err) => (Value::Null, json!(err.to_string())),
|
||||
};
|
||||
json!({
|
||||
"type": "mft",
|
||||
"decode": {"profileValid": true},
|
||||
"eContentType": mft.econtent_type,
|
||||
"signedObject": signed_object_json(&mft.signed_object),
|
||||
"manifest": {
|
||||
"version": mft.manifest.version,
|
||||
"manifestNumberHex": mft.manifest.manifest_number.to_hex_upper(),
|
||||
"thisUpdate": format_time(mft.manifest.this_update),
|
||||
"nextUpdate": format_time(mft.manifest.next_update),
|
||||
"fileHashAlg": mft.manifest.file_hash_alg,
|
||||
"fileCount": mft.manifest.file_count(),
|
||||
"fileList": file_sample,
|
||||
"fileListError": file_list_error,
|
||||
},
|
||||
"embeddedEeProfile": result_json(mft.validate_embedded_ee_cert().map_err(|e| e.to_string())),
|
||||
"cmsSignature": result_json(mft.signed_object.verify_signature().map_err(|e| e.to_string())),
|
||||
})
|
||||
}
|
||||
Err(err) => json!({
|
||||
"type": "mft",
|
||||
"decode": {"profileValid": false, "error": err.to_string()},
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_crl_json(bytes: &[u8], entry_limit: usize) -> Value {
|
||||
match RpkixCrl::decode_der(bytes) {
|
||||
Ok(crl) => json!({
|
||||
"type": "crl",
|
||||
"decode": {"profileValid": true},
|
||||
"rawDer": bytes_summary(&crl.raw_der),
|
||||
"version": crl.version,
|
||||
"issuer": crl.issuer_dn,
|
||||
"signatureAlgorithm": crl.signature_algorithm_oid,
|
||||
"thisUpdate": format_time(crl.this_update.utc),
|
||||
"nextUpdate": format_time(crl.next_update.utc),
|
||||
"extensions": {
|
||||
"authorityKeyIdentifier": hex::encode(&crl.extensions.authority_key_identifier),
|
||||
"crlNumberHex": crl.extensions.crl_number.to_hex_upper(),
|
||||
"crlNumber": crl.extensions.crl_number.to_u64(),
|
||||
},
|
||||
"revokedCertificates": {
|
||||
"count": crl.revoked_certs.len(),
|
||||
"truncated": crl.revoked_certs.len() > entry_limit,
|
||||
"entries": crl.revoked_certs.iter().take(entry_limit).map(|item| {
|
||||
json!({
|
||||
"serialNumberHex": item.serial_number.to_hex_upper(),
|
||||
"serialNumber": item.serial_number.to_u64(),
|
||||
"revocationDate": format_time(item.revocation_date.utc),
|
||||
})
|
||||
}).collect::<Vec<_>>(),
|
||||
},
|
||||
}),
|
||||
Err(err) => json!({
|
||||
"type": "crl",
|
||||
"decode": {"profileValid": false, "error": err.to_string()},
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn manifest_file_entries_page(
|
||||
bytes: &[u8],
|
||||
offset: usize,
|
||||
limit: usize,
|
||||
) -> Result<(usize, Vec<Value>), String> {
|
||||
let mft = ManifestObject::decode_der(bytes).map_err(|err| err.to_string())?;
|
||||
let entries = mft.manifest.parse_files().map_err(|err| err.to_string())?;
|
||||
let total = entries.len();
|
||||
let end = (offset + limit).min(total);
|
||||
let page = entries[offset.min(total)..end]
|
||||
.iter()
|
||||
.map(|item| json!({"fileName": item.file_name, "hashHex": hex::encode(&item.hash_bytes)}))
|
||||
.collect::<Vec<_>>();
|
||||
Ok((total, page))
|
||||
}
|
||||
|
||||
pub fn crl_revoked_entries_page(
|
||||
bytes: &[u8],
|
||||
offset: usize,
|
||||
limit: usize,
|
||||
) -> Result<(usize, Vec<Value>), String> {
|
||||
let crl = RpkixCrl::decode_der(bytes).map_err(|err| err.to_string())?;
|
||||
let total = crl.revoked_certs.len();
|
||||
let end = (offset + limit).min(total);
|
||||
let page = crl.revoked_certs[offset.min(total)..end]
|
||||
.iter()
|
||||
.map(|item| {
|
||||
json!({
|
||||
"serialNumberHex": item.serial_number.to_hex_upper(),
|
||||
"serialNumber": item.serial_number.to_u64(),
|
||||
"revocationDate": format_time(item.revocation_date.utc),
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
Ok((total, page))
|
||||
}
|
||||
|
||||
pub fn parse_roa_json(bytes: &[u8], entry_limit: usize) -> Value {
|
||||
match RoaObject::decode_der(bytes) {
|
||||
Ok(roa) => json!({
|
||||
"type": "roa",
|
||||
"decode": {"profileValid": true},
|
||||
"eContentType": roa.econtent_type,
|
||||
"signedObject": signed_object_json(&roa.signed_object),
|
||||
"roa": {
|
||||
"version": roa.roa.version,
|
||||
"asId": roa.roa.as_id,
|
||||
"ipAddressFamilies": roa.roa.ip_addr_blocks.iter().map(|family| {
|
||||
json!({
|
||||
"afi": format!("{:?}", family.afi),
|
||||
"addressCount": family.addresses.len(),
|
||||
"truncated": family.addresses.len() > entry_limit,
|
||||
"addresses": family.addresses.iter().take(entry_limit).map(|entry| {
|
||||
json!({
|
||||
"prefix": roa_prefix_string(&entry.prefix),
|
||||
"maxLength": entry.max_length,
|
||||
})
|
||||
}).collect::<Vec<_>>(),
|
||||
})
|
||||
}).collect::<Vec<_>>(),
|
||||
},
|
||||
"embeddedEeProfile": result_json(roa.validate_embedded_ee_cert().map_err(|e| e.to_string())),
|
||||
"cmsSignature": result_json(roa.signed_object.verify_signature().map_err(|e| e.to_string())),
|
||||
}),
|
||||
Err(err) => json!({
|
||||
"type": "roa",
|
||||
"decode": {"profileValid": false, "error": err.to_string()},
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_aspa_json(bytes: &[u8], entry_limit: usize) -> Value {
|
||||
match AspaObject::decode_der(bytes) {
|
||||
Ok(aspa) => json!({
|
||||
"type": "aspa",
|
||||
"decode": {"profileValid": true},
|
||||
"eContentType": aspa.econtent_type,
|
||||
"signedObject": signed_object_json(&aspa.signed_object),
|
||||
"aspa": {
|
||||
"version": aspa.aspa.version,
|
||||
"customerAsId": aspa.aspa.customer_as_id,
|
||||
"providerCount": aspa.aspa.provider_as_ids.len(),
|
||||
"providersTruncated": aspa.aspa.provider_as_ids.len() > entry_limit,
|
||||
"providerAsIds": aspa.aspa.provider_as_ids.iter().take(entry_limit).copied().collect::<Vec<_>>(),
|
||||
},
|
||||
"embeddedEeProfile": result_json(aspa.validate_embedded_ee_cert().map_err(|e| e.to_string())),
|
||||
"cmsSignature": result_json(aspa.signed_object.verify_signature().map_err(|e| e.to_string())),
|
||||
}),
|
||||
Err(err) => json!({
|
||||
"type": "aspa",
|
||||
"decode": {"profileValid": false, "error": err.to_string()},
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
fn resource_certificate_json(cert: &ResourceCertificate) -> Value {
|
||||
let tbs = &cert.tbs;
|
||||
json!({
|
||||
"rawDer": bytes_summary(&cert.raw_der),
|
||||
"kind": format!("{:?}", cert.kind),
|
||||
"version": tbs.version,
|
||||
"serialNumberHex": hex::encode(tbs.serial_number.to_bytes_be()),
|
||||
"signatureAlgorithm": tbs.signature_algorithm,
|
||||
"issuer": tbs.issuer_name.to_string(),
|
||||
"subject": tbs.subject_name.to_string(),
|
||||
"validity": {
|
||||
"notBefore": format_time(tbs.validity_not_before),
|
||||
"notAfter": format_time(tbs.validity_not_after),
|
||||
},
|
||||
"subjectPublicKeyInfo": bytes_summary(&tbs.subject_public_key_info),
|
||||
"extensions": rc_extensions_json(&tbs.extensions),
|
||||
})
|
||||
}
|
||||
|
||||
fn rc_extensions_json(ext: &RcExtensions) -> Value {
|
||||
json!({
|
||||
"basicConstraintsCa": ext.basic_constraints_ca,
|
||||
"subjectKeyIdentifier": ext.subject_key_identifier.as_ref().map(|v| hex::encode(v)),
|
||||
"authorityKeyIdentifier": ext.authority_key_identifier.as_ref().map(|v| hex::encode(v)),
|
||||
"crlDistributionPointsUris": ext.crl_distribution_points_uris,
|
||||
"caIssuersUris": ext.ca_issuers_uris,
|
||||
"subjectInfoAccess": subject_info_access_json(ext.subject_info_access.as_ref()),
|
||||
"certificatePoliciesOid": ext.certificate_policies_oid,
|
||||
"ipResources": serde_json::to_value(&ext.ip_resources).unwrap_or(Value::Null),
|
||||
"asResources": serde_json::to_value(&ext.as_resources).unwrap_or(Value::Null),
|
||||
})
|
||||
}
|
||||
|
||||
fn subject_info_access_json(value: Option<&SubjectInfoAccess>) -> Value {
|
||||
match value {
|
||||
None => Value::Null,
|
||||
Some(SubjectInfoAccess::Ca(ca)) => json!({
|
||||
"kind": "ca",
|
||||
"accessDescriptions": ca.access_descriptions.iter().map(access_description_json).collect::<Vec<_>>(),
|
||||
}),
|
||||
Some(SubjectInfoAccess::Ee(ee)) => json!({
|
||||
"kind": "ee",
|
||||
"signedObjectUris": ee.signed_object_uris,
|
||||
"accessDescriptions": ee.access_descriptions.iter().map(access_description_json).collect::<Vec<_>>(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
fn access_description_json(value: &AccessDescription) -> Value {
|
||||
json!({
|
||||
"accessMethodOid": value.access_method_oid,
|
||||
"accessLocation": value.access_location,
|
||||
})
|
||||
}
|
||||
|
||||
fn signed_object_json(signed_object: &RpkiSignedObject) -> Value {
|
||||
let signed_data = &signed_object.signed_data;
|
||||
json!({
|
||||
"rawDer": bytes_summary(&signed_object.raw_der),
|
||||
"contentInfoContentType": signed_object.content_info_content_type,
|
||||
"signedData": {
|
||||
"version": signed_data.version,
|
||||
"digestAlgorithms": signed_data.digest_algorithms,
|
||||
"encapContentInfo": {
|
||||
"eContentType": signed_data.encap_content_info.econtent_type,
|
||||
"eContent": bytes_summary(&signed_data.encap_content_info.econtent),
|
||||
},
|
||||
"certificates": signed_data.certificates.iter().map(ee_certificate_json).collect::<Vec<_>>(),
|
||||
"crlsPresent": signed_data.crls_present,
|
||||
"signerInfos": signed_data.signer_infos.iter().map(signer_info_json).collect::<Vec<_>>(),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
fn ee_certificate_json(cert: &ResourceEeCertificate) -> Value {
|
||||
json!({
|
||||
"rawDer": bytes_summary(&cert.raw_der),
|
||||
"subjectKeyIdentifier": hex::encode(&cert.subject_key_identifier),
|
||||
"spkiDer": bytes_summary(&cert.spki_der),
|
||||
"rsaPublicKey": {
|
||||
"modulus": bytes_summary(&cert.rsa_public_modulus),
|
||||
"exponent": bytes_summary(&cert.rsa_public_exponent),
|
||||
},
|
||||
"tbsCertificate": bytes_summary(&cert.tbs_certificate_der),
|
||||
"certificateSignature": bytes_summary(&cert.signature_bytes),
|
||||
"keyUsageSummary": format!("{:?}", cert.key_usage_summary),
|
||||
"siaSignedObjectUris": cert.sia_signed_object_uris,
|
||||
"resourceCertificate": resource_certificate_json(&cert.resource_cert),
|
||||
})
|
||||
}
|
||||
|
||||
fn signer_info_json(info: &SignerInfoProfiled) -> Value {
|
||||
json!({
|
||||
"version": info.version,
|
||||
"sidSki": hex::encode(&info.sid_ski),
|
||||
"digestAlgorithm": info.digest_algorithm,
|
||||
"signatureAlgorithm": info.signature_algorithm,
|
||||
"signedAttrs": signed_attrs_json(&info.signed_attrs),
|
||||
"unsignedAttrsPresent": info.unsigned_attrs_present,
|
||||
"signature": bytes_summary(&info.signature),
|
||||
"signedAttrsDerForSignature": bytes_summary(&info.signed_attrs_der_for_signature),
|
||||
})
|
||||
}
|
||||
|
||||
fn signed_attrs_json(attrs: &SignedAttrsProfiled) -> Value {
|
||||
json!({
|
||||
"contentType": attrs.content_type,
|
||||
"messageDigest": hex::encode(&attrs.message_digest),
|
||||
"signingTime": {
|
||||
"utc": format_time(attrs.signing_time.utc),
|
||||
"encoding": format!("{:?}", attrs.signing_time.encoding),
|
||||
},
|
||||
"otherAttrsPresent": attrs.other_attrs_present,
|
||||
})
|
||||
}
|
||||
|
||||
fn result_json(result: Result<(), String>) -> Value {
|
||||
match result {
|
||||
Ok(()) => json!({"valid": true}),
|
||||
Err(err) => json!({"valid": false, "error": err}),
|
||||
}
|
||||
}
|
||||
|
||||
fn object_type_label(object_type: ObjectType) -> &'static str {
|
||||
match object_type {
|
||||
ObjectType::Auto => "auto",
|
||||
ObjectType::Cer => "cer",
|
||||
ObjectType::Mft => "mft",
|
||||
ObjectType::Crl => "crl",
|
||||
ObjectType::Roa => "roa",
|
||||
ObjectType::Aspa => "aspa",
|
||||
}
|
||||
}
|
||||
|
||||
fn bytes_summary(bytes: &[u8]) -> Value {
|
||||
let head_len = bytes.len().min(16);
|
||||
let tail_len = bytes.len().min(16);
|
||||
json!({
|
||||
"len": bytes.len(),
|
||||
"sha256": sha256_hex(bytes),
|
||||
"headHex": hex::encode(&bytes[..head_len]),
|
||||
"tailHex": hex::encode(&bytes[bytes.len().saturating_sub(tail_len)..]),
|
||||
})
|
||||
}
|
||||
|
||||
fn sha256_hex(bytes: &[u8]) -> String {
|
||||
hex::encode(Sha256::digest(bytes))
|
||||
}
|
||||
|
||||
fn format_time(value: time::OffsetDateTime) -> String {
|
||||
value
|
||||
.to_offset(time::UtcOffset::UTC)
|
||||
.format(&time::format_description::well_known::Rfc3339)
|
||||
.unwrap_or_else(|_| value.unix_timestamp().to_string())
|
||||
}
|
||||
|
||||
fn roa_prefix_string(prefix: &RoaIpPrefix) -> String {
|
||||
let bytes = prefix.addr_bytes();
|
||||
match prefix.afi {
|
||||
RoaAfi::Ipv4 => {
|
||||
let octets = [bytes[0], bytes[1], bytes[2], bytes[3]];
|
||||
format!("{}/{}", Ipv4Addr::from(octets), prefix.prefix_len)
|
||||
}
|
||||
RoaAfi::Ipv6 => {
|
||||
let mut octets = [0u8; 16];
|
||||
octets.copy_from_slice(bytes);
|
||||
format!("{}/{}", Ipv6Addr::from(octets), prefix.prefix_len)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::path::Path;
|
||||
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn object_type_parser_and_resolver_cover_aliases() {
|
||||
assert_eq!(ObjectType::parse("auto").unwrap(), ObjectType::Auto);
|
||||
assert_eq!(ObjectType::parse(".cer").unwrap(), ObjectType::Cer);
|
||||
assert_eq!(ObjectType::parse("certificate").unwrap(), ObjectType::Cer);
|
||||
assert_eq!(ObjectType::parse("manifest").unwrap(), ObjectType::Mft);
|
||||
assert_eq!(ObjectType::parse(".crl").unwrap(), ObjectType::Crl);
|
||||
assert_eq!(ObjectType::parse("roa").unwrap(), ObjectType::Roa);
|
||||
assert_eq!(ObjectType::parse("aspa").unwrap(), ObjectType::Aspa);
|
||||
assert_eq!(ObjectType::parse(".asa").unwrap(), ObjectType::Aspa);
|
||||
assert!(ObjectType::parse("unknown").is_err());
|
||||
assert_eq!(ObjectType::Aspa.label(), "aspa");
|
||||
|
||||
assert_eq!(
|
||||
resolve_object_type(ObjectType::Auto, Path::new("repo/a.cer")).unwrap(),
|
||||
ObjectType::Cer
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_object_type(ObjectType::Auto, Path::new("repo/a.mft")).unwrap(),
|
||||
ObjectType::Mft
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_object_type(ObjectType::Auto, Path::new("repo/a.crl")).unwrap(),
|
||||
ObjectType::Crl
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_object_type(ObjectType::Auto, Path::new("repo/a.roa")).unwrap(),
|
||||
ObjectType::Roa
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_object_type(ObjectType::Auto, Path::new("repo/a.asa")).unwrap(),
|
||||
ObjectType::Aspa
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_object_type(ObjectType::Roa, Path::new("repo/a.bin")).unwrap(),
|
||||
ObjectType::Roa
|
||||
);
|
||||
assert!(resolve_object_type(ObjectType::Auto, Path::new("repo/a.bin")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_der_returns_error_projection_for_all_object_types() {
|
||||
let bytes = b"not der";
|
||||
for object_type in [
|
||||
ObjectType::Cer,
|
||||
ObjectType::Mft,
|
||||
ObjectType::Crl,
|
||||
ObjectType::Roa,
|
||||
ObjectType::Aspa,
|
||||
] {
|
||||
let value = parse_object_json(object_type, Path::new("bad.der"), bytes, 1);
|
||||
assert_eq!(
|
||||
value["object"]["decode"]["profileValid"].as_bool(),
|
||||
Some(false)
|
||||
);
|
||||
assert!(value["object"]["decode"]["error"].as_str().is_some());
|
||||
}
|
||||
|
||||
let record = build_object_projection(ObjectType::Auto, Path::new("bad.bin"), bytes, 1);
|
||||
assert_eq!(record.object_type, "unknown");
|
||||
assert_eq!(record.parse_status, "error");
|
||||
assert!(record.error_summary.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_fixture_objects_into_human_readable_projection() {
|
||||
let cases = [
|
||||
(
|
||||
ObjectType::Cer,
|
||||
"tests/fixtures/ta/apnic-ta.cer",
|
||||
"cer",
|
||||
"resourceCertificate",
|
||||
),
|
||||
(
|
||||
ObjectType::Mft,
|
||||
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft",
|
||||
"mft",
|
||||
"manifest",
|
||||
),
|
||||
(
|
||||
ObjectType::Crl,
|
||||
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.crl",
|
||||
"crl",
|
||||
"revokedCertificates",
|
||||
),
|
||||
(
|
||||
ObjectType::Roa,
|
||||
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/AS4538.roa",
|
||||
"roa",
|
||||
"roa",
|
||||
),
|
||||
(
|
||||
ObjectType::Aspa,
|
||||
"tests/fixtures/repository/chloe.sobornost.net/rpki/RIPE-nljobsnijders/5m80fwYws_3FiFD7JiQjAqZ1RYQ.asa",
|
||||
"aspa",
|
||||
"aspa",
|
||||
),
|
||||
];
|
||||
|
||||
for (object_type, path, expected_type, expected_section) in cases {
|
||||
let bytes = std::fs::read(path).expect("fixture");
|
||||
let record = build_object_projection(object_type, Path::new(path), &bytes, 1);
|
||||
assert_eq!(record.object_type, expected_type);
|
||||
assert_eq!(record.parse_status, "ok");
|
||||
assert_eq!(
|
||||
record.projection["object"]["decode"]["profileValid"].as_bool(),
|
||||
Some(true)
|
||||
);
|
||||
assert!(record.projection["object"][expected_section].is_object());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn large_projection_lists_are_paged_from_raw_bytes() {
|
||||
let mft_bytes = std::fs::read(
|
||||
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft",
|
||||
)
|
||||
.expect("mft");
|
||||
let (total, page) = manifest_file_entries_page(&mft_bytes, 1, 3).expect("mft page");
|
||||
assert!(total >= 3);
|
||||
assert_eq!(page.len(), 3);
|
||||
assert!(page[0]["fileName"].as_str().is_some());
|
||||
let (_, empty_page) =
|
||||
manifest_file_entries_page(&mft_bytes, total + 10, 3).expect("empty page");
|
||||
assert!(empty_page.is_empty());
|
||||
|
||||
let crl_bytes = std::fs::read(
|
||||
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.crl",
|
||||
)
|
||||
.expect("crl");
|
||||
let (total, page) = crl_revoked_entries_page(&crl_bytes, 0, 5).expect("crl page");
|
||||
assert!(page.len() <= total);
|
||||
let (_, empty_page) =
|
||||
crl_revoked_entries_page(&crl_bytes, total + 10, 5).expect("empty crl page");
|
||||
assert!(empty_page.is_empty());
|
||||
}
|
||||
}
|
||||
79
crates/panda-rpki-validator/src/parallel/config.rs
Normal file
79
crates/panda-rpki-validator/src/parallel/config.rs
Normal file
@ -0,0 +1,79 @@
|
||||
use crate::parallel::dead_repo_blacklist::DeadRepoBlacklistConfig;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ParallelPhase1Config {
|
||||
pub max_repo_sync_workers_global: usize,
|
||||
pub max_inflight_snapshot_bytes_global: usize,
|
||||
pub max_pending_repo_results: usize,
|
||||
/// Dead-repo transport blacklist (#141). `None` disables the feature
|
||||
/// entirely (default, behavior unchanged).
|
||||
pub dead_repo_blacklist: Option<DeadRepoBlacklistConfig>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ParallelPhase2Config {
|
||||
pub object_workers: usize,
|
||||
pub worker_queue_capacity: usize,
|
||||
pub ready_batch_size: usize,
|
||||
pub ready_batch_wall_time_budget_ms: u64,
|
||||
pub object_result_drain_batch_size: usize,
|
||||
pub publication_point_finalize_batch_size: usize,
|
||||
pub publication_point_finalize_wall_time_budget_ms: u64,
|
||||
pub publication_point_finalize_queue_capacity: usize,
|
||||
/// Experimental: workers of the ready publication point stage pool. `0`
|
||||
/// disables the pool and keeps the inline compute+apply staging path.
|
||||
pub stage_workers: usize,
|
||||
}
|
||||
|
||||
impl Default for ParallelPhase2Config {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
object_workers: 8,
|
||||
worker_queue_capacity: 256,
|
||||
ready_batch_size: 256,
|
||||
ready_batch_wall_time_budget_ms: 100,
|
||||
object_result_drain_batch_size: 2048,
|
||||
publication_point_finalize_batch_size: 256,
|
||||
publication_point_finalize_wall_time_budget_ms: 100,
|
||||
publication_point_finalize_queue_capacity: 32768,
|
||||
stage_workers: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for ParallelPhase1Config {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_repo_sync_workers_global: 4,
|
||||
max_inflight_snapshot_bytes_global: 512 * 1024 * 1024,
|
||||
max_pending_repo_results: 1024,
|
||||
dead_repo_blacklist: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ParallelPhase1Config, ParallelPhase2Config};
|
||||
|
||||
#[test]
|
||||
fn default_parallel_phase1_config_is_bounded() {
|
||||
let cfg = ParallelPhase1Config::default();
|
||||
assert!(cfg.max_repo_sync_workers_global > 0);
|
||||
assert!(cfg.max_inflight_snapshot_bytes_global > 0);
|
||||
assert!(cfg.max_pending_repo_results > 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_parallel_phase2_config_is_bounded() {
|
||||
let cfg = ParallelPhase2Config::default();
|
||||
assert!(cfg.object_workers > 0);
|
||||
assert!(cfg.worker_queue_capacity > 0);
|
||||
assert!(cfg.ready_batch_size > 0);
|
||||
assert!(cfg.ready_batch_wall_time_budget_ms > 0);
|
||||
assert!(cfg.object_result_drain_batch_size > 0);
|
||||
assert!(cfg.publication_point_finalize_batch_size > 0);
|
||||
assert!(cfg.publication_point_finalize_wall_time_budget_ms > 0);
|
||||
assert!(cfg.publication_point_finalize_queue_capacity > 0);
|
||||
}
|
||||
}
|
||||
391
crates/panda-rpki-validator/src/parallel/dead_repo_blacklist.rs
Normal file
391
crates/panda-rpki-validator/src/parallel/dead_repo_blacklist.rs
Normal file
@ -0,0 +1,391 @@
|
||||
//! Persistent dead-repo transport blacklist (#141).
|
||||
//!
|
||||
//! Tracks per-(repo, transport) consecutive transport-layer fetch failures
|
||||
//! across runs. Once an entry reaches the configured failure threshold it is
|
||||
//! admitted to the blacklist; the live transport scheduler then skips the
|
||||
//! dead transport (rrdp-blacklisted -> straight to rsync, dual-blacklisted ->
|
||||
//! immediate terminal failure). Entries are removed either by a successful
|
||||
//! fetch (self-heal) or by the daemon-side health probe.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fs;
|
||||
use std::io;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::parallel::types::{RepoTransportErrorClass, RepoTransportMode};
|
||||
|
||||
pub const DEAD_REPO_BLACKLIST_SCHEMA_VERSION: u32 = 1;
|
||||
pub const DEAD_REPO_BLACKLIST_DEFAULT_CAPACITY: usize = 256;
|
||||
/// Probe backoff is capped at interval * 2^PROBE_BACKOFF_CAP_SHIFT.
|
||||
pub const PROBE_BACKOFF_CAP_SHIFT: u32 = 4;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct DeadRepoBlacklistConfig {
|
||||
pub path: PathBuf,
|
||||
pub fail_threshold: u32,
|
||||
pub capacity: usize,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct DeadRepoBlacklistEntry {
|
||||
pub transport: RepoTransportMode,
|
||||
pub uri: String,
|
||||
/// Consecutive runs with transport-class fetch failure. Reset on success.
|
||||
pub consecutive_failures: u32,
|
||||
/// True once `consecutive_failures` reached the admission threshold.
|
||||
#[serde(default)]
|
||||
pub blacklisted: bool,
|
||||
#[serde(default)]
|
||||
pub first_failure_at_unix: u64,
|
||||
#[serde(default)]
|
||||
pub last_failure_at_unix: u64,
|
||||
/// When the entry was admitted to the blacklist.
|
||||
#[serde(default)]
|
||||
pub added_at_unix: u64,
|
||||
#[serde(default)]
|
||||
pub last_probe_at_unix: Option<u64>,
|
||||
#[serde(default)]
|
||||
pub probe_failures: u32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum DeadRepoFailureOutcome {
|
||||
/// Failure counted, entry not (or not yet) blacklisted.
|
||||
Counted { consecutive_failures: u32 },
|
||||
/// Entry just reached the threshold and was admitted.
|
||||
Admitted,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum DeadRepoSuccessOutcome {
|
||||
/// A counting (not yet admitted) entry was reset.
|
||||
CounterReset,
|
||||
/// A blacklisted entry was removed (repo revived).
|
||||
BlacklistRemoved,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
struct DeadRepoBlacklistFile {
|
||||
schema_version: u32,
|
||||
updated_at_unix: u64,
|
||||
#[serde(default)]
|
||||
entries: Vec<DeadRepoBlacklistEntry>,
|
||||
}
|
||||
|
||||
/// In-memory blacklist state for one writer at a time (child run, or daemon
|
||||
/// supervisor between runs). Lookups go through a HashMap keyed by
|
||||
/// (transport, uri).
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct DeadRepoBlacklist {
|
||||
entries: HashMap<(RepoTransportMode, String), DeadRepoBlacklistEntry>,
|
||||
}
|
||||
|
||||
impl DeadRepoBlacklist {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Load from disk. A missing file or an unreadable/incompatible file
|
||||
/// degrades to an empty blacklist (logged by the caller via `warning`).
|
||||
pub fn load(path: &Path) -> (Self, Option<String>) {
|
||||
let bytes = match fs::read(path) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(err) if err.kind() == io::ErrorKind::NotFound => return (Self::new(), None),
|
||||
Err(err) => {
|
||||
return (
|
||||
Self::new(),
|
||||
Some(format!(
|
||||
"dead repo blacklist {} unreadable ({err}); starting empty",
|
||||
path.display()
|
||||
)),
|
||||
);
|
||||
}
|
||||
};
|
||||
match serde_json::from_slice::<DeadRepoBlacklistFile>(&bytes) {
|
||||
Ok(file) if file.schema_version == DEAD_REPO_BLACKLIST_SCHEMA_VERSION => {
|
||||
let entries = file
|
||||
.entries
|
||||
.into_iter()
|
||||
.map(|entry| ((entry.transport, entry.uri.clone()), entry))
|
||||
.collect();
|
||||
(Self { entries }, None)
|
||||
}
|
||||
Ok(file) => (
|
||||
Self::new(),
|
||||
Some(format!(
|
||||
"dead repo blacklist {} schema_version {} != {}; starting empty",
|
||||
path.display(),
|
||||
file.schema_version,
|
||||
DEAD_REPO_BLACKLIST_SCHEMA_VERSION
|
||||
)),
|
||||
),
|
||||
Err(err) => (
|
||||
Self::new(),
|
||||
Some(format!(
|
||||
"dead repo blacklist {} unparsable ({err}); starting empty",
|
||||
path.display()
|
||||
)),
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// Persist atomically (tmp file + rename).
|
||||
pub fn store_atomic(&self, path: &Path, now_unix: u64) -> io::Result<()> {
|
||||
if let Some(parent) = path.parent() {
|
||||
fs::create_dir_all(parent)?;
|
||||
}
|
||||
let mut entries: Vec<&DeadRepoBlacklistEntry> = self.entries.values().collect();
|
||||
entries.sort_by(|a, b| a.uri.cmp(&b.uri).then(a.transport.cmp(&b.transport)));
|
||||
let file = DeadRepoBlacklistFile {
|
||||
schema_version: DEAD_REPO_BLACKLIST_SCHEMA_VERSION,
|
||||
updated_at_unix: now_unix,
|
||||
entries: entries.into_iter().cloned().collect(),
|
||||
};
|
||||
let bytes = serde_json::to_vec_pretty(&file)
|
||||
.map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err))?;
|
||||
let mut tmp_name = path
|
||||
.file_name()
|
||||
.map(|name| name.to_os_string())
|
||||
.unwrap_or_default();
|
||||
tmp_name.push(".tmp");
|
||||
let tmp_path = path.with_file_name(tmp_name);
|
||||
fs::write(&tmp_path, &bytes)?;
|
||||
// Best-effort durability before the rename.
|
||||
if let Ok(file) = fs::File::open(&tmp_path) {
|
||||
let _ = file.sync_all();
|
||||
}
|
||||
fs::rename(&tmp_path, path)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn is_blacklisted(&self, transport: RepoTransportMode, uri: &str) -> bool {
|
||||
self.entries
|
||||
.get(&(transport, uri.to_string()))
|
||||
.map(|entry| entry.blacklisted)
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
/// Record a transport-class fetch failure. Returns whether the failure
|
||||
/// newly admitted the entry to the blacklist.
|
||||
pub fn record_transport_failure(
|
||||
&mut self,
|
||||
transport: RepoTransportMode,
|
||||
uri: &str,
|
||||
now_unix: u64,
|
||||
fail_threshold: u32,
|
||||
capacity: usize,
|
||||
) -> DeadRepoFailureOutcome {
|
||||
let key = (transport, uri.to_string());
|
||||
if !self.entries.contains_key(&key) {
|
||||
self.evict_if_full(capacity);
|
||||
}
|
||||
let threshold = fail_threshold.max(1);
|
||||
let entry = self
|
||||
.entries
|
||||
.entry(key)
|
||||
.or_insert_with(|| DeadRepoBlacklistEntry {
|
||||
transport,
|
||||
uri: uri.to_string(),
|
||||
consecutive_failures: 0,
|
||||
blacklisted: false,
|
||||
first_failure_at_unix: now_unix,
|
||||
last_failure_at_unix: now_unix,
|
||||
added_at_unix: 0,
|
||||
last_probe_at_unix: None,
|
||||
probe_failures: 0,
|
||||
});
|
||||
entry.consecutive_failures = entry.consecutive_failures.saturating_add(1);
|
||||
entry.last_failure_at_unix = now_unix;
|
||||
if entry.first_failure_at_unix == 0 {
|
||||
entry.first_failure_at_unix = now_unix;
|
||||
}
|
||||
if !entry.blacklisted && entry.consecutive_failures >= threshold {
|
||||
entry.blacklisted = true;
|
||||
entry.added_at_unix = now_unix;
|
||||
// A fresh admission restarts probe bookkeeping.
|
||||
entry.last_probe_at_unix = None;
|
||||
entry.probe_failures = 0;
|
||||
return DeadRepoFailureOutcome::Admitted;
|
||||
}
|
||||
DeadRepoFailureOutcome::Counted {
|
||||
consecutive_failures: entry.consecutive_failures,
|
||||
}
|
||||
}
|
||||
|
||||
/// Record a successful fetch: reset a counting entry, or remove a
|
||||
/// blacklisted entry entirely (self-heal).
|
||||
pub fn record_transport_success(
|
||||
&mut self,
|
||||
transport: RepoTransportMode,
|
||||
uri: &str,
|
||||
) -> Option<DeadRepoSuccessOutcome> {
|
||||
match self.entries.remove(&(transport, uri.to_string())) {
|
||||
Some(entry) if entry.blacklisted => Some(DeadRepoSuccessOutcome::BlacklistRemoved),
|
||||
Some(_) => Some(DeadRepoSuccessOutcome::CounterReset),
|
||||
None => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Blacklisted entries whose next probe is due at `now_unix`.
|
||||
pub fn probe_due_entries(
|
||||
&self,
|
||||
now_unix: u64,
|
||||
probe_interval_secs: u64,
|
||||
) -> Vec<DeadRepoBlacklistEntry> {
|
||||
let mut due: Vec<DeadRepoBlacklistEntry> = self
|
||||
.entries
|
||||
.values()
|
||||
.filter(|entry| entry.blacklisted)
|
||||
.filter(|entry| {
|
||||
let interval = probe_backoff_secs(probe_interval_secs, entry.probe_failures);
|
||||
match entry.last_probe_at_unix {
|
||||
None => true,
|
||||
Some(last) => now_unix.saturating_sub(last) >= interval,
|
||||
}
|
||||
})
|
||||
.cloned()
|
||||
.collect();
|
||||
due.sort_by(|a, b| a.uri.cmp(&b.uri));
|
||||
due
|
||||
}
|
||||
|
||||
/// Probe succeeded: remove the entry. Returns true if it was present.
|
||||
pub fn record_probe_success(&mut self, transport: RepoTransportMode, uri: &str) -> bool {
|
||||
self.entries.remove(&(transport, uri.to_string())).is_some()
|
||||
}
|
||||
|
||||
pub fn record_probe_failure(&mut self, transport: RepoTransportMode, uri: &str, now_unix: u64) {
|
||||
if let Some(entry) = self.entries.get_mut(&(transport, uri.to_string())) {
|
||||
entry.last_probe_at_unix = Some(now_unix);
|
||||
entry.probe_failures = entry.probe_failures.saturating_add(1);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn len(&self) -> usize {
|
||||
self.entries.len()
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.entries.is_empty()
|
||||
}
|
||||
|
||||
pub fn blacklisted_len(&self) -> usize {
|
||||
self.entries
|
||||
.values()
|
||||
.filter(|entry| entry.blacklisted)
|
||||
.count()
|
||||
}
|
||||
|
||||
/// Deterministic snapshot of all entries (for status/observability).
|
||||
pub fn entries_sorted(&self) -> Vec<DeadRepoBlacklistEntry> {
|
||||
let mut entries: Vec<DeadRepoBlacklistEntry> = self.entries.values().cloned().collect();
|
||||
entries.sort_by(|a, b| a.uri.cmp(&b.uri).then(a.transport.cmp(&b.transport)));
|
||||
entries
|
||||
}
|
||||
|
||||
fn evict_if_full(&mut self, capacity: usize) {
|
||||
let capacity = capacity.max(1);
|
||||
if self.entries.len() < capacity {
|
||||
return;
|
||||
}
|
||||
// Prefer evicting the stalest counting entry; fall back to the oldest
|
||||
// blacklisted entry.
|
||||
let victim = self
|
||||
.entries
|
||||
.values()
|
||||
.filter(|entry| !entry.blacklisted)
|
||||
.min_by_key(|entry| entry.last_failure_at_unix)
|
||||
.map(|entry| (entry.transport, entry.uri.clone()))
|
||||
.or_else(|| {
|
||||
self.entries
|
||||
.values()
|
||||
.min_by_key(|entry| entry.added_at_unix)
|
||||
.map(|entry| (entry.transport, entry.uri.clone()))
|
||||
});
|
||||
if let Some(key) = victim {
|
||||
self.entries.remove(&key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn probe_backoff_secs(base_interval_secs: u64, probe_failures: u32) -> u64 {
|
||||
// First retry after a probe failure still uses the base interval;
|
||||
// exponential backoff kicks in from the second consecutive failure.
|
||||
let shift = probe_failures
|
||||
.saturating_sub(1)
|
||||
.min(PROBE_BACKOFF_CAP_SHIFT);
|
||||
base_interval_secs.saturating_mul(1u64 << shift)
|
||||
}
|
||||
|
||||
/// Classify an HTTP fetch error string (formats from
|
||||
/// `fetch/http.rs::BlockingHttpFetcher::fetch_bytes`).
|
||||
pub fn classify_http_fetch_error(detail: &str) -> RepoTransportErrorClass {
|
||||
if detail.starts_with("http status") {
|
||||
// The peer answered: transport is fine, the failure is protocol-level.
|
||||
RepoTransportErrorClass::Protocol
|
||||
} else if detail.starts_with("http request failed")
|
||||
|| detail.starts_with("http read body failed")
|
||||
{
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
} else {
|
||||
RepoTransportErrorClass::Unknown
|
||||
}
|
||||
}
|
||||
|
||||
/// Classify an rsync fetch error string using the same host-level substring
|
||||
/// set as the run-local rsync failure scope short-circuit.
|
||||
pub fn classify_rsync_fetch_error(detail: &str) -> RepoTransportErrorClass {
|
||||
if crate::parallel::repo_scheduler::is_host_level_rsync_failure(detail)
|
||||
// Stall failures are transport death too: the fail-fast mechanism
|
||||
// gives up when wall-clock windows pass with no (additional) progress
|
||||
// (#141 M9). Content-level errors fail immediately and never carry
|
||||
// this marker, so they still do not count.
|
||||
|| detail.contains("rsync fail-fast gave up")
|
||||
{
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
} else {
|
||||
// Content/protocol-level rsync failures (digest mismatch, vanished
|
||||
// files, partial transfers) never count toward the blacklist.
|
||||
RepoTransportErrorClass::Unknown
|
||||
}
|
||||
}
|
||||
|
||||
/// Map a structured RRDP sync error to a transport error class.
|
||||
pub fn classify_rrdp_sync_error(err: &crate::sync::rrdp::RrdpSyncError) -> RepoTransportErrorClass {
|
||||
match err {
|
||||
crate::sync::rrdp::RrdpSyncError::Fetch(detail) => classify_http_fetch_error(detail),
|
||||
crate::sync::rrdp::RrdpSyncError::Rrdp(_) => RepoTransportErrorClass::Protocol,
|
||||
crate::sync::rrdp::RrdpSyncError::Storage(_) => RepoTransportErrorClass::Storage,
|
||||
}
|
||||
}
|
||||
|
||||
/// Map a structured rsync-side repo sync error to a transport error class.
|
||||
pub fn classify_rsync_repo_sync_error(
|
||||
err: &crate::sync::repo::RepoSyncError,
|
||||
) -> RepoTransportErrorClass {
|
||||
match err {
|
||||
crate::sync::repo::RepoSyncError::Rsync(crate::fetch::rsync::RsyncFetchError::Fetch(
|
||||
detail,
|
||||
)) => classify_rsync_fetch_error(detail),
|
||||
crate::sync::repo::RepoSyncError::Storage(_) => RepoTransportErrorClass::Storage,
|
||||
_ => RepoTransportErrorClass::Protocol,
|
||||
}
|
||||
}
|
||||
|
||||
impl RepoTransportMode {
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
RepoTransportMode::Rrdp => "rrdp",
|
||||
RepoTransportMode::Rsync => "rsync",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keep the public surface free of internal key types; the (transport, uri)
|
||||
// tuple key is an implementation detail of the lookup map.
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "dead_repo_blacklist_tests.rs"]
|
||||
mod tests;
|
||||
@ -0,0 +1,430 @@
|
||||
use super::*;
|
||||
use crate::parallel::types::RepoTransportMode;
|
||||
|
||||
const RRDP_URI: &str = "https://dead.example.com/notification.xml";
|
||||
const RSYNC_URI: &str = "rsync://dead.example.com/repo/";
|
||||
|
||||
fn now() -> u64 {
|
||||
1_700_000_000
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn admission_requires_threshold_consecutive_failures() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
assert!(!bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
|
||||
let out1 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
|
||||
assert_eq!(
|
||||
out1,
|
||||
DeadRepoFailureOutcome::Counted {
|
||||
consecutive_failures: 1
|
||||
}
|
||||
);
|
||||
assert!(!bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
|
||||
let out2 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 1, 3, 256);
|
||||
assert_eq!(
|
||||
out2,
|
||||
DeadRepoFailureOutcome::Counted {
|
||||
consecutive_failures: 2
|
||||
}
|
||||
);
|
||||
assert!(!bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
|
||||
let out3 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 2, 3, 256);
|
||||
assert_eq!(out3, DeadRepoFailureOutcome::Admitted);
|
||||
assert!(bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
|
||||
// Further failures keep counting but do not re-admit.
|
||||
let out4 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 3, 3, 256);
|
||||
assert!(matches!(
|
||||
out4,
|
||||
DeadRepoFailureOutcome::Counted {
|
||||
consecutive_failures: 4
|
||||
}
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn threshold_of_one_admits_immediately() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
let out = bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now(), 1, 256);
|
||||
assert_eq!(out, DeadRepoFailureOutcome::Admitted);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn success_resets_counting_entry() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
|
||||
assert_eq!(bl.len(), 1);
|
||||
let out = bl.record_transport_success(RepoTransportMode::Rrdp, RRDP_URI);
|
||||
assert_eq!(out, Some(DeadRepoSuccessOutcome::CounterReset));
|
||||
assert!(bl.is_empty());
|
||||
// Next failure starts from zero again.
|
||||
let out = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
|
||||
assert_eq!(
|
||||
out,
|
||||
DeadRepoFailureOutcome::Counted {
|
||||
consecutive_failures: 1
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn success_on_blacklisted_entry_self_heals() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now() + i, 3, 256);
|
||||
}
|
||||
assert!(bl.is_blacklisted(RepoTransportMode::Rsync, RSYNC_URI));
|
||||
let out = bl.record_transport_success(RepoTransportMode::Rsync, RSYNC_URI);
|
||||
assert_eq!(out, Some(DeadRepoSuccessOutcome::BlacklistRemoved));
|
||||
assert!(!bl.is_blacklisted(RepoTransportMode::Rsync, RSYNC_URI));
|
||||
assert!(bl.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn success_on_unknown_entry_is_none() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
assert_eq!(
|
||||
bl.record_transport_success(RepoTransportMode::Rrdp, RRDP_URI),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entries_are_per_transport() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
|
||||
}
|
||||
assert!(bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
assert!(!bl.is_blacklisted(RepoTransportMode::Rsync, RRDP_URI));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn capacity_evicts_stale_counting_entries_first() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
// One blacklisted entry (old) and two counting entries.
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://a.example/n.xml",
|
||||
100 + i,
|
||||
1,
|
||||
3,
|
||||
);
|
||||
}
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://b.example/n.xml",
|
||||
200,
|
||||
3,
|
||||
3,
|
||||
);
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://c.example/n.xml",
|
||||
300,
|
||||
3,
|
||||
3,
|
||||
);
|
||||
assert_eq!(bl.len(), 3);
|
||||
assert_eq!(bl.blacklisted_len(), 1);
|
||||
|
||||
// Inserting a fourth entry evicts the stalest counting entry (b).
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://d.example/n.xml",
|
||||
400,
|
||||
3,
|
||||
3,
|
||||
);
|
||||
assert_eq!(bl.len(), 3);
|
||||
assert!(
|
||||
bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://a.example/n.xml")
|
||||
);
|
||||
assert!(
|
||||
!bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://b.example/n.xml")
|
||||
);
|
||||
assert!(
|
||||
bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://c.example/n.xml")
|
||||
);
|
||||
assert!(
|
||||
bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://d.example/n.xml")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn capacity_evicts_oldest_blacklisted_when_no_counting_entries() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://a.example/n.xml",
|
||||
100,
|
||||
1,
|
||||
2,
|
||||
);
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://b.example/n.xml",
|
||||
200,
|
||||
1,
|
||||
2,
|
||||
);
|
||||
assert_eq!(bl.blacklisted_len(), 2);
|
||||
bl.record_transport_failure(
|
||||
RepoTransportMode::Rrdp,
|
||||
"https://c.example/n.xml",
|
||||
300,
|
||||
1,
|
||||
2,
|
||||
);
|
||||
assert_eq!(bl.len(), 2);
|
||||
assert!(
|
||||
!bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://a.example/n.xml")
|
||||
);
|
||||
assert!(
|
||||
bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://b.example/n.xml")
|
||||
);
|
||||
assert!(
|
||||
bl.entries_sorted()
|
||||
.iter()
|
||||
.any(|e| e.uri == "https://c.example/n.xml")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn store_load_roundtrip() {
|
||||
let dir = std::env::temp_dir().join(format!(
|
||||
"dead_repo_blacklist_test_roundtrip_{}_{}",
|
||||
std::process::id(),
|
||||
now()
|
||||
));
|
||||
let path = dir.join("state").join("dead-repo-blacklist.json");
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
|
||||
}
|
||||
bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now(), 3, 256);
|
||||
bl.store_atomic(&path, now() + 10).unwrap();
|
||||
|
||||
let (loaded, warning) = DeadRepoBlacklist::load(&path);
|
||||
assert!(warning.is_none());
|
||||
assert!(loaded.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
assert!(!loaded.is_blacklisted(RepoTransportMode::Rsync, RSYNC_URI));
|
||||
assert_eq!(loaded.len(), 2);
|
||||
let entry = loaded
|
||||
.entries_sorted()
|
||||
.into_iter()
|
||||
.find(|e| e.uri == RRDP_URI)
|
||||
.unwrap();
|
||||
assert_eq!(entry.consecutive_failures, 3);
|
||||
assert!(entry.blacklisted);
|
||||
assert_eq!(entry.first_failure_at_unix, now());
|
||||
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn load_missing_file_is_empty_without_warning() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"dead_repo_blacklist_test_missing_{}_{}.json",
|
||||
std::process::id(),
|
||||
now()
|
||||
));
|
||||
let (bl, warning) = DeadRepoBlacklist::load(&path);
|
||||
assert!(bl.is_empty());
|
||||
assert!(warning.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn load_corrupted_file_degrades_to_empty_with_warning() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"dead_repo_blacklist_test_corrupt_{}_{}.json",
|
||||
std::process::id(),
|
||||
now()
|
||||
));
|
||||
std::fs::write(&path, b"{not json").unwrap();
|
||||
let (bl, warning) = DeadRepoBlacklist::load(&path);
|
||||
assert!(bl.is_empty());
|
||||
assert!(warning.is_some());
|
||||
let _ = std::fs::remove_file(&path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn load_wrong_schema_version_degrades_to_empty() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"dead_repo_blacklist_test_schema_{}_{}.json",
|
||||
std::process::id(),
|
||||
now()
|
||||
));
|
||||
std::fs::write(
|
||||
&path,
|
||||
r#"{"schema_version": 99, "updated_at_unix": 0, "entries": []}"#,
|
||||
)
|
||||
.unwrap();
|
||||
let (bl, warning) = DeadRepoBlacklist::load(&path);
|
||||
assert!(bl.is_empty());
|
||||
assert!(warning.unwrap().contains("schema_version"));
|
||||
let _ = std::fs::remove_file(&path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn probe_due_uses_exponential_backoff() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
|
||||
}
|
||||
// Never probed: due immediately.
|
||||
let due = bl.probe_due_entries(now() + 100, 600);
|
||||
assert_eq!(due.len(), 1);
|
||||
|
||||
// Probe failure: not due until base interval has passed.
|
||||
bl.record_probe_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 100);
|
||||
assert!(bl.probe_due_entries(now() + 100 + 599, 600).is_empty());
|
||||
assert_eq!(bl.probe_due_entries(now() + 100 + 600, 600).len(), 1);
|
||||
|
||||
// Second probe failure: backoff doubles to 1200s.
|
||||
bl.record_probe_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 700);
|
||||
assert!(bl.probe_due_entries(now() + 700 + 1199, 600).is_empty());
|
||||
assert_eq!(bl.probe_due_entries(now() + 700 + 1200, 600).len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn probe_success_removes_entry() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now() + i, 3, 256);
|
||||
}
|
||||
assert!(bl.record_probe_success(RepoTransportMode::Rsync, RSYNC_URI));
|
||||
assert!(bl.is_empty());
|
||||
assert!(!bl.record_probe_success(RepoTransportMode::Rsync, RSYNC_URI));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn counting_entries_are_not_probed() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
|
||||
assert!(bl.probe_due_entries(now() + 10_000, 600).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn probe_backoff_caps_at_max_shift() {
|
||||
assert_eq!(probe_backoff_secs(600, 0), 600);
|
||||
assert_eq!(probe_backoff_secs(600, 1), 600);
|
||||
assert_eq!(probe_backoff_secs(600, 2), 1200);
|
||||
assert_eq!(probe_backoff_secs(600, 5), 9600);
|
||||
assert_eq!(probe_backoff_secs(600, 100), 9600);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn http_fetch_error_classification() {
|
||||
use crate::parallel::types::RepoTransportErrorClass;
|
||||
assert_eq!(
|
||||
classify_http_fetch_error("http request failed: reqwest::Error { kind: Connect, .. }"),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
assert_eq!(
|
||||
classify_http_fetch_error("http read body failed: connection closed; status=200"),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
assert_eq!(
|
||||
classify_http_fetch_error("http status 404 Not Found; content_type=; .."),
|
||||
RepoTransportErrorClass::Protocol
|
||||
);
|
||||
assert_eq!(
|
||||
classify_http_fetch_error("something unexpected"),
|
||||
RepoTransportErrorClass::Unknown
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rsync_fetch_error_classification() {
|
||||
use crate::parallel::types::RepoTransportErrorClass;
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error("rsync error: timeout waiting for daemon connection"),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error("rsync: failed to connect to host: Connection refused"),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error("temporary failure in name resolution"),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
// Content-level failures must not count.
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error("rsync file digest mismatch after download"),
|
||||
RepoTransportErrorClass::Unknown
|
||||
);
|
||||
// Stall failures (#141 M9): fail-fast give-up means the host delivered no
|
||||
// bytes inside the wall-clock window — transport death, must count.
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error(
|
||||
"rsync fail-fast gave up after 1 attempts with no progress: rsync wall-clock window expired"
|
||||
),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error(
|
||||
"rsync fail-fast gave up after 3 attempts with no additional progress: rsync wall-clock window expired"
|
||||
),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
// Content-level give-up text without the fail-fast marker stays unknown.
|
||||
assert_eq!(
|
||||
classify_rsync_fetch_error("rsync error: some files vanished before transfer"),
|
||||
RepoTransportErrorClass::Unknown
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rrdp_sync_error_classification() {
|
||||
use crate::parallel::types::RepoTransportErrorClass;
|
||||
use crate::sync::rrdp::RrdpSyncError;
|
||||
assert_eq!(
|
||||
classify_rrdp_sync_error(&RrdpSyncError::Fetch(
|
||||
"http request failed: connect timeout".to_string()
|
||||
)),
|
||||
RepoTransportErrorClass::TransportFetch
|
||||
);
|
||||
assert_eq!(
|
||||
classify_rrdp_sync_error(&RrdpSyncError::Fetch("http status 500".to_string())),
|
||||
RepoTransportErrorClass::Protocol
|
||||
);
|
||||
assert_eq!(
|
||||
classify_rrdp_sync_error(&RrdpSyncError::Storage("db full".to_string())),
|
||||
RepoTransportErrorClass::Storage
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn readmission_after_probe_removal_starts_fresh() {
|
||||
let mut bl = DeadRepoBlacklist::new();
|
||||
for i in 0..3 {
|
||||
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
|
||||
}
|
||||
assert!(bl.record_probe_success(RepoTransportMode::Rrdp, RRDP_URI));
|
||||
// No half-open: needs the full threshold again.
|
||||
let out = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 10, 3, 256);
|
||||
assert_eq!(
|
||||
out,
|
||||
DeadRepoFailureOutcome::Counted {
|
||||
consecutive_failures: 1
|
||||
}
|
||||
);
|
||||
}
|
||||
11
crates/panda-rpki-validator/src/parallel/mod.rs
Normal file
11
crates/panda-rpki-validator/src/parallel/mod.rs
Normal file
@ -0,0 +1,11 @@
|
||||
pub mod config;
|
||||
pub mod dead_repo_blacklist;
|
||||
pub mod object_worker;
|
||||
pub mod phase2_scheduler;
|
||||
pub mod repo_runtime;
|
||||
pub mod repo_scheduler;
|
||||
pub mod repo_worker;
|
||||
pub mod run_coordinator;
|
||||
pub mod stats;
|
||||
pub mod transport_prefetch;
|
||||
pub mod types;
|
||||
485
crates/panda-rpki-validator/src/parallel/object_worker.rs
Normal file
485
crates/panda-rpki-validator/src/parallel/object_worker.rs
Normal file
@ -0,0 +1,485 @@
|
||||
use std::marker::PhantomData;
|
||||
use std::sync::Arc;
|
||||
use std::sync::mpsc::{self, Receiver, RecvTimeoutError, SyncSender, TrySendError};
|
||||
use std::thread::{self, JoinHandle, Scope};
|
||||
use std::time::Duration;
|
||||
|
||||
/// Executor shared by every worker of a pool. The `'static` bound required to
|
||||
/// move the executor into detached threads is expressed on the pool types
|
||||
/// instead of this trait so scoped pools can borrow their environment.
|
||||
pub trait ObjectTaskExecutor<T, R>: Send + Sync {
|
||||
fn execute(&self, worker_index: usize, task: T) -> R;
|
||||
}
|
||||
|
||||
enum ObjectWorkerMessage<T> {
|
||||
Task(T),
|
||||
Shutdown,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub enum ObjectWorkerSubmitError<T> {
|
||||
QueueFull { worker_index: usize, task: T },
|
||||
Disconnected { worker_index: usize, task: T },
|
||||
}
|
||||
|
||||
pub struct ObjectWorkerPool<T, R, E>
|
||||
where
|
||||
T: Send + 'static,
|
||||
R: Send + 'static,
|
||||
E: ObjectTaskExecutor<T, R> + 'static,
|
||||
{
|
||||
task_txs: Vec<SyncSender<ObjectWorkerMessage<T>>>,
|
||||
result_rx: Receiver<R>,
|
||||
workers: Vec<JoinHandle<()>>,
|
||||
next_worker_idx: usize,
|
||||
_executor: Arc<E>,
|
||||
}
|
||||
|
||||
impl<T, R, E> ObjectWorkerPool<T, R, E>
|
||||
where
|
||||
T: Send + 'static,
|
||||
R: Send + 'static,
|
||||
E: ObjectTaskExecutor<T, R> + 'static,
|
||||
{
|
||||
pub fn new(worker_count: usize, queue_capacity: usize, executor: E) -> Result<Self, String> {
|
||||
if worker_count == 0 {
|
||||
return Err("ObjectWorkerPool requires at least one worker".to_string());
|
||||
}
|
||||
if queue_capacity == 0 {
|
||||
return Err("ObjectWorkerPool requires queue_capacity > 0".to_string());
|
||||
}
|
||||
|
||||
let executor = Arc::new(executor);
|
||||
let (result_tx, result_rx) = mpsc::channel::<R>();
|
||||
let mut task_txs = Vec::with_capacity(worker_count);
|
||||
let mut workers = Vec::with_capacity(worker_count);
|
||||
|
||||
for worker_index in 0..worker_count {
|
||||
let (task_tx, task_rx) = mpsc::sync_channel::<ObjectWorkerMessage<T>>(queue_capacity);
|
||||
let result_tx = result_tx.clone();
|
||||
let executor = Arc::clone(&executor);
|
||||
let handle = thread::Builder::new()
|
||||
.name(format!("object-validation-worker-{worker_index}"))
|
||||
.spawn(move || object_worker_loop(worker_index, task_rx, result_tx, executor))
|
||||
.map_err(|e| format!("spawn object worker failed: {e}"))?;
|
||||
task_txs.push(task_tx);
|
||||
workers.push(handle);
|
||||
}
|
||||
|
||||
Ok(Self {
|
||||
task_txs,
|
||||
result_rx,
|
||||
workers,
|
||||
next_worker_idx: 0,
|
||||
_executor: executor,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn worker_count(&self) -> usize {
|
||||
self.task_txs.len()
|
||||
}
|
||||
|
||||
pub fn next_worker_index(&self) -> usize {
|
||||
self.next_worker_idx
|
||||
}
|
||||
|
||||
pub fn try_submit_round_robin(&mut self, task: T) -> Result<usize, ObjectWorkerSubmitError<T>> {
|
||||
let worker_index = self.next_worker_idx % self.task_txs.len();
|
||||
match self.task_txs[worker_index].try_send(ObjectWorkerMessage::Task(task)) {
|
||||
Ok(()) => {
|
||||
self.next_worker_idx = (worker_index + 1) % self.task_txs.len();
|
||||
Ok(worker_index)
|
||||
}
|
||||
Err(TrySendError::Full(ObjectWorkerMessage::Task(task))) => {
|
||||
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task })
|
||||
}
|
||||
Err(TrySendError::Disconnected(ObjectWorkerMessage::Task(task))) => {
|
||||
Err(ObjectWorkerSubmitError::Disconnected { worker_index, task })
|
||||
}
|
||||
Err(TrySendError::Full(ObjectWorkerMessage::Shutdown))
|
||||
| Err(TrySendError::Disconnected(ObjectWorkerMessage::Shutdown)) => {
|
||||
unreachable!("shutdown is never submitted via try_submit_round_robin")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn recv_result_timeout(&self, timeout: Duration) -> Result<Option<R>, String> {
|
||||
match self.result_rx.recv_timeout(timeout) {
|
||||
Ok(result) => Ok(Some(result)),
|
||||
Err(RecvTimeoutError::Timeout) => Ok(None),
|
||||
Err(RecvTimeoutError::Disconnected) => {
|
||||
Err("object worker result channel disconnected".to_string())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn shutdown(mut self) -> Result<(), String> {
|
||||
self.shutdown_inner()
|
||||
}
|
||||
|
||||
fn shutdown_inner(&mut self) -> Result<(), String> {
|
||||
if self.workers.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
for tx in &self.task_txs {
|
||||
tx.send(ObjectWorkerMessage::Shutdown)
|
||||
.map_err(|e| format!("send shutdown to object worker failed: {e}"))?;
|
||||
}
|
||||
let mut first_err = None;
|
||||
for handle in self.workers.drain(..) {
|
||||
if let Err(e) = handle.join() {
|
||||
if first_err.is_none() {
|
||||
first_err = Some(format!("join object worker failed: {e:?}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(err) = first_err {
|
||||
return Err(err);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl<T, R, E> Drop for ObjectWorkerPool<T, R, E>
|
||||
where
|
||||
T: Send + 'static,
|
||||
R: Send + 'static,
|
||||
E: ObjectTaskExecutor<T, R> + 'static,
|
||||
{
|
||||
fn drop(&mut self) {
|
||||
let _ = self.shutdown_inner();
|
||||
}
|
||||
}
|
||||
|
||||
fn object_worker_loop<T, R, E>(
|
||||
worker_index: usize,
|
||||
task_rx: Receiver<ObjectWorkerMessage<T>>,
|
||||
result_tx: mpsc::Sender<R>,
|
||||
executor: Arc<E>,
|
||||
) where
|
||||
T: Send,
|
||||
R: Send,
|
||||
E: ObjectTaskExecutor<T, R>,
|
||||
{
|
||||
loop {
|
||||
match task_rx.recv() {
|
||||
Ok(ObjectWorkerMessage::Task(task)) => {
|
||||
let result = executor.execute(worker_index, task);
|
||||
if result_tx.send(result).is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(ObjectWorkerMessage::Shutdown) | Err(_) => break,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Scoped variant of `ObjectWorkerPool`: workers are spawned on a
|
||||
/// `std::thread::Scope`, so tasks, results and the executor may borrow their
|
||||
/// environment (`'env`) instead of being `'static`. The pool never sends
|
||||
/// `Shutdown`; workers exit when every task sender is dropped, which happens
|
||||
/// when the pool itself is dropped ahead of the scope join.
|
||||
pub struct ScopedObjectWorkerPool<'scope, 'env, T, R, E>
|
||||
where
|
||||
T: Send + 'env,
|
||||
R: Send + 'env,
|
||||
E: ObjectTaskExecutor<T, R> + 'env,
|
||||
{
|
||||
task_txs: Vec<SyncSender<ObjectWorkerMessage<T>>>,
|
||||
result_rx: Receiver<R>,
|
||||
next_worker_idx: usize,
|
||||
_executor: Arc<E>,
|
||||
// Join handles are intentionally not stored: dropping a `ScopedJoinHandle`
|
||||
// detaches the worker and the enclosing scope joins it on exit, after the
|
||||
// dropped task senders have made every worker return.
|
||||
_marker: PhantomData<(&'scope (), &'env ())>,
|
||||
}
|
||||
|
||||
impl<'scope, 'env, T, R, E> ScopedObjectWorkerPool<'scope, 'env, T, R, E>
|
||||
where
|
||||
T: Send + 'env,
|
||||
R: Send + 'env,
|
||||
E: ObjectTaskExecutor<T, R> + 'env,
|
||||
{
|
||||
pub fn new(
|
||||
scope: &'scope Scope<'scope, 'env>,
|
||||
worker_count: usize,
|
||||
queue_capacity: usize,
|
||||
executor: E,
|
||||
) -> Result<Self, String> {
|
||||
if worker_count == 0 {
|
||||
return Err("ScopedObjectWorkerPool requires at least one worker".to_string());
|
||||
}
|
||||
if queue_capacity == 0 {
|
||||
return Err("ScopedObjectWorkerPool requires queue_capacity > 0".to_string());
|
||||
}
|
||||
|
||||
let executor = Arc::new(executor);
|
||||
let (result_tx, result_rx) = mpsc::channel::<R>();
|
||||
let mut task_txs = Vec::with_capacity(worker_count);
|
||||
|
||||
for worker_index in 0..worker_count {
|
||||
let (task_tx, task_rx) = mpsc::sync_channel::<ObjectWorkerMessage<T>>(queue_capacity);
|
||||
let result_tx = result_tx.clone();
|
||||
let executor = Arc::clone(&executor);
|
||||
thread::Builder::new()
|
||||
.name(format!("object-validation-worker-{worker_index}"))
|
||||
.spawn_scoped(scope, move || {
|
||||
object_worker_loop(worker_index, task_rx, result_tx, executor)
|
||||
})
|
||||
.map_err(|e| format!("spawn scoped object worker failed: {e}"))?;
|
||||
task_txs.push(task_tx);
|
||||
}
|
||||
|
||||
Ok(Self {
|
||||
task_txs,
|
||||
result_rx,
|
||||
next_worker_idx: 0,
|
||||
_executor: executor,
|
||||
_marker: PhantomData,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn worker_count(&self) -> usize {
|
||||
self.task_txs.len()
|
||||
}
|
||||
|
||||
pub fn try_submit_round_robin(&mut self, task: T) -> Result<usize, ObjectWorkerSubmitError<T>> {
|
||||
let worker_index = self.next_worker_idx % self.task_txs.len();
|
||||
match self.task_txs[worker_index].try_send(ObjectWorkerMessage::Task(task)) {
|
||||
Ok(()) => {
|
||||
self.next_worker_idx = (worker_index + 1) % self.task_txs.len();
|
||||
Ok(worker_index)
|
||||
}
|
||||
Err(TrySendError::Full(ObjectWorkerMessage::Task(task))) => {
|
||||
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task })
|
||||
}
|
||||
Err(TrySendError::Disconnected(ObjectWorkerMessage::Task(task))) => {
|
||||
Err(ObjectWorkerSubmitError::Disconnected { worker_index, task })
|
||||
}
|
||||
Err(TrySendError::Full(ObjectWorkerMessage::Shutdown))
|
||||
| Err(TrySendError::Disconnected(ObjectWorkerMessage::Shutdown)) => {
|
||||
unreachable!("shutdown is never submitted via try_submit_round_robin")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn recv_result_timeout(&self, timeout: Duration) -> Result<Option<R>, String> {
|
||||
match self.result_rx.recv_timeout(timeout) {
|
||||
Ok(result) => Ok(Some(result)),
|
||||
Err(RecvTimeoutError::Timeout) => Ok(None),
|
||||
Err(RecvTimeoutError::Disconnected) => {
|
||||
Err("scoped object worker result channel disconnected".to_string())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<'scope, 'env, T, R, E> Drop for ScopedObjectWorkerPool<'scope, 'env, T, R, E>
|
||||
where
|
||||
T: Send + 'env,
|
||||
R: Send + 'env,
|
||||
E: ObjectTaskExecutor<T, R> + 'env,
|
||||
{
|
||||
fn drop(&mut self) {
|
||||
// Close every worker input queue so blocked `recv` calls return and
|
||||
// the scoped workers exit before the enclosing scope joins them.
|
||||
self.task_txs.clear();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ObjectTaskExecutor, ObjectWorkerPool, ObjectWorkerSubmitError};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Barrier};
|
||||
use std::time::Duration;
|
||||
|
||||
#[derive(Clone)]
|
||||
struct EchoExecutor;
|
||||
|
||||
impl ObjectTaskExecutor<u32, (usize, u32)> for EchoExecutor {
|
||||
fn execute(&self, worker_index: usize, task: u32) -> (usize, u32) {
|
||||
(worker_index, task)
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_worker_pool_rejects_invalid_config_and_shutdowns_explicitly() {
|
||||
let err = match ObjectWorkerPool::new(0, 1, EchoExecutor) {
|
||||
Ok(_) => panic!("zero workers should be rejected"),
|
||||
Err(err) => err,
|
||||
};
|
||||
assert!(err.contains("at least one worker"));
|
||||
let err = match ObjectWorkerPool::new(1, 0, EchoExecutor) {
|
||||
Ok(_) => panic!("zero queue should be rejected"),
|
||||
Err(err) => err,
|
||||
};
|
||||
assert!(err.contains("queue_capacity > 0"));
|
||||
|
||||
let pool = ObjectWorkerPool::new(2, 1, EchoExecutor).expect("pool");
|
||||
assert_eq!(pool.worker_count(), 2);
|
||||
assert_eq!(pool.next_worker_index(), 0);
|
||||
pool.shutdown().expect("shutdown");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_worker_pool_round_robin_submits_to_worker_queues() {
|
||||
let mut pool = ObjectWorkerPool::new(3, 4, EchoExecutor).expect("pool");
|
||||
assert_eq!(pool.try_submit_round_robin(10).expect("submit 10"), 0);
|
||||
assert_eq!(pool.try_submit_round_robin(11).expect("submit 11"), 1);
|
||||
assert_eq!(pool.try_submit_round_robin(12).expect("submit 12"), 2);
|
||||
assert_eq!(pool.try_submit_round_robin(13).expect("submit 13"), 0);
|
||||
|
||||
let mut results = Vec::new();
|
||||
for _ in 0..4 {
|
||||
results.push(
|
||||
pool.recv_result_timeout(Duration::from_secs(1))
|
||||
.expect("result channel")
|
||||
.expect("result"),
|
||||
);
|
||||
}
|
||||
results.sort_by_key(|(_, task)| *task);
|
||||
assert_eq!(results, vec![(0, 10), (1, 11), (2, 12), (0, 13)]);
|
||||
}
|
||||
|
||||
struct BlockingExecutor {
|
||||
barrier: Arc<Barrier>,
|
||||
started: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
impl ObjectTaskExecutor<u32, u32> for BlockingExecutor {
|
||||
fn execute(&self, _worker_index: usize, task: u32) -> u32 {
|
||||
self.started.store(true, Ordering::SeqCst);
|
||||
self.barrier.wait();
|
||||
task
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_worker_pool_reports_full_worker_queue_without_advancing_round_robin() {
|
||||
let barrier = Arc::new(Barrier::new(2));
|
||||
let started = Arc::new(AtomicBool::new(false));
|
||||
let mut pool = ObjectWorkerPool::new(
|
||||
1,
|
||||
1,
|
||||
BlockingExecutor {
|
||||
barrier: Arc::clone(&barrier),
|
||||
started: Arc::clone(&started),
|
||||
},
|
||||
)
|
||||
.expect("pool");
|
||||
|
||||
assert_eq!(pool.try_submit_round_robin(1).expect("first task"), 0);
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(1);
|
||||
while !started.load(Ordering::SeqCst) {
|
||||
assert!(
|
||||
std::time::Instant::now() < deadline,
|
||||
"worker did not start first task"
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(1));
|
||||
}
|
||||
assert_eq!(pool.try_submit_round_robin(2).expect("queued task"), 0);
|
||||
match pool.try_submit_round_robin(3) {
|
||||
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task }) => {
|
||||
assert_eq!(worker_index, 0);
|
||||
assert_eq!(task, 3);
|
||||
}
|
||||
other => panic!("expected queue full, got {other:?}"),
|
||||
}
|
||||
assert_eq!(pool.next_worker_index(), 0);
|
||||
|
||||
barrier.wait();
|
||||
assert_eq!(
|
||||
pool.recv_result_timeout(Duration::from_secs(1))
|
||||
.expect("result channel"),
|
||||
Some(1)
|
||||
);
|
||||
barrier.wait();
|
||||
assert_eq!(
|
||||
pool.recv_result_timeout(Duration::from_secs(1))
|
||||
.expect("result channel"),
|
||||
Some(2)
|
||||
);
|
||||
}
|
||||
|
||||
struct BorrowingEchoExecutor<'a> {
|
||||
base: &'a u32,
|
||||
}
|
||||
|
||||
impl<'a> ObjectTaskExecutor<u32, u32> for BorrowingEchoExecutor<'a> {
|
||||
fn execute(&self, _worker_index: usize, task: u32) -> u32 {
|
||||
task + *self.base
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scoped_object_worker_pool_borrows_environment_and_processes_tasks() {
|
||||
let base = 100u32;
|
||||
std::thread::scope(|scope| {
|
||||
let mut pool = super::ScopedObjectWorkerPool::new(
|
||||
scope,
|
||||
2,
|
||||
2,
|
||||
BorrowingEchoExecutor { base: &base },
|
||||
)
|
||||
.expect("scoped pool");
|
||||
assert_eq!(pool.worker_count(), 2);
|
||||
pool.try_submit_round_robin(1).expect("submit 1");
|
||||
pool.try_submit_round_robin(2).expect("submit 2");
|
||||
let mut results = Vec::new();
|
||||
for _ in 0..2 {
|
||||
results.push(
|
||||
pool.recv_result_timeout(Duration::from_secs(1))
|
||||
.expect("result channel")
|
||||
.expect("result"),
|
||||
);
|
||||
}
|
||||
results.sort();
|
||||
assert_eq!(results, vec![101, 102]);
|
||||
// Dropping the pool inside the scope closes the task queues; the
|
||||
// workers exit on their own and the scope join below must not hang.
|
||||
drop(pool);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scoped_object_worker_pool_reports_full_queue() {
|
||||
std::thread::scope(|scope| {
|
||||
let barrier = Arc::new(Barrier::new(2));
|
||||
let started = Arc::new(AtomicBool::new(false));
|
||||
let mut pool = super::ScopedObjectWorkerPool::new(
|
||||
scope,
|
||||
1,
|
||||
1,
|
||||
BlockingExecutor {
|
||||
barrier: Arc::clone(&barrier),
|
||||
started: Arc::clone(&started),
|
||||
},
|
||||
)
|
||||
.expect("scoped pool");
|
||||
pool.try_submit_round_robin(1).expect("first task");
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(1);
|
||||
while !started.load(Ordering::SeqCst) {
|
||||
assert!(
|
||||
std::time::Instant::now() < deadline,
|
||||
"scoped worker did not start first task"
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(1));
|
||||
}
|
||||
pool.try_submit_round_robin(2).expect("queued task");
|
||||
match pool.try_submit_round_robin(3) {
|
||||
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task }) => {
|
||||
assert_eq!(worker_index, 0);
|
||||
assert_eq!(task, 3);
|
||||
}
|
||||
other => panic!("expected queue full, got {other:?}"),
|
||||
}
|
||||
// Dropping the pool closes the task queues; releasing the barrier
|
||||
// afterwards lets the blocked worker finish task 1, observe the
|
||||
// closed result channel, and exit before the scope join.
|
||||
drop(pool);
|
||||
barrier.wait();
|
||||
});
|
||||
}
|
||||
}
|
||||
332
crates/panda-rpki-validator/src/parallel/phase2_scheduler.rs
Normal file
332
crates/panda-rpki-validator/src/parallel/phase2_scheduler.rs
Normal file
@ -0,0 +1,332 @@
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
|
||||
use crate::parallel::types::RepoIdentity;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||
pub struct CaInstanceId(pub u64);
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||
pub struct PublicationPointId(pub u64);
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct PublicationPointState {
|
||||
pub ca_instance_id: CaInstanceId,
|
||||
pub pending_roa_tasks: usize,
|
||||
pub child_discovery_released: bool,
|
||||
pub finalized: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct Phase2CompletionSnapshot {
|
||||
pub ca_ready_queue_empty: bool,
|
||||
pub ca_waiting_repo_empty: bool,
|
||||
pub repo_tasks_idle: bool,
|
||||
pub pending_roa_dispatch_empty: bool,
|
||||
pub worker_queues_empty: bool,
|
||||
pub object_result_queue_empty: bool,
|
||||
pub object_workers_idle: bool,
|
||||
pub inflight_publication_points_empty: bool,
|
||||
}
|
||||
|
||||
impl Phase2CompletionSnapshot {
|
||||
pub fn is_complete(&self) -> bool {
|
||||
self.ca_ready_queue_empty
|
||||
&& self.ca_waiting_repo_empty
|
||||
&& self.repo_tasks_idle
|
||||
&& self.pending_roa_dispatch_empty
|
||||
&& self.worker_queues_empty
|
||||
&& self.object_result_queue_empty
|
||||
&& self.object_workers_idle
|
||||
&& self.inflight_publication_points_empty
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct Phase2SchedulerState<T> {
|
||||
ca_waiting_repo_by_identity: HashMap<RepoIdentity, Vec<CaInstanceId>>,
|
||||
ca_ready_queue: VecDeque<CaInstanceId>,
|
||||
inflight_publication_points: HashMap<PublicationPointId, PublicationPointState>,
|
||||
pending_roa_dispatch: VecDeque<T>,
|
||||
}
|
||||
|
||||
impl<T> Phase2SchedulerState<T> {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
ca_waiting_repo_by_identity: HashMap::new(),
|
||||
ca_ready_queue: VecDeque::new(),
|
||||
inflight_publication_points: HashMap::new(),
|
||||
pending_roa_dispatch: VecDeque::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn wait_for_repo(&mut self, identity: RepoIdentity, ca_id: CaInstanceId) {
|
||||
self.ca_waiting_repo_by_identity
|
||||
.entry(identity)
|
||||
.or_default()
|
||||
.push(ca_id);
|
||||
}
|
||||
|
||||
pub fn release_repo_waiters(&mut self, identity: &RepoIdentity) -> Vec<CaInstanceId> {
|
||||
let released = self
|
||||
.ca_waiting_repo_by_identity
|
||||
.remove(identity)
|
||||
.unwrap_or_default();
|
||||
for ca_id in &released {
|
||||
self.ca_ready_queue.push_back(*ca_id);
|
||||
}
|
||||
released
|
||||
}
|
||||
|
||||
pub fn push_ready_ca(&mut self, ca_id: CaInstanceId) {
|
||||
self.ca_ready_queue.push_back(ca_id);
|
||||
}
|
||||
|
||||
pub fn pop_ready_ca(&mut self) -> Option<CaInstanceId> {
|
||||
self.ca_ready_queue.pop_front()
|
||||
}
|
||||
|
||||
pub fn start_publication_point(
|
||||
&mut self,
|
||||
pp_id: PublicationPointId,
|
||||
ca_id: CaInstanceId,
|
||||
pending_roa_tasks: usize,
|
||||
) {
|
||||
self.inflight_publication_points.insert(
|
||||
pp_id,
|
||||
PublicationPointState {
|
||||
ca_instance_id: ca_id,
|
||||
pending_roa_tasks,
|
||||
child_discovery_released: false,
|
||||
finalized: false,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
pub fn mark_child_discovery_released(&mut self, pp_id: PublicationPointId) {
|
||||
if let Some(state) = self.inflight_publication_points.get_mut(&pp_id) {
|
||||
state.child_discovery_released = true;
|
||||
}
|
||||
}
|
||||
|
||||
pub fn enqueue_roa_task(&mut self, task: T) {
|
||||
self.pending_roa_dispatch.push_back(task);
|
||||
}
|
||||
|
||||
pub fn pop_pending_roa_dispatch(&mut self) -> Option<T> {
|
||||
self.pending_roa_dispatch.pop_front()
|
||||
}
|
||||
|
||||
pub fn push_front_pending_roa_dispatch(&mut self, task: T) {
|
||||
self.pending_roa_dispatch.push_front(task);
|
||||
}
|
||||
|
||||
pub fn record_roa_result(&mut self, pp_id: PublicationPointId) -> Option<PublicationPointId> {
|
||||
let state = self.inflight_publication_points.get_mut(&pp_id)?;
|
||||
state.pending_roa_tasks = state.pending_roa_tasks.saturating_sub(1);
|
||||
if state.pending_roa_tasks == 0 {
|
||||
state.finalized = true;
|
||||
self.inflight_publication_points.remove(&pp_id);
|
||||
Some(pp_id)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
pub fn waiting_repo_len(&self) -> usize {
|
||||
self.ca_waiting_repo_by_identity
|
||||
.values()
|
||||
.map(Vec::len)
|
||||
.sum()
|
||||
}
|
||||
|
||||
pub fn ready_queue_len(&self) -> usize {
|
||||
self.ca_ready_queue.len()
|
||||
}
|
||||
|
||||
pub fn inflight_len(&self) -> usize {
|
||||
self.inflight_publication_points.len()
|
||||
}
|
||||
|
||||
pub fn pending_roa_dispatch_len(&self) -> usize {
|
||||
self.pending_roa_dispatch.len()
|
||||
}
|
||||
|
||||
pub fn publication_point_state(
|
||||
&self,
|
||||
pp_id: PublicationPointId,
|
||||
) -> Option<&PublicationPointState> {
|
||||
self.inflight_publication_points.get(&pp_id)
|
||||
}
|
||||
|
||||
pub fn completion_snapshot(
|
||||
&self,
|
||||
repo_tasks_idle: bool,
|
||||
worker_queues_empty: bool,
|
||||
object_result_queue_empty: bool,
|
||||
object_workers_idle: bool,
|
||||
) -> Phase2CompletionSnapshot {
|
||||
Phase2CompletionSnapshot {
|
||||
ca_ready_queue_empty: self.ca_ready_queue.is_empty(),
|
||||
ca_waiting_repo_empty: self.ca_waiting_repo_by_identity.is_empty(),
|
||||
repo_tasks_idle,
|
||||
pending_roa_dispatch_empty: self.pending_roa_dispatch.is_empty(),
|
||||
worker_queues_empty,
|
||||
object_result_queue_empty,
|
||||
object_workers_idle,
|
||||
inflight_publication_points_empty: self.inflight_publication_points.is_empty(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{CaInstanceId, Phase2SchedulerState, PublicationPointId};
|
||||
use crate::parallel::object_worker::{ObjectTaskExecutor, ObjectWorkerPool};
|
||||
use crate::parallel::types::RepoIdentity;
|
||||
use std::time::Duration;
|
||||
|
||||
fn identity(name: &str) -> RepoIdentity {
|
||||
RepoIdentity::new(
|
||||
Some(format!("https://example.test/{name}/notification.xml")),
|
||||
format!("rsync://example.test/{name}/"),
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduler_repo_ready_moves_waiting_ca_to_ready_queue() {
|
||||
let mut state = Phase2SchedulerState::<u64>::new();
|
||||
let repo = identity("arin");
|
||||
state.wait_for_repo(repo.clone(), CaInstanceId(1));
|
||||
state.wait_for_repo(repo.clone(), CaInstanceId(2));
|
||||
assert_eq!(state.waiting_repo_len(), 2);
|
||||
|
||||
let released = state.release_repo_waiters(&repo);
|
||||
assert_eq!(released, vec![CaInstanceId(1), CaInstanceId(2)]);
|
||||
assert_eq!(state.waiting_repo_len(), 0);
|
||||
assert_eq!(state.ready_queue_len(), 2);
|
||||
assert_eq!(state.pop_ready_ca(), Some(CaInstanceId(1)));
|
||||
assert_eq!(state.pop_ready_ca(), Some(CaInstanceId(2)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduler_releases_child_before_roa_results_finalize_parent() {
|
||||
let mut state = Phase2SchedulerState::<u64>::new();
|
||||
let pp = PublicationPointId(10);
|
||||
state.start_publication_point(pp, CaInstanceId(1), 2);
|
||||
state.mark_child_discovery_released(pp);
|
||||
state.push_ready_ca(CaInstanceId(2));
|
||||
|
||||
let pp_state = state.publication_point_state(pp).expect("inflight pp");
|
||||
assert!(pp_state.child_discovery_released);
|
||||
assert_eq!(pp_state.pending_roa_tasks, 2);
|
||||
assert_eq!(state.pop_ready_ca(), Some(CaInstanceId(2)));
|
||||
assert_eq!(state.record_roa_result(pp), None);
|
||||
assert_eq!(state.inflight_len(), 1);
|
||||
assert_eq!(state.record_roa_result(pp), Some(pp));
|
||||
assert_eq!(state.inflight_len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduler_completion_requires_all_queues_and_inflight_to_be_empty() {
|
||||
let mut state = Phase2SchedulerState::new();
|
||||
assert!(
|
||||
state
|
||||
.completion_snapshot(true, true, true, true)
|
||||
.is_complete()
|
||||
);
|
||||
|
||||
state.enqueue_roa_task(1u64);
|
||||
assert!(
|
||||
!state
|
||||
.completion_snapshot(true, true, true, true)
|
||||
.is_complete()
|
||||
);
|
||||
assert_eq!(state.pop_pending_roa_dispatch(), Some(1));
|
||||
assert!(
|
||||
state
|
||||
.completion_snapshot(true, true, true, true)
|
||||
.is_complete()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduler_can_retry_pending_roa_task_at_front() {
|
||||
let mut state = Phase2SchedulerState::new();
|
||||
state.enqueue_roa_task(1u64);
|
||||
state.enqueue_roa_task(2u64);
|
||||
|
||||
assert_eq!(state.pop_pending_roa_dispatch(), Some(1));
|
||||
state.push_front_pending_roa_dispatch(3);
|
||||
assert_eq!(state.pop_pending_roa_dispatch(), Some(3));
|
||||
assert_eq!(state.pop_pending_roa_dispatch(), Some(2));
|
||||
assert_eq!(state.pop_pending_roa_dispatch(), None);
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
struct TestRoaTask {
|
||||
pp_id: PublicationPointId,
|
||||
value: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
struct TestRoaResult {
|
||||
worker_index: usize,
|
||||
pp_id: PublicationPointId,
|
||||
value: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct TestRoaExecutor;
|
||||
|
||||
impl ObjectTaskExecutor<TestRoaTask, TestRoaResult> for TestRoaExecutor {
|
||||
fn execute(&self, worker_index: usize, task: TestRoaTask) -> TestRoaResult {
|
||||
TestRoaResult {
|
||||
worker_index,
|
||||
pp_id: task.pp_id,
|
||||
value: task.value,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduler_dispatches_pending_roa_tasks_to_workers_and_finalizes_on_results() {
|
||||
let pp = PublicationPointId(42);
|
||||
let mut state = Phase2SchedulerState::new();
|
||||
state.start_publication_point(pp, CaInstanceId(7), 3);
|
||||
state.mark_child_discovery_released(pp);
|
||||
for value in 0..3 {
|
||||
state.enqueue_roa_task(TestRoaTask { pp_id: pp, value });
|
||||
}
|
||||
|
||||
let mut pool = ObjectWorkerPool::new(2, 4, TestRoaExecutor).expect("object pool");
|
||||
while let Some(task) = state.pop_pending_roa_dispatch() {
|
||||
pool.try_submit_round_robin(task).expect("submit task");
|
||||
}
|
||||
assert_eq!(state.pending_roa_dispatch_len(), 0);
|
||||
|
||||
let mut results = Vec::new();
|
||||
for _ in 0..3 {
|
||||
let result = pool
|
||||
.recv_result_timeout(Duration::from_secs(1))
|
||||
.expect("result channel")
|
||||
.expect("result");
|
||||
let finalized = state.record_roa_result(result.pp_id);
|
||||
results.push(result);
|
||||
if results.len() < 3 {
|
||||
assert_eq!(finalized, None);
|
||||
} else {
|
||||
assert_eq!(finalized, Some(pp));
|
||||
}
|
||||
}
|
||||
results.sort_by_key(|result| result.value);
|
||||
assert_eq!(results[0].worker_index, 0);
|
||||
assert_eq!(results[1].worker_index, 1);
|
||||
assert_eq!(results[2].worker_index, 0);
|
||||
assert_eq!(state.inflight_len(), 0);
|
||||
assert!(
|
||||
state
|
||||
.completion_snapshot(true, true, true, true)
|
||||
.is_complete()
|
||||
);
|
||||
}
|
||||
}
|
||||
1514
crates/panda-rpki-validator/src/parallel/repo_runtime.rs
Normal file
1514
crates/panda-rpki-validator/src/parallel/repo_runtime.rs
Normal file
File diff suppressed because it is too large
Load Diff
2244
crates/panda-rpki-validator/src/parallel/repo_scheduler.rs
Normal file
2244
crates/panda-rpki-validator/src/parallel/repo_scheduler.rs
Normal file
File diff suppressed because it is too large
Load Diff
1270
crates/panda-rpki-validator/src/parallel/repo_worker.rs
Normal file
1270
crates/panda-rpki-validator/src/parallel/repo_worker.rs
Normal file
File diff suppressed because it is too large
Load Diff
879
crates/panda-rpki-validator/src/parallel/run_coordinator.rs
Normal file
879
crates/panda-rpki-validator/src/parallel/run_coordinator.rs
Normal file
@ -0,0 +1,879 @@
|
||||
use std::collections::VecDeque;
|
||||
|
||||
use crate::current_repo_index::{CurrentRepoIndex, CurrentRepoIndexHandle};
|
||||
use crate::parallel::config::ParallelPhase1Config;
|
||||
use crate::parallel::dead_repo_blacklist::{
|
||||
DeadRepoBlacklist, DeadRepoFailureOutcome, DeadRepoSuccessOutcome,
|
||||
};
|
||||
use crate::parallel::repo_scheduler::{
|
||||
InFlightRepoTable, RepoCompletion, RepoRequestAction, TransportCompletion,
|
||||
TransportRequestAction, TransportStateTables,
|
||||
};
|
||||
use crate::parallel::stats::ParallelRunStats;
|
||||
use crate::parallel::types::{
|
||||
RepoDedupKey, RepoIdentity, RepoKey, RepoRequester, RepoSyncResultEnvelope, RepoSyncTask,
|
||||
RepoTransportErrorClass, RepoTransportMode, RepoTransportResultEnvelope,
|
||||
RepoTransportResultKind, RepoTransportTask, TalInputSpec,
|
||||
};
|
||||
use crate::policy::SyncPreference;
|
||||
|
||||
pub struct GlobalRunCoordinator {
|
||||
pub config: ParallelPhase1Config,
|
||||
pub tal_inputs: Vec<TalInputSpec>,
|
||||
pub current_repo_index: CurrentRepoIndexHandle,
|
||||
pub in_flight_repos: InFlightRepoTable,
|
||||
pub transport_tables: TransportStateTables,
|
||||
pub pending_repo_tasks: VecDeque<RepoSyncTask>,
|
||||
pub pending_transport_tasks: VecDeque<RepoTransportTask>,
|
||||
pub stats: ParallelRunStats,
|
||||
/// Mutable working copy of the dead-repo blacklist (#141): failure/success
|
||||
/// counters update during the run and persist at run end. Skip decisions
|
||||
/// inside `transport_tables` use the frozen run-start snapshot.
|
||||
pub dead_repo_blacklist: Option<DeadRepoBlacklist>,
|
||||
}
|
||||
|
||||
impl GlobalRunCoordinator {
|
||||
pub fn new(config: ParallelPhase1Config, tal_inputs: Vec<TalInputSpec>) -> Self {
|
||||
let dead_repo_blacklist = config.dead_repo_blacklist.as_ref().map(|blacklist_config| {
|
||||
let (blacklist, warning) = DeadRepoBlacklist::load(&blacklist_config.path);
|
||||
if let Some(warning) = warning {
|
||||
crate::progress_log::emit(
|
||||
"dead_repo_blacklist_load_warning",
|
||||
serde_json::json!({ "warning": warning }),
|
||||
);
|
||||
}
|
||||
blacklist
|
||||
});
|
||||
let mut transport_tables = TransportStateTables::new();
|
||||
if let Some(blacklist) = dead_repo_blacklist.as_ref() {
|
||||
// Frozen run-start snapshot for skip decisions.
|
||||
transport_tables.set_dead_repo_blacklist(blacklist.clone());
|
||||
}
|
||||
Self {
|
||||
config,
|
||||
tal_inputs,
|
||||
current_repo_index: CurrentRepoIndex::shared(),
|
||||
in_flight_repos: InFlightRepoTable::new(),
|
||||
transport_tables,
|
||||
pending_repo_tasks: VecDeque::new(),
|
||||
pending_transport_tasks: VecDeque::new(),
|
||||
stats: ParallelRunStats::default(),
|
||||
dead_repo_blacklist,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn current_repo_index_handle(&self) -> CurrentRepoIndexHandle {
|
||||
self.current_repo_index.clone()
|
||||
}
|
||||
|
||||
pub fn register_repo_request(
|
||||
&mut self,
|
||||
repo_key: RepoKey,
|
||||
requester: RepoRequester,
|
||||
validation_time: time::OffsetDateTime,
|
||||
sync_preference: SyncPreference,
|
||||
priority: u8,
|
||||
) -> RepoRequestAction {
|
||||
let action = self.in_flight_repos.register_request(
|
||||
repo_key,
|
||||
requester,
|
||||
validation_time,
|
||||
sync_preference,
|
||||
priority,
|
||||
);
|
||||
|
||||
match &action {
|
||||
RepoRequestAction::Enqueued(task) => {
|
||||
self.stats.repo_tasks_total += 1;
|
||||
self.pending_repo_tasks.push_back(task.clone());
|
||||
self.stats.repo_queue_depth = self.pending_repo_tasks.len();
|
||||
}
|
||||
RepoRequestAction::Reused(_) | RepoRequestAction::FailedReuse { .. } => {
|
||||
self.stats.repo_tasks_reused += 1;
|
||||
}
|
||||
RepoRequestAction::Waiting => {}
|
||||
}
|
||||
|
||||
action
|
||||
}
|
||||
|
||||
pub fn pop_next_repo_task(&mut self) -> Option<RepoSyncTask> {
|
||||
let next = self.pending_repo_tasks.pop_front();
|
||||
self.stats.repo_queue_depth = self.pending_repo_tasks.len();
|
||||
next
|
||||
}
|
||||
|
||||
pub fn mark_repo_running(
|
||||
&mut self,
|
||||
repo_key: &RepoKey,
|
||||
started_at: time::OffsetDateTime,
|
||||
) -> Result<(), String> {
|
||||
self.in_flight_repos.mark_running(repo_key, started_at)?;
|
||||
self.stats.repo_tasks_running += 1;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn complete_repo_success(
|
||||
&mut self,
|
||||
result: RepoSyncResultEnvelope,
|
||||
finished_at: time::OffsetDateTime,
|
||||
) -> Result<RepoCompletion, String> {
|
||||
let repo_key = result.repo_key.clone();
|
||||
let completion = self
|
||||
.in_flight_repos
|
||||
.complete_success(&repo_key, result, finished_at)?;
|
||||
self.stats.repo_tasks_running = self.stats.repo_tasks_running.saturating_sub(1);
|
||||
Ok(completion)
|
||||
}
|
||||
|
||||
pub fn complete_repo_failure(
|
||||
&mut self,
|
||||
result: RepoSyncResultEnvelope,
|
||||
finished_at: time::OffsetDateTime,
|
||||
) -> Result<RepoCompletion, String> {
|
||||
let repo_key = result.repo_key.clone();
|
||||
let completion = self
|
||||
.in_flight_repos
|
||||
.complete_failure(&repo_key, result, finished_at)?;
|
||||
self.stats.repo_tasks_running = self.stats.repo_tasks_running.saturating_sub(1);
|
||||
self.stats.repo_tasks_failed += 1;
|
||||
Ok(completion)
|
||||
}
|
||||
|
||||
pub fn register_transport_request(
|
||||
&mut self,
|
||||
identity: RepoIdentity,
|
||||
requester: RepoRequester,
|
||||
validation_time: time::OffsetDateTime,
|
||||
priority: u8,
|
||||
rsync_scope_uri: String,
|
||||
rsync_failure_scope_uri: Option<String>,
|
||||
sync_preference: SyncPreference,
|
||||
retry_short_timeout: bool,
|
||||
) -> TransportRequestAction {
|
||||
let mut action = self.transport_tables.register_transport_request(
|
||||
identity,
|
||||
requester,
|
||||
validation_time,
|
||||
priority,
|
||||
rsync_scope_uri,
|
||||
rsync_failure_scope_uri,
|
||||
sync_preference,
|
||||
);
|
||||
match &mut action {
|
||||
TransportRequestAction::Enqueue(task) => {
|
||||
if retry_short_timeout {
|
||||
task.retry_short_timeout = true;
|
||||
}
|
||||
self.stats.repo_tasks_total += 1;
|
||||
self.pending_transport_tasks.push_back(task.clone());
|
||||
self.stats.repo_queue_depth = self.pending_transport_tasks.len();
|
||||
}
|
||||
TransportRequestAction::ReusedSuccess(_)
|
||||
| TransportRequestAction::ReusedTerminalFailure(_) => {
|
||||
self.stats.repo_tasks_reused += 1;
|
||||
}
|
||||
TransportRequestAction::Waiting { .. } => {}
|
||||
}
|
||||
action
|
||||
}
|
||||
|
||||
pub fn push_transport_task(&mut self, task: RepoTransportTask) {
|
||||
self.stats.repo_tasks_total += 1;
|
||||
self.pending_transport_tasks.push_back(task);
|
||||
self.stats.repo_queue_depth = self.pending_transport_tasks.len();
|
||||
}
|
||||
|
||||
pub fn pop_next_transport_task(&mut self) -> Option<RepoTransportTask> {
|
||||
let next = self.pending_transport_tasks.pop_front();
|
||||
self.stats.repo_queue_depth = self.pending_transport_tasks.len();
|
||||
next
|
||||
}
|
||||
|
||||
pub fn mark_transport_running(
|
||||
&mut self,
|
||||
dedup_key: &crate::parallel::types::RepoDedupKey,
|
||||
started_at: time::OffsetDateTime,
|
||||
) -> Result<(), String> {
|
||||
self.transport_tables
|
||||
.mark_transport_running(dedup_key, started_at)?;
|
||||
self.stats.repo_tasks_running += 1;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn complete_transport_result(
|
||||
&mut self,
|
||||
result: RepoTransportResultEnvelope,
|
||||
finished_at: time::OffsetDateTime,
|
||||
) -> Result<TransportCompletion, String> {
|
||||
let completion = self
|
||||
.transport_tables
|
||||
.complete_transport_result(result.clone(), finished_at)?;
|
||||
self.stats.repo_tasks_running = self.stats.repo_tasks_running.saturating_sub(1);
|
||||
if matches!(
|
||||
result.result,
|
||||
crate::parallel::types::RepoTransportResultKind::Failed { .. }
|
||||
) && result.mode == crate::parallel::types::RepoTransportMode::Rsync
|
||||
{
|
||||
self.stats.repo_tasks_failed += 1;
|
||||
}
|
||||
self.update_dead_repo_blacklist(&result, finished_at);
|
||||
Ok(completion)
|
||||
}
|
||||
|
||||
/// Update dead-repo blacklist counters (#141) from a real worker result.
|
||||
/// Runs on the single pump thread; synthesized terminal envelopes never
|
||||
/// pass through here, so no double counting is possible.
|
||||
fn update_dead_repo_blacklist(
|
||||
&mut self,
|
||||
result: &RepoTransportResultEnvelope,
|
||||
finished_at: time::OffsetDateTime,
|
||||
) {
|
||||
let Some(blacklist_config) = self.config.dead_repo_blacklist.clone() else {
|
||||
return;
|
||||
};
|
||||
let Some(blacklist) = self.dead_repo_blacklist.as_mut() else {
|
||||
return;
|
||||
};
|
||||
let now_unix = finished_at.unix_timestamp().max(0) as u64;
|
||||
let key: Option<(RepoTransportMode, String)> = match (&result.dedup_key, &result.result) {
|
||||
(
|
||||
RepoDedupKey::RrdpNotify { notification_uri },
|
||||
RepoTransportResultKind::Success { .. },
|
||||
) => Some((RepoTransportMode::Rrdp, notification_uri.clone())),
|
||||
(RepoDedupKey::RsyncScope { .. }, RepoTransportResultKind::Success { .. }) => Some((
|
||||
RepoTransportMode::Rsync,
|
||||
result.repo_identity.rsync_base_uri.clone(),
|
||||
)),
|
||||
(
|
||||
RepoDedupKey::RrdpNotify { notification_uri },
|
||||
RepoTransportResultKind::Failed {
|
||||
error_class: RepoTransportErrorClass::TransportFetch,
|
||||
..
|
||||
},
|
||||
) => Some((RepoTransportMode::Rrdp, notification_uri.clone())),
|
||||
(
|
||||
RepoDedupKey::RsyncScope { .. },
|
||||
RepoTransportResultKind::Failed {
|
||||
error_class: RepoTransportErrorClass::TransportFetch,
|
||||
..
|
||||
},
|
||||
) => Some((
|
||||
RepoTransportMode::Rsync,
|
||||
result.repo_identity.rsync_base_uri.clone(),
|
||||
)),
|
||||
_ => None,
|
||||
};
|
||||
let Some((transport, uri)) = key else {
|
||||
return;
|
||||
};
|
||||
match &result.result {
|
||||
RepoTransportResultKind::Success { .. } => {
|
||||
if let Some(outcome) = blacklist.record_transport_success(transport, &uri) {
|
||||
crate::progress_log::emit(
|
||||
"dead_repo_blacklist_remove",
|
||||
serde_json::json!({
|
||||
"transport": transport.as_str(),
|
||||
"uri": uri,
|
||||
"reason": match outcome {
|
||||
DeadRepoSuccessOutcome::CounterReset => "counter_reset",
|
||||
DeadRepoSuccessOutcome::BlacklistRemoved => "fetch_success",
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
RepoTransportResultKind::Failed { .. } => {
|
||||
if let DeadRepoFailureOutcome::Admitted = blacklist.record_transport_failure(
|
||||
transport,
|
||||
&uri,
|
||||
now_unix,
|
||||
blacklist_config.fail_threshold,
|
||||
blacklist_config.capacity,
|
||||
) {
|
||||
crate::progress_log::emit(
|
||||
"dead_repo_blacklist_admit",
|
||||
serde_json::json!({
|
||||
"transport": transport.as_str(),
|
||||
"uri": uri,
|
||||
"fail_threshold": blacklist_config.fail_threshold,
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn dead_repo_blacklist(&self) -> Option<&DeadRepoBlacklist> {
|
||||
self.dead_repo_blacklist.as_ref()
|
||||
}
|
||||
|
||||
pub fn runtime_record(
|
||||
&self,
|
||||
identity: &RepoIdentity,
|
||||
) -> Option<&crate::parallel::repo_scheduler::RepoRuntimeRecord> {
|
||||
self.transport_tables.runtime_record(identity)
|
||||
}
|
||||
|
||||
pub fn finalized_runtime_records_for_transport(
|
||||
&self,
|
||||
dedup_key: &RepoDedupKey,
|
||||
) -> Vec<crate::parallel::repo_scheduler::RepoRuntimeRecord> {
|
||||
self.transport_tables
|
||||
.finalized_runtime_records_for_transport(dedup_key)
|
||||
}
|
||||
|
||||
pub fn finalized_runtime_records_for_transport_result(
|
||||
&self,
|
||||
result: &RepoTransportResultEnvelope,
|
||||
) -> Vec<crate::parallel::repo_scheduler::RepoRuntimeRecord> {
|
||||
self.transport_tables
|
||||
.finalized_runtime_records_for_transport_result(result)
|
||||
}
|
||||
|
||||
pub fn reset_run_state(&mut self) {
|
||||
self.in_flight_repos.reset_run_state();
|
||||
self.transport_tables.reset_run_state();
|
||||
self.pending_repo_tasks.clear();
|
||||
self.pending_transport_tasks.clear();
|
||||
self.stats = ParallelRunStats::default();
|
||||
if let Ok(mut index) = self.current_repo_index.write() {
|
||||
index.clear();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::parallel::config::ParallelPhase1Config;
|
||||
use crate::parallel::repo_scheduler::RepoRequestAction;
|
||||
use crate::parallel::run_coordinator::GlobalRunCoordinator;
|
||||
use crate::parallel::types::{
|
||||
RepoIdentity, RepoKey, RepoRequester, RepoSyncResultEnvelope, RepoSyncResultKind,
|
||||
RepoSyncResultRef, TalInputSpec,
|
||||
};
|
||||
use crate::policy::SyncPreference;
|
||||
use crate::storage::{RepositoryViewEntry, RepositoryViewState};
|
||||
|
||||
fn requester(tal_id: &str, rir_id: &str, manifest: &str) -> RepoRequester {
|
||||
RepoRequester {
|
||||
tal_id: tal_id.to_string(),
|
||||
rir_id: rir_id.to_string(),
|
||||
parent_node_id: None,
|
||||
ca_instance_handle_id: format!("{tal_id}:{manifest}"),
|
||||
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
|
||||
manifest_rsync_uri: manifest.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn coordinator_holds_tal_inputs_and_default_stats() {
|
||||
let coordinator = GlobalRunCoordinator::new(
|
||||
ParallelPhase1Config::default(),
|
||||
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
|
||||
);
|
||||
assert_eq!(coordinator.tal_inputs.len(), 1);
|
||||
assert_eq!(coordinator.tal_inputs[0].tal_id, "arin");
|
||||
assert_eq!(coordinator.stats.repo_tasks_total, 0);
|
||||
assert!(coordinator.in_flight_repos.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn coordinator_enqueues_new_repo_task_once() {
|
||||
let mut coordinator = GlobalRunCoordinator::new(
|
||||
ParallelPhase1Config::default(),
|
||||
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
|
||||
);
|
||||
let key = RepoKey::new("rsync://example.test/repo/", None);
|
||||
let action = coordinator.register_repo_request(
|
||||
key.clone(),
|
||||
requester("arin", "arin", "rsync://example.test/repo/root.mft"),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
7,
|
||||
);
|
||||
let task = match action {
|
||||
RepoRequestAction::Enqueued(task) => task,
|
||||
other => panic!("expected enqueue, got {other:?}"),
|
||||
};
|
||||
assert_eq!(task.priority, 7);
|
||||
assert_eq!(coordinator.stats.repo_tasks_total, 1);
|
||||
assert_eq!(coordinator.stats.repo_queue_depth, 1);
|
||||
|
||||
let popped = coordinator.pop_next_repo_task().expect("task queued");
|
||||
assert_eq!(popped.repo_key, key);
|
||||
assert_eq!(coordinator.stats.repo_queue_depth, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn coordinator_reset_run_state_clears_runtime_only_state() {
|
||||
let mut coordinator = GlobalRunCoordinator::new(
|
||||
ParallelPhase1Config::default(),
|
||||
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
|
||||
);
|
||||
let identity = RepoIdentity::new(
|
||||
Some("https://example.test/notify.xml".to_string()),
|
||||
"rsync://example.test/repo/",
|
||||
);
|
||||
let requester = requester("arin", "arin", "rsync://example.test/repo/root.mft");
|
||||
let action = coordinator.register_transport_request(
|
||||
identity.clone(),
|
||||
requester,
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
0,
|
||||
"rsync://example.test/repo/".to_string(),
|
||||
None,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
false,
|
||||
);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
assert!(coordinator.runtime_record(&identity).is_some());
|
||||
assert_eq!(coordinator.pending_transport_tasks.len(), 1);
|
||||
|
||||
{
|
||||
let mut index = coordinator
|
||||
.current_repo_index
|
||||
.write()
|
||||
.expect("index write lock");
|
||||
index
|
||||
.apply_repository_view_entries(&[RepositoryViewEntry {
|
||||
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
|
||||
current_hash: Some("aa".repeat(32)),
|
||||
repository_source: Some("rsync://example.test/repo/".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
}])
|
||||
.expect("apply current object");
|
||||
assert_eq!(index.active_uri_count(), 1);
|
||||
}
|
||||
|
||||
coordinator.reset_run_state();
|
||||
|
||||
assert!(coordinator.runtime_record(&identity).is_none());
|
||||
assert!(coordinator.pending_transport_tasks.is_empty());
|
||||
assert!(coordinator.pending_repo_tasks.is_empty());
|
||||
assert_eq!(
|
||||
coordinator
|
||||
.current_repo_index
|
||||
.read()
|
||||
.expect("index read lock")
|
||||
.active_uri_count(),
|
||||
0
|
||||
);
|
||||
assert_eq!(coordinator.stats.repo_tasks_total, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn coordinator_merges_waiting_requesters_and_reuses_success() {
|
||||
let mut coordinator = GlobalRunCoordinator::new(
|
||||
ParallelPhase1Config::default(),
|
||||
vec![
|
||||
TalInputSpec::from_url("https://example.test/arin.tal"),
|
||||
TalInputSpec::from_url("https://example.test/apnic.tal"),
|
||||
],
|
||||
);
|
||||
let key = RepoKey::new("rsync://shared.example/repo/", None);
|
||||
let _ = coordinator.register_repo_request(
|
||||
key.clone(),
|
||||
requester("arin", "arin", "rsync://shared.example/repo/root.mft"),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
0,
|
||||
);
|
||||
let _ = coordinator.pop_next_repo_task();
|
||||
coordinator
|
||||
.mark_repo_running(&key, time::OffsetDateTime::UNIX_EPOCH)
|
||||
.expect("running");
|
||||
|
||||
let wait_action = coordinator.register_repo_request(
|
||||
key.clone(),
|
||||
requester("apnic", "apnic", "rsync://shared.example/repo/child.mft"),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
0,
|
||||
);
|
||||
assert_eq!(wait_action, RepoRequestAction::Waiting);
|
||||
|
||||
let completion = coordinator
|
||||
.complete_repo_success(
|
||||
RepoSyncResultEnvelope {
|
||||
repo_key: key.clone(),
|
||||
tal_id: "arin".to_string(),
|
||||
rir_id: "arin".to_string(),
|
||||
result: RepoSyncResultKind::Success(RepoSyncResultRef {
|
||||
repo_key: key.clone(),
|
||||
source: "rrdp".to_string(),
|
||||
}),
|
||||
phase: Some("rrdp_ok".to_string()),
|
||||
timing_ms: 12,
|
||||
warnings: Vec::new(),
|
||||
},
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
)
|
||||
.expect("success");
|
||||
assert_eq!(completion.released_requesters.len(), 1);
|
||||
assert_eq!(coordinator.stats.repo_tasks_running, 0);
|
||||
|
||||
let reuse_action = coordinator.register_repo_request(
|
||||
key,
|
||||
requester("ripe", "ripe", "rsync://shared.example/repo/again.mft"),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
0,
|
||||
);
|
||||
assert!(matches!(reuse_action, RepoRequestAction::Reused(_)));
|
||||
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn coordinator_reuses_failures_without_enqueuing_again() {
|
||||
let mut coordinator = GlobalRunCoordinator::new(
|
||||
ParallelPhase1Config::default(),
|
||||
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
|
||||
);
|
||||
let key = RepoKey::new("rsync://shared.example/repo/", None);
|
||||
let _ = coordinator.register_repo_request(
|
||||
key.clone(),
|
||||
requester("arin", "arin", "rsync://shared.example/repo/root.mft"),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
0,
|
||||
);
|
||||
let _ = coordinator.pop_next_repo_task();
|
||||
coordinator
|
||||
.mark_repo_running(&key, time::OffsetDateTime::UNIX_EPOCH)
|
||||
.expect("running");
|
||||
coordinator
|
||||
.complete_repo_failure(
|
||||
RepoSyncResultEnvelope {
|
||||
repo_key: key.clone(),
|
||||
tal_id: "arin".to_string(),
|
||||
rir_id: "arin".to_string(),
|
||||
result: RepoSyncResultKind::Failed {
|
||||
detail: "timeout".to_string(),
|
||||
},
|
||||
phase: Some("rrdp_failed_rsync_failed".to_string()),
|
||||
timing_ms: 12,
|
||||
warnings: Vec::new(),
|
||||
},
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
)
|
||||
.expect("failure");
|
||||
assert_eq!(coordinator.stats.repo_tasks_failed, 1);
|
||||
|
||||
let action = coordinator.register_repo_request(
|
||||
key,
|
||||
requester("ripe", "ripe", "rsync://shared.example/repo/child.mft"),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
0,
|
||||
);
|
||||
assert_eq!(
|
||||
action,
|
||||
RepoRequestAction::FailedReuse {
|
||||
detail: "timeout".to_string()
|
||||
}
|
||||
);
|
||||
assert_eq!(coordinator.stats.repo_tasks_total, 1);
|
||||
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
|
||||
}
|
||||
|
||||
// ---- Dead-repo blacklist (#141) integration tests ----
|
||||
|
||||
use crate::parallel::dead_repo_blacklist::{DeadRepoBlacklist, DeadRepoBlacklistConfig};
|
||||
use crate::parallel::types::{
|
||||
RepoDedupKey, RepoTransportErrorClass, RepoTransportMode, RepoTransportResultEnvelope,
|
||||
RepoTransportResultKind,
|
||||
};
|
||||
|
||||
const DEAD_NOTIFY: &str = "https://dead.example/notification.xml";
|
||||
const DEAD_BASE: &str = "rsync://dead.example/repo/";
|
||||
|
||||
fn blacklist_config(fail_threshold: u32) -> DeadRepoBlacklistConfig {
|
||||
DeadRepoBlacklistConfig {
|
||||
path: std::env::temp_dir().join(format!(
|
||||
"dead_repo_blacklist_coordinator_test_{}.json",
|
||||
std::process::id()
|
||||
)),
|
||||
fail_threshold,
|
||||
capacity: 256,
|
||||
}
|
||||
}
|
||||
|
||||
fn coordinator_with_blacklist(
|
||||
fail_threshold: u32,
|
||||
seed: DeadRepoBlacklist,
|
||||
) -> GlobalRunCoordinator {
|
||||
let mut config = ParallelPhase1Config::default();
|
||||
config.dead_repo_blacklist = Some(blacklist_config(fail_threshold));
|
||||
let mut coordinator = GlobalRunCoordinator::new(
|
||||
config,
|
||||
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
|
||||
);
|
||||
coordinator.dead_repo_blacklist = Some(seed.clone());
|
||||
coordinator.transport_tables.set_dead_repo_blacklist(seed);
|
||||
coordinator
|
||||
}
|
||||
|
||||
fn dead_identity() -> RepoIdentity {
|
||||
RepoIdentity::new(Some(DEAD_NOTIFY.to_string()), DEAD_BASE)
|
||||
}
|
||||
|
||||
fn dead_requester() -> RepoRequester {
|
||||
requester("arin", "arin", "rsync://dead.example/repo/root.mft")
|
||||
}
|
||||
|
||||
fn transport_envelope(
|
||||
mode: RepoTransportMode,
|
||||
error_class: RepoTransportErrorClass,
|
||||
) -> RepoTransportResultEnvelope {
|
||||
let (dedup_key, detail) = match mode {
|
||||
RepoTransportMode::Rrdp => (
|
||||
RepoDedupKey::RrdpNotify {
|
||||
notification_uri: DEAD_NOTIFY.to_string(),
|
||||
},
|
||||
"http request failed: connect timeout".to_string(),
|
||||
),
|
||||
RepoTransportMode::Rsync => (
|
||||
RepoDedupKey::RsyncScope {
|
||||
rsync_scope_uri: DEAD_BASE.to_string(),
|
||||
},
|
||||
"rsync error: timeout waiting for daemon connection".to_string(),
|
||||
),
|
||||
};
|
||||
RepoTransportResultEnvelope {
|
||||
dedup_key,
|
||||
rsync_failure_scope_uri: None,
|
||||
repo_identity: dead_identity(),
|
||||
mode,
|
||||
tal_id: "arin".to_string(),
|
||||
rir_id: "arin".to_string(),
|
||||
timing_ms: 1,
|
||||
result: RepoTransportResultKind::Failed {
|
||||
detail,
|
||||
warnings: Vec::new(),
|
||||
error_class,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn register_dead(
|
||||
coordinator: &mut GlobalRunCoordinator,
|
||||
) -> crate::parallel::repo_scheduler::TransportRequestAction {
|
||||
coordinator.register_transport_request(
|
||||
dead_identity(),
|
||||
dead_requester(),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
0,
|
||||
DEAD_BASE.to_string(),
|
||||
None,
|
||||
SyncPreference::RrdpThenRsync,
|
||||
false,
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blacklist_admission_after_threshold_transport_failures() {
|
||||
let mut coordinator = coordinator_with_blacklist(2, DeadRepoBlacklist::new());
|
||||
for _ in 0..2 {
|
||||
let action = register_dead(&mut coordinator);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
coordinator
|
||||
.complete_transport_result(
|
||||
transport_envelope(
|
||||
RepoTransportMode::Rrdp,
|
||||
RepoTransportErrorClass::TransportFetch,
|
||||
),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
)
|
||||
.expect("complete");
|
||||
// Simulate the next run: run-local scheduler state resets, the
|
||||
// persistent blacklist working copy carries over.
|
||||
coordinator.reset_run_state();
|
||||
}
|
||||
let blacklist = coordinator.dead_repo_blacklist().expect("blacklist");
|
||||
assert!(blacklist.is_blacklisted(RepoTransportMode::Rrdp, DEAD_NOTIFY));
|
||||
// Admission mid-run does not affect this run's frozen snapshot: the
|
||||
// scheduler still enqueues rrdp for a fresh identity registration.
|
||||
coordinator.reset_run_state();
|
||||
let action = register_dead(&mut coordinator);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blacklist_ignores_protocol_and_unknown_failures() {
|
||||
let mut coordinator = coordinator_with_blacklist(1, DeadRepoBlacklist::new());
|
||||
for error_class in [
|
||||
RepoTransportErrorClass::Protocol,
|
||||
RepoTransportErrorClass::Unknown,
|
||||
RepoTransportErrorClass::Storage,
|
||||
] {
|
||||
let action = register_dead(&mut coordinator);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
coordinator
|
||||
.complete_transport_result(
|
||||
transport_envelope(RepoTransportMode::Rrdp, error_class),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
)
|
||||
.expect("complete");
|
||||
coordinator.reset_run_state();
|
||||
}
|
||||
assert!(
|
||||
coordinator
|
||||
.dead_repo_blacklist()
|
||||
.expect("blacklist")
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blacklist_skip_rrdp_routes_straight_to_rsync() {
|
||||
let mut seed = DeadRepoBlacklist::new();
|
||||
seed.record_transport_failure(RepoTransportMode::Rrdp, DEAD_NOTIFY, 100, 1, 256);
|
||||
let mut coordinator = coordinator_with_blacklist(3, seed);
|
||||
let action = register_dead(&mut coordinator);
|
||||
let task = match action {
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(task) => task,
|
||||
other => panic!("expected rsync enqueue, got {other:?}"),
|
||||
};
|
||||
assert_eq!(task.mode, RepoTransportMode::Rsync);
|
||||
// register_dead passes retry_short_timeout=false (mirroring a prefetch
|
||||
// snapshot that recorded no real rrdp failure for skipped runs); the
|
||||
// skip path must still keep the short retry profile (#141).
|
||||
assert!(task.retry_short_timeout);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blacklist_dual_dead_terminates_immediately() {
|
||||
let mut seed = DeadRepoBlacklist::new();
|
||||
seed.record_transport_failure(RepoTransportMode::Rrdp, DEAD_NOTIFY, 100, 1, 256);
|
||||
seed.record_transport_failure(RepoTransportMode::Rsync, DEAD_BASE, 100, 1, 256);
|
||||
let mut coordinator = coordinator_with_blacklist(3, seed);
|
||||
let action = register_dead(&mut coordinator);
|
||||
let envelope = match action {
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::ReusedTerminalFailure(
|
||||
envelope,
|
||||
) => envelope,
|
||||
other => panic!("expected terminal failure, got {other:?}"),
|
||||
};
|
||||
match &envelope.result {
|
||||
RepoTransportResultKind::Failed {
|
||||
detail, warnings, ..
|
||||
} => {
|
||||
assert!(detail.contains("dead repo blacklist"));
|
||||
assert!(
|
||||
warnings
|
||||
.iter()
|
||||
.any(|warning| warning.message.contains("dead_repo_blacklist_skip_all"))
|
||||
);
|
||||
}
|
||||
other => panic!("expected failed result, got {other:?}"),
|
||||
}
|
||||
let record = coordinator
|
||||
.runtime_record(&dead_identity())
|
||||
.expect("record");
|
||||
assert_eq!(
|
||||
record.state,
|
||||
crate::parallel::types::RepoRuntimeState::FailedTerminal
|
||||
);
|
||||
assert_eq!(coordinator.stats.repo_tasks_total, 0);
|
||||
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blacklist_rsync_dead_terminates_after_rrdp_failure() {
|
||||
// Only rsync blacklisted: rrdp is attempted; when it fails, the rsync
|
||||
// fallback is short-circuited into an instant terminal failure.
|
||||
let mut seed = DeadRepoBlacklist::new();
|
||||
seed.record_transport_failure(RepoTransportMode::Rsync, DEAD_BASE, 100, 1, 256);
|
||||
let mut coordinator = coordinator_with_blacklist(3, seed);
|
||||
let action = register_dead(&mut coordinator);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
let completion = coordinator
|
||||
.complete_transport_result(
|
||||
transport_envelope(
|
||||
RepoTransportMode::Rrdp,
|
||||
RepoTransportErrorClass::TransportFetch,
|
||||
),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
)
|
||||
.expect("complete");
|
||||
assert!(completion.follow_up_tasks.is_empty());
|
||||
let record = coordinator
|
||||
.runtime_record(&dead_identity())
|
||||
.expect("record");
|
||||
assert_eq!(
|
||||
record.state,
|
||||
crate::parallel::types::RepoRuntimeState::FailedTerminal
|
||||
);
|
||||
let terminal = record.terminal_failure.as_ref().expect("terminal failure");
|
||||
match &terminal.result {
|
||||
RepoTransportResultKind::Failed { detail, .. } => {
|
||||
assert!(detail.contains("dead repo blacklist"));
|
||||
}
|
||||
other => panic!("expected failed result, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn blacklist_success_resets_failure_counter() {
|
||||
let mut coordinator = coordinator_with_blacklist(3, DeadRepoBlacklist::new());
|
||||
// One transport failure starts counting.
|
||||
let action = register_dead(&mut coordinator);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
coordinator
|
||||
.complete_transport_result(
|
||||
transport_envelope(
|
||||
RepoTransportMode::Rrdp,
|
||||
RepoTransportErrorClass::TransportFetch,
|
||||
),
|
||||
time::OffsetDateTime::UNIX_EPOCH,
|
||||
)
|
||||
.expect("complete");
|
||||
assert_eq!(
|
||||
coordinator.dead_repo_blacklist().expect("blacklist").len(),
|
||||
1
|
||||
);
|
||||
|
||||
// A later success clears the entry.
|
||||
coordinator.reset_run_state();
|
||||
let action = register_dead(&mut coordinator);
|
||||
assert!(matches!(
|
||||
action,
|
||||
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
|
||||
));
|
||||
let mut success =
|
||||
transport_envelope(RepoTransportMode::Rrdp, RepoTransportErrorClass::Unknown);
|
||||
success.result = RepoTransportResultKind::Success {
|
||||
source: "rrdp".to_string(),
|
||||
warnings: Vec::new(),
|
||||
};
|
||||
coordinator
|
||||
.complete_transport_result(success, time::OffsetDateTime::UNIX_EPOCH)
|
||||
.expect("complete");
|
||||
assert!(
|
||||
coordinator
|
||||
.dead_repo_blacklist()
|
||||
.expect("blacklist")
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
}
|
||||
25
crates/panda-rpki-validator/src/parallel/stats.rs
Normal file
25
crates/panda-rpki-validator/src/parallel/stats.rs
Normal file
@ -0,0 +1,25 @@
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct ParallelRunStats {
|
||||
pub repo_tasks_total: usize,
|
||||
pub repo_tasks_reused: usize,
|
||||
pub repo_tasks_running: usize,
|
||||
pub repo_tasks_failed: usize,
|
||||
pub inflight_snapshot_bytes: usize,
|
||||
pub repo_queue_depth: usize,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::ParallelRunStats;
|
||||
|
||||
#[test]
|
||||
fn parallel_run_stats_default_to_zero() {
|
||||
let stats = ParallelRunStats::default();
|
||||
assert_eq!(stats.repo_tasks_total, 0);
|
||||
assert_eq!(stats.repo_tasks_reused, 0);
|
||||
assert_eq!(stats.repo_tasks_running, 0);
|
||||
assert_eq!(stats.repo_tasks_failed, 0);
|
||||
assert_eq!(stats.inflight_snapshot_bytes, 0);
|
||||
assert_eq!(stats.repo_queue_depth, 0);
|
||||
}
|
||||
}
|
||||
512
crates/panda-rpki-validator/src/parallel/transport_prefetch.rs
Normal file
512
crates/panda-rpki-validator/src/parallel/transport_prefetch.rs
Normal file
@ -0,0 +1,512 @@
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::parallel::types::{
|
||||
RepoDedupKey, RepoIdentity, RepoRequester, RepoTransportMode, RepoTransportTask,
|
||||
};
|
||||
use crate::policy::SyncPreference;
|
||||
|
||||
pub const TRANSPORT_PREFETCH_SCHEMA_VERSION: u32 = 1;
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TransportPrefetchSnapshot {
|
||||
pub schema_version: u32,
|
||||
pub generated_at_unix: i64,
|
||||
pub sync_preference: SyncPreference,
|
||||
pub requests: Vec<TransportPrefetchRequest>,
|
||||
}
|
||||
|
||||
impl TransportPrefetchSnapshot {
|
||||
pub fn new(sync_preference: SyncPreference, requests: Vec<TransportPrefetchRequest>) -> Self {
|
||||
Self {
|
||||
schema_version: TRANSPORT_PREFETCH_SCHEMA_VERSION,
|
||||
generated_at_unix: time::OffsetDateTime::now_utc().unix_timestamp(),
|
||||
sync_preference,
|
||||
requests,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_compatible_with(&self, sync_preference: SyncPreference) -> bool {
|
||||
self.schema_version == TRANSPORT_PREFETCH_SCHEMA_VERSION
|
||||
&& self.sync_preference == sync_preference
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TransportPrefetchRequest {
|
||||
pub dedup_key: TransportPrefetchDedupKey,
|
||||
pub rsync_scope_uri: String,
|
||||
pub rsync_failure_scope_uri: Option<String>,
|
||||
pub repo_identity: TransportPrefetchRepoIdentity,
|
||||
pub mode: TransportPrefetchMode,
|
||||
#[serde(default)]
|
||||
pub last_result: Option<TransportPrefetchLastResult>,
|
||||
#[serde(default)]
|
||||
pub last_rsync_result: Option<TransportPrefetchLastResult>,
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub priority: u8,
|
||||
pub requesters: Vec<TransportPrefetchRequester>,
|
||||
}
|
||||
|
||||
impl TransportPrefetchRequest {
|
||||
pub fn from_registered_request(
|
||||
identity: &RepoIdentity,
|
||||
requester: &RepoRequester,
|
||||
priority: u8,
|
||||
rsync_scope_uri: String,
|
||||
rsync_failure_scope_uri: Option<String>,
|
||||
sync_preference: SyncPreference,
|
||||
) -> Self {
|
||||
let (dedup_key, mode) = if sync_preference == SyncPreference::RrdpThenRsync {
|
||||
if let Some(notification_uri) = identity.notification_uri.clone() {
|
||||
(
|
||||
TransportPrefetchDedupKey::RrdpNotify { notification_uri },
|
||||
TransportPrefetchMode::Rrdp,
|
||||
)
|
||||
} else {
|
||||
(
|
||||
TransportPrefetchDedupKey::RsyncScope {
|
||||
rsync_scope_uri: rsync_scope_uri.clone(),
|
||||
},
|
||||
TransportPrefetchMode::Rsync,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
(
|
||||
TransportPrefetchDedupKey::RsyncScope {
|
||||
rsync_scope_uri: rsync_scope_uri.clone(),
|
||||
},
|
||||
TransportPrefetchMode::Rsync,
|
||||
)
|
||||
};
|
||||
|
||||
Self {
|
||||
dedup_key,
|
||||
rsync_scope_uri,
|
||||
rsync_failure_scope_uri,
|
||||
repo_identity: TransportPrefetchRepoIdentity::from_identity(identity),
|
||||
mode,
|
||||
last_result: None,
|
||||
last_rsync_result: None,
|
||||
tal_id: requester.tal_id.clone(),
|
||||
rir_id: requester.rir_id.clone(),
|
||||
priority,
|
||||
requesters: vec![TransportPrefetchRequester::from_requester(requester)],
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_task(task: &RepoTransportTask, rsync_scope_uri: String) -> Self {
|
||||
Self {
|
||||
dedup_key: TransportPrefetchDedupKey::from_repo_key(&task.dedup_key),
|
||||
rsync_scope_uri,
|
||||
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
|
||||
repo_identity: TransportPrefetchRepoIdentity::from_identity(&task.repo_identity),
|
||||
mode: TransportPrefetchMode::from_mode(task.mode),
|
||||
last_result: None,
|
||||
last_rsync_result: None,
|
||||
tal_id: task.tal_id.clone(),
|
||||
rir_id: task.rir_id.clone(),
|
||||
priority: task.priority,
|
||||
requesters: task
|
||||
.requesters
|
||||
.iter()
|
||||
.map(TransportPrefetchRequester::from_requester)
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn to_identity(&self) -> RepoIdentity {
|
||||
self.repo_identity.to_identity()
|
||||
}
|
||||
|
||||
pub fn to_requester(&self) -> RepoRequester {
|
||||
self.requesters
|
||||
.first()
|
||||
.map(TransportPrefetchRequester::to_requester)
|
||||
.unwrap_or_else(|| RepoRequester {
|
||||
tal_id: self.tal_id.clone(),
|
||||
rir_id: self.rir_id.clone(),
|
||||
parent_node_id: None,
|
||||
ca_instance_handle_id: format!(
|
||||
"{}:{}",
|
||||
self.tal_id, self.repo_identity.rsync_base_uri
|
||||
),
|
||||
publication_point_rsync_uri: self.repo_identity.rsync_base_uri.clone(),
|
||||
manifest_rsync_uri: format!("{}prefetch.mft", self.repo_identity.rsync_base_uri),
|
||||
})
|
||||
}
|
||||
|
||||
fn recorder_key(&self) -> String {
|
||||
self.dedup_key.stable_key()
|
||||
}
|
||||
|
||||
pub fn retry_short_timeout(&self) -> bool {
|
||||
matches!(
|
||||
self.last_result,
|
||||
Some(TransportPrefetchLastResult { ok: false })
|
||||
)
|
||||
}
|
||||
|
||||
pub fn retry_short_rsync_timeout(&self) -> bool {
|
||||
matches!(
|
||||
self.last_rsync_result,
|
||||
Some(TransportPrefetchLastResult { ok: false })
|
||||
) || (self.mode == TransportPrefetchMode::Rsync && self.retry_short_timeout())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TransportPrefetchDedupKey {
|
||||
RrdpNotify { notification_uri: String },
|
||||
RsyncScope { rsync_scope_uri: String },
|
||||
}
|
||||
|
||||
impl TransportPrefetchDedupKey {
|
||||
fn from_repo_key(key: &RepoDedupKey) -> Self {
|
||||
match key {
|
||||
RepoDedupKey::RrdpNotify { notification_uri } => Self::RrdpNotify {
|
||||
notification_uri: notification_uri.clone(),
|
||||
},
|
||||
RepoDedupKey::RsyncScope { rsync_scope_uri } => Self::RsyncScope {
|
||||
rsync_scope_uri: rsync_scope_uri.clone(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn stable_key(&self) -> String {
|
||||
match self {
|
||||
Self::RrdpNotify { notification_uri } => format!("rrdp:{notification_uri}"),
|
||||
Self::RsyncScope { rsync_scope_uri } => format!("rsync:{rsync_scope_uri}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TransportPrefetchRepoIdentity {
|
||||
pub notification_uri: Option<String>,
|
||||
pub rsync_base_uri: String,
|
||||
}
|
||||
|
||||
impl TransportPrefetchRepoIdentity {
|
||||
fn from_identity(identity: &RepoIdentity) -> Self {
|
||||
Self {
|
||||
notification_uri: identity.notification_uri.clone(),
|
||||
rsync_base_uri: identity.rsync_base_uri.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn to_identity(&self) -> RepoIdentity {
|
||||
RepoIdentity::new(self.notification_uri.clone(), self.rsync_base_uri.clone())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TransportPrefetchMode {
|
||||
Rrdp,
|
||||
Rsync,
|
||||
}
|
||||
|
||||
impl TransportPrefetchMode {
|
||||
fn from_mode(mode: RepoTransportMode) -> Self {
|
||||
match mode {
|
||||
RepoTransportMode::Rrdp => Self::Rrdp,
|
||||
RepoTransportMode::Rsync => Self::Rsync,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TransportPrefetchLastResult {
|
||||
pub ok: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TransportPrefetchRequester {
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub parent_node_id: Option<u64>,
|
||||
pub ca_instance_handle_id: String,
|
||||
pub publication_point_rsync_uri: String,
|
||||
pub manifest_rsync_uri: String,
|
||||
}
|
||||
|
||||
impl TransportPrefetchRequester {
|
||||
fn from_requester(requester: &RepoRequester) -> Self {
|
||||
Self {
|
||||
tal_id: requester.tal_id.clone(),
|
||||
rir_id: requester.rir_id.clone(),
|
||||
parent_node_id: requester.parent_node_id,
|
||||
ca_instance_handle_id: requester.ca_instance_handle_id.clone(),
|
||||
publication_point_rsync_uri: requester.publication_point_rsync_uri.clone(),
|
||||
manifest_rsync_uri: requester.manifest_rsync_uri.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn to_requester(&self) -> RepoRequester {
|
||||
RepoRequester {
|
||||
tal_id: self.tal_id.clone(),
|
||||
rir_id: self.rir_id.clone(),
|
||||
parent_node_id: self.parent_node_id,
|
||||
ca_instance_handle_id: self.ca_instance_handle_id.clone(),
|
||||
publication_point_rsync_uri: self.publication_point_rsync_uri.clone(),
|
||||
manifest_rsync_uri: self.manifest_rsync_uri.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct TransportPrefetchRecorder {
|
||||
requests_by_key: BTreeMap<String, TransportPrefetchRequest>,
|
||||
results_by_key: BTreeMap<String, TransportPrefetchLastResult>,
|
||||
rsync_results_by_scope: BTreeMap<String, TransportPrefetchLastResult>,
|
||||
request_order: Vec<String>,
|
||||
}
|
||||
|
||||
impl TransportPrefetchRecorder {
|
||||
pub fn record_registered_request(
|
||||
&mut self,
|
||||
identity: &RepoIdentity,
|
||||
requester: &RepoRequester,
|
||||
priority: u8,
|
||||
rsync_scope_uri: String,
|
||||
rsync_failure_scope_uri: Option<String>,
|
||||
sync_preference: SyncPreference,
|
||||
) {
|
||||
let request = TransportPrefetchRequest::from_registered_request(
|
||||
identity,
|
||||
requester,
|
||||
priority,
|
||||
rsync_scope_uri,
|
||||
rsync_failure_scope_uri,
|
||||
sync_preference,
|
||||
);
|
||||
self.record_request(request);
|
||||
}
|
||||
|
||||
pub fn record_task(&mut self, task: &RepoTransportTask, rsync_scope_uri: String) {
|
||||
let request = TransportPrefetchRequest::from_task(task, rsync_scope_uri);
|
||||
self.record_request(request);
|
||||
}
|
||||
|
||||
pub fn record_result(&mut self, result: &crate::parallel::types::RepoTransportResultEnvelope) {
|
||||
let key = TransportPrefetchDedupKey::from_repo_key(&result.dedup_key).stable_key();
|
||||
let last_result = TransportPrefetchLastResult {
|
||||
ok: matches!(
|
||||
result.result,
|
||||
crate::parallel::types::RepoTransportResultKind::Success { .. }
|
||||
),
|
||||
};
|
||||
self.results_by_key.insert(key.clone(), last_result);
|
||||
if let Some(request) = self.requests_by_key.get_mut(&key) {
|
||||
request.last_result = Some(last_result);
|
||||
}
|
||||
if let crate::parallel::types::RepoDedupKey::RsyncScope { rsync_scope_uri } =
|
||||
&result.dedup_key
|
||||
{
|
||||
self.rsync_results_by_scope
|
||||
.insert(rsync_scope_uri.clone(), last_result);
|
||||
for request in self.requests_by_key.values_mut() {
|
||||
if request.rsync_scope_uri == *rsync_scope_uri {
|
||||
request.last_rsync_result = Some(last_result);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn snapshot(&self, sync_preference: SyncPreference) -> TransportPrefetchSnapshot {
|
||||
TransportPrefetchSnapshot::new(
|
||||
sync_preference,
|
||||
self.request_order
|
||||
.iter()
|
||||
.filter_map(|key| self.requests_by_key.get(key))
|
||||
.cloned()
|
||||
.collect(),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn len(&self) -> usize {
|
||||
self.requests_by_key.len()
|
||||
}
|
||||
|
||||
fn record_request(&mut self, request: TransportPrefetchRequest) {
|
||||
let key = request.recorder_key();
|
||||
if !self.requests_by_key.contains_key(&key) {
|
||||
self.request_order.push(key.clone());
|
||||
let mut request = request;
|
||||
if request.last_result.is_none() {
|
||||
request.last_result = self.results_by_key.get(&key).copied();
|
||||
}
|
||||
if request.last_rsync_result.is_none() {
|
||||
request.last_rsync_result = self
|
||||
.rsync_results_by_scope
|
||||
.get(&request.rsync_scope_uri)
|
||||
.copied();
|
||||
}
|
||||
self.requests_by_key.insert(key, request);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct TransportPrefetchDispatchStats {
|
||||
pub loaded_requests: u64,
|
||||
pub enqueued_tasks: u64,
|
||||
pub waiting_requests: u64,
|
||||
pub reused_results: u64,
|
||||
pub skipped_incompatible: u64,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn requester(uri: &str) -> RepoRequester {
|
||||
RepoRequester {
|
||||
tal_id: "apnic".to_string(),
|
||||
rir_id: "apnic".to_string(),
|
||||
parent_node_id: None,
|
||||
ca_instance_handle_id: format!("apnic:{uri}"),
|
||||
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
|
||||
manifest_rsync_uri: uri.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn task(notification_uri: &str, manifest_uri: &str) -> RepoTransportTask {
|
||||
RepoTransportTask {
|
||||
dedup_key: RepoDedupKey::RrdpNotify {
|
||||
notification_uri: notification_uri.to_string(),
|
||||
},
|
||||
rsync_failure_scope_uri: Some("rsync://example.test/".to_string()),
|
||||
repo_identity: RepoIdentity::new(
|
||||
Some(notification_uri.to_string()),
|
||||
"rsync://example.test/repo/",
|
||||
),
|
||||
mode: RepoTransportMode::Rrdp,
|
||||
retry_short_timeout: false,
|
||||
tal_id: "apnic".to_string(),
|
||||
rir_id: "apnic".to_string(),
|
||||
validation_time: time::OffsetDateTime::UNIX_EPOCH,
|
||||
priority: 0,
|
||||
requesters: vec![requester(manifest_uri)],
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recorder_deduplicates_by_transport_key() {
|
||||
let mut recorder = TransportPrefetchRecorder::default();
|
||||
recorder.record_task(
|
||||
&task(
|
||||
"https://example.test/notification.xml",
|
||||
"rsync://example.test/repo/a.mft",
|
||||
),
|
||||
"rsync://example.test/repo/".to_string(),
|
||||
);
|
||||
recorder.record_task(
|
||||
&task(
|
||||
"https://example.test/notification.xml",
|
||||
"rsync://example.test/repo/b.mft",
|
||||
),
|
||||
"rsync://example.test/repo/".to_string(),
|
||||
);
|
||||
|
||||
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
|
||||
assert_eq!(snapshot.requests.len(), 1);
|
||||
assert_eq!(
|
||||
snapshot.requests[0].requesters[0].manifest_rsync_uri,
|
||||
"rsync://example.test/repo/a.mft"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recorder_preserves_first_discovery_order() {
|
||||
let mut recorder = TransportPrefetchRecorder::default();
|
||||
recorder.record_task(
|
||||
&task(
|
||||
"https://z.example.test/notification.xml",
|
||||
"rsync://z.example.test/repo/root.mft",
|
||||
),
|
||||
"rsync://z.example.test/repo/".to_string(),
|
||||
);
|
||||
recorder.record_task(
|
||||
&task(
|
||||
"https://a.example.test/notification.xml",
|
||||
"rsync://a.example.test/repo/root.mft",
|
||||
),
|
||||
"rsync://a.example.test/repo/".to_string(),
|
||||
);
|
||||
|
||||
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
|
||||
assert_eq!(
|
||||
snapshot.requests[0]
|
||||
.repo_identity
|
||||
.notification_uri
|
||||
.as_deref(),
|
||||
Some("https://z.example.test/notification.xml")
|
||||
);
|
||||
assert_eq!(
|
||||
snapshot.requests[1]
|
||||
.repo_identity
|
||||
.notification_uri
|
||||
.as_deref(),
|
||||
Some("https://a.example.test/notification.xml")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn snapshot_checks_schema_and_sync_preference() {
|
||||
let snapshot = TransportPrefetchSnapshot::new(SyncPreference::RrdpThenRsync, Vec::new());
|
||||
assert!(snapshot.is_compatible_with(SyncPreference::RrdpThenRsync));
|
||||
assert!(!snapshot.is_compatible_with(SyncPreference::RsyncOnly));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recorder_marks_failed_rrdp_and_rsync_for_next_run_short_timeouts() {
|
||||
let mut recorder = TransportPrefetchRecorder::default();
|
||||
let rrdp_task = task(
|
||||
"https://example.test/notification.xml",
|
||||
"rsync://example.test/repo/a.mft",
|
||||
);
|
||||
recorder.record_task(&rrdp_task, "rsync://example.test/repo/".to_string());
|
||||
recorder.record_result(&crate::parallel::types::RepoTransportResultEnvelope {
|
||||
dedup_key: RepoDedupKey::RrdpNotify {
|
||||
notification_uri: "https://example.test/notification.xml".to_string(),
|
||||
},
|
||||
rsync_failure_scope_uri: None,
|
||||
repo_identity: rrdp_task.repo_identity.clone(),
|
||||
mode: RepoTransportMode::Rrdp,
|
||||
tal_id: "apnic".to_string(),
|
||||
rir_id: "apnic".to_string(),
|
||||
timing_ms: 1,
|
||||
result: crate::parallel::types::RepoTransportResultKind::Failed {
|
||||
detail: "timeout".to_string(),
|
||||
warnings: Vec::new(),
|
||||
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
|
||||
},
|
||||
});
|
||||
recorder.record_result(&crate::parallel::types::RepoTransportResultEnvelope {
|
||||
dedup_key: RepoDedupKey::RsyncScope {
|
||||
rsync_scope_uri: "rsync://example.test/repo/".to_string(),
|
||||
},
|
||||
rsync_failure_scope_uri: None,
|
||||
repo_identity: rrdp_task.repo_identity.clone(),
|
||||
mode: RepoTransportMode::Rsync,
|
||||
tal_id: "apnic".to_string(),
|
||||
rir_id: "apnic".to_string(),
|
||||
timing_ms: 1,
|
||||
result: crate::parallel::types::RepoTransportResultKind::Failed {
|
||||
detail: "timeout".to_string(),
|
||||
warnings: Vec::new(),
|
||||
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
|
||||
},
|
||||
});
|
||||
|
||||
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
|
||||
assert_eq!(snapshot.requests.len(), 1);
|
||||
assert!(snapshot.requests[0].retry_short_timeout());
|
||||
assert!(snapshot.requests[0].retry_short_rsync_timeout());
|
||||
}
|
||||
}
|
||||
537
crates/panda-rpki-validator/src/parallel/types.rs
Normal file
537
crates/panda-rpki-validator/src/parallel/types.rs
Normal file
@ -0,0 +1,537 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::policy::SyncPreference;
|
||||
use crate::report::Warning;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum TalSource {
|
||||
Url(String),
|
||||
DerBytes {
|
||||
tal_url: String,
|
||||
tal_bytes: Vec<u8>,
|
||||
ta_der: Vec<u8>,
|
||||
},
|
||||
FilePath(PathBuf),
|
||||
FilePathWithTa {
|
||||
tal_path: PathBuf,
|
||||
ta_path: PathBuf,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct TalInputSpec {
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub source: TalSource,
|
||||
}
|
||||
|
||||
impl TalInputSpec {
|
||||
pub fn from_url(url: impl Into<String>) -> Self {
|
||||
let url = url.into();
|
||||
let tal_id = derive_tal_id_from_url_like(&url);
|
||||
Self {
|
||||
rir_id: tal_id.clone(),
|
||||
tal_id,
|
||||
source: TalSource::Url(url),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_file_path(path: impl Into<PathBuf>) -> Self {
|
||||
let path = path.into();
|
||||
let tal_id = derive_tal_id_from_path(&path);
|
||||
Self {
|
||||
rir_id: tal_id.clone(),
|
||||
tal_id,
|
||||
source: TalSource::FilePath(path),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_file_path_with_ta(
|
||||
tal_path: impl Into<PathBuf>,
|
||||
ta_path: impl Into<PathBuf>,
|
||||
) -> Self {
|
||||
let tal_path = tal_path.into();
|
||||
let ta_path = ta_path.into();
|
||||
let tal_id = derive_tal_id_from_path(&tal_path);
|
||||
Self {
|
||||
rir_id: tal_id.clone(),
|
||||
tal_id,
|
||||
source: TalSource::FilePathWithTa { tal_path, ta_path },
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_ta_der(tal_url: impl Into<String>, tal_bytes: Vec<u8>, ta_der: Vec<u8>) -> Self {
|
||||
let tal_url = tal_url.into();
|
||||
let tal_id = derive_tal_id_from_url_like(&tal_url);
|
||||
Self {
|
||||
rir_id: tal_id.clone(),
|
||||
tal_id,
|
||||
source: TalSource::DerBytes {
|
||||
tal_url,
|
||||
tal_bytes,
|
||||
ta_der,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
pub struct RepoIdentity {
|
||||
pub notification_uri: Option<String>,
|
||||
pub rsync_base_uri: String,
|
||||
}
|
||||
|
||||
impl RepoIdentity {
|
||||
pub fn new(notification_uri: Option<String>, rsync_base_uri: impl Into<String>) -> Self {
|
||||
Self {
|
||||
notification_uri,
|
||||
rsync_base_uri: rsync_base_uri.into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
pub enum RepoDedupKey {
|
||||
RrdpNotify { notification_uri: String },
|
||||
RsyncScope { rsync_scope_uri: String },
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RepoTransportMode {
|
||||
Rrdp,
|
||||
Rsync,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoTransportTask {
|
||||
pub dedup_key: RepoDedupKey,
|
||||
pub rsync_failure_scope_uri: Option<String>,
|
||||
pub repo_identity: RepoIdentity,
|
||||
pub mode: RepoTransportMode,
|
||||
pub retry_short_timeout: bool,
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub validation_time: time::OffsetDateTime,
|
||||
pub priority: u8,
|
||||
pub requesters: Vec<RepoRequester>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RepoTransportErrorClass {
|
||||
/// Transport-layer fetch failure (connect timeout/refused/TLS/DNS/broken
|
||||
/// connection). Only this class counts toward the dead-repo blacklist.
|
||||
TransportFetch,
|
||||
/// Protocol-level failure (HTTP status, XML/parse, content mismatch).
|
||||
Protocol,
|
||||
/// Local storage failure.
|
||||
Storage,
|
||||
/// Unclassified (synthesized or legacy results); never counted.
|
||||
Unknown,
|
||||
}
|
||||
|
||||
impl Default for RepoTransportErrorClass {
|
||||
fn default() -> Self {
|
||||
RepoTransportErrorClass::Unknown
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum RepoTransportResultKind {
|
||||
Success {
|
||||
source: String,
|
||||
warnings: Vec<Warning>,
|
||||
},
|
||||
Failed {
|
||||
detail: String,
|
||||
warnings: Vec<Warning>,
|
||||
error_class: RepoTransportErrorClass,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoTransportResultEnvelope {
|
||||
pub dedup_key: RepoDedupKey,
|
||||
pub rsync_failure_scope_uri: Option<String>,
|
||||
pub repo_identity: RepoIdentity,
|
||||
pub mode: RepoTransportMode,
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub timing_ms: u64,
|
||||
pub result: RepoTransportResultKind,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum RepoRuntimeState {
|
||||
Init,
|
||||
WaitingRrdp,
|
||||
RrdpOk,
|
||||
RrdpFailedPendingRsync,
|
||||
WaitingRsync,
|
||||
RsyncOk,
|
||||
FailedTerminal,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
pub struct RepoKey {
|
||||
pub rsync_base_uri: String,
|
||||
pub notification_uri: Option<String>,
|
||||
}
|
||||
|
||||
impl RepoKey {
|
||||
pub fn new(rsync_base_uri: impl Into<String>, notification_uri: Option<String>) -> Self {
|
||||
Self {
|
||||
rsync_base_uri: rsync_base_uri.into(),
|
||||
notification_uri,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn as_identity(&self) -> RepoIdentity {
|
||||
RepoIdentity {
|
||||
notification_uri: self.notification_uri.clone(),
|
||||
rsync_base_uri: self.rsync_base_uri.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoRequester {
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub parent_node_id: Option<u64>,
|
||||
pub ca_instance_handle_id: String,
|
||||
pub publication_point_rsync_uri: String,
|
||||
pub manifest_rsync_uri: String,
|
||||
}
|
||||
|
||||
impl RepoRequester {
|
||||
pub fn with_tal_rir(
|
||||
tal_id: impl Into<String>,
|
||||
rir_id: impl Into<String>,
|
||||
manifest_rsync_uri: impl Into<String>,
|
||||
publication_point_rsync_uri: impl Into<String>,
|
||||
ca_instance_handle_id: impl Into<String>,
|
||||
) -> Self {
|
||||
Self {
|
||||
tal_id: tal_id.into(),
|
||||
rir_id: rir_id.into(),
|
||||
parent_node_id: None,
|
||||
ca_instance_handle_id: ca_instance_handle_id.into(),
|
||||
publication_point_rsync_uri: publication_point_rsync_uri.into(),
|
||||
manifest_rsync_uri: manifest_rsync_uri.into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoSyncTask {
|
||||
pub repo_key: RepoKey,
|
||||
pub validation_time: time::OffsetDateTime,
|
||||
pub sync_preference: SyncPreference,
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub priority: u8,
|
||||
pub requesters: Vec<RepoRequester>,
|
||||
}
|
||||
|
||||
impl RepoSyncTask {
|
||||
pub fn as_transport_task(
|
||||
&self,
|
||||
dedup_key: RepoDedupKey,
|
||||
mode: RepoTransportMode,
|
||||
) -> RepoTransportTask {
|
||||
RepoTransportTask {
|
||||
dedup_key,
|
||||
rsync_failure_scope_uri: None,
|
||||
repo_identity: self.repo_key.as_identity(),
|
||||
mode,
|
||||
retry_short_timeout: false,
|
||||
tal_id: self.tal_id.clone(),
|
||||
rir_id: self.rir_id.clone(),
|
||||
validation_time: self.validation_time,
|
||||
priority: self.priority,
|
||||
requesters: self.requesters.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum RepoTaskState {
|
||||
Pending,
|
||||
Running,
|
||||
Succeeded,
|
||||
Failed,
|
||||
Reused,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoSyncResultRef {
|
||||
pub repo_key: RepoKey,
|
||||
pub source: String,
|
||||
}
|
||||
|
||||
impl RepoSyncResultRef {
|
||||
pub fn as_identity(&self) -> RepoIdentity {
|
||||
self.repo_key.as_identity()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct InFlightRepoEntry {
|
||||
pub state: RepoTaskState,
|
||||
pub task_ref: Option<RepoSyncTask>,
|
||||
pub waiting_requesters: Vec<RepoRequester>,
|
||||
pub result_ref: Option<RepoSyncResultRef>,
|
||||
pub last_result: Option<RepoSyncResultEnvelope>,
|
||||
pub last_error: Option<String>,
|
||||
pub started_at: Option<time::OffsetDateTime>,
|
||||
pub finished_at: Option<time::OffsetDateTime>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoSyncResultEnvelope {
|
||||
pub repo_key: RepoKey,
|
||||
pub tal_id: String,
|
||||
pub rir_id: String,
|
||||
pub result: RepoSyncResultKind,
|
||||
pub phase: Option<String>,
|
||||
pub timing_ms: u64,
|
||||
pub warnings: Vec<Warning>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum RepoSyncResultKind {
|
||||
Success(RepoSyncResultRef),
|
||||
Failed { detail: String },
|
||||
Reused(RepoSyncResultRef),
|
||||
}
|
||||
|
||||
fn derive_tal_id_from_url_like(s: &str) -> String {
|
||||
if let Ok(url) = url::Url::parse(s) {
|
||||
if let Some(last) = url
|
||||
.path_segments()
|
||||
.and_then(|segments| segments.filter(|seg| !seg.is_empty()).next_back())
|
||||
{
|
||||
let stem = last.rsplit_once('.').map(|(stem, _)| stem).unwrap_or(last);
|
||||
let trimmed = stem.trim();
|
||||
if !trimmed.is_empty() {
|
||||
return trimmed.to_string();
|
||||
}
|
||||
}
|
||||
if let Some(host) = url.host_str() {
|
||||
return host.to_string();
|
||||
}
|
||||
}
|
||||
"unknown-tal".to_string()
|
||||
}
|
||||
|
||||
fn derive_tal_id_from_path(path: &Path) -> String {
|
||||
path.file_stem()
|
||||
.and_then(|stem| stem.to_str())
|
||||
.map(|s| s.trim())
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or("unknown-tal")
|
||||
.to_string()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::path::Path;
|
||||
|
||||
use crate::policy::SyncPreference;
|
||||
use crate::report::Warning;
|
||||
|
||||
use super::{
|
||||
RepoDedupKey, RepoIdentity, RepoKey, RepoRequester, RepoRuntimeState, RepoSyncTask,
|
||||
RepoTaskState, RepoTransportMode, RepoTransportResultEnvelope, RepoTransportResultKind,
|
||||
TalInputSpec, TalSource, derive_tal_id_from_path, derive_tal_id_from_url_like,
|
||||
};
|
||||
|
||||
#[test]
|
||||
fn tal_input_spec_from_url_derives_tal_and_rir_ids() {
|
||||
let spec = TalInputSpec::from_url("https://example.test/tals/apnic.tal");
|
||||
assert_eq!(spec.tal_id, "apnic");
|
||||
assert_eq!(spec.rir_id, "apnic");
|
||||
assert_eq!(
|
||||
spec.source,
|
||||
TalSource::Url("https://example.test/tals/apnic.tal".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tal_input_spec_from_file_path_derives_file_stem() {
|
||||
let spec = TalInputSpec::from_file_path("local/arin.tal");
|
||||
assert_eq!(spec.tal_id, "arin");
|
||||
assert_eq!(spec.rir_id, "arin");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tal_input_spec_from_ta_der_preserves_payload() {
|
||||
let spec = TalInputSpec::from_ta_der(
|
||||
"https://example.test/ripe.tal",
|
||||
vec![4, 5, 6],
|
||||
vec![1, 2, 3],
|
||||
);
|
||||
assert_eq!(spec.tal_id, "ripe");
|
||||
assert_eq!(spec.rir_id, "ripe");
|
||||
assert_eq!(
|
||||
spec.source,
|
||||
TalSource::DerBytes {
|
||||
tal_url: "https://example.test/ripe.tal".to_string(),
|
||||
tal_bytes: vec![4, 5, 6],
|
||||
ta_der: vec![1, 2, 3],
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_key_equality_uses_rsync_base_and_notification() {
|
||||
let a = RepoKey::new(
|
||||
"rsync://example.test/repo/",
|
||||
Some("https://example.test/notify.xml".to_string()),
|
||||
);
|
||||
let b = RepoKey::new(
|
||||
"rsync://example.test/repo/",
|
||||
Some("https://example.test/notify.xml".to_string()),
|
||||
);
|
||||
let c = RepoKey::new("rsync://example.test/repo/", None);
|
||||
assert_eq!(a, b);
|
||||
assert_ne!(a, c);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_task_state_variants_are_distinct() {
|
||||
assert_ne!(RepoTaskState::Pending, RepoTaskState::Running);
|
||||
assert_ne!(RepoTaskState::Succeeded, RepoTaskState::Failed);
|
||||
assert_ne!(RepoTaskState::Failed, RepoTaskState::Reused);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_identity_preserves_raw_inputs() {
|
||||
let ident = RepoIdentity::new(
|
||||
Some("https://example.test/notify.xml".to_string()),
|
||||
"rsync://example.test/repo/",
|
||||
);
|
||||
assert_eq!(
|
||||
ident.notification_uri.as_deref(),
|
||||
Some("https://example.test/notify.xml")
|
||||
);
|
||||
assert_eq!(ident.rsync_base_uri, "rsync://example.test/repo/");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_key_can_be_viewed_as_repo_identity() {
|
||||
let key = RepoKey::new(
|
||||
"rsync://example.test/repo/",
|
||||
Some("https://example.test/notify.xml".to_string()),
|
||||
);
|
||||
let ident = key.as_identity();
|
||||
assert_eq!(ident.rsync_base_uri, "rsync://example.test/repo/");
|
||||
assert_eq!(
|
||||
ident.notification_uri.as_deref(),
|
||||
Some("https://example.test/notify.xml")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_sync_task_maps_to_rrdp_transport_task() {
|
||||
let task = RepoSyncTask {
|
||||
repo_key: RepoKey::new(
|
||||
"rsync://example.test/repo/",
|
||||
Some("https://example.test/notify.xml".to_string()),
|
||||
),
|
||||
validation_time: time::OffsetDateTime::UNIX_EPOCH,
|
||||
sync_preference: SyncPreference::RrdpThenRsync,
|
||||
tal_id: "apnic".to_string(),
|
||||
rir_id: "apnic".to_string(),
|
||||
priority: 1,
|
||||
requesters: vec![RepoRequester::with_tal_rir(
|
||||
"apnic",
|
||||
"apnic",
|
||||
"rsync://example.test/repo/root.mft",
|
||||
"rsync://example.test/repo/",
|
||||
"node:1",
|
||||
)],
|
||||
};
|
||||
let transport = task.as_transport_task(
|
||||
RepoDedupKey::RrdpNotify {
|
||||
notification_uri: "https://example.test/notify.xml".to_string(),
|
||||
},
|
||||
RepoTransportMode::Rrdp,
|
||||
);
|
||||
assert_eq!(transport.mode, RepoTransportMode::Rrdp);
|
||||
assert_eq!(transport.tal_id, "apnic");
|
||||
assert_eq!(transport.rir_id, "apnic");
|
||||
assert_eq!(transport.requesters.len(), 1);
|
||||
assert_eq!(
|
||||
transport.repo_identity.notification_uri.as_deref(),
|
||||
Some("https://example.test/notify.xml")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_transport_result_envelope_supports_success_and_failure_shapes() {
|
||||
let identity = RepoIdentity::new(None, "rsync://example.test/repo/");
|
||||
let ok = RepoTransportResultEnvelope {
|
||||
dedup_key: RepoDedupKey::RsyncScope {
|
||||
rsync_scope_uri: "rsync://example.test/module/".to_string(),
|
||||
},
|
||||
rsync_failure_scope_uri: None,
|
||||
repo_identity: identity.clone(),
|
||||
mode: RepoTransportMode::Rsync,
|
||||
tal_id: "arin".to_string(),
|
||||
rir_id: "arin".to_string(),
|
||||
timing_ms: 12,
|
||||
result: RepoTransportResultKind::Success {
|
||||
source: "rsync".to_string(),
|
||||
warnings: vec![Warning::new("ok")],
|
||||
},
|
||||
};
|
||||
let fail = RepoTransportResultEnvelope {
|
||||
dedup_key: RepoDedupKey::RsyncScope {
|
||||
rsync_scope_uri: "rsync://example.test/module/".to_string(),
|
||||
},
|
||||
rsync_failure_scope_uri: None,
|
||||
repo_identity: identity,
|
||||
mode: RepoTransportMode::Rsync,
|
||||
tal_id: "arin".to_string(),
|
||||
rir_id: "arin".to_string(),
|
||||
timing_ms: 30,
|
||||
result: RepoTransportResultKind::Failed {
|
||||
detail: "timeout".to_string(),
|
||||
warnings: vec![Warning::new("timeout")],
|
||||
error_class: super::RepoTransportErrorClass::Unknown,
|
||||
},
|
||||
};
|
||||
assert!(matches!(ok.result, RepoTransportResultKind::Success { .. }));
|
||||
assert!(matches!(
|
||||
fail.result,
|
||||
RepoTransportResultKind::Failed { .. }
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repo_runtime_state_variants_are_distinct() {
|
||||
assert_ne!(RepoRuntimeState::Init, RepoRuntimeState::WaitingRrdp);
|
||||
assert_ne!(RepoRuntimeState::RrdpOk, RepoRuntimeState::RsyncOk);
|
||||
assert_ne!(
|
||||
RepoRuntimeState::RrdpFailedPendingRsync,
|
||||
RepoRuntimeState::FailedTerminal
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn derive_tal_id_helpers_fall_back_safely() {
|
||||
assert_eq!(
|
||||
derive_tal_id_from_url_like("https://example.test/path/afrinic.tal"),
|
||||
"afrinic"
|
||||
);
|
||||
assert_eq!(
|
||||
derive_tal_id_from_path(Path::new("foo/lacnic.tal")),
|
||||
"lacnic"
|
||||
);
|
||||
}
|
||||
}
|
||||
235
crates/panda-rpki-validator/src/policy.rs
Normal file
235
crates/panda-rpki-validator/src/policy.rs
Normal file
@ -0,0 +1,235 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::ta_constraints::TaConstraintsByTal;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SyncPreference {
|
||||
RrdpThenRsync,
|
||||
RsyncOnly,
|
||||
}
|
||||
|
||||
impl Default for SyncPreference {
|
||||
fn default() -> Self {
|
||||
Self::RrdpThenRsync
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CaFailedFetchPolicy {
|
||||
ReuseCurrentInstanceVcir,
|
||||
StopAllOutput,
|
||||
}
|
||||
|
||||
impl Default for CaFailedFetchPolicy {
|
||||
fn default() -> Self {
|
||||
Self::ReuseCurrentInstanceVcir
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SignedObjectFailurePolicy {
|
||||
DropObject,
|
||||
DropPublicationPoint,
|
||||
}
|
||||
|
||||
impl Default for SignedObjectFailurePolicy {
|
||||
fn default() -> Self {
|
||||
Self::DropObject
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum ResourceValidationMode {
|
||||
#[serde(rename = "validation-update-03")]
|
||||
ValidationUpdate03,
|
||||
#[serde(rename = "rfc6487")]
|
||||
Rfc6487,
|
||||
}
|
||||
|
||||
impl Default for ResourceValidationMode {
|
||||
fn default() -> Self {
|
||||
Self::ValidationUpdate03
|
||||
}
|
||||
}
|
||||
|
||||
impl ResourceValidationMode {
|
||||
pub fn parse_cli_value(raw: &str) -> Result<Self, String> {
|
||||
match raw.trim() {
|
||||
"validation-update-03" => Ok(Self::ValidationUpdate03),
|
||||
"rfc6487" => Ok(Self::Rfc6487),
|
||||
value => Err(format!(
|
||||
"unknown resource validation mode: {value}; supported: validation-update-03,rfc6487"
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct StrictPolicy {
|
||||
pub name: bool,
|
||||
pub cms_der: bool,
|
||||
pub signed_attrs: bool,
|
||||
}
|
||||
|
||||
impl StrictPolicy {
|
||||
pub fn none() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
pub fn all() -> Self {
|
||||
Self {
|
||||
name: true,
|
||||
cms_der: true,
|
||||
signed_attrs: true,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_cli_spec(spec: Option<&str>) -> Result<Self, String> {
|
||||
let Some(spec) = spec else {
|
||||
return Ok(Self::all());
|
||||
};
|
||||
let spec = spec.trim();
|
||||
if spec.is_empty() || spec == "all" {
|
||||
return Ok(Self::all());
|
||||
}
|
||||
if spec == "none" {
|
||||
return Ok(Self::none());
|
||||
}
|
||||
|
||||
let mut out = Self::none();
|
||||
for raw in spec.split(',') {
|
||||
let item = raw.trim();
|
||||
match item {
|
||||
"name" => out.name = true,
|
||||
"cms-der" | "cms_der" => out.cms_der = true,
|
||||
"signed-attrs" | "signed_attrs" => out.signed_attrs = true,
|
||||
"all" => out = Self::all(),
|
||||
"none" => out = Self::none(),
|
||||
"" => return Err("empty strict policy name".to_string()),
|
||||
_ => {
|
||||
return Err(format!(
|
||||
"unknown strict policy: {item}; supported: name,cms-der,signed-attrs,all,none"
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct Policy {
|
||||
pub sync_preference: SyncPreference,
|
||||
pub ca_failed_fetch_policy: CaFailedFetchPolicy,
|
||||
pub signed_object_failure_policy: SignedObjectFailurePolicy,
|
||||
pub resource_validation_mode: ResourceValidationMode,
|
||||
pub strict: StrictPolicy,
|
||||
/// Locally configured, per-TAL EE certificate resource constraints. They
|
||||
/// are intentionally CLI/runtime-only rather than policy-file input.
|
||||
#[serde(skip)]
|
||||
pub ta_constraints: TaConstraintsByTal,
|
||||
}
|
||||
|
||||
impl Default for Policy {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
sync_preference: SyncPreference::default(),
|
||||
ca_failed_fetch_policy: CaFailedFetchPolicy::default(),
|
||||
signed_object_failure_policy: SignedObjectFailurePolicy::default(),
|
||||
resource_validation_mode: ResourceValidationMode::default(),
|
||||
strict: StrictPolicy::default(),
|
||||
ta_constraints: TaConstraintsByTal::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum PolicyParseError {
|
||||
#[error("policy TOML parse error: {0}")]
|
||||
Toml(String),
|
||||
}
|
||||
|
||||
impl Policy {
|
||||
pub fn from_toml_str(s: &str) -> Result<Self, PolicyParseError> {
|
||||
toml::from_str(s).map_err(|e| PolicyParseError::Toml(e.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn strict_policy_parses_cli_specs() {
|
||||
assert_eq!(
|
||||
StrictPolicy::parse_cli_spec(None).unwrap(),
|
||||
StrictPolicy::all()
|
||||
);
|
||||
assert_eq!(
|
||||
StrictPolicy::parse_cli_spec(Some("name,signed-attrs")).unwrap(),
|
||||
StrictPolicy {
|
||||
name: true,
|
||||
cms_der: false,
|
||||
signed_attrs: true,
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
StrictPolicy::parse_cli_spec(Some("none")).unwrap(),
|
||||
StrictPolicy::none()
|
||||
);
|
||||
assert!(StrictPolicy::parse_cli_spec(Some("bogus")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resource_validation_mode_parses_cli_values() {
|
||||
assert_eq!(
|
||||
ResourceValidationMode::parse_cli_value("validation-update-03").unwrap(),
|
||||
ResourceValidationMode::ValidationUpdate03
|
||||
);
|
||||
assert_eq!(
|
||||
ResourceValidationMode::parse_cli_value("rfc6487").unwrap(),
|
||||
ResourceValidationMode::Rfc6487
|
||||
);
|
||||
assert!(ResourceValidationMode::parse_cli_value("bogus").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn policy_toml_accepts_strict_table() {
|
||||
let policy = Policy::from_toml_str(
|
||||
r#"
|
||||
[strict]
|
||||
name = true
|
||||
cms_der = true
|
||||
signed_attrs = false
|
||||
"#,
|
||||
)
|
||||
.expect("parse policy");
|
||||
assert_eq!(
|
||||
policy.strict,
|
||||
StrictPolicy {
|
||||
name: true,
|
||||
cms_der: true,
|
||||
signed_attrs: false,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn policy_toml_accepts_resource_validation_mode() {
|
||||
let policy = Policy::from_toml_str(
|
||||
r#"
|
||||
resource_validation_mode = "rfc6487"
|
||||
"#,
|
||||
)
|
||||
.expect("parse policy");
|
||||
assert_eq!(
|
||||
policy.resource_validation_mode,
|
||||
ResourceValidationMode::Rfc6487
|
||||
);
|
||||
}
|
||||
}
|
||||
68
crates/panda-rpki-validator/src/progress_log.rs
Normal file
68
crates/panda-rpki-validator/src/progress_log.rs
Normal file
@ -0,0 +1,68 @@
|
||||
use serde_json::Value;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
fn progress_enabled() -> bool {
|
||||
std::env::var("RPKI_PROGRESS_LOG")
|
||||
.ok()
|
||||
.map(|v| matches!(v.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"))
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
pub fn slow_threshold_secs() -> f64 {
|
||||
std::env::var("RPKI_PROGRESS_SLOW_SECS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<f64>().ok())
|
||||
.filter(|v| *v >= 0.0)
|
||||
.unwrap_or(30.0)
|
||||
}
|
||||
|
||||
pub fn stage_fresh_slow_threshold_ms() -> u64 {
|
||||
std::env::var("RPKI_PROGRESS_STAGE_FRESH_SLOW_MS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.unwrap_or(1_000)
|
||||
}
|
||||
|
||||
pub fn pp_control_slow_threshold_ms() -> u64 {
|
||||
std::env::var("RPKI_PROGRESS_PP_CONTROL_SLOW_MS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.unwrap_or(100)
|
||||
}
|
||||
|
||||
pub fn pp_cache_slow_threshold_ms() -> u64 {
|
||||
std::env::var("RPKI_PROGRESS_PP_CACHE_SLOW_MS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.unwrap_or(50)
|
||||
}
|
||||
|
||||
pub fn control_loop_slow_threshold_ms() -> u64 {
|
||||
std::env::var("RPKI_PROGRESS_CONTROL_LOOP_SLOW_MS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.unwrap_or(1_000)
|
||||
}
|
||||
|
||||
pub fn emit(kind: &str, payload: Value) {
|
||||
if !progress_enabled() {
|
||||
return;
|
||||
}
|
||||
let ts = time::OffsetDateTime::now_utc()
|
||||
.format(&Rfc3339)
|
||||
.unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_string());
|
||||
let mut map = serde_json::Map::new();
|
||||
map.insert("ts".to_string(), Value::String(ts));
|
||||
map.insert("kind".to_string(), Value::String(kind.to_string()));
|
||||
match payload {
|
||||
Value::Object(obj) => {
|
||||
for (k, v) in obj {
|
||||
map.insert(k, v);
|
||||
}
|
||||
}
|
||||
other => {
|
||||
map.insert("value".to_string(), other);
|
||||
}
|
||||
}
|
||||
eprintln!("[progress] {}", Value::Object(map));
|
||||
}
|
||||
986
crates/panda-rpki-validator/src/replay/archive.rs
Normal file
986
crates/panda-rpki-validator/src/replay/archive.rs
Normal file
@ -0,0 +1,986 @@
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use serde::Deserialize;
|
||||
use sha2::Digest;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ReplayArchiveError {
|
||||
#[error("read {entity} failed: {path}: {detail}")]
|
||||
ReadFile {
|
||||
entity: &'static str,
|
||||
path: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("parse {entity} JSON failed: {path}: {detail}")]
|
||||
ParseJson {
|
||||
entity: &'static str,
|
||||
path: String,
|
||||
detail: String,
|
||||
},
|
||||
|
||||
#[error("unsupported {entity} version: expected 1, got {version}")]
|
||||
UnsupportedVersion { entity: &'static str, version: u32 },
|
||||
|
||||
#[error("capture directory not found: {0}")]
|
||||
MissingCaptureDirectory(String),
|
||||
|
||||
#[error("capture.json captureId mismatch: locks={locks_capture}, capture={capture_json}")]
|
||||
CaptureIdMismatch {
|
||||
locks_capture: String,
|
||||
capture_json: String,
|
||||
},
|
||||
|
||||
#[error("RRDP lock entry invalid for {notify_uri}: {detail}")]
|
||||
InvalidRrdpLock { notify_uri: String, detail: String },
|
||||
|
||||
#[error("RRDP repo bucket not found for {notify_uri}: {path}")]
|
||||
MissingRrdpRepoBucket { notify_uri: String, path: String },
|
||||
|
||||
#[error("RRDP repo meta rpkiNotify mismatch: expected {expected}, actual {actual}")]
|
||||
RrdpMetaMismatch { expected: String, actual: String },
|
||||
|
||||
#[error("RRDP session directory not found for {notify_uri}: {path}")]
|
||||
MissingRrdpSessionDir { notify_uri: String, path: String },
|
||||
|
||||
#[error("locked notification file not found for {notify_uri}: {path}")]
|
||||
MissingLockedNotification { notify_uri: String, path: String },
|
||||
|
||||
#[error("locked snapshot file not found for {notify_uri} at serial {serial} in {session_dir}")]
|
||||
MissingLockedSnapshot {
|
||||
notify_uri: String,
|
||||
serial: u64,
|
||||
session_dir: String,
|
||||
},
|
||||
|
||||
#[error(
|
||||
"multiple locked snapshot files found for {notify_uri} at serial {serial} in {session_dir}"
|
||||
)]
|
||||
AmbiguousLockedSnapshot {
|
||||
notify_uri: String,
|
||||
serial: u64,
|
||||
session_dir: String,
|
||||
},
|
||||
|
||||
#[error("rsync URI is not a valid module/base URI: {uri}: {detail}")]
|
||||
InvalidRsyncUri { uri: String, detail: String },
|
||||
|
||||
#[error("rsync module bucket not found for {module_uri}: {path}")]
|
||||
MissingRsyncModuleBucket { module_uri: String, path: String },
|
||||
|
||||
#[error("rsync module meta mismatch: expected {expected}, actual {actual}")]
|
||||
RsyncMetaMismatch { expected: String, actual: String },
|
||||
|
||||
#[error("rsync module tree not found for {module_uri}: {path}")]
|
||||
MissingRsyncTree { module_uri: String, path: String },
|
||||
|
||||
#[error("no replay lock found for RRDP notification URI: {0}")]
|
||||
MissingRrdpLock(String),
|
||||
|
||||
#[error("no replay lock found for rsync module: {0}")]
|
||||
MissingRsyncLock(String),
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ReplayTransport {
|
||||
Rrdp,
|
||||
Rsync,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
|
||||
pub struct ReplayRrdpLock {
|
||||
pub transport: ReplayTransport,
|
||||
pub session: Option<String>,
|
||||
pub serial: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
|
||||
pub struct ReplayRsyncLock {
|
||||
pub transport: ReplayTransport,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
|
||||
pub struct ReplayLocks {
|
||||
pub version: u32,
|
||||
pub capture: String,
|
||||
pub rrdp: BTreeMap<String, ReplayRrdpLock>,
|
||||
pub rsync: BTreeMap<String, ReplayRsyncLock>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
|
||||
pub struct ReplayCaptureMeta {
|
||||
pub version: u32,
|
||||
#[serde(rename = "captureId")]
|
||||
pub capture_id: String,
|
||||
#[serde(rename = "createdAt")]
|
||||
pub created_at: String,
|
||||
pub notes: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
|
||||
pub struct ReplayRrdpRepoMeta {
|
||||
pub version: u32,
|
||||
#[serde(rename = "rpkiNotify")]
|
||||
pub rpki_notify: String,
|
||||
#[serde(rename = "createdAt")]
|
||||
pub created_at: String,
|
||||
#[serde(rename = "lastSeenAt")]
|
||||
pub last_seen_at: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
|
||||
pub struct ReplayRsyncModuleMeta {
|
||||
pub version: u32,
|
||||
pub module: String,
|
||||
#[serde(rename = "createdAt")]
|
||||
pub created_at: String,
|
||||
#[serde(rename = "lastSeenAt")]
|
||||
pub last_seen_at: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ReplayRrdpRepo {
|
||||
pub notify_uri: String,
|
||||
pub bucket_hash: String,
|
||||
pub bucket_dir: PathBuf,
|
||||
pub meta: ReplayRrdpRepoMeta,
|
||||
pub locked_session: String,
|
||||
pub locked_serial: u64,
|
||||
pub session_dir: PathBuf,
|
||||
pub locked_notification_path: PathBuf,
|
||||
pub locked_snapshot_path: PathBuf,
|
||||
pub available_delta_paths: Vec<PathBuf>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ReplayRsyncModule {
|
||||
pub module_uri: String,
|
||||
pub bucket_hash: String,
|
||||
pub bucket_dir: PathBuf,
|
||||
pub meta: ReplayRsyncModuleMeta,
|
||||
pub tree_dir: PathBuf,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ReplayArchiveIndex {
|
||||
pub archive_root: PathBuf,
|
||||
pub capture_root: PathBuf,
|
||||
pub locks_path: PathBuf,
|
||||
pub locks: ReplayLocks,
|
||||
pub capture_meta: ReplayCaptureMeta,
|
||||
pub rrdp_repos: BTreeMap<String, ReplayRrdpRepo>,
|
||||
pub rsync_modules: BTreeMap<String, ReplayRsyncModule>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum ReplayArchiveLoadMode {
|
||||
Strict,
|
||||
AllowMissingRsyncModules,
|
||||
}
|
||||
|
||||
impl ReplayArchiveIndex {
|
||||
pub fn load(
|
||||
archive_root: impl AsRef<Path>,
|
||||
locks_path: impl AsRef<Path>,
|
||||
) -> Result<Self, ReplayArchiveError> {
|
||||
Self::load_with_mode(archive_root, locks_path, ReplayArchiveLoadMode::Strict)
|
||||
}
|
||||
|
||||
pub fn load_allow_missing_rsync_modules(
|
||||
archive_root: impl AsRef<Path>,
|
||||
locks_path: impl AsRef<Path>,
|
||||
) -> Result<Self, ReplayArchiveError> {
|
||||
Self::load_with_mode(
|
||||
archive_root,
|
||||
locks_path,
|
||||
ReplayArchiveLoadMode::AllowMissingRsyncModules,
|
||||
)
|
||||
}
|
||||
|
||||
fn load_with_mode(
|
||||
archive_root: impl AsRef<Path>,
|
||||
locks_path: impl AsRef<Path>,
|
||||
load_mode: ReplayArchiveLoadMode,
|
||||
) -> Result<Self, ReplayArchiveError> {
|
||||
let archive_root = archive_root.as_ref().to_path_buf();
|
||||
let locks_path = locks_path.as_ref().to_path_buf();
|
||||
|
||||
let locks: ReplayLocks = read_json_file(&locks_path, "payload replay locks")?;
|
||||
ensure_version("payload replay locks", locks.version)?;
|
||||
|
||||
let capture_root = archive_root
|
||||
.join("v1")
|
||||
.join("captures")
|
||||
.join(&locks.capture);
|
||||
if !capture_root.is_dir() {
|
||||
return Err(ReplayArchiveError::MissingCaptureDirectory(
|
||||
capture_root.display().to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let capture_meta_path = capture_root.join("capture.json");
|
||||
let capture_meta: ReplayCaptureMeta = read_json_file(&capture_meta_path, "capture meta")?;
|
||||
ensure_version("capture meta", capture_meta.version)?;
|
||||
if capture_meta.capture_id != locks.capture {
|
||||
return Err(ReplayArchiveError::CaptureIdMismatch {
|
||||
locks_capture: locks.capture.clone(),
|
||||
capture_json: capture_meta.capture_id.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
let mut rrdp_repos = BTreeMap::new();
|
||||
for (notify_uri, lock) in &locks.rrdp {
|
||||
match lock.transport {
|
||||
ReplayTransport::Rrdp => {
|
||||
let repo = load_rrdp_repo(&capture_root, notify_uri, lock)?;
|
||||
rrdp_repos.insert(notify_uri.clone(), repo);
|
||||
}
|
||||
ReplayTransport::Rsync => {
|
||||
validate_rsync_transport_lock(notify_uri, lock)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut rsync_modules = BTreeMap::new();
|
||||
for (module_uri, lock) in &locks.rsync {
|
||||
if lock.transport != ReplayTransport::Rsync {
|
||||
return Err(ReplayArchiveError::InvalidRrdpLock {
|
||||
notify_uri: module_uri.clone(),
|
||||
detail: "rsync lock transport must be rsync".to_string(),
|
||||
});
|
||||
}
|
||||
match load_rsync_module(&capture_root, module_uri) {
|
||||
Ok(module) => {
|
||||
rsync_modules.insert(module.module_uri.clone(), module);
|
||||
}
|
||||
Err(ReplayArchiveError::MissingRsyncModuleBucket { .. })
|
||||
if load_mode == ReplayArchiveLoadMode::AllowMissingRsyncModules => {}
|
||||
Err(err) => return Err(err),
|
||||
}
|
||||
}
|
||||
|
||||
Ok(Self {
|
||||
archive_root,
|
||||
capture_root,
|
||||
locks_path,
|
||||
locks,
|
||||
capture_meta,
|
||||
rrdp_repos,
|
||||
rsync_modules,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn rrdp_lock(&self, notify_uri: &str) -> Option<&ReplayRrdpLock> {
|
||||
self.locks.rrdp.get(notify_uri)
|
||||
}
|
||||
|
||||
pub fn rrdp_repo(&self, notify_uri: &str) -> Option<&ReplayRrdpRepo> {
|
||||
self.rrdp_repos.get(notify_uri)
|
||||
}
|
||||
|
||||
pub fn require_rrdp_repo(
|
||||
&self,
|
||||
notify_uri: &str,
|
||||
) -> Result<&ReplayRrdpRepo, ReplayArchiveError> {
|
||||
self.rrdp_repos
|
||||
.get(notify_uri)
|
||||
.ok_or_else(|| ReplayArchiveError::MissingRrdpLock(notify_uri.to_string()))
|
||||
}
|
||||
|
||||
pub fn rsync_module(&self, module_uri: &str) -> Option<&ReplayRsyncModule> {
|
||||
self.rsync_modules.get(module_uri)
|
||||
}
|
||||
|
||||
pub fn resolve_rsync_module_for_base_uri(
|
||||
&self,
|
||||
rsync_base_uri: &str,
|
||||
) -> Result<&ReplayRsyncModule, ReplayArchiveError> {
|
||||
let module_uri = canonical_rsync_module(rsync_base_uri)?;
|
||||
self.rsync_modules
|
||||
.get(&module_uri)
|
||||
.ok_or(ReplayArchiveError::MissingRsyncLock(module_uri))
|
||||
}
|
||||
}
|
||||
|
||||
fn load_rrdp_repo(
|
||||
capture_root: &Path,
|
||||
notify_uri: &str,
|
||||
lock: &ReplayRrdpLock,
|
||||
) -> Result<ReplayRrdpRepo, ReplayArchiveError> {
|
||||
let session = lock
|
||||
.session
|
||||
.as_deref()
|
||||
.ok_or_else(|| ReplayArchiveError::InvalidRrdpLock {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
detail: "transport=rrdp requires non-null session".to_string(),
|
||||
})?;
|
||||
let serial = lock
|
||||
.serial
|
||||
.ok_or_else(|| ReplayArchiveError::InvalidRrdpLock {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
detail: "transport=rrdp requires non-null serial".to_string(),
|
||||
})?;
|
||||
|
||||
let bucket_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let bucket_dir = capture_root.join("rrdp").join("repos").join(&bucket_hash);
|
||||
if !bucket_dir.is_dir() {
|
||||
return Err(ReplayArchiveError::MissingRrdpRepoBucket {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
path: bucket_dir.display().to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let meta_path = bucket_dir.join("meta.json");
|
||||
let meta: ReplayRrdpRepoMeta = read_json_file(&meta_path, "RRDP repo meta")?;
|
||||
ensure_version("RRDP repo meta", meta.version)?;
|
||||
if meta.rpki_notify != notify_uri {
|
||||
return Err(ReplayArchiveError::RrdpMetaMismatch {
|
||||
expected: notify_uri.to_string(),
|
||||
actual: meta.rpki_notify.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
let session_dir = bucket_dir.join(session);
|
||||
if !session_dir.is_dir() {
|
||||
return Err(ReplayArchiveError::MissingRrdpSessionDir {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
path: session_dir.display().to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let locked_notification_path = session_dir.join(format!("notification-{serial}.xml"));
|
||||
if !locked_notification_path.is_file() {
|
||||
return Err(ReplayArchiveError::MissingLockedNotification {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
path: locked_notification_path.display().to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let mut snapshot_candidates: Vec<PathBuf> = fs::read_dir(&session_dir)
|
||||
.map_err(|e| ReplayArchiveError::ReadFile {
|
||||
entity: "RRDP session directory",
|
||||
path: session_dir.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?
|
||||
.filter_map(|entry| entry.ok().map(|e| e.path()))
|
||||
.filter(|path| path.is_file())
|
||||
.filter(|path| {
|
||||
path.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.is_some_and(|name| {
|
||||
name.starts_with(&format!("snapshot-{serial}-")) && name.ends_with(".xml")
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
snapshot_candidates.sort();
|
||||
let locked_snapshot_path = match snapshot_candidates.len() {
|
||||
0 => {
|
||||
return Err(ReplayArchiveError::MissingLockedSnapshot {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
serial,
|
||||
session_dir: session_dir.display().to_string(),
|
||||
});
|
||||
}
|
||||
1 => snapshot_candidates.remove(0),
|
||||
_ => {
|
||||
return Err(ReplayArchiveError::AmbiguousLockedSnapshot {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
serial,
|
||||
session_dir: session_dir.display().to_string(),
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
let mut available_delta_paths: Vec<PathBuf> = fs::read_dir(&session_dir)
|
||||
.map_err(|e| ReplayArchiveError::ReadFile {
|
||||
entity: "RRDP session directory",
|
||||
path: session_dir.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?
|
||||
.filter_map(|entry| entry.ok().map(|e| e.path()))
|
||||
.filter(|path| path.is_file())
|
||||
.filter(|path| {
|
||||
path.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.is_some_and(|name| name.starts_with("delta-") && name.ends_with(".xml"))
|
||||
})
|
||||
.collect();
|
||||
available_delta_paths.sort();
|
||||
|
||||
Ok(ReplayRrdpRepo {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
bucket_hash,
|
||||
bucket_dir,
|
||||
meta,
|
||||
locked_session: session.to_string(),
|
||||
locked_serial: serial,
|
||||
session_dir,
|
||||
locked_notification_path,
|
||||
locked_snapshot_path,
|
||||
available_delta_paths,
|
||||
})
|
||||
}
|
||||
|
||||
fn validate_rsync_transport_lock(
|
||||
notify_uri: &str,
|
||||
lock: &ReplayRrdpLock,
|
||||
) -> Result<(), ReplayArchiveError> {
|
||||
if lock.session.is_some() || lock.serial.is_some() {
|
||||
return Err(ReplayArchiveError::InvalidRrdpLock {
|
||||
notify_uri: notify_uri.to_string(),
|
||||
detail: "transport=rsync requires null session and serial".to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn load_rsync_module(
|
||||
capture_root: &Path,
|
||||
module_uri: &str,
|
||||
) -> Result<ReplayRsyncModule, ReplayArchiveError> {
|
||||
let canonical_module = canonical_rsync_module(module_uri)?;
|
||||
let bucket_hash = sha256_hex(canonical_module.as_bytes());
|
||||
let bucket_dir = capture_root
|
||||
.join("rsync")
|
||||
.join("modules")
|
||||
.join(&bucket_hash);
|
||||
if !bucket_dir.is_dir() {
|
||||
return Err(ReplayArchiveError::MissingRsyncModuleBucket {
|
||||
module_uri: canonical_module.clone(),
|
||||
path: bucket_dir.display().to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let meta_path = bucket_dir.join("meta.json");
|
||||
let meta: ReplayRsyncModuleMeta = if meta_path.is_file() {
|
||||
let meta: ReplayRsyncModuleMeta = read_json_file(&meta_path, "rsync module meta")?;
|
||||
ensure_version("rsync module meta", meta.version)?;
|
||||
if meta.module != canonical_module {
|
||||
return Err(ReplayArchiveError::RsyncMetaMismatch {
|
||||
expected: canonical_module.clone(),
|
||||
actual: meta.module.clone(),
|
||||
});
|
||||
}
|
||||
meta
|
||||
} else {
|
||||
ReplayRsyncModuleMeta {
|
||||
version: 1,
|
||||
module: canonical_module.clone(),
|
||||
created_at: String::new(),
|
||||
last_seen_at: String::new(),
|
||||
}
|
||||
};
|
||||
|
||||
let tree_dir = bucket_dir.join("tree");
|
||||
if !tree_dir.is_dir() {
|
||||
return Err(ReplayArchiveError::MissingRsyncTree {
|
||||
module_uri: canonical_module.clone(),
|
||||
path: tree_dir.display().to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(ReplayRsyncModule {
|
||||
module_uri: canonical_module,
|
||||
bucket_hash,
|
||||
bucket_dir,
|
||||
meta,
|
||||
tree_dir,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn canonical_rsync_module(rsync_uri: &str) -> Result<String, ReplayArchiveError> {
|
||||
let raw = rsync_uri.trim();
|
||||
let prefix = "rsync://";
|
||||
let rest = raw
|
||||
.strip_prefix(prefix)
|
||||
.ok_or_else(|| ReplayArchiveError::InvalidRsyncUri {
|
||||
uri: rsync_uri.to_string(),
|
||||
detail: "URI must start with rsync://".to_string(),
|
||||
})?;
|
||||
|
||||
let mut parts = rest.split('/');
|
||||
let authority = parts.next().unwrap_or_default();
|
||||
let module = parts.next().unwrap_or_default();
|
||||
if authority.is_empty() || module.is_empty() {
|
||||
return Err(ReplayArchiveError::InvalidRsyncUri {
|
||||
uri: rsync_uri.to_string(),
|
||||
detail: "URI must contain authority and module".to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(format!("rsync://{authority}/{module}/"))
|
||||
}
|
||||
|
||||
pub fn sha256_hex(bytes: &[u8]) -> String {
|
||||
let digest = sha2::Sha256::digest(bytes);
|
||||
hex::encode(digest)
|
||||
}
|
||||
|
||||
fn ensure_version(entity: &'static str, version: u32) -> Result<(), ReplayArchiveError> {
|
||||
if version == 1 {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(ReplayArchiveError::UnsupportedVersion { entity, version })
|
||||
}
|
||||
}
|
||||
|
||||
fn read_json_file<T: for<'de> Deserialize<'de>>(
|
||||
path: &Path,
|
||||
entity: &'static str,
|
||||
) -> Result<T, ReplayArchiveError> {
|
||||
let bytes = fs::read(path).map_err(|e| ReplayArchiveError::ReadFile {
|
||||
entity,
|
||||
path: path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
serde_json::from_slice(&bytes).map_err(|e| ReplayArchiveError::ParseJson {
|
||||
entity,
|
||||
path: path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn build_minimal_archive() -> (tempfile::TempDir, PathBuf, PathBuf, String, String) {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let archive_root = temp.path().join("payload-archive");
|
||||
let capture = "capture-001";
|
||||
let capture_root = archive_root.join("v1").join("captures").join(capture);
|
||||
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
|
||||
std::fs::write(
|
||||
capture_root.join("capture.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
|
||||
),
|
||||
)
|
||||
.expect("write capture json");
|
||||
|
||||
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
|
||||
let session = "11111111-1111-1111-1111-111111111111".to_string();
|
||||
let serial = 42u64;
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let repo_dir = capture_root.join("rrdp").join("repos").join(&repo_hash);
|
||||
let session_dir = repo_dir.join(&session);
|
||||
std::fs::create_dir_all(&session_dir).expect("mkdir session dir");
|
||||
std::fs::write(
|
||||
repo_dir.join("meta.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
|
||||
),
|
||||
)
|
||||
.expect("write repo meta");
|
||||
std::fs::write(session_dir.join("notification-42.xml"), b"<notification/>")
|
||||
.expect("write notification");
|
||||
std::fs::write(session_dir.join("snapshot-42-deadbeef.xml"), b"<snapshot/>")
|
||||
.expect("write snapshot");
|
||||
|
||||
let module_uri = "rsync://rsync.example.test/repo/".to_string();
|
||||
let mod_hash = sha256_hex(module_uri.as_bytes());
|
||||
let mod_dir = capture_root.join("rsync").join("modules").join(&mod_hash);
|
||||
let tree_dir = mod_dir.join("tree").join("rsync.example.test").join("repo");
|
||||
std::fs::create_dir_all(&tree_dir).expect("mkdir tree dir");
|
||||
std::fs::write(
|
||||
mod_dir.join("meta.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
|
||||
),
|
||||
)
|
||||
.expect("write rsync meta");
|
||||
std::fs::write(tree_dir.join("a.roa"), b"roa").expect("write rsync object");
|
||||
|
||||
let locks_path = temp.path().join("locks.json");
|
||||
std::fs::write(
|
||||
&locks_path,
|
||||
format!(
|
||||
r#"{{
|
||||
"version":1,
|
||||
"capture":"{capture}",
|
||||
"rrdp":{{
|
||||
"{notify_uri}":{{"transport":"rrdp","session":"{session}","serial":{serial}}},
|
||||
"https://rrdp-fallback.example.test/notification.xml":{{"transport":"rsync","session":null,"serial":null}}
|
||||
}},
|
||||
"rsync":{{
|
||||
"{module_uri}":{{"transport":"rsync"}}
|
||||
}}
|
||||
}}"#
|
||||
),
|
||||
)
|
||||
.expect("write locks");
|
||||
|
||||
(temp, archive_root, locks_path, notify_uri, module_uri)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn canonical_rsync_module_normalizes_subpaths_and_trailing_slash() {
|
||||
let normalized = canonical_rsync_module("rsync://example.test/repo/sub/dir/file.roa")
|
||||
.expect("normalize module");
|
||||
assert_eq!(normalized, "rsync://example.test/repo/");
|
||||
|
||||
let normalized = canonical_rsync_module("rsync://example.test/repo")
|
||||
.expect("normalize module without slash");
|
||||
assert_eq!(normalized, "rsync://example.test/repo/");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_loads_rrdp_and_rsync_entries() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, module_uri) = build_minimal_archive();
|
||||
let index =
|
||||
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
|
||||
|
||||
assert_eq!(index.capture_meta.capture_id, "capture-001");
|
||||
assert_eq!(index.rrdp_repos.len(), 1);
|
||||
assert_eq!(index.locks.rrdp.len(), 2);
|
||||
assert_eq!(index.rsync_modules.len(), 1);
|
||||
|
||||
let repo = index.require_rrdp_repo(¬ify_uri).expect("rrdp repo");
|
||||
assert_eq!(repo.locked_serial, 42);
|
||||
assert!(
|
||||
repo.locked_notification_path
|
||||
.ends_with("notification-42.xml")
|
||||
);
|
||||
assert!(
|
||||
repo.locked_snapshot_path
|
||||
.ends_with("snapshot-42-deadbeef.xml")
|
||||
);
|
||||
assert!(repo.available_delta_paths.is_empty());
|
||||
|
||||
let module = index
|
||||
.resolve_rsync_module_for_base_uri("rsync://rsync.example.test/repo/sub/path")
|
||||
.expect("resolve module from base");
|
||||
assert_eq!(module.module_uri, module_uri);
|
||||
assert!(
|
||||
module
|
||||
.tree_dir
|
||||
.join("rsync.example.test")
|
||||
.join("repo")
|
||||
.join("a.roa")
|
||||
.is_file()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_rsync_transport_with_session_or_serial() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
|
||||
let bad = locks_path.with_file_name("bad-locks.json");
|
||||
std::fs::write(
|
||||
&bad,
|
||||
r#"{
|
||||
"version":1,
|
||||
"capture":"capture-001",
|
||||
"rrdp":{
|
||||
"https://rrdp-fallback.example.test/notification.xml":{"transport":"rsync","session":"oops","serial":1}
|
||||
},
|
||||
"rsync":{}
|
||||
}"#,
|
||||
)
|
||||
.expect("write bad locks");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &bad).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::InvalidRrdpLock { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_mismatched_rrdp_meta() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let meta_path = archive_root
|
||||
.join("v1/captures/capture-001/rrdp/repos")
|
||||
.join(repo_hash)
|
||||
.join("meta.json");
|
||||
std::fs::write(
|
||||
&meta_path,
|
||||
r#"{"version":1,"rpkiNotify":"https://other.example/notification.xml","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}"#,
|
||||
)
|
||||
.expect("rewrite meta");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::RrdpMetaMismatch { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_missing_snapshot() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let session_dir = archive_root
|
||||
.join("v1/captures/capture-001/rrdp/repos")
|
||||
.join(repo_hash)
|
||||
.join("11111111-1111-1111-1111-111111111111");
|
||||
std::fs::remove_file(session_dir.join("snapshot-42-deadbeef.xml"))
|
||||
.expect("remove snapshot");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingLockedSnapshot { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_ambiguous_snapshot_candidates() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let session_dir = archive_root
|
||||
.join("v1/captures/capture-001/rrdp/repos")
|
||||
.join(repo_hash)
|
||||
.join("11111111-1111-1111-1111-111111111111");
|
||||
std::fs::write(session_dir.join("snapshot-42-another.xml"), b"<snapshot/>")
|
||||
.expect("write second snapshot");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::AmbiguousLockedSnapshot { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn canonical_rsync_module_rejects_invalid_uris() {
|
||||
let err = canonical_rsync_module("https://example.test/repo/").unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::InvalidRsyncUri { .. }),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let err = canonical_rsync_module("rsync://example.test").unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::InvalidRsyncUri { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_missing_capture_directory() {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let archive_root = temp.path().join("payload-archive");
|
||||
std::fs::create_dir_all(&archive_root).expect("mkdir archive root");
|
||||
let locks_path = temp.path().join("locks.json");
|
||||
std::fs::write(
|
||||
&locks_path,
|
||||
r#"{"version":1,"capture":"missing","rrdp":{},"rsync":{}}"#,
|
||||
)
|
||||
.expect("write locks");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingCaptureDirectory(_)),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_missing_rrdp_bucket_and_session_and_notification() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let repo_dir = archive_root
|
||||
.join("v1/captures/capture-001/rrdp/repos")
|
||||
.join(&repo_hash);
|
||||
|
||||
std::fs::remove_dir_all(&repo_dir).expect("remove repo dir");
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingRrdpRepoBucket { .. }),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let session_dir = archive_root
|
||||
.join("v1/captures/capture-001/rrdp/repos")
|
||||
.join(&repo_hash)
|
||||
.join("11111111-1111-1111-1111-111111111111");
|
||||
std::fs::remove_dir_all(&session_dir).expect("remove session dir");
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingRrdpSessionDir { .. }),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let session_dir = archive_root
|
||||
.join("v1/captures/capture-001/rrdp/repos")
|
||||
.join(&repo_hash)
|
||||
.join("11111111-1111-1111-1111-111111111111");
|
||||
std::fs::remove_file(session_dir.join("notification-42.xml")).expect("remove notification");
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingLockedNotification { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_invalid_json_and_missing_rsync_bucket() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
|
||||
std::fs::write(&locks_path, b"not json").expect("corrupt locks");
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(
|
||||
err,
|
||||
ReplayArchiveError::ParseJson {
|
||||
entity: "payload replay locks",
|
||||
..
|
||||
}
|
||||
),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
|
||||
let mod_hash = sha256_hex(module_uri.as_bytes());
|
||||
let mod_dir = archive_root
|
||||
.join("v1/captures/capture-001/rsync/modules")
|
||||
.join(&mod_hash);
|
||||
std::fs::remove_dir_all(&mod_dir).expect("remove module dir");
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingRsyncModuleBucket { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_can_skip_missing_rsync_modules_in_lenient_mode() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
|
||||
let mod_hash = sha256_hex(module_uri.as_bytes());
|
||||
let mod_dir = archive_root
|
||||
.join("v1/captures/capture-001/rsync/modules")
|
||||
.join(mod_hash);
|
||||
std::fs::remove_dir_all(&mod_dir).expect("remove module dir");
|
||||
|
||||
let index =
|
||||
ReplayArchiveIndex::load_allow_missing_rsync_modules(&archive_root, &locks_path)
|
||||
.expect("load lenient replay index");
|
||||
assert!(index.rsync_modules.is_empty());
|
||||
}
|
||||
#[test]
|
||||
fn replay_archive_index_accepts_missing_rsync_module_meta_when_tree_exists() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
|
||||
let mod_hash = sha256_hex(module_uri.as_bytes());
|
||||
let meta_path = archive_root
|
||||
.join("v1/captures/capture-001/rsync/modules")
|
||||
.join(mod_hash)
|
||||
.join("meta.json");
|
||||
std::fs::remove_file(&meta_path).expect("remove rsync module meta");
|
||||
|
||||
let index =
|
||||
ReplayArchiveIndex::load_allow_missing_rsync_modules(&archive_root, &locks_path)
|
||||
.expect("load replay index without rsync meta");
|
||||
let module = index
|
||||
.rsync_modules
|
||||
.get(&module_uri)
|
||||
.expect("module present");
|
||||
assert_eq!(module.meta.module, module_uri);
|
||||
assert_eq!(module.meta.version, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_capture_id_mismatch() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
|
||||
let capture_json = archive_root.join("v1/captures/capture-001/capture.json");
|
||||
std::fs::write(
|
||||
&capture_json,
|
||||
r#"{"version":1,"captureId":"other-capture","createdAt":"2026-03-13T00:00:00Z","notes":""}"#,
|
||||
)
|
||||
.expect("rewrite capture json");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::CaptureIdMismatch { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_unsupported_locks_version() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
|
||||
std::fs::write(
|
||||
&locks_path,
|
||||
r#"{"version":2,"capture":"capture-001","rrdp":{},"rsync":{}}"#,
|
||||
)
|
||||
.expect("rewrite locks version");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(
|
||||
err,
|
||||
ReplayArchiveError::UnsupportedVersion {
|
||||
entity: "payload replay locks",
|
||||
version: 2
|
||||
}
|
||||
),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_reports_missing_rrdp_lock_and_missing_rsync_lock() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
|
||||
let index =
|
||||
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
|
||||
|
||||
let err = index
|
||||
.require_rrdp_repo("https://missing.example/notification.xml")
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingRrdpLock(_)),
|
||||
"{err}"
|
||||
);
|
||||
|
||||
let err = index
|
||||
.resolve_rsync_module_for_base_uri("rsync://missing.example/repo/path")
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingRsyncLock(_)),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_rsync_meta_mismatch() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
|
||||
let mod_hash = sha256_hex(module_uri.as_bytes());
|
||||
let meta_path = archive_root
|
||||
.join("v1/captures/capture-001/rsync/modules")
|
||||
.join(mod_hash)
|
||||
.join("meta.json");
|
||||
std::fs::write(
|
||||
&meta_path,
|
||||
r#"{"version":1,"module":"rsync://other.example/repo/","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}"#,
|
||||
)
|
||||
.expect("rewrite rsync meta");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::RsyncMetaMismatch { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn replay_archive_index_rejects_missing_rsync_module_tree() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
|
||||
let mod_hash = sha256_hex(module_uri.as_bytes());
|
||||
let tree_dir = archive_root
|
||||
.join("v1/captures/capture-001/rsync/modules")
|
||||
.join(mod_hash)
|
||||
.join("tree");
|
||||
std::fs::remove_dir_all(&tree_dir).expect("remove tree dir");
|
||||
|
||||
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayArchiveError::MissingRsyncTree { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
}
|
||||
1161
crates/panda-rpki-validator/src/replay/delta_archive.rs
Normal file
1161
crates/panda-rpki-validator/src/replay/delta_archive.rs
Normal file
File diff suppressed because it is too large
Load Diff
368
crates/panda-rpki-validator/src/replay/delta_fetch_http.rs
Normal file
368
crates/panda-rpki-validator/src/replay/delta_fetch_http.rs
Normal file
@ -0,0 +1,368 @@
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::replay::delta_archive::{
|
||||
ReplayDeltaArchiveError, ReplayDeltaArchiveIndex, ReplayDeltaRrdpKind,
|
||||
};
|
||||
use crate::sync::rrdp::{Fetcher, parse_notification};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum PayloadDeltaReplayHttpFetcherError {
|
||||
#[error(transparent)]
|
||||
DeltaIndex(#[from] ReplayDeltaArchiveError),
|
||||
|
||||
#[error("read delta replay RRDP file failed: {path}: {detail}")]
|
||||
ReadFile { path: String, detail: String },
|
||||
|
||||
#[error("parse target notification failed for {notify_uri}: {detail}")]
|
||||
ParseNotification { notify_uri: String, detail: String },
|
||||
|
||||
#[error(
|
||||
"target notification session/serial mismatch for {notify_uri}: expected session={expected_session} serial={expected_serial}, got session={actual_session} serial={actual_serial}"
|
||||
)]
|
||||
NotificationTargetMismatch {
|
||||
notify_uri: String,
|
||||
expected_session: String,
|
||||
expected_serial: u64,
|
||||
actual_session: String,
|
||||
actual_serial: u64,
|
||||
},
|
||||
|
||||
#[error(
|
||||
"delta serial list mismatch between target notification and transition for {notify_uri}"
|
||||
)]
|
||||
DeltaSerialMismatch { notify_uri: String },
|
||||
|
||||
#[error("duplicate delta replay HTTP URI mapping for {uri}: {first_path} vs {second_path}")]
|
||||
DuplicateUriMapping {
|
||||
uri: String,
|
||||
first_path: String,
|
||||
second_path: String,
|
||||
},
|
||||
|
||||
#[error(
|
||||
"delta replay notification URI is {kind} and should not be fetched as RRDP: {notify_uri}"
|
||||
)]
|
||||
NotificationKindNotFetchable { notify_uri: String, kind: String },
|
||||
|
||||
#[error("delta replay HTTP URI not found in archive: {0}")]
|
||||
MissingUri(String),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PayloadDeltaReplayHttpFetcher {
|
||||
index: Arc<ReplayDeltaArchiveIndex>,
|
||||
routes: BTreeMap<String, PathBuf>,
|
||||
repo_kinds: BTreeMap<String, ReplayDeltaRrdpKind>,
|
||||
}
|
||||
|
||||
impl PayloadDeltaReplayHttpFetcher {
|
||||
pub fn new(
|
||||
index: Arc<ReplayDeltaArchiveIndex>,
|
||||
) -> Result<Self, PayloadDeltaReplayHttpFetcherError> {
|
||||
let mut routes = BTreeMap::new();
|
||||
let mut repo_kinds = BTreeMap::new();
|
||||
for (notify_uri, repo) in &index.rrdp_repos {
|
||||
repo_kinds.insert(notify_uri.clone(), repo.transition.kind);
|
||||
if repo.transition.kind != ReplayDeltaRrdpKind::Delta {
|
||||
continue;
|
||||
}
|
||||
let notification_path = repo
|
||||
.target_notification_path
|
||||
.as_ref()
|
||||
.expect("delta repo target notification indexed");
|
||||
insert_unique_route(&mut routes, notify_uri, notification_path)?;
|
||||
|
||||
let notification_xml = fs::read(notification_path).map_err(|e| {
|
||||
PayloadDeltaReplayHttpFetcherError::ReadFile {
|
||||
path: notification_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
let notification = parse_notification(¬ification_xml).map_err(|e| {
|
||||
PayloadDeltaReplayHttpFetcherError::ParseNotification {
|
||||
notify_uri: notify_uri.clone(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
let expected_session = repo.transition.target.session.as_deref().unwrap_or("");
|
||||
let expected_serial = repo.transition.target.serial.unwrap_or_default();
|
||||
let actual_session = notification.session_id.to_string();
|
||||
if actual_session != expected_session || notification.serial != expected_serial {
|
||||
return Err(
|
||||
PayloadDeltaReplayHttpFetcherError::NotificationTargetMismatch {
|
||||
notify_uri: notify_uri.clone(),
|
||||
expected_session: expected_session.to_string(),
|
||||
expected_serial,
|
||||
actual_session,
|
||||
actual_serial: notification.serial,
|
||||
},
|
||||
);
|
||||
}
|
||||
let transition_serials = repo
|
||||
.delta_paths
|
||||
.iter()
|
||||
.map(|(serial, _)| *serial)
|
||||
.collect::<Vec<_>>();
|
||||
let mut notification_delta_map = BTreeMap::new();
|
||||
for dref in notification.deltas {
|
||||
notification_delta_map.insert(dref.serial, dref.uri);
|
||||
}
|
||||
for serial in &transition_serials {
|
||||
if !notification_delta_map.contains_key(serial) {
|
||||
return Err(PayloadDeltaReplayHttpFetcherError::DeltaSerialMismatch {
|
||||
notify_uri: notify_uri.clone(),
|
||||
});
|
||||
}
|
||||
}
|
||||
for (serial, path) in &repo.delta_paths {
|
||||
let uri = notification_delta_map
|
||||
.get(serial)
|
||||
.expect("delta uri present for transition serial");
|
||||
insert_unique_route(&mut routes, uri, path)?;
|
||||
}
|
||||
}
|
||||
Ok(Self {
|
||||
index,
|
||||
routes,
|
||||
repo_kinds,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn from_index(
|
||||
index: Arc<ReplayDeltaArchiveIndex>,
|
||||
) -> Result<Self, PayloadDeltaReplayHttpFetcherError> {
|
||||
Self::new(index)
|
||||
}
|
||||
|
||||
pub fn archive_index(&self) -> &ReplayDeltaArchiveIndex {
|
||||
self.index.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
impl Fetcher for PayloadDeltaReplayHttpFetcher {
|
||||
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
|
||||
if let Some(path) = self.routes.get(uri) {
|
||||
return fs::read(path).map_err(|e| {
|
||||
PayloadDeltaReplayHttpFetcherError::ReadFile {
|
||||
path: path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
.to_string()
|
||||
});
|
||||
}
|
||||
if let Some(kind) = self.repo_kinds.get(uri) {
|
||||
return Err(
|
||||
PayloadDeltaReplayHttpFetcherError::NotificationKindNotFetchable {
|
||||
notify_uri: uri.to_string(),
|
||||
kind: kind.as_str().to_string(),
|
||||
}
|
||||
.to_string(),
|
||||
);
|
||||
}
|
||||
Err(PayloadDeltaReplayHttpFetcherError::MissingUri(uri.to_string()).to_string())
|
||||
}
|
||||
}
|
||||
|
||||
fn insert_unique_route(
|
||||
routes: &mut BTreeMap<String, PathBuf>,
|
||||
uri: &str,
|
||||
path: &Path,
|
||||
) -> Result<(), PayloadDeltaReplayHttpFetcherError> {
|
||||
if let Some(existing) = routes.get(uri) {
|
||||
if existing != path {
|
||||
return Err(PayloadDeltaReplayHttpFetcherError::DuplicateUriMapping {
|
||||
uri: uri.to_string(),
|
||||
first_path: existing.display().to_string(),
|
||||
second_path: path.display().to_string(),
|
||||
});
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
routes.insert(uri.to_string(), path.to_path_buf());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::replay::archive::sha256_hex;
|
||||
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
|
||||
|
||||
fn build_delta_http_fixture(
|
||||
kind: ReplayDeltaRrdpKind,
|
||||
) -> (tempfile::TempDir, PathBuf, PathBuf, String, String, String) {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let archive_root = temp.path().join("payload-delta-archive");
|
||||
let capture = "delta-http";
|
||||
let capture_root = archive_root.join("v1").join("captures").join(capture);
|
||||
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
|
||||
std::fs::write(
|
||||
capture_root.join("capture.json"),
|
||||
format!(r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-16T00:00:00Z","notes":""}}"#),
|
||||
)
|
||||
.expect("write capture meta");
|
||||
std::fs::write(
|
||||
capture_root.join("base.json"),
|
||||
r#"{"version":1,"baseCapture":"base-cap","baseLocksSha256":"deadbeef","createdAt":"2026-03-16T00:00:00Z"}"#,
|
||||
)
|
||||
.expect("write base meta");
|
||||
|
||||
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
|
||||
let snapshot_uri = "https://rrdp.example.test/snapshot.xml".to_string();
|
||||
let delta1_uri = "https://rrdp.example.test/d1.xml".to_string();
|
||||
let delta2_uri = "https://rrdp.example.test/d2.xml".to_string();
|
||||
let session = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string();
|
||||
let target_serial = 12u64;
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let session_dir = capture_root
|
||||
.join("rrdp/repos")
|
||||
.join(&repo_hash)
|
||||
.join(&session);
|
||||
let deltas_dir = session_dir.join("deltas");
|
||||
std::fs::create_dir_all(&deltas_dir).expect("mkdir deltas dir");
|
||||
std::fs::write(
|
||||
session_dir.parent().unwrap().join("meta.json"),
|
||||
format!(r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
|
||||
)
|
||||
.expect("write repo meta");
|
||||
|
||||
let notification_xml = format!(
|
||||
r#"<?xml version="1.0" encoding="UTF-8"?>
|
||||
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="{target_serial}">
|
||||
<snapshot uri="{snapshot_uri}" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
|
||||
<delta serial="11" uri="{delta1_uri}" hash="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" />
|
||||
<delta serial="12" uri="{delta2_uri}" hash="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" />
|
||||
</notification>"#
|
||||
);
|
||||
std::fs::write(
|
||||
session_dir.join("notification-target-12.xml"),
|
||||
notification_xml,
|
||||
)
|
||||
.expect("write target notification");
|
||||
std::fs::write(
|
||||
deltas_dir.join("delta-11-aaaa.xml"),
|
||||
b"<delta serial='11'/>",
|
||||
)
|
||||
.expect("write delta1");
|
||||
std::fs::write(
|
||||
deltas_dir.join("delta-12-bbbb.xml"),
|
||||
b"<delta serial='12'/>",
|
||||
)
|
||||
.expect("write delta2");
|
||||
std::fs::write(
|
||||
session_dir.parent().unwrap().join("transition.json"),
|
||||
format!(
|
||||
r#"{{"kind":"{}","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}"#,
|
||||
kind.as_str()
|
||||
),
|
||||
)
|
||||
.expect("write transition");
|
||||
let locks_path = temp.path().join("locks-delta.json");
|
||||
std::fs::write(
|
||||
&locks_path,
|
||||
format!(
|
||||
r#"{{"version":1,"capture":"{capture}","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{{"{notify_uri}":{{"kind":"{}","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}}},"rsync":{{}}}}"#,
|
||||
kind.as_str()
|
||||
),
|
||||
)
|
||||
.expect("write locks");
|
||||
(
|
||||
temp,
|
||||
archive_root,
|
||||
locks_path,
|
||||
notify_uri,
|
||||
delta1_uri,
|
||||
delta2_uri,
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_http_fetcher_rejects_session_reset_and_gap_notification_kind() {
|
||||
for kind in [ReplayDeltaRrdpKind::SessionReset, ReplayDeltaRrdpKind::Gap] {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _delta1_uri, _delta2_uri) =
|
||||
build_delta_http_fixture(kind);
|
||||
let index = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path)
|
||||
.expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayHttpFetcher::from_index(index).expect("build fetcher");
|
||||
let err = fetcher.fetch(¬ify_uri).unwrap_err();
|
||||
assert!(err.contains(kind.as_str()), "{err}");
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn delta_http_fetcher_reads_target_notification_and_delta_files() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, delta1_uri, delta2_uri) =
|
||||
build_delta_http_fixture(ReplayDeltaRrdpKind::Delta);
|
||||
let index = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayHttpFetcher::from_index(index).expect("build fetcher");
|
||||
let notification = fetcher.fetch(¬ify_uri).expect("fetch notification");
|
||||
assert!(
|
||||
std::str::from_utf8(¬ification)
|
||||
.unwrap()
|
||||
.contains("notification")
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher.fetch(&delta1_uri).expect("fetch delta1"),
|
||||
b"<delta serial='11'/>".to_vec()
|
||||
);
|
||||
assert_eq!(
|
||||
fetcher.fetch(&delta2_uri).expect("fetch delta2"),
|
||||
b"<delta serial='12'/>".to_vec()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_http_fetcher_rejects_non_delta_notification_kinds_and_missing_uri() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _delta1_uri, _delta2_uri) =
|
||||
build_delta_http_fixture(ReplayDeltaRrdpKind::Unchanged);
|
||||
let index = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayHttpFetcher::from_index(index).expect("build fetcher");
|
||||
let err = fetcher.fetch(¬ify_uri).unwrap_err();
|
||||
assert!(err.contains("unchanged"), "{err}");
|
||||
let err = fetcher
|
||||
.fetch("https://missing.example/test.xml")
|
||||
.unwrap_err();
|
||||
assert!(err.contains("not found in archive"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_http_fetcher_rejects_target_notification_mismatch() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _delta1_uri, _delta2_uri) =
|
||||
build_delta_http_fixture(ReplayDeltaRrdpKind::Delta);
|
||||
let repo_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let notification = archive_root
|
||||
.join("v1/captures/delta-http/rrdp/repos")
|
||||
.join(repo_hash)
|
||||
.join("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
|
||||
.join("notification-target-12.xml");
|
||||
std::fs::write(
|
||||
¬ification,
|
||||
r#"<?xml version="1.0" encoding="UTF-8"?>
|
||||
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" serial="12">
|
||||
<snapshot uri="https://rrdp.example.test/snapshot.xml" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
|
||||
<delta serial="11" uri="https://rrdp.example.test/d1.xml" hash="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" />
|
||||
<delta serial="12" uri="https://rrdp.example.test/d2.xml" hash="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" />
|
||||
</notification>"#,
|
||||
)
|
||||
.expect("rewrite notification");
|
||||
let index = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index"),
|
||||
);
|
||||
let err = PayloadDeltaReplayHttpFetcher::from_index(index).unwrap_err();
|
||||
assert!(
|
||||
matches!(
|
||||
err,
|
||||
PayloadDeltaReplayHttpFetcherError::NotificationTargetMismatch { .. }
|
||||
),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
}
|
||||
471
crates/panda-rpki-validator/src/replay/delta_fetch_rsync.rs
Normal file
471
crates/panda-rpki-validator/src/replay/delta_fetch_rsync.rs
Normal file
@ -0,0 +1,471 @@
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::fetch::rsync::{RsyncFetchError, RsyncFetchResult, RsyncFetcher};
|
||||
use crate::replay::archive::{ReplayArchiveIndex, canonical_rsync_module};
|
||||
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
|
||||
use crate::storage::{RepositoryViewState, RocksStore};
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PayloadDeltaReplayRsyncFetcher {
|
||||
base_index: Arc<ReplayArchiveIndex>,
|
||||
delta_index: Arc<ReplayDeltaArchiveIndex>,
|
||||
}
|
||||
|
||||
impl PayloadDeltaReplayRsyncFetcher {
|
||||
pub fn new(
|
||||
base_index: Arc<ReplayArchiveIndex>,
|
||||
delta_index: Arc<ReplayDeltaArchiveIndex>,
|
||||
) -> Self {
|
||||
Self {
|
||||
base_index,
|
||||
delta_index,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn base_index(&self) -> &ReplayArchiveIndex {
|
||||
self.base_index.as_ref()
|
||||
}
|
||||
|
||||
pub fn delta_index(&self) -> &ReplayDeltaArchiveIndex {
|
||||
self.delta_index.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
pub struct PayloadDeltaReplayCurrentStoreRsyncFetcher<'a> {
|
||||
store: &'a RocksStore,
|
||||
delta_index: Arc<ReplayDeltaArchiveIndex>,
|
||||
}
|
||||
|
||||
impl<'a> PayloadDeltaReplayCurrentStoreRsyncFetcher<'a> {
|
||||
pub fn new(store: &'a RocksStore, delta_index: Arc<ReplayDeltaArchiveIndex>) -> Self {
|
||||
Self { store, delta_index }
|
||||
}
|
||||
|
||||
pub fn delta_index(&self) -> &ReplayDeltaArchiveIndex {
|
||||
self.delta_index.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncFetcher for PayloadDeltaReplayRsyncFetcher {
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
|
||||
let module_uri = canonical_rsync_module(rsync_base_uri)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
let normalized_base = if rsync_base_uri.ends_with('/') {
|
||||
rsync_base_uri.to_string()
|
||||
} else {
|
||||
format!("{rsync_base_uri}/")
|
||||
};
|
||||
|
||||
let mut merged: BTreeMap<String, Vec<u8>> = BTreeMap::new();
|
||||
let mut saw_base = false;
|
||||
let overlay_only = self
|
||||
.delta_index
|
||||
.rsync_module(&module_uri)
|
||||
.map(|module| module.overlay_only)
|
||||
.unwrap_or(false);
|
||||
if !overlay_only {
|
||||
if let Ok(base_module) = self
|
||||
.base_index
|
||||
.resolve_rsync_module_for_base_uri(rsync_base_uri)
|
||||
{
|
||||
let base_tree_root = module_tree_root(&module_uri, &base_module.tree_dir)
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
if base_tree_root.is_dir() {
|
||||
let mut base_objects = Vec::new();
|
||||
walk_dir_collect(
|
||||
&base_tree_root,
|
||||
&base_tree_root,
|
||||
&module_uri,
|
||||
&mut base_objects,
|
||||
)
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
for (uri, bytes) in base_objects {
|
||||
merged.insert(uri, bytes);
|
||||
}
|
||||
saw_base = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut saw_overlay = false;
|
||||
if let Some(delta_module) = self.delta_index.rsync_module(&module_uri) {
|
||||
for (uri, path) in &delta_module.overlay_files {
|
||||
let bytes = fs::read(path).map_err(|e| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"read delta rsync overlay failed: {}: {e}",
|
||||
path.display()
|
||||
))
|
||||
})?;
|
||||
merged.insert(uri.clone(), bytes);
|
||||
saw_overlay = true;
|
||||
}
|
||||
}
|
||||
|
||||
if !saw_base && !saw_overlay {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"delta replay base rsync module not found and no delta overlay exists for module: {module_uri}"
|
||||
)));
|
||||
}
|
||||
|
||||
let filtered: Vec<(String, Vec<u8>)> = merged
|
||||
.into_iter()
|
||||
.filter(|(uri, _)| uri.starts_with(&normalized_base))
|
||||
.collect();
|
||||
if filtered.is_empty() {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"delta replay rsync subtree not found: {normalized_base}"
|
||||
)));
|
||||
}
|
||||
Ok(filtered)
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncFetcher for PayloadDeltaReplayCurrentStoreRsyncFetcher<'_> {
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
|
||||
let module_uri = canonical_rsync_module(rsync_base_uri)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
let normalized_base = if rsync_base_uri.ends_with('/') {
|
||||
rsync_base_uri.to_string()
|
||||
} else {
|
||||
format!("{rsync_base_uri}/")
|
||||
};
|
||||
|
||||
let mut merged: BTreeMap<String, Vec<u8>> = BTreeMap::new();
|
||||
let mut saw_base = false;
|
||||
let overlay_only = self
|
||||
.delta_index
|
||||
.rsync_module(&module_uri)
|
||||
.map(|module| module.overlay_only)
|
||||
.unwrap_or(false);
|
||||
|
||||
if !overlay_only {
|
||||
let entries = self
|
||||
.store
|
||||
.list_repository_view_entries_with_prefix(&module_uri)
|
||||
.map_err(|e| RsyncFetchError::Fetch(format!("list repository view failed: {e}")))?;
|
||||
for entry in entries {
|
||||
if entry.state != RepositoryViewState::Present {
|
||||
continue;
|
||||
}
|
||||
let bytes = self
|
||||
.store
|
||||
.load_current_object_bytes_by_uri(&entry.rsync_uri)
|
||||
.map_err(|e| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"load current object failed for {}: {e}",
|
||||
entry.rsync_uri
|
||||
))
|
||||
})?
|
||||
.ok_or_else(|| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"current object missing for {}",
|
||||
entry.rsync_uri
|
||||
))
|
||||
})?;
|
||||
merged.insert(entry.rsync_uri, bytes);
|
||||
saw_base = true;
|
||||
}
|
||||
}
|
||||
|
||||
let mut saw_overlay = false;
|
||||
if let Some(delta_module) = self.delta_index.rsync_module(&module_uri) {
|
||||
for (uri, path) in &delta_module.overlay_files {
|
||||
let bytes = fs::read(path).map_err(|e| {
|
||||
RsyncFetchError::Fetch(format!(
|
||||
"read delta rsync overlay failed: {}: {e}",
|
||||
path.display()
|
||||
))
|
||||
})?;
|
||||
merged.insert(uri.clone(), bytes);
|
||||
saw_overlay = true;
|
||||
}
|
||||
}
|
||||
|
||||
if !saw_base && !saw_overlay {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"delta replay current-store base not found and no delta overlay exists for module: {module_uri}"
|
||||
)));
|
||||
}
|
||||
|
||||
let filtered: Vec<(String, Vec<u8>)> = merged
|
||||
.into_iter()
|
||||
.filter(|(uri, _)| uri.starts_with(&normalized_base))
|
||||
.collect();
|
||||
if filtered.is_empty() {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"delta replay rsync subtree not found: {normalized_base}"
|
||||
)));
|
||||
}
|
||||
Ok(filtered)
|
||||
}
|
||||
}
|
||||
|
||||
fn module_tree_root(module_uri: &str, tree_dir: &Path) -> Result<PathBuf, String> {
|
||||
let rest = module_uri
|
||||
.strip_prefix("rsync://")
|
||||
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
|
||||
let mut parts = rest.trim_end_matches('/').split('/');
|
||||
let authority = parts
|
||||
.next()
|
||||
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
|
||||
let module = parts
|
||||
.next()
|
||||
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
|
||||
Ok(tree_dir.join(authority).join(module))
|
||||
}
|
||||
|
||||
fn walk_dir_collect(
|
||||
root: &Path,
|
||||
current: &Path,
|
||||
rsync_base_uri: &str,
|
||||
out: &mut Vec<(String, Vec<u8>)>,
|
||||
) -> Result<(), String> {
|
||||
let rd = fs::read_dir(current).map_err(|e| e.to_string())?;
|
||||
for entry in rd {
|
||||
let entry = entry.map_err(|e| e.to_string())?;
|
||||
let path = entry.path();
|
||||
let meta = entry.metadata().map_err(|e| e.to_string())?;
|
||||
if meta.is_dir() {
|
||||
walk_dir_collect(root, &path, rsync_base_uri, out)?;
|
||||
continue;
|
||||
}
|
||||
if !meta.is_file() {
|
||||
continue;
|
||||
}
|
||||
let rel = path
|
||||
.strip_prefix(root)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
let base = if rsync_base_uri.ends_with('/') {
|
||||
rsync_base_uri.to_string()
|
||||
} else {
|
||||
format!("{rsync_base_uri}/")
|
||||
};
|
||||
let uri = format!("{base}{rel}");
|
||||
let bytes = fs::read(&path).map_err(|e| e.to_string())?;
|
||||
out.push((uri, bytes));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::replay::archive::ReplayArchiveIndex;
|
||||
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
|
||||
use crate::storage::{RawByHashEntry, RepositoryViewEntry, RepositoryViewState, RocksStore};
|
||||
|
||||
fn build_base_and_delta_rsync_fixture()
|
||||
-> (tempfile::TempDir, PathBuf, PathBuf, PathBuf, PathBuf) {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let base_archive = temp.path().join("payload-archive");
|
||||
let base_capture_root = base_archive.join("v1/captures/base-cap");
|
||||
std::fs::create_dir_all(&base_capture_root).expect("mkdir base capture");
|
||||
std::fs::write(
|
||||
base_capture_root.join("capture.json"),
|
||||
r#"{"version":1,"captureId":"base-cap","createdAt":"2026-03-16T00:00:00Z","notes":""}"#,
|
||||
)
|
||||
.expect("write base capture meta");
|
||||
let module_uri = "rsync://rsync.example.test/repo/";
|
||||
let module_hash = crate::replay::archive::sha256_hex(module_uri.as_bytes());
|
||||
let base_bucket = base_capture_root.join("rsync/modules").join(&module_hash);
|
||||
let base_tree = base_bucket.join("tree/rsync.example.test/repo");
|
||||
std::fs::create_dir_all(base_tree.join("sub")).expect("mkdir base tree");
|
||||
std::fs::write(base_bucket.join("meta.json"), format!(r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#)).expect("write base meta");
|
||||
std::fs::write(base_tree.join("a.roa"), b"base-a").expect("write base a");
|
||||
std::fs::write(base_tree.join("sub").join("b.cer"), b"base-b").expect("write base b");
|
||||
let base_locks = temp.path().join("base-locks.json");
|
||||
std::fs::write(&base_locks, format!(r#"{{"version":1,"capture":"base-cap","rrdp":{{}},"rsync":{{"{module_uri}":{{"transport":"rsync"}}}}}}"#)).expect("write base locks");
|
||||
|
||||
let delta_archive = temp.path().join("payload-delta-archive");
|
||||
let delta_capture_root = delta_archive.join("v1/captures/delta-cap");
|
||||
std::fs::create_dir_all(&delta_capture_root).expect("mkdir delta capture");
|
||||
std::fs::write(delta_capture_root.join("capture.json"), r#"{"version":1,"captureId":"delta-cap","createdAt":"2026-03-16T00:00:00Z","notes":""}"#).expect("write delta capture meta");
|
||||
std::fs::write(delta_capture_root.join("base.json"), r#"{"version":1,"baseCapture":"base-cap","baseLocksSha256":"deadbeef","createdAt":"2026-03-16T00:00:00Z"}"#).expect("write delta base meta");
|
||||
let delta_bucket = delta_capture_root.join("rsync/modules").join(&module_hash);
|
||||
let delta_tree = delta_bucket.join("tree/rsync.example.test/repo");
|
||||
std::fs::create_dir_all(delta_tree.join("sub")).expect("mkdir delta tree");
|
||||
std::fs::write(delta_bucket.join("meta.json"), format!(r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#)).expect("write delta meta");
|
||||
std::fs::write(delta_bucket.join("files.json"), format!(r#"{{"version":1,"module":"{module_uri}","fileCount":1,"files":["{module_uri}sub/b.cer"]}}"#)).expect("write files json");
|
||||
std::fs::write(delta_tree.join("sub").join("b.cer"), b"delta-b")
|
||||
.expect("write delta overlay");
|
||||
let delta_locks = temp.path().join("locks-delta.json");
|
||||
std::fs::write(&delta_locks, format!(r#"{{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{{}},"rsync":{{"{module_uri}":{{"file_count":1,"overlay_only":false}}}}}}"#)).expect("write delta locks");
|
||||
(temp, base_archive, base_locks, delta_archive, delta_locks)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_rsync_current_store_fetcher_merges_current_store_and_overlay() {
|
||||
let (_temp, _base_archive, _base_locks, delta_archive, delta_locks) =
|
||||
build_base_and_delta_rsync_fixture();
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let store = RocksStore::open(temp.path()).expect("open rocksdb");
|
||||
store
|
||||
.put_raw_by_hash_entry(&RawByHashEntry {
|
||||
sha256_hex: crate::replay::archive::sha256_hex(b"base-a"),
|
||||
bytes: b"base-a".to_vec(),
|
||||
origin_uris: vec!["rsync://rsync.example.test/repo/a.roa".to_string()],
|
||||
object_type: Some("roa".to_string()),
|
||||
encoding: None,
|
||||
})
|
||||
.expect("put raw by hash");
|
||||
store
|
||||
.put_repository_view_entry(&RepositoryViewEntry {
|
||||
rsync_uri: "rsync://rsync.example.test/repo/a.roa".to_string(),
|
||||
current_hash: Some(crate::replay::archive::sha256_hex(b"base-a")),
|
||||
repository_source: Some("rsync".to_string()),
|
||||
object_type: Some("roa".to_string()),
|
||||
state: RepositoryViewState::Present,
|
||||
})
|
||||
.expect("put repository view");
|
||||
|
||||
let delta = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayCurrentStoreRsyncFetcher::new(&store, delta);
|
||||
let mut objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/")
|
||||
.expect("fetch current-store objects");
|
||||
objects.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
assert_eq!(objects.len(), 2);
|
||||
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/a.roa");
|
||||
assert_eq!(objects[0].1, b"base-a");
|
||||
assert_eq!(objects[1].0, "rsync://rsync.example.test/repo/sub/b.cer");
|
||||
assert_eq!(objects[1].1, b"delta-b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_rsync_fetcher_uses_base_only_when_delta_has_no_module_entry() {
|
||||
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
|
||||
build_base_and_delta_rsync_fixture();
|
||||
std::fs::write(
|
||||
&delta_locks,
|
||||
r#"{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{},"rsync":{}}"#,
|
||||
)
|
||||
.expect("rewrite delta locks no rsync");
|
||||
let base = Arc::new(
|
||||
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
|
||||
);
|
||||
let delta = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
|
||||
let mut objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/")
|
||||
.expect("fetch base only objects");
|
||||
objects.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
assert_eq!(objects.len(), 2);
|
||||
assert_eq!(objects[0].1, b"base-a");
|
||||
assert_eq!(objects[1].1, b"base-b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_rsync_fetcher_reports_missing_base_subtree_and_exposes_indexes() {
|
||||
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
|
||||
build_base_and_delta_rsync_fixture();
|
||||
let base = Arc::new(
|
||||
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
|
||||
);
|
||||
let delta = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base.clone(), delta.clone());
|
||||
assert_eq!(fetcher.base_index().rsync_modules.len(), 1);
|
||||
assert_eq!(fetcher.delta_index().rsync_modules.len(), 1);
|
||||
let err = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/missing/")
|
||||
.unwrap_err();
|
||||
match err {
|
||||
RsyncFetchError::Fetch(msg) => {
|
||||
assert!(
|
||||
msg.contains("delta replay rsync subtree not found"),
|
||||
"{msg}"
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_rsync_fetcher_can_use_overlay_without_base_module() {
|
||||
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
|
||||
build_base_and_delta_rsync_fixture();
|
||||
let module_hash =
|
||||
crate::replay::archive::sha256_hex("rsync://rsync.example.test/repo/".as_bytes());
|
||||
std::fs::remove_dir_all(
|
||||
base_archive
|
||||
.join("v1/captures/base-cap/rsync/modules")
|
||||
.join(&module_hash),
|
||||
)
|
||||
.expect("remove base module dir");
|
||||
let base = Arc::new(
|
||||
ReplayArchiveIndex::load_allow_missing_rsync_modules(&base_archive, &base_locks)
|
||||
.expect("load lenient base index"),
|
||||
);
|
||||
let delta = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
|
||||
let objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/sub/")
|
||||
.expect("fetch subtree from overlay only");
|
||||
assert_eq!(objects.len(), 1);
|
||||
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/sub/b.cer");
|
||||
assert_eq!(objects[0].1, b"delta-b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_rsync_fetcher_merges_base_and_overlay() {
|
||||
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
|
||||
build_base_and_delta_rsync_fixture();
|
||||
let base = Arc::new(
|
||||
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
|
||||
);
|
||||
let delta = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
|
||||
let mut objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/")
|
||||
.expect("fetch merged objects");
|
||||
objects.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
assert_eq!(objects.len(), 2);
|
||||
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/a.roa");
|
||||
assert_eq!(objects[0].1, b"base-a");
|
||||
assert_eq!(objects[1].0, "rsync://rsync.example.test/repo/sub/b.cer");
|
||||
assert_eq!(objects[1].1, b"delta-b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delta_rsync_fetcher_reads_subtree_and_rejects_missing_base_module() {
|
||||
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
|
||||
build_base_and_delta_rsync_fixture();
|
||||
let base = Arc::new(
|
||||
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
|
||||
);
|
||||
let delta = Arc::new(
|
||||
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
|
||||
);
|
||||
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
|
||||
let objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/sub/")
|
||||
.expect("fetch subtree");
|
||||
assert_eq!(objects.len(), 1);
|
||||
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/sub/b.cer");
|
||||
assert_eq!(objects[0].1, b"delta-b");
|
||||
|
||||
let err = fetcher
|
||||
.fetch_objects("rsync://missing.example/repo/")
|
||||
.unwrap_err();
|
||||
match err {
|
||||
RsyncFetchError::Fetch(msg) => assert!(
|
||||
msg.contains("delta replay base rsync module not found")
|
||||
|| msg.contains("no replay lock found for rsync module"),
|
||||
"{msg}"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
399
crates/panda-rpki-validator/src/replay/fetch_http.rs
Normal file
399
crates/panda-rpki-validator/src/replay/fetch_http.rs
Normal file
@ -0,0 +1,399 @@
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::replay::archive::{ReplayArchiveError, ReplayArchiveIndex, ReplayTransport};
|
||||
use crate::sync::rrdp::{Fetcher, parse_notification};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ReplayHttpFetcherError {
|
||||
#[error(transparent)]
|
||||
Archive(#[from] ReplayArchiveError),
|
||||
|
||||
#[error("read replay RRDP file failed: {path}: {detail}")]
|
||||
ReadFile { path: String, detail: String },
|
||||
|
||||
#[error("parse locked notification failed for {notify_uri}: {detail}")]
|
||||
ParseNotification { notify_uri: String, detail: String },
|
||||
|
||||
#[error(
|
||||
"locked notification session/serial mismatch for {notify_uri}: expected session={expected_session} serial={expected_serial}, got session={actual_session} serial={actual_serial}"
|
||||
)]
|
||||
NotificationLockMismatch {
|
||||
notify_uri: String,
|
||||
expected_session: String,
|
||||
expected_serial: u64,
|
||||
actual_session: String,
|
||||
actual_serial: u64,
|
||||
},
|
||||
|
||||
#[error("locked delta file not found for {notify_uri} at serial {serial}: {path}")]
|
||||
MissingLockedDelta {
|
||||
notify_uri: String,
|
||||
serial: u64,
|
||||
path: String,
|
||||
},
|
||||
|
||||
#[error("duplicate replay HTTP URI mapping for {uri}: {first_path} vs {second_path}")]
|
||||
DuplicateUriMapping {
|
||||
uri: String,
|
||||
first_path: String,
|
||||
second_path: String,
|
||||
},
|
||||
|
||||
#[error("RRDP notification URI is locked to rsync transport in replay: {0}")]
|
||||
LockedToRsync(String),
|
||||
|
||||
#[error("replay HTTP URI not found in archive: {0}")]
|
||||
MissingUri(String),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PayloadReplayHttpFetcher {
|
||||
index: Arc<ReplayArchiveIndex>,
|
||||
routes: BTreeMap<String, PathBuf>,
|
||||
}
|
||||
|
||||
impl PayloadReplayHttpFetcher {
|
||||
pub fn new(index: Arc<ReplayArchiveIndex>) -> Result<Self, ReplayHttpFetcherError> {
|
||||
let mut routes = BTreeMap::new();
|
||||
for (notify_uri, lock) in &index.locks.rrdp {
|
||||
if lock.transport != ReplayTransport::Rrdp {
|
||||
continue;
|
||||
}
|
||||
let repo = index.require_rrdp_repo(notify_uri)?;
|
||||
insert_unique_route(&mut routes, notify_uri, &repo.locked_notification_path)?;
|
||||
|
||||
let notification_xml = fs::read(&repo.locked_notification_path).map_err(|e| {
|
||||
ReplayHttpFetcherError::ReadFile {
|
||||
path: repo.locked_notification_path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
let notification = parse_notification(¬ification_xml).map_err(|e| {
|
||||
ReplayHttpFetcherError::ParseNotification {
|
||||
notify_uri: notify_uri.clone(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
let actual_session = notification.session_id.to_string();
|
||||
if actual_session != repo.locked_session || notification.serial != repo.locked_serial {
|
||||
return Err(ReplayHttpFetcherError::NotificationLockMismatch {
|
||||
notify_uri: notify_uri.clone(),
|
||||
expected_session: repo.locked_session.clone(),
|
||||
expected_serial: repo.locked_serial,
|
||||
actual_session,
|
||||
actual_serial: notification.serial,
|
||||
});
|
||||
}
|
||||
|
||||
insert_unique_route(
|
||||
&mut routes,
|
||||
¬ification.snapshot_uri,
|
||||
&repo.locked_snapshot_path,
|
||||
)?;
|
||||
|
||||
for delta in notification.deltas {
|
||||
let expected = repo.session_dir.join(format!(
|
||||
"delta-{}-{}.xml",
|
||||
delta.serial,
|
||||
hex::encode(delta.hash_sha256)
|
||||
));
|
||||
if expected.is_file() {
|
||||
insert_unique_route(&mut routes, &delta.uri, &expected)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Self { index, routes })
|
||||
}
|
||||
|
||||
pub fn from_paths(
|
||||
archive_root: impl AsRef<Path>,
|
||||
locks_path: impl AsRef<Path>,
|
||||
) -> Result<Self, ReplayHttpFetcherError> {
|
||||
let index = Arc::new(ReplayArchiveIndex::load(archive_root, locks_path)?);
|
||||
Self::new(index)
|
||||
}
|
||||
|
||||
pub fn archive_index(&self) -> &ReplayArchiveIndex {
|
||||
self.index.as_ref()
|
||||
}
|
||||
|
||||
pub fn mapped_route(&self, uri: &str) -> Option<&Path> {
|
||||
self.routes.get(uri).map(PathBuf::as_path)
|
||||
}
|
||||
}
|
||||
|
||||
impl Fetcher for PayloadReplayHttpFetcher {
|
||||
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
|
||||
if let Some(path) = self.routes.get(uri) {
|
||||
return fs::read(path).map_err(|e| {
|
||||
ReplayHttpFetcherError::ReadFile {
|
||||
path: path.display().to_string(),
|
||||
detail: e.to_string(),
|
||||
}
|
||||
.to_string()
|
||||
});
|
||||
}
|
||||
|
||||
if self
|
||||
.index
|
||||
.rrdp_lock(uri)
|
||||
.is_some_and(|lock| lock.transport == ReplayTransport::Rsync)
|
||||
{
|
||||
return Err(ReplayHttpFetcherError::LockedToRsync(uri.to_string()).to_string());
|
||||
}
|
||||
|
||||
Err(ReplayHttpFetcherError::MissingUri(uri.to_string()).to_string())
|
||||
}
|
||||
}
|
||||
|
||||
fn insert_unique_route(
|
||||
routes: &mut BTreeMap<String, PathBuf>,
|
||||
uri: &str,
|
||||
path: &Path,
|
||||
) -> Result<(), ReplayHttpFetcherError> {
|
||||
if let Some(existing) = routes.get(uri) {
|
||||
if existing != path {
|
||||
return Err(ReplayHttpFetcherError::DuplicateUriMapping {
|
||||
uri: uri.to_string(),
|
||||
first_path: existing.display().to_string(),
|
||||
second_path: path.display().to_string(),
|
||||
});
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
routes.insert(uri.to_string(), path.to_path_buf());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::replay::archive::sha256_hex;
|
||||
|
||||
fn build_archive_with_rrdp_and_rsync(
|
||||
with_delta: bool,
|
||||
) -> (
|
||||
tempfile::TempDir,
|
||||
PathBuf,
|
||||
PathBuf,
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
Option<String>,
|
||||
) {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let archive_root = temp.path().join("payload-archive");
|
||||
let capture = "capture-http";
|
||||
let capture_root = archive_root.join("v1").join("captures").join(capture);
|
||||
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
|
||||
std::fs::write(
|
||||
capture_root.join("capture.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
|
||||
),
|
||||
)
|
||||
.expect("write capture meta");
|
||||
|
||||
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
|
||||
let snapshot_uri = "https://rrdp.example.test/snapshot.xml".to_string();
|
||||
let delta_uri = "https://rrdp.example.test/delta.xml".to_string();
|
||||
let rsync_locked_notify = "https://rrdp-fallback.example.test/notification.xml".to_string();
|
||||
let session = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string();
|
||||
let serial = 99u64;
|
||||
let bucket_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let session_dir = capture_root
|
||||
.join("rrdp/repos")
|
||||
.join(bucket_hash)
|
||||
.join(&session);
|
||||
std::fs::create_dir_all(&session_dir).expect("mkdir session dir");
|
||||
std::fs::write(
|
||||
session_dir.parent().unwrap().join("meta.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
|
||||
),
|
||||
)
|
||||
.expect("write repo meta");
|
||||
|
||||
let snapshot_hash = "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff";
|
||||
let delta_hash = "ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100";
|
||||
let notification_xml = if with_delta {
|
||||
format!(
|
||||
r#"<?xml version="1.0" encoding="UTF-8"?>
|
||||
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="{serial}">
|
||||
<snapshot uri="{snapshot_uri}" hash="{snapshot_hash}" />
|
||||
<delta serial="99" uri="{delta_uri}" hash="{delta_hash}" />
|
||||
</notification>"#
|
||||
)
|
||||
} else {
|
||||
format!(
|
||||
r#"<?xml version="1.0" encoding="UTF-8"?>
|
||||
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="{serial}">
|
||||
<snapshot uri="{snapshot_uri}" hash="{snapshot_hash}" />
|
||||
</notification>"#
|
||||
)
|
||||
};
|
||||
std::fs::write(session_dir.join("notification-99.xml"), notification_xml)
|
||||
.expect("write notification");
|
||||
std::fs::write(
|
||||
session_dir.join(format!("snapshot-99-{snapshot_hash}.xml")),
|
||||
b"<snapshot/>",
|
||||
)
|
||||
.expect("write snapshot");
|
||||
if with_delta {
|
||||
std::fs::write(
|
||||
session_dir.join(format!("delta-99-{delta_hash}.xml")),
|
||||
b"<delta/>",
|
||||
)
|
||||
.expect("write delta");
|
||||
}
|
||||
|
||||
let module_uri = "rsync://rsync.example.test/repo/".to_string();
|
||||
let module_hash = sha256_hex(module_uri.as_bytes());
|
||||
let tree_root = capture_root
|
||||
.join("rsync/modules")
|
||||
.join(&module_hash)
|
||||
.join("tree")
|
||||
.join("rsync.example.test")
|
||||
.join("repo");
|
||||
std::fs::create_dir_all(&tree_root).expect("mkdir tree root");
|
||||
let module_bucket_dir = capture_root.join("rsync/modules").join(&module_hash);
|
||||
std::fs::write(
|
||||
module_bucket_dir.join("meta.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
|
||||
),
|
||||
)
|
||||
.expect("write rsync meta");
|
||||
std::fs::write(tree_root.join("x.cer"), b"cer").expect("write tree file");
|
||||
|
||||
let locks_path = temp.path().join("locks.json");
|
||||
std::fs::write(
|
||||
&locks_path,
|
||||
format!(
|
||||
r#"{{
|
||||
"version":1,
|
||||
"capture":"{capture}",
|
||||
"rrdp":{{
|
||||
"{notify_uri}":{{"transport":"rrdp","session":"{session}","serial":{serial}}},
|
||||
"{rsync_locked_notify}":{{"transport":"rsync","session":null,"serial":null}}
|
||||
}},
|
||||
"rsync":{{
|
||||
"{module_uri}":{{"transport":"rsync"}}
|
||||
}}
|
||||
}}"#
|
||||
),
|
||||
)
|
||||
.expect("write locks");
|
||||
|
||||
(
|
||||
temp,
|
||||
archive_root,
|
||||
locks_path,
|
||||
notify_uri,
|
||||
snapshot_uri,
|
||||
rsync_locked_notify,
|
||||
with_delta.then_some(delta_uri),
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_http_fetcher_reads_locked_notification_and_snapshot_and_delta() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, snapshot_uri, _rsync_locked, delta_uri) =
|
||||
build_archive_with_rrdp_and_rsync(true);
|
||||
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay http fetcher");
|
||||
|
||||
let notification = fetcher.fetch(¬ify_uri).expect("fetch notification");
|
||||
assert!(
|
||||
std::str::from_utf8(¬ification)
|
||||
.expect("utf8")
|
||||
.contains("<notification")
|
||||
);
|
||||
|
||||
let snapshot = fetcher.fetch(&snapshot_uri).expect("fetch snapshot");
|
||||
assert_eq!(snapshot, b"<snapshot/>");
|
||||
|
||||
let delta = fetcher
|
||||
.fetch(delta_uri.as_deref().expect("delta uri"))
|
||||
.expect("fetch delta");
|
||||
assert_eq!(delta, b"<delta/>");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_http_fetcher_rejects_notification_locked_to_rsync() {
|
||||
let (_temp, archive_root, locks_path, _notify_uri, _snapshot_uri, rsync_locked, _delta_uri) =
|
||||
build_archive_with_rrdp_and_rsync(false);
|
||||
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay http fetcher");
|
||||
let err = fetcher.fetch(&rsync_locked).unwrap_err();
|
||||
assert!(err.contains("locked to rsync transport"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_http_fetcher_rejects_unknown_uri() {
|
||||
let (
|
||||
_temp,
|
||||
archive_root,
|
||||
locks_path,
|
||||
_notify_uri,
|
||||
_snapshot_uri,
|
||||
_rsync_locked,
|
||||
_delta_uri,
|
||||
) = build_archive_with_rrdp_and_rsync(false);
|
||||
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay http fetcher");
|
||||
let err = fetcher
|
||||
.fetch("https://unknown.example/test.xml")
|
||||
.unwrap_err();
|
||||
assert!(err.contains("not found in archive"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_http_fetcher_rejects_notification_lock_mismatch() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _snapshot_uri, _rsync_locked, _delta_uri) =
|
||||
build_archive_with_rrdp_and_rsync(false);
|
||||
let bucket_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let notification = archive_root
|
||||
.join("v1/captures/capture-http/rrdp/repos")
|
||||
.join(bucket_hash)
|
||||
.join("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
|
||||
.join("notification-99.xml");
|
||||
std::fs::write(
|
||||
¬ification,
|
||||
r#"<?xml version="1.0" encoding="UTF-8"?>
|
||||
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" serial="99">
|
||||
<snapshot uri="https://rrdp.example.test/snapshot.xml" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
|
||||
</notification>"#,
|
||||
)
|
||||
.expect("rewrite notification");
|
||||
|
||||
let err = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path).unwrap_err();
|
||||
assert!(
|
||||
matches!(err, ReplayHttpFetcherError::NotificationLockMismatch { .. }),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_http_fetcher_skips_missing_locked_delta_route() {
|
||||
let (_temp, archive_root, locks_path, notify_uri, _snapshot_uri, _rsync_locked, _delta_uri) =
|
||||
build_archive_with_rrdp_and_rsync(true);
|
||||
let bucket_hash = sha256_hex(notify_uri.as_bytes());
|
||||
let delta = archive_root
|
||||
.join("v1/captures/capture-http/rrdp/repos")
|
||||
.join(bucket_hash)
|
||||
.join("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
|
||||
.join("delta-99-ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100.xml");
|
||||
std::fs::remove_file(delta).expect("remove delta");
|
||||
|
||||
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay http fetcher without delta route");
|
||||
let err = fetcher
|
||||
.fetch("https://rrdp.example.test/delta.xml")
|
||||
.unwrap_err();
|
||||
assert!(err.contains("not found in archive"), "{err}");
|
||||
}
|
||||
}
|
||||
253
crates/panda-rpki-validator/src/replay/fetch_rsync.rs
Normal file
253
crates/panda-rpki-validator/src/replay/fetch_rsync.rs
Normal file
@ -0,0 +1,253 @@
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::fetch::rsync::{RsyncFetchError, RsyncFetchResult, RsyncFetcher};
|
||||
use crate::replay::archive::{ReplayArchiveIndex, canonical_rsync_module};
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PayloadReplayRsyncFetcher {
|
||||
index: Arc<ReplayArchiveIndex>,
|
||||
}
|
||||
|
||||
impl PayloadReplayRsyncFetcher {
|
||||
pub fn new(index: Arc<ReplayArchiveIndex>) -> Self {
|
||||
Self { index }
|
||||
}
|
||||
|
||||
pub fn from_paths(
|
||||
archive_root: impl AsRef<Path>,
|
||||
locks_path: impl AsRef<Path>,
|
||||
) -> Result<Self, String> {
|
||||
let index = Arc::new(
|
||||
ReplayArchiveIndex::load(archive_root, locks_path).map_err(|e| e.to_string())?,
|
||||
);
|
||||
Ok(Self::new(index))
|
||||
}
|
||||
|
||||
pub fn archive_index(&self) -> &ReplayArchiveIndex {
|
||||
self.index.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
impl RsyncFetcher for PayloadReplayRsyncFetcher {
|
||||
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
|
||||
let (module_uri, relative_path) = split_rsync_base_uri(rsync_base_uri)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
let module = self
|
||||
.index
|
||||
.resolve_rsync_module_for_base_uri(rsync_base_uri)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
let module_root = module_tree_root(&module_uri, &module.tree_dir)
|
||||
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
|
||||
let start = if relative_path.as_os_str().is_empty() {
|
||||
module_root
|
||||
} else {
|
||||
module_root.join(&relative_path)
|
||||
};
|
||||
if !start.is_dir() {
|
||||
return Err(RsyncFetchError::Fetch(format!(
|
||||
"replay rsync subtree not found: {}",
|
||||
start.display()
|
||||
)));
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
walk_dir_collect(&start, &start, rsync_base_uri, &mut out)
|
||||
.map_err(RsyncFetchError::Fetch)?;
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
|
||||
fn split_rsync_base_uri(rsync_base_uri: &str) -> Result<(String, PathBuf), String> {
|
||||
let module_uri = canonical_rsync_module(rsync_base_uri).map_err(|e| e.to_string())?;
|
||||
let rest = rsync_base_uri
|
||||
.strip_prefix(&module_uri)
|
||||
.unwrap_or_default()
|
||||
.trim_matches('/');
|
||||
let relative_path = if rest.is_empty() {
|
||||
PathBuf::new()
|
||||
} else {
|
||||
let mut path = PathBuf::new();
|
||||
for segment in rest.split('/') {
|
||||
if !segment.is_empty() {
|
||||
path.push(segment);
|
||||
}
|
||||
}
|
||||
path
|
||||
};
|
||||
Ok((module_uri, relative_path))
|
||||
}
|
||||
|
||||
fn module_tree_root(module_uri: &str, tree_dir: &Path) -> Result<PathBuf, String> {
|
||||
let rest = module_uri
|
||||
.strip_prefix("rsync://")
|
||||
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
|
||||
let mut parts = rest.trim_end_matches('/').split('/');
|
||||
let authority = parts
|
||||
.next()
|
||||
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
|
||||
let module = parts
|
||||
.next()
|
||||
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
|
||||
Ok(tree_dir.join(authority).join(module))
|
||||
}
|
||||
|
||||
fn walk_dir_collect(
|
||||
root: &Path,
|
||||
current: &Path,
|
||||
rsync_base_uri: &str,
|
||||
out: &mut Vec<(String, Vec<u8>)>,
|
||||
) -> Result<(), String> {
|
||||
let rd = fs::read_dir(current).map_err(|e| e.to_string())?;
|
||||
for entry in rd {
|
||||
let entry = entry.map_err(|e| e.to_string())?;
|
||||
let path = entry.path();
|
||||
let meta = entry.metadata().map_err(|e| e.to_string())?;
|
||||
if meta.is_dir() {
|
||||
walk_dir_collect(root, &path, rsync_base_uri, out)?;
|
||||
continue;
|
||||
}
|
||||
if !meta.is_file() {
|
||||
continue;
|
||||
}
|
||||
let rel = path
|
||||
.strip_prefix(root)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
let base = if rsync_base_uri.ends_with('/') {
|
||||
rsync_base_uri.to_string()
|
||||
} else {
|
||||
format!("{rsync_base_uri}/")
|
||||
};
|
||||
let uri = format!("{base}{rel}");
|
||||
let bytes = fs::read(&path).map_err(|e| e.to_string())?;
|
||||
out.push((uri, bytes));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn build_rsync_archive() -> (tempfile::TempDir, PathBuf, PathBuf) {
|
||||
let temp = tempfile::tempdir().expect("tempdir");
|
||||
let archive_root = temp.path().join("payload-archive");
|
||||
let capture = "capture-rsync";
|
||||
let capture_root = archive_root.join("v1").join("captures").join(capture);
|
||||
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
|
||||
std::fs::write(
|
||||
capture_root.join("capture.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
|
||||
),
|
||||
)
|
||||
.expect("write capture meta");
|
||||
|
||||
let module_uri = "rsync://rsync.example.test/repo/";
|
||||
let mod_hash = crate::replay::archive::sha256_hex(module_uri.as_bytes());
|
||||
let module_root = capture_root
|
||||
.join("rsync/modules")
|
||||
.join(&mod_hash)
|
||||
.join("tree")
|
||||
.join("rsync.example.test")
|
||||
.join("repo");
|
||||
std::fs::create_dir_all(module_root.join("child")).expect("mkdir module tree");
|
||||
let module_bucket_dir = capture_root.join("rsync/modules").join(&mod_hash);
|
||||
std::fs::write(
|
||||
module_bucket_dir.join("meta.json"),
|
||||
format!(
|
||||
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
|
||||
),
|
||||
)
|
||||
.expect("write module meta");
|
||||
std::fs::write(module_root.join("a.roa"), b"a").expect("write a.roa");
|
||||
std::fs::write(module_root.join("child").join("b.cer"), b"b").expect("write b.cer");
|
||||
|
||||
let locks_path = temp.path().join("locks.json");
|
||||
std::fs::write(
|
||||
&locks_path,
|
||||
format!(
|
||||
r#"{{"version":1,"capture":"{capture}","rrdp":{{}},"rsync":{{"{module_uri}":{{"transport":"rsync"}}}}}}"#
|
||||
),
|
||||
)
|
||||
.expect("write locks");
|
||||
|
||||
(temp, archive_root, locks_path)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_rsync_fetcher_reads_module_root() {
|
||||
let (_temp, archive_root, locks_path) = build_rsync_archive();
|
||||
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay rsync fetcher");
|
||||
let mut objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/")
|
||||
.expect("fetch root objects");
|
||||
objects.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
|
||||
assert_eq!(objects.len(), 2);
|
||||
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/a.roa");
|
||||
assert_eq!(objects[0].1, b"a");
|
||||
assert_eq!(objects[1].0, "rsync://rsync.example.test/repo/child/b.cer");
|
||||
assert_eq!(objects[1].1, b"b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_rsync_fetcher_reads_subtree_only() {
|
||||
let (_temp, archive_root, locks_path) = build_rsync_archive();
|
||||
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay rsync fetcher");
|
||||
let objects = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/child/")
|
||||
.expect("fetch child subtree");
|
||||
|
||||
assert_eq!(objects.len(), 1);
|
||||
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/child/b.cer");
|
||||
assert_eq!(objects[0].1, b"b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_rsync_fetcher_rejects_missing_module_and_subtree() {
|
||||
let (_temp, archive_root, locks_path) = build_rsync_archive();
|
||||
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay rsync fetcher");
|
||||
|
||||
let err = fetcher
|
||||
.fetch_objects("rsync://missing.example/repo/")
|
||||
.unwrap_err();
|
||||
match err {
|
||||
RsyncFetchError::Fetch(msg) => assert!(
|
||||
msg.contains("no replay lock found for rsync module"),
|
||||
"{msg}"
|
||||
),
|
||||
}
|
||||
|
||||
let err = fetcher
|
||||
.fetch_objects("rsync://rsync.example.test/repo/missing/")
|
||||
.unwrap_err();
|
||||
match err {
|
||||
RsyncFetchError::Fetch(msg) => {
|
||||
assert!(msg.contains("replay rsync subtree not found"), "{msg}")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_replay_rsync_fetcher_rejects_invalid_uri_and_exposes_index() {
|
||||
let (_temp, archive_root, locks_path) = build_rsync_archive();
|
||||
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
|
||||
.expect("build replay rsync fetcher");
|
||||
assert_eq!(fetcher.archive_index().rsync_modules.len(), 1);
|
||||
|
||||
let err = fetcher
|
||||
.fetch_objects("https://not-rsync.example/repo/")
|
||||
.unwrap_err();
|
||||
match err {
|
||||
RsyncFetchError::Fetch(msg) => {
|
||||
assert!(msg.contains("URI must start with rsync://"), "{msg}")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
6
crates/panda-rpki-validator/src/replay/mod.rs
Normal file
6
crates/panda-rpki-validator/src/replay/mod.rs
Normal file
@ -0,0 +1,6 @@
|
||||
pub mod archive;
|
||||
pub mod delta_archive;
|
||||
pub mod delta_fetch_http;
|
||||
pub mod delta_fetch_rsync;
|
||||
pub mod fetch_http;
|
||||
pub mod fetch_rsync;
|
||||
57
crates/panda-rpki-validator/src/report.rs
Normal file
57
crates/panda-rpki-validator/src/report.rs
Normal file
@ -0,0 +1,57 @@
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct RfcRef(pub &'static str);
|
||||
|
||||
/// Stable categories for warnings written to audit reports and exported metrics.
|
||||
///
|
||||
/// New warning sites should use a specific category when one is available. The
|
||||
/// default preserves the behaviour and schema of existing callers while making
|
||||
/// their metric label explicit.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub enum WarningCategory {
|
||||
#[default]
|
||||
Unclassified,
|
||||
BerCompatibleCmsEncoding,
|
||||
}
|
||||
|
||||
impl WarningCategory {
|
||||
pub const fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Unclassified => "unclassified",
|
||||
Self::BerCompatibleCmsEncoding => "ber_compatible_cms_encoding",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct Warning {
|
||||
pub message: String,
|
||||
pub category: WarningCategory,
|
||||
pub rfc_refs: Vec<RfcRef>,
|
||||
pub context: Option<String>,
|
||||
}
|
||||
|
||||
impl Warning {
|
||||
pub fn new(message: impl Into<String>) -> Self {
|
||||
Self {
|
||||
message: message.into(),
|
||||
category: WarningCategory::Unclassified,
|
||||
rfc_refs: Vec::new(),
|
||||
context: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_category(mut self, category: WarningCategory) -> Self {
|
||||
self.category = category;
|
||||
self
|
||||
}
|
||||
|
||||
pub fn with_rfc_refs(mut self, refs: &[RfcRef]) -> Self {
|
||||
self.rfc_refs.extend_from_slice(refs);
|
||||
self
|
||||
}
|
||||
|
||||
pub fn with_context(mut self, context: impl Into<String>) -> Self {
|
||||
self.context = Some(context.into());
|
||||
self
|
||||
}
|
||||
}
|
||||
4862
crates/panda-rpki-validator/src/storage.rs
Normal file
4862
crates/panda-rpki-validator/src/storage.rs
Normal file
File diff suppressed because it is too large
Load Diff
178
crates/panda-rpki-validator/src/storage/config.rs
Normal file
178
crates/panda-rpki-validator/src/storage/config.rs
Normal file
@ -0,0 +1,178 @@
|
||||
use rocksdb::{ColumnFamilyDescriptor, DBCompressionType, Options};
|
||||
|
||||
pub const CF_REPOSITORY_VIEW: &str = "repository_view";
|
||||
pub const CF_RAW_BY_HASH: &str = "raw_by_hash";
|
||||
pub const CF_RAW_BLOB: &str = "raw_blob";
|
||||
pub const CF_VCIR: &str = "vcir";
|
||||
pub const CF_VCIR_FAILED_FETCH_REUSE_IDENTITY: &str = "vcir_failed_fetch_reuse_identity";
|
||||
pub const CF_MANIFEST_REPLAY_META: &str = "manifest_replay_meta";
|
||||
pub const CF_ROA_CACHE_PROJECTION: &str = "roa_cache_projection";
|
||||
pub const CF_PUBLICATION_POINT_CACHE_PROJECTION: &str = "publication_point_cache_projection";
|
||||
pub const CF_CHILD_CERTIFICATE_CACHE_PROJECTION: &str = "child_certificate_cache_projection";
|
||||
pub const CF_RRDP_SOURCE: &str = "rrdp_source";
|
||||
pub const CF_RRDP_SOURCE_MEMBER: &str = "rrdp_source_member";
|
||||
pub const CF_RRDP_URI_OWNER: &str = "rrdp_uri_owner";
|
||||
pub const CF_TRANSPORT_PREFETCH: &str = "transport_prefetch";
|
||||
|
||||
pub const ALL_COLUMN_FAMILY_NAMES: &[&str] = &[
|
||||
CF_REPOSITORY_VIEW,
|
||||
CF_RAW_BY_HASH,
|
||||
CF_RAW_BLOB,
|
||||
CF_VCIR,
|
||||
CF_VCIR_FAILED_FETCH_REUSE_IDENTITY,
|
||||
CF_MANIFEST_REPLAY_META,
|
||||
CF_ROA_CACHE_PROJECTION,
|
||||
CF_PUBLICATION_POINT_CACHE_PROJECTION,
|
||||
CF_CHILD_CERTIFICATE_CACHE_PROJECTION,
|
||||
CF_RRDP_SOURCE,
|
||||
CF_RRDP_SOURCE_MEMBER,
|
||||
CF_RRDP_URI_OWNER,
|
||||
CF_TRANSPORT_PREFETCH,
|
||||
];
|
||||
|
||||
pub(super) const REPOSITORY_VIEW_KEY_PREFIX: &str = "repo_view:";
|
||||
pub(super) const RAW_BY_HASH_KEY_PREFIX: &str = "rawbyhash:";
|
||||
pub(super) const RAW_BLOB_KEY_PREFIX: &str = "rawblob:";
|
||||
pub(super) const VCIR_KEY_PREFIX: &str = "vcir:";
|
||||
pub(super) const VCIR_FAILED_FETCH_REUSE_IDENTITY_KEY_PREFIX: &str =
|
||||
"vcir_failed_fetch_reuse_identity:";
|
||||
pub(super) const MANIFEST_REPLAY_META_KEY_PREFIX: &str = "manifest_replay_meta:";
|
||||
pub(super) const ROA_CACHE_PROJECTION_KEY_PREFIX: &str = "roa_cache_projection:";
|
||||
pub(super) const PUBLICATION_POINT_CACHE_PROJECTION_KEY_PREFIX: &str =
|
||||
"publication_point_cache_projection:";
|
||||
pub(super) const CHILD_CERTIFICATE_CACHE_PROJECTION_KEY_PREFIX: &str =
|
||||
"child_certificate_cache_projection:";
|
||||
pub(super) const RRDP_SOURCE_KEY_PREFIX: &str = "rrdp_source:";
|
||||
pub(super) const RRDP_SOURCE_MEMBER_KEY_PREFIX: &str = "rrdp_source_member:";
|
||||
pub(super) const RRDP_URI_OWNER_KEY_PREFIX: &str = "rrdp_uri_owner:";
|
||||
pub(super) const TRANSPORT_PREFETCH_LAST_SNAPSHOT_KEY: &str = "transport_prefetch:last_snapshot";
|
||||
|
||||
const WORK_DB_BLOB_MODE_ENV: &str = "RPKI_WORK_DB_BLOB_MODE";
|
||||
const WORK_DB_MEMORY_PROFILE_ENV: &str = "RPKI_WORK_DB_MEMORY_PROFILE";
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) enum WorkDbBlobMode {
|
||||
Current,
|
||||
Disabled,
|
||||
Lz4,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) enum WorkDbMemoryProfile {
|
||||
Default,
|
||||
Compact,
|
||||
}
|
||||
|
||||
pub(super) fn parse_work_db_blob_mode(raw: &str) -> Option<WorkDbBlobMode> {
|
||||
match raw.trim().to_ascii_lowercase().as_str() {
|
||||
"" | "default" => Some(default_work_db_blob_mode()),
|
||||
"current" | "legacy" => Some(WorkDbBlobMode::Current),
|
||||
"disabled" | "disable" | "off" | "none" | "no_blob" | "no-blob" => {
|
||||
Some(WorkDbBlobMode::Disabled)
|
||||
}
|
||||
"lz4" | "blob_lz4" | "blob-lz4" => Some(WorkDbBlobMode::Lz4),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn default_work_db_blob_mode() -> WorkDbBlobMode {
|
||||
WorkDbBlobMode::Disabled
|
||||
}
|
||||
|
||||
pub(super) fn work_db_blob_mode_from_env() -> WorkDbBlobMode {
|
||||
let Ok(raw) = std::env::var(WORK_DB_BLOB_MODE_ENV) else {
|
||||
return default_work_db_blob_mode();
|
||||
};
|
||||
match parse_work_db_blob_mode(&raw) {
|
||||
Some(mode) => mode,
|
||||
None => {
|
||||
eprintln!(
|
||||
"warning: unsupported {WORK_DB_BLOB_MODE_ENV}={raw:?}; using default work-db blobdb mode"
|
||||
);
|
||||
default_work_db_blob_mode()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn parse_work_db_memory_profile(raw: &str) -> Option<WorkDbMemoryProfile> {
|
||||
match raw.trim().to_ascii_lowercase().as_str() {
|
||||
"" | "default" | "off" | "none" => Some(WorkDbMemoryProfile::Default),
|
||||
"compact" | "low" | "low_memory" | "low-memory" => Some(WorkDbMemoryProfile::Compact),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn work_db_memory_profile_from_env() -> WorkDbMemoryProfile {
|
||||
let Ok(raw) = std::env::var(WORK_DB_MEMORY_PROFILE_ENV) else {
|
||||
return WorkDbMemoryProfile::Default;
|
||||
};
|
||||
match parse_work_db_memory_profile(&raw) {
|
||||
Some(profile) => profile,
|
||||
None => {
|
||||
eprintln!(
|
||||
"warning: unsupported {WORK_DB_MEMORY_PROFILE_ENV}={raw:?}; using default work-db memory profile"
|
||||
);
|
||||
WorkDbMemoryProfile::Default
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn configure_work_db_options(
|
||||
opts: &mut Options,
|
||||
blob_mode: WorkDbBlobMode,
|
||||
memory_profile: WorkDbMemoryProfile,
|
||||
) {
|
||||
opts.set_compression_type(DBCompressionType::Lz4);
|
||||
apply_work_db_memory_profile(opts, memory_profile);
|
||||
match blob_mode {
|
||||
WorkDbBlobMode::Current => enable_blobdb_current(opts),
|
||||
WorkDbBlobMode::Disabled => {}
|
||||
WorkDbBlobMode::Lz4 => {
|
||||
enable_blobdb_current(opts);
|
||||
opts.set_blob_compression_type(DBCompressionType::Lz4);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn cf_opts(blob_mode: WorkDbBlobMode, memory_profile: WorkDbMemoryProfile) -> Options {
|
||||
let mut opts = Options::default();
|
||||
configure_work_db_options(&mut opts, blob_mode, memory_profile);
|
||||
opts
|
||||
}
|
||||
|
||||
pub fn column_family_descriptors() -> Vec<ColumnFamilyDescriptor> {
|
||||
column_family_descriptors_for_blob_mode(work_db_blob_mode_from_env())
|
||||
}
|
||||
|
||||
pub(super) fn column_family_descriptors_for_blob_mode(
|
||||
blob_mode: WorkDbBlobMode,
|
||||
) -> Vec<ColumnFamilyDescriptor> {
|
||||
let memory_profile = work_db_memory_profile_from_env();
|
||||
ALL_COLUMN_FAMILY_NAMES
|
||||
.iter()
|
||||
.map(|name| ColumnFamilyDescriptor::new(*name, cf_opts(blob_mode, memory_profile)))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn apply_work_db_memory_profile(opts: &mut Options, memory_profile: WorkDbMemoryProfile) {
|
||||
match memory_profile {
|
||||
WorkDbMemoryProfile::Default => {}
|
||||
WorkDbMemoryProfile::Compact => {
|
||||
opts.set_write_buffer_size(16 * 1024 * 1024);
|
||||
opts.set_max_write_buffer_number(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn enable_blobdb_current(opts: &mut Options) {
|
||||
#[allow(unused_mut)]
|
||||
let mut _enabled = false;
|
||||
|
||||
#[allow(dead_code)]
|
||||
fn _set(opts: &mut Options) {
|
||||
opts.set_enable_blob_files(true);
|
||||
opts.set_min_blob_size(1024);
|
||||
}
|
||||
|
||||
_set(opts);
|
||||
}
|
||||
217
crates/panda-rpki-validator/src/storage/keys.rs
Normal file
217
crates/panda-rpki-validator/src/storage/keys.rs
Normal file
@ -0,0 +1,217 @@
|
||||
use serde::{Serialize, de::DeserializeOwned};
|
||||
|
||||
use crate::data_model::common::der_take_tlv;
|
||||
|
||||
use super::config::*;
|
||||
use super::pack::PackTime;
|
||||
use super::{StorageError, StorageResult};
|
||||
|
||||
pub(super) fn repository_view_key(rsync_uri: &str) -> String {
|
||||
format!("{REPOSITORY_VIEW_KEY_PREFIX}{rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn repository_view_prefix(rsync_uri_prefix: &str) -> String {
|
||||
format!("{REPOSITORY_VIEW_KEY_PREFIX}{rsync_uri_prefix}")
|
||||
}
|
||||
|
||||
pub(super) fn raw_by_hash_key(sha256_hex: &str) -> String {
|
||||
format!("{RAW_BY_HASH_KEY_PREFIX}{sha256_hex}")
|
||||
}
|
||||
|
||||
pub(super) fn raw_blob_key(sha256_hex: &str) -> String {
|
||||
format!("{RAW_BLOB_KEY_PREFIX}{sha256_hex}")
|
||||
}
|
||||
|
||||
pub(super) fn vcir_key(manifest_rsync_uri: &str) -> String {
|
||||
format!("{VCIR_KEY_PREFIX}{manifest_rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn vcir_failed_fetch_reuse_identity_key(manifest_rsync_uri: &str) -> String {
|
||||
format!("{VCIR_FAILED_FETCH_REUSE_IDENTITY_KEY_PREFIX}{manifest_rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn manifest_replay_meta_key(manifest_rsync_uri: &str) -> String {
|
||||
format!("{MANIFEST_REPLAY_META_KEY_PREFIX}{manifest_rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn roa_cache_projection_key(manifest_rsync_uri: &str) -> String {
|
||||
format!("{ROA_CACHE_PROJECTION_KEY_PREFIX}{manifest_rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn publication_point_cache_projection_key(manifest_rsync_uri: &str) -> String {
|
||||
format!("{PUBLICATION_POINT_CACHE_PROJECTION_KEY_PREFIX}{manifest_rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn child_certificate_cache_projection_key(cache_key_sha256_hex: &str) -> String {
|
||||
format!("{CHILD_CERTIFICATE_CACHE_PROJECTION_KEY_PREFIX}{cache_key_sha256_hex}")
|
||||
}
|
||||
|
||||
pub(super) fn publication_point_cache_projection_key_manifest_uri(key: &[u8]) -> Option<String> {
|
||||
let key = std::str::from_utf8(key).ok()?;
|
||||
key.strip_prefix(PUBLICATION_POINT_CACHE_PROJECTION_KEY_PREFIX)
|
||||
.map(str::to_string)
|
||||
}
|
||||
|
||||
pub(super) fn rrdp_source_key(notify_uri: &str) -> String {
|
||||
format!("{RRDP_SOURCE_KEY_PREFIX}{notify_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn rrdp_source_member_key(notify_uri: &str, rsync_uri: &str) -> String {
|
||||
format!("{RRDP_SOURCE_MEMBER_KEY_PREFIX}{notify_uri}:{rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn rrdp_source_member_prefix(notify_uri: &str) -> String {
|
||||
format!("{RRDP_SOURCE_MEMBER_KEY_PREFIX}{notify_uri}:")
|
||||
}
|
||||
|
||||
pub(super) fn rrdp_uri_owner_key(rsync_uri: &str) -> String {
|
||||
format!("{RRDP_URI_OWNER_KEY_PREFIX}{rsync_uri}")
|
||||
}
|
||||
|
||||
pub(super) fn encode_cbor<T: Serialize>(value: &T, entity: &'static str) -> StorageResult<Vec<u8>> {
|
||||
serde_cbor::to_vec(value).map_err(|e| StorageError::Codec {
|
||||
entity,
|
||||
detail: e.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn decode_cbor<T: DeserializeOwned>(
|
||||
bytes: &[u8],
|
||||
entity: &'static str,
|
||||
) -> StorageResult<T> {
|
||||
serde_cbor::from_slice(bytes).map_err(|e| StorageError::Codec {
|
||||
entity,
|
||||
detail: e.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn validate_non_empty(field: &'static str, value: &str) -> StorageResult<()> {
|
||||
if value.is_empty() {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "must not be empty".to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn validate_sha256_hex(field: &'static str, value: &str) -> StorageResult<()> {
|
||||
if value.len() != 64 || !value.as_bytes().iter().all(u8::is_ascii_hexdigit) {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "must be a 64-character lowercase or uppercase SHA-256 hex string".to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn decode_sha256_hex_32(field: &'static str, value: &str) -> StorageResult<[u8; 32]> {
|
||||
validate_sha256_hex(field, value)?;
|
||||
let mut out = [0u8; 32];
|
||||
hex::decode_to_slice(value, &mut out).map_err(|e| StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: format!("hex decode failed: {e}"),
|
||||
})?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
pub(super) fn validate_manifest_number_be(field: &'static str, value: &[u8]) -> StorageResult<()> {
|
||||
if value.is_empty() {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "must not be empty".to_string(),
|
||||
});
|
||||
}
|
||||
if value.len() > 20 {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "must be at most 20 octets".to_string(),
|
||||
});
|
||||
}
|
||||
if value.len() > 1 && value[0] == 0 {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "must be minimal big-endian without leading zeros".to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn validate_sha256_digest_bytes(field: &'static str, value: &[u8]) -> StorageResult<()> {
|
||||
if value.len() != 32 {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: format!("must be 32 bytes, got {}", value.len()),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn validate_fixed_len_bytes(
|
||||
field: &'static str,
|
||||
value: &[u8],
|
||||
expected_len: usize,
|
||||
) -> StorageResult<()> {
|
||||
if value.len() != expected_len {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: format!("must be {expected_len} bytes, got {}", value.len()),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn validate_sorted_unique_fixed_len_bytes(
|
||||
field: &'static str,
|
||||
values: &[Vec<u8>],
|
||||
expected_len: usize,
|
||||
) -> StorageResult<()> {
|
||||
for value in values {
|
||||
validate_fixed_len_bytes(field, value, expected_len)?;
|
||||
}
|
||||
for window in values.windows(2) {
|
||||
if window[0] >= window[1] {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "must be strictly sorted and unique".to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn validate_full_der_with_tag(
|
||||
field: &'static str,
|
||||
der: &[u8],
|
||||
expected_tag: Option<u8>,
|
||||
) -> StorageResult<()> {
|
||||
let (tag, _value, rem) = der_take_tlv(der).map_err(|detail| StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail,
|
||||
})?;
|
||||
if !rem.is_empty() {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: "trailing bytes after DER object".to_string(),
|
||||
});
|
||||
}
|
||||
if let Some(expected_tag) = expected_tag {
|
||||
if tag != expected_tag {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail: format!("unexpected tag 0x{tag:02X}, expected 0x{expected_tag:02X}"),
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn parse_time(
|
||||
field: &'static str,
|
||||
value: &PackTime,
|
||||
) -> StorageResult<time::OffsetDateTime> {
|
||||
value.parse().map_err(|detail| StorageError::InvalidData {
|
||||
entity: field,
|
||||
detail,
|
||||
})
|
||||
}
|
||||
200
crates/panda-rpki-validator/src/storage/pack.rs
Normal file
200
crates/panda-rpki-validator/src/storage/pack.rs
Normal file
@ -0,0 +1,200 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::Digest;
|
||||
|
||||
use crate::blob_store::{ExternalRawStoreDb, ExternalRepoBytesDb, RawObjectStore};
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum PackBytes {
|
||||
Eager(std::sync::Arc<[u8]>),
|
||||
LazyExternal {
|
||||
sha256_hex: String,
|
||||
store: std::sync::Arc<ExternalRawStoreDb>,
|
||||
cache: std::sync::Arc<std::sync::OnceLock<std::sync::Arc<[u8]>>>,
|
||||
},
|
||||
LazyRepoBytes {
|
||||
sha256_hex: String,
|
||||
store: std::sync::Arc<ExternalRepoBytesDb>,
|
||||
cache: std::sync::Arc<std::sync::OnceLock<std::sync::Arc<[u8]>>>,
|
||||
},
|
||||
}
|
||||
|
||||
impl PackBytes {
|
||||
pub fn eager(bytes: Vec<u8>) -> Self {
|
||||
Self::Eager(std::sync::Arc::from(bytes))
|
||||
}
|
||||
|
||||
pub fn lazy_external(sha256_hex: String, store: std::sync::Arc<ExternalRawStoreDb>) -> Self {
|
||||
Self::LazyExternal {
|
||||
sha256_hex,
|
||||
store,
|
||||
cache: std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn lazy_repo_bytes(sha256_hex: String, store: std::sync::Arc<ExternalRepoBytesDb>) -> Self {
|
||||
Self::LazyRepoBytes {
|
||||
sha256_hex,
|
||||
store,
|
||||
cache: std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn as_slice(&self) -> Result<&[u8], String> {
|
||||
match self {
|
||||
Self::Eager(bytes) => Ok(bytes.as_ref()),
|
||||
Self::LazyExternal {
|
||||
sha256_hex,
|
||||
store,
|
||||
cache,
|
||||
} => {
|
||||
if cache.get().is_none() {
|
||||
let bytes = store
|
||||
.get_blob_bytes(sha256_hex)
|
||||
.map_err(|e| e.to_string())?
|
||||
.ok_or_else(|| format!("missing raw blob for sha256={sha256_hex}"))?;
|
||||
let _ = cache.set(std::sync::Arc::from(bytes));
|
||||
}
|
||||
let bytes = cache
|
||||
.get()
|
||||
.ok_or_else(|| format!("missing raw blob cache for sha256={sha256_hex}"))?;
|
||||
Ok(bytes.as_ref())
|
||||
}
|
||||
Self::LazyRepoBytes {
|
||||
sha256_hex,
|
||||
store,
|
||||
cache,
|
||||
} => {
|
||||
if cache.get().is_none() {
|
||||
let bytes = store
|
||||
.get_blob_bytes(sha256_hex)
|
||||
.map_err(|e| e.to_string())?
|
||||
.ok_or_else(|| format!("missing repo bytes for sha256={sha256_hex}"))?;
|
||||
let _ = cache.set(std::sync::Arc::from(bytes));
|
||||
}
|
||||
let bytes = cache
|
||||
.get()
|
||||
.ok_or_else(|| format!("missing repo bytes cache for sha256={sha256_hex}"))?;
|
||||
Ok(bytes.as_ref())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn to_vec(&self) -> Result<Vec<u8>, String> {
|
||||
Ok(self.as_slice()?.to_vec())
|
||||
}
|
||||
}
|
||||
|
||||
impl PartialEq for PackBytes {
|
||||
fn eq(&self, other: &Self) -> bool {
|
||||
match (self.as_slice(), other.as_slice()) {
|
||||
(Ok(a), Ok(b)) => a == b,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Eq for PackBytes {}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PackFile {
|
||||
pub rsync_uri: String,
|
||||
pub bytes: PackBytes,
|
||||
pub sha256: [u8; 32],
|
||||
}
|
||||
|
||||
impl PackFile {
|
||||
pub fn new(rsync_uri: impl Into<String>, bytes: PackBytes, sha256: [u8; 32]) -> Self {
|
||||
Self {
|
||||
rsync_uri: rsync_uri.into(),
|
||||
bytes,
|
||||
sha256,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn from_bytes_with_sha256(
|
||||
rsync_uri: impl Into<String>,
|
||||
bytes: Vec<u8>,
|
||||
sha256: [u8; 32],
|
||||
) -> Self {
|
||||
Self::new(rsync_uri, PackBytes::eager(bytes), sha256)
|
||||
}
|
||||
|
||||
pub fn from_lazy_external_raw_store(
|
||||
rsync_uri: impl Into<String>,
|
||||
sha256_hex: String,
|
||||
sha256: [u8; 32],
|
||||
store: std::sync::Arc<ExternalRawStoreDb>,
|
||||
) -> Self {
|
||||
Self::new(
|
||||
rsync_uri,
|
||||
PackBytes::lazy_external(sha256_hex, store),
|
||||
sha256,
|
||||
)
|
||||
}
|
||||
|
||||
pub fn from_lazy_repo_bytes(
|
||||
rsync_uri: impl Into<String>,
|
||||
sha256_hex: String,
|
||||
sha256: [u8; 32],
|
||||
store: std::sync::Arc<ExternalRepoBytesDb>,
|
||||
) -> Self {
|
||||
Self::new(
|
||||
rsync_uri,
|
||||
PackBytes::lazy_repo_bytes(sha256_hex, store),
|
||||
sha256,
|
||||
)
|
||||
}
|
||||
|
||||
pub fn from_bytes_compute_sha256(rsync_uri: impl Into<String>, bytes: Vec<u8>) -> Self {
|
||||
let sha256 = compute_sha256_32(&bytes);
|
||||
Self::new(rsync_uri, PackBytes::eager(bytes), sha256)
|
||||
}
|
||||
|
||||
pub fn bytes(&self) -> Result<&[u8], String> {
|
||||
self.bytes.as_slice()
|
||||
}
|
||||
|
||||
pub fn bytes_cloned(&self) -> Result<Vec<u8>, String> {
|
||||
self.bytes.to_vec()
|
||||
}
|
||||
|
||||
pub fn compute_sha256(&self) -> Result<[u8; 32], String> {
|
||||
Ok(compute_sha256_32(self.bytes()?))
|
||||
}
|
||||
}
|
||||
|
||||
impl PartialEq for PackFile {
|
||||
fn eq(&self, other: &Self) -> bool {
|
||||
self.rsync_uri == other.rsync_uri
|
||||
&& self.sha256 == other.sha256
|
||||
&& self.bytes == other.bytes
|
||||
}
|
||||
}
|
||||
|
||||
impl Eq for PackFile {}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct PackTime {
|
||||
pub rfc3339_utc: String,
|
||||
}
|
||||
|
||||
impl PackTime {
|
||||
pub fn from_utc_offset_datetime(t: time::OffsetDateTime) -> Self {
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
let utc = t.to_offset(time::UtcOffset::UTC);
|
||||
let s = utc.format(&Rfc3339).expect("format RFC 3339 UTC time");
|
||||
Self { rfc3339_utc: s }
|
||||
}
|
||||
|
||||
pub fn parse(&self) -> Result<time::OffsetDateTime, String> {
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
time::OffsetDateTime::parse(&self.rfc3339_utc, &Rfc3339).map_err(|e| e.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn compute_sha256_32(bytes: &[u8]) -> [u8; 32] {
|
||||
let digest = sha2::Sha256::digest(bytes);
|
||||
let mut out = [0u8; 32];
|
||||
out.copy_from_slice(&digest);
|
||||
out
|
||||
}
|
||||
671
crates/panda-rpki-validator/src/storage/pp_cache_index.rs
Normal file
671
crates/panda-rpki-validator/src/storage/pp_cache_index.rs
Normal file
@ -0,0 +1,671 @@
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::fs::{self, File};
|
||||
use std::io::Write;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Instant, SystemTime, UNIX_EPOCH};
|
||||
|
||||
use memmap2::Mmap;
|
||||
use serde::Serialize;
|
||||
|
||||
use super::{StorageError, StorageResult};
|
||||
|
||||
const MAGIC: &[u8; 12] = b"RPKIPPIDX\0\0\0";
|
||||
const VERSION: u32 = 1;
|
||||
const HEADER_LEN: usize = 64;
|
||||
const ENTRY_LEN: usize = 24;
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct PpCacheIndexLoadStats {
|
||||
pub source: String,
|
||||
pub entries: usize,
|
||||
pub bytes: usize,
|
||||
pub file_bytes: u64,
|
||||
pub load_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct PpCacheIndexRefreshStats {
|
||||
pub state: String,
|
||||
pub old_entries: usize,
|
||||
pub dirty_entries: usize,
|
||||
pub new_entries: usize,
|
||||
pub file_bytes: u64,
|
||||
pub write_ms: u64,
|
||||
pub compaction_triggered: bool,
|
||||
pub compaction_reason: Option<String>,
|
||||
pub compaction_segments_before: usize,
|
||||
pub compaction_total_file_bytes_before: u64,
|
||||
pub compaction_live_entries: usize,
|
||||
pub compaction_file_bytes: u64,
|
||||
pub compaction_reclaimed_bytes: u64,
|
||||
pub compaction_ms: u64,
|
||||
pub compaction_deleted_segments: usize,
|
||||
pub compaction_error: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
|
||||
pub struct PpCacheIndexDirectoryStats {
|
||||
pub segment_count: usize,
|
||||
pub total_file_bytes: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
struct PpCacheIndexEntry {
|
||||
value_offset: usize,
|
||||
value_len: usize,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum PpCacheIndexLookup<'a> {
|
||||
Hit(&'a [u8]),
|
||||
Deleted,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct PpCacheMmapIndex {
|
||||
mmap: Arc<Mmap>,
|
||||
entries: HashMap<String, PpCacheIndexEntry>,
|
||||
bytes: usize,
|
||||
file_bytes: u64,
|
||||
}
|
||||
|
||||
impl PpCacheMmapIndex {
|
||||
pub fn open(path: &Path) -> StorageResult<Self> {
|
||||
let file = File::open(path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
let file_len = file
|
||||
.metadata()
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.len() as usize;
|
||||
if file_len < HEADER_LEN {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index",
|
||||
detail: "file too small".to_string(),
|
||||
});
|
||||
}
|
||||
// SAFETY: The mmap is read-only, held by Arc for all returned slices, and the file
|
||||
// format is bounds-checked before any slice is exposed.
|
||||
let mmap = unsafe { Mmap::map(&file) }.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
if &mmap[0..MAGIC.len()] != MAGIC {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index",
|
||||
detail: "invalid magic".to_string(),
|
||||
});
|
||||
}
|
||||
let version = read_u32(&mmap, 12)?;
|
||||
if version != VERSION {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index",
|
||||
detail: format!("unsupported version {version}"),
|
||||
});
|
||||
}
|
||||
let entry_count = read_u64(&mmap, 16)? as usize;
|
||||
let entry_table_offset = read_u64(&mmap, 24)? as usize;
|
||||
let entry_table_len = read_u64(&mmap, 32)? as usize;
|
||||
let blob_offset = read_u64(&mmap, 40)? as usize;
|
||||
let blob_len = read_u64(&mmap, 48)? as usize;
|
||||
checked_range(file_len, entry_table_offset, entry_table_len)?;
|
||||
checked_range(file_len, blob_offset, blob_len)?;
|
||||
if entry_table_len != entry_count.saturating_mul(ENTRY_LEN) {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index",
|
||||
detail: "entry table length mismatch".to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let mut entries = HashMap::with_capacity(entry_count);
|
||||
let mut bytes = 0usize;
|
||||
for offset in (entry_table_offset..entry_table_offset + entry_table_len).step_by(ENTRY_LEN)
|
||||
{
|
||||
let key_offset = read_u64(&mmap, offset)? as usize;
|
||||
let key_len = read_u32(&mmap, offset + 8)? as usize;
|
||||
let value_offset = read_u64(&mmap, offset + 12)? as usize;
|
||||
let value_len = read_u32(&mmap, offset + 20)? as usize;
|
||||
checked_range(blob_len, key_offset, key_len)?;
|
||||
checked_range(blob_len, value_offset, value_len)?;
|
||||
let key_start = blob_offset + key_offset;
|
||||
let key_end = key_start + key_len;
|
||||
let key = std::str::from_utf8(&mmap[key_start..key_end])
|
||||
.map_err(|e| StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index.key",
|
||||
detail: e.to_string(),
|
||||
})?
|
||||
.to_string();
|
||||
bytes = bytes.saturating_add(value_len);
|
||||
entries.insert(
|
||||
key,
|
||||
PpCacheIndexEntry {
|
||||
value_offset,
|
||||
value_len,
|
||||
},
|
||||
);
|
||||
}
|
||||
Ok(Self {
|
||||
mmap: Arc::new(mmap),
|
||||
entries,
|
||||
bytes,
|
||||
file_bytes: file_len as u64,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn lookup(&self, manifest_rsync_uri: &str) -> Option<PpCacheIndexLookup<'_>> {
|
||||
let entry = self.entries.get(manifest_rsync_uri)?;
|
||||
if entry.value_len == 0 {
|
||||
return Some(PpCacheIndexLookup::Deleted);
|
||||
}
|
||||
let blob_offset = read_u64(self.mmap.as_ref(), 40).ok()? as usize;
|
||||
let start = blob_offset + entry.value_offset;
|
||||
let end = start + entry.value_len;
|
||||
Some(PpCacheIndexLookup::Hit(&self.mmap[start..end]))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn get(&self, manifest_rsync_uri: &str) -> Option<&[u8]> {
|
||||
match self.lookup(manifest_rsync_uri)? {
|
||||
PpCacheIndexLookup::Hit(bytes) => Some(bytes),
|
||||
PpCacheIndexLookup::Deleted => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn entries(&self) -> usize {
|
||||
self.entries.len()
|
||||
}
|
||||
|
||||
pub fn bytes(&self) -> usize {
|
||||
self.bytes
|
||||
}
|
||||
|
||||
pub fn file_bytes(&self) -> u64 {
|
||||
self.file_bytes
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct PpCacheMmapIndexSet {
|
||||
indexes: Vec<PpCacheMmapIndex>,
|
||||
entries: usize,
|
||||
bytes: usize,
|
||||
file_bytes: u64,
|
||||
}
|
||||
|
||||
impl PpCacheMmapIndexSet {
|
||||
pub fn lookup(&self, manifest_rsync_uri: &str) -> Option<PpCacheIndexLookup<'_>> {
|
||||
self.indexes
|
||||
.iter()
|
||||
.find_map(|index| index.lookup(manifest_rsync_uri))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn get(&self, manifest_rsync_uri: &str) -> Option<&[u8]> {
|
||||
match self.lookup(manifest_rsync_uri)? {
|
||||
PpCacheIndexLookup::Hit(bytes) => Some(bytes),
|
||||
PpCacheIndexLookup::Deleted => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn entries(&self) -> usize {
|
||||
self.entries
|
||||
}
|
||||
|
||||
pub fn bytes(&self) -> usize {
|
||||
self.bytes
|
||||
}
|
||||
|
||||
pub fn file_bytes(&self) -> u64 {
|
||||
self.file_bytes
|
||||
}
|
||||
}
|
||||
|
||||
pub fn default_pp_cache_index_dir(db_path: &Path) -> PathBuf {
|
||||
let file_name = db_path
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.unwrap_or("work-db");
|
||||
db_path.with_file_name(format!("{file_name}.pp-cache-index"))
|
||||
}
|
||||
|
||||
pub fn load_pp_cache_mmap_index(
|
||||
path: &Path,
|
||||
) -> StorageResult<(PpCacheMmapIndex, PpCacheIndexLoadStats)> {
|
||||
let started = Instant::now();
|
||||
let index = PpCacheMmapIndex::open(path)?;
|
||||
let file_bytes = fs::metadata(path)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.len();
|
||||
let stats = PpCacheIndexLoadStats {
|
||||
source: "mmap".to_string(),
|
||||
entries: index.entries(),
|
||||
bytes: index.bytes(),
|
||||
file_bytes,
|
||||
load_ms: started.elapsed().as_millis() as u64,
|
||||
};
|
||||
Ok((index, stats))
|
||||
}
|
||||
|
||||
pub fn load_pp_cache_mmap_index_set(
|
||||
dir: &Path,
|
||||
) -> StorageResult<(PpCacheMmapIndexSet, PpCacheIndexLoadStats)> {
|
||||
let started = Instant::now();
|
||||
let mut paths = Vec::new();
|
||||
if dir.exists() {
|
||||
let mut segments = fs::read_dir(dir)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.filter_map(Result::ok)
|
||||
.map(|entry| entry.path())
|
||||
.filter(|path| {
|
||||
path.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.is_some_and(|name| name.starts_with("segment-") && name.ends_with(".idx"))
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
segments.sort();
|
||||
segments.reverse();
|
||||
paths.extend(segments);
|
||||
let current = dir.join("current.idx");
|
||||
if current.exists() {
|
||||
paths.push(current);
|
||||
}
|
||||
}
|
||||
if paths.is_empty() {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index",
|
||||
detail: "index file missing".to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let mut indexes = Vec::new();
|
||||
let mut entries = 0usize;
|
||||
let mut bytes = 0usize;
|
||||
let mut file_bytes = 0u64;
|
||||
for path in paths {
|
||||
let (index, _) = load_pp_cache_mmap_index(&path)?;
|
||||
entries = entries.saturating_add(index.entries());
|
||||
bytes = bytes.saturating_add(index.bytes());
|
||||
file_bytes = file_bytes.saturating_add(index.file_bytes());
|
||||
indexes.push(index);
|
||||
}
|
||||
let set = PpCacheMmapIndexSet {
|
||||
indexes,
|
||||
entries,
|
||||
bytes,
|
||||
file_bytes,
|
||||
};
|
||||
let stats = PpCacheIndexLoadStats {
|
||||
source: "mmap".to_string(),
|
||||
entries: set.entries(),
|
||||
bytes: set.bytes(),
|
||||
file_bytes: set.file_bytes(),
|
||||
load_ms: started.elapsed().as_millis() as u64,
|
||||
};
|
||||
Ok((set, stats))
|
||||
}
|
||||
|
||||
pub fn pp_cache_index_directory_stats(dir: &Path) -> StorageResult<PpCacheIndexDirectoryStats> {
|
||||
let mut stats = PpCacheIndexDirectoryStats::default();
|
||||
if !dir.exists() {
|
||||
return Ok(stats);
|
||||
}
|
||||
for entry in fs::read_dir(dir).map_err(|e| StorageError::RocksDb(e.to_string()))? {
|
||||
let path = entry
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.path();
|
||||
let Some(name) = path.file_name().and_then(|name| name.to_str()) else {
|
||||
continue;
|
||||
};
|
||||
let is_index =
|
||||
name == "current.idx" || (name.starts_with("segment-") && name.ends_with(".idx"));
|
||||
if !is_index {
|
||||
continue;
|
||||
}
|
||||
let len = fs::metadata(&path)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.len();
|
||||
stats.total_file_bytes = stats.total_file_bytes.saturating_add(len);
|
||||
if name.starts_with("segment-") && name.ends_with(".idx") {
|
||||
stats.segment_count += 1;
|
||||
}
|
||||
}
|
||||
Ok(stats)
|
||||
}
|
||||
|
||||
pub fn compact_pp_cache_index(dir: &Path) -> StorageResult<PpCacheIndexRefreshStats> {
|
||||
let started = Instant::now();
|
||||
fs::create_dir_all(dir).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
let before = pp_cache_index_directory_stats(dir)?;
|
||||
let (index_set, _) = load_pp_cache_mmap_index_set(dir)?;
|
||||
let mut compact_entries = Vec::with_capacity(index_set.entries());
|
||||
let mut seen = HashMap::<String, ()>::with_capacity(index_set.entries());
|
||||
for index in &index_set.indexes {
|
||||
for (key, entry) in &index.entries {
|
||||
if seen.insert(key.clone(), ()).is_some() {
|
||||
continue;
|
||||
}
|
||||
if entry.value_len == 0 {
|
||||
continue;
|
||||
}
|
||||
let blob_offset = read_u64(index.mmap.as_ref(), 40)? as usize;
|
||||
let start = blob_offset + entry.value_offset;
|
||||
let end = start + entry.value_len;
|
||||
compact_entries.push((key.clone(), index.mmap[start..end].to_vec()));
|
||||
}
|
||||
}
|
||||
|
||||
let current = dir.join("current.idx");
|
||||
let mut stats = write_pp_cache_index_atomic(¤t, compact_entries)?;
|
||||
stats.state = "compacted".to_string();
|
||||
stats.compaction_triggered = true;
|
||||
stats.compaction_segments_before = before.segment_count;
|
||||
stats.compaction_total_file_bytes_before = before.total_file_bytes;
|
||||
stats.compaction_live_entries = stats.new_entries;
|
||||
stats.compaction_file_bytes = stats.file_bytes;
|
||||
stats.compaction_reclaimed_bytes = before.total_file_bytes.saturating_sub(stats.file_bytes);
|
||||
|
||||
let mut deleted_segments = 0usize;
|
||||
for entry in fs::read_dir(dir).map_err(|e| StorageError::RocksDb(e.to_string()))? {
|
||||
let path = entry
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.path();
|
||||
let Some(name) = path.file_name().and_then(|name| name.to_str()) else {
|
||||
continue;
|
||||
};
|
||||
if name.starts_with("segment-") && name.ends_with(".idx") {
|
||||
fs::remove_file(&path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
deleted_segments += 1;
|
||||
}
|
||||
}
|
||||
stats.compaction_deleted_segments = deleted_segments;
|
||||
stats.compaction_ms = started.elapsed().as_millis() as u64;
|
||||
Ok(stats)
|
||||
}
|
||||
|
||||
pub fn write_pp_cache_index_segment<I>(
|
||||
dir: &Path,
|
||||
entries: I,
|
||||
) -> StorageResult<PpCacheIndexRefreshStats>
|
||||
where
|
||||
I: IntoIterator<Item = (String, Vec<u8>)>,
|
||||
{
|
||||
fs::create_dir_all(dir).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
let current = dir.join("current.idx");
|
||||
if !current.exists() {
|
||||
return write_pp_cache_index_atomic(¤t, entries);
|
||||
}
|
||||
let now = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.as_nanos();
|
||||
let segment = dir.join(format!("segment-{now:020}-{}.idx", std::process::id()));
|
||||
let mut stats = write_pp_cache_index_atomic(&segment, entries)?;
|
||||
stats.state = "segment_written".to_string();
|
||||
Ok(stats)
|
||||
}
|
||||
|
||||
pub fn write_pp_cache_index_atomic<I>(
|
||||
path: &Path,
|
||||
entries: I,
|
||||
) -> StorageResult<PpCacheIndexRefreshStats>
|
||||
where
|
||||
I: IntoIterator<Item = (String, Vec<u8>)>,
|
||||
{
|
||||
let started = Instant::now();
|
||||
let parent = path.parent().ok_or_else(|| StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index.path",
|
||||
detail: "missing parent".to_string(),
|
||||
})?;
|
||||
fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
|
||||
let mut ordered = BTreeMap::<String, Vec<u8>>::new();
|
||||
for (key, value) in entries {
|
||||
ordered.insert(key, value);
|
||||
}
|
||||
|
||||
let tmp_path = parent.join("next.tmp");
|
||||
write_pp_cache_index_file(
|
||||
&tmp_path,
|
||||
ordered.iter().map(|(k, v)| (k.as_str(), v.as_slice())),
|
||||
)?;
|
||||
let file_bytes = fs::metadata(&tmp_path)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?
|
||||
.len();
|
||||
fs::rename(&tmp_path, path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(PpCacheIndexRefreshStats {
|
||||
state: "written".to_string(),
|
||||
old_entries: 0,
|
||||
dirty_entries: 0,
|
||||
new_entries: ordered.len(),
|
||||
file_bytes,
|
||||
write_ms: started.elapsed().as_millis() as u64,
|
||||
..PpCacheIndexRefreshStats::default()
|
||||
})
|
||||
}
|
||||
|
||||
fn write_pp_cache_index_file<'a, I>(path: &Path, entries: I) -> StorageResult<()>
|
||||
where
|
||||
I: IntoIterator<Item = (&'a str, &'a [u8])>,
|
||||
{
|
||||
let entries = entries.into_iter().collect::<Vec<_>>();
|
||||
let entry_table_offset = HEADER_LEN;
|
||||
let entry_table_len = entries.len() * ENTRY_LEN;
|
||||
let blob_offset = entry_table_offset + entry_table_len;
|
||||
let mut table = Vec::with_capacity(entry_table_len);
|
||||
let mut blob = Vec::new();
|
||||
for (key, value) in entries.iter() {
|
||||
let key_offset = blob.len();
|
||||
blob.extend_from_slice(key.as_bytes());
|
||||
let value_offset = blob.len();
|
||||
blob.extend_from_slice(value);
|
||||
write_u64(&mut table, key_offset as u64);
|
||||
write_u32(&mut table, key.len() as u32);
|
||||
write_u64(&mut table, value_offset as u64);
|
||||
write_u32(&mut table, value.len() as u32);
|
||||
}
|
||||
|
||||
let mut header = Vec::with_capacity(HEADER_LEN);
|
||||
header.extend_from_slice(MAGIC);
|
||||
write_u32(&mut header, VERSION);
|
||||
write_u64(&mut header, entries.len() as u64);
|
||||
write_u64(&mut header, entry_table_offset as u64);
|
||||
write_u64(&mut header, entry_table_len as u64);
|
||||
write_u64(&mut header, blob_offset as u64);
|
||||
write_u64(&mut header, blob.len() as u64);
|
||||
header.resize(HEADER_LEN, 0);
|
||||
|
||||
let mut file = File::create(path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
file.write_all(&header)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
file.write_all(&table)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
file.write_all(&blob)
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
file.sync_all()
|
||||
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn checked_range(total: usize, offset: usize, len: usize) -> StorageResult<()> {
|
||||
if offset.checked_add(len).is_none_or(|end| end > total) {
|
||||
return Err(StorageError::InvalidData {
|
||||
entity: "publication_point_cache_mmap_index.range",
|
||||
detail: "out of bounds".to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn read_u32(bytes: &[u8], offset: usize) -> StorageResult<u32> {
|
||||
checked_range(bytes.len(), offset, 4)?;
|
||||
Ok(u32::from_le_bytes(
|
||||
bytes[offset..offset + 4].try_into().unwrap(),
|
||||
))
|
||||
}
|
||||
|
||||
fn read_u64(bytes: &[u8], offset: usize) -> StorageResult<u64> {
|
||||
checked_range(bytes.len(), offset, 8)?;
|
||||
Ok(u64::from_le_bytes(
|
||||
bytes[offset..offset + 8].try_into().unwrap(),
|
||||
))
|
||||
}
|
||||
|
||||
fn write_u32(out: &mut Vec<u8>, value: u32) {
|
||||
out.extend_from_slice(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
fn write_u64(out: &mut Vec<u8>, value: u64) {
|
||||
out.extend_from_slice(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn pp_cache_index_roundtrips_multiple_entries() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join("current.idx");
|
||||
let stats = write_pp_cache_index_atomic(
|
||||
&path,
|
||||
vec![
|
||||
("rsync://example.test/a.mft".to_string(), b"aaa".to_vec()),
|
||||
("rsync://example.test/b.mft".to_string(), b"bbbb".to_vec()),
|
||||
],
|
||||
)
|
||||
.expect("write index");
|
||||
assert_eq!(stats.new_entries, 2);
|
||||
|
||||
let (index, load) = load_pp_cache_mmap_index(&path).expect("load index");
|
||||
assert_eq!(load.entries, 2);
|
||||
assert_eq!(index.get("rsync://example.test/a.mft"), Some(&b"aaa"[..]));
|
||||
assert_eq!(index.get("rsync://example.test/b.mft"), Some(&b"bbbb"[..]));
|
||||
assert_eq!(index.get("rsync://example.test/missing.mft"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pp_cache_index_rejects_bad_magic() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join("current.idx");
|
||||
fs::write(&path, b"bad").expect("write bad");
|
||||
let err = load_pp_cache_mmap_index(&path).expect_err("bad index rejected");
|
||||
assert!(err.to_string().contains("file too small"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pp_cache_index_last_duplicate_wins() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = dir.path().join("current.idx");
|
||||
write_pp_cache_index_atomic(
|
||||
&path,
|
||||
vec![
|
||||
("rsync://example.test/a.mft".to_string(), b"old".to_vec()),
|
||||
("rsync://example.test/a.mft".to_string(), b"new".to_vec()),
|
||||
],
|
||||
)
|
||||
.expect("write index");
|
||||
|
||||
let (index, _) = load_pp_cache_mmap_index(&path).expect("load index");
|
||||
assert_eq!(index.entries(), 1);
|
||||
assert_eq!(index.get("rsync://example.test/a.mft"), Some(&b"new"[..]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pp_cache_index_set_prefers_newest_segment() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let current = dir.path().join("current.idx");
|
||||
write_pp_cache_index_atomic(
|
||||
¤t,
|
||||
vec![("rsync://example.test/a.mft".to_string(), b"old".to_vec())],
|
||||
)
|
||||
.expect("write current index");
|
||||
write_pp_cache_index_segment(
|
||||
dir.path(),
|
||||
vec![("rsync://example.test/a.mft".to_string(), b"new".to_vec())],
|
||||
)
|
||||
.expect("write segment index");
|
||||
|
||||
let (set, stats) = load_pp_cache_mmap_index_set(dir.path()).expect("load index set");
|
||||
assert_eq!(stats.entries, 2);
|
||||
assert_eq!(set.get("rsync://example.test/a.mft"), Some(&b"new"[..]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pp_cache_index_set_tombstone_shadows_older_entry() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let current = dir.path().join("current.idx");
|
||||
write_pp_cache_index_atomic(
|
||||
¤t,
|
||||
vec![("rsync://example.test/a.mft".to_string(), b"old".to_vec())],
|
||||
)
|
||||
.expect("write current index");
|
||||
write_pp_cache_index_segment(
|
||||
dir.path(),
|
||||
vec![("rsync://example.test/a.mft".to_string(), Vec::new())],
|
||||
)
|
||||
.expect("write tombstone segment index");
|
||||
|
||||
let (set, stats) = load_pp_cache_mmap_index_set(dir.path()).expect("load index set");
|
||||
assert_eq!(stats.entries, 2);
|
||||
assert_eq!(
|
||||
set.lookup("rsync://example.test/a.mft"),
|
||||
Some(PpCacheIndexLookup::Deleted)
|
||||
);
|
||||
assert_eq!(set.get("rsync://example.test/a.mft"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pp_cache_index_compaction_keeps_latest_values_and_removes_segments() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let current = dir.path().join("current.idx");
|
||||
write_pp_cache_index_atomic(
|
||||
¤t,
|
||||
vec![
|
||||
("rsync://example.test/a.mft".to_string(), b"old-a".to_vec()),
|
||||
("rsync://example.test/b.mft".to_string(), b"old-b".to_vec()),
|
||||
("rsync://example.test/c.mft".to_string(), b"old-c".to_vec()),
|
||||
],
|
||||
)
|
||||
.expect("write current index");
|
||||
write_pp_cache_index_segment(
|
||||
dir.path(),
|
||||
vec![
|
||||
("rsync://example.test/a.mft".to_string(), b"new-a".to_vec()),
|
||||
("rsync://example.test/b.mft".to_string(), Vec::new()),
|
||||
],
|
||||
)
|
||||
.expect("write first segment");
|
||||
write_pp_cache_index_segment(
|
||||
dir.path(),
|
||||
vec![("rsync://example.test/d.mft".to_string(), b"new-d".to_vec())],
|
||||
)
|
||||
.expect("write second segment");
|
||||
|
||||
let before = pp_cache_index_directory_stats(dir.path()).expect("stats before");
|
||||
assert_eq!(before.segment_count, 2);
|
||||
|
||||
let stats = compact_pp_cache_index(dir.path()).expect("compact");
|
||||
assert!(stats.compaction_triggered);
|
||||
assert_eq!(stats.compaction_segments_before, 2);
|
||||
assert_eq!(stats.compaction_live_entries, 3);
|
||||
assert_eq!(stats.compaction_deleted_segments, 2);
|
||||
assert!(stats.compaction_reclaimed_bytes > 0);
|
||||
|
||||
let after = pp_cache_index_directory_stats(dir.path()).expect("stats after");
|
||||
assert_eq!(after.segment_count, 0);
|
||||
let segments = fs::read_dir(dir.path())
|
||||
.expect("read dir")
|
||||
.filter_map(Result::ok)
|
||||
.filter(|entry| {
|
||||
entry
|
||||
.file_name()
|
||||
.to_str()
|
||||
.is_some_and(|name| name.starts_with("segment-"))
|
||||
})
|
||||
.count();
|
||||
assert_eq!(segments, 0);
|
||||
|
||||
let (index, _) = load_pp_cache_mmap_index(¤t).expect("load compacted current");
|
||||
assert_eq!(index.entries(), 3);
|
||||
assert_eq!(index.get("rsync://example.test/a.mft"), Some(&b"new-a"[..]));
|
||||
assert_eq!(index.get("rsync://example.test/b.mft"), None);
|
||||
assert_eq!(index.get("rsync://example.test/c.mft"), Some(&b"old-c"[..]));
|
||||
assert_eq!(index.get("rsync://example.test/d.mft"), Some(&b"new-d"[..]));
|
||||
}
|
||||
}
|
||||
2460
crates/panda-rpki-validator/src/storage/tests.rs
Normal file
2460
crates/panda-rpki-validator/src/storage/tests.rs
Normal file
File diff suppressed because it is too large
Load Diff
3
crates/panda-rpki-validator/src/sync/mod.rs
Normal file
3
crates/panda-rpki-validator/src/sync/mod.rs
Normal file
@ -0,0 +1,3 @@
|
||||
pub mod repo;
|
||||
pub mod rrdp;
|
||||
pub(crate) mod store_projection;
|
||||
728
crates/panda-rpki-validator/src/sync/repo.rs
Normal file
728
crates/panda-rpki-validator/src/sync/repo.rs
Normal file
@ -0,0 +1,728 @@
|
||||
use crate::analysis::timing::TimingHandle;
|
||||
use crate::audit::AuditDownloadKind;
|
||||
use crate::audit_downloads::DownloadLogHandle;
|
||||
use crate::current_repo_index::CurrentRepoIndexHandle;
|
||||
use crate::fetch::rsync::{RsyncFetchError, RsyncFetcher};
|
||||
use crate::policy::{Policy, SyncPreference};
|
||||
use crate::replay::archive::{ReplayArchiveIndex, ReplayTransport};
|
||||
use crate::replay::delta_archive::{ReplayDeltaArchiveIndex, ReplayDeltaRrdpKind};
|
||||
use crate::report::{RfcRef, Warning};
|
||||
use crate::storage::RocksStore;
|
||||
use crate::sync::rrdp::sync_from_notification_with_timing_and_download_log;
|
||||
use crate::sync::rrdp::{Fetcher as HttpFetcher, RrdpSyncError, load_rrdp_local_state};
|
||||
use crate::sync::store_projection::{
|
||||
build_repository_view_present_entry, build_repository_view_withdrawn_entry,
|
||||
prepare_repo_bytes_batch_owned,
|
||||
};
|
||||
use std::collections::HashSet;
|
||||
|
||||
#[cfg(test)]
|
||||
use crate::storage::RrdpSourceSyncState;
|
||||
#[cfg(test)]
|
||||
use crate::sync::rrdp::persist_rrdp_local_state;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum RepoSyncSource {
|
||||
Rrdp,
|
||||
Rsync,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum RepoSyncPhase {
|
||||
RrdpOk,
|
||||
RrdpFailedRsyncOk,
|
||||
RsyncOnlyOk,
|
||||
ReplayRrdpOk,
|
||||
ReplayRsyncOk,
|
||||
ReplayNoopRrdp,
|
||||
ReplayNoopRsync,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct RepoSyncResult {
|
||||
pub source: RepoSyncSource,
|
||||
pub phase: RepoSyncPhase,
|
||||
pub objects_written: usize,
|
||||
pub warnings: Vec<Warning>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum RepoSyncError {
|
||||
#[error("RRDP sync failed: {0}")]
|
||||
Rrdp(#[from] RrdpSyncError),
|
||||
|
||||
#[error("rsync fallback failed: {0}")]
|
||||
Rsync(#[from] RsyncFetchError),
|
||||
|
||||
#[error("replay sync error: {0}")]
|
||||
Replay(String),
|
||||
|
||||
#[error("storage error: {0}")]
|
||||
Storage(String),
|
||||
}
|
||||
|
||||
/// Sync a publication point into the current repository view.
|
||||
///
|
||||
/// v1 behavior:
|
||||
/// - If `rrdp_notification_uri` is present and `policy.sync_preference` is `rrdp_then_rsync`,
|
||||
/// try RRDP snapshot sync first (RFC 8182 §3.4.1-§3.4.3).
|
||||
/// - On RRDP failure, fall back to rsync (RFC 8182 §3.4.5).
|
||||
/// - If `sync_preference` is `rsync_only` or there is no RRDP URI, use rsync.
|
||||
pub fn sync_publication_point(
|
||||
store: &RocksStore,
|
||||
policy: &Policy,
|
||||
rrdp_notification_uri: Option<&str>,
|
||||
rsync_base_uri: &str,
|
||||
http_fetcher: &dyn HttpFetcher,
|
||||
rsync_fetcher: &dyn RsyncFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<RepoSyncResult, RepoSyncError> {
|
||||
match (policy.sync_preference, rrdp_notification_uri) {
|
||||
(SyncPreference::RrdpThenRsync, Some(notification_uri)) => {
|
||||
match try_rrdp_sync_with_retry(
|
||||
store,
|
||||
notification_uri,
|
||||
None,
|
||||
http_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
) {
|
||||
Ok(written) => {
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rrdp_ok_total", 1);
|
||||
t.record_count("repo_sync_rrdp_objects_written_total", written as u64);
|
||||
}
|
||||
crate::progress_log::emit(
|
||||
"repo_sync_rrdp_ok",
|
||||
serde_json::json!({
|
||||
"notify_uri": notification_uri,
|
||||
"objects_written": written,
|
||||
}),
|
||||
);
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rrdp,
|
||||
phase: RepoSyncPhase::RrdpOk,
|
||||
objects_written: written,
|
||||
warnings: Vec::new(),
|
||||
})
|
||||
}
|
||||
Err(err) => {
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rrdp_failed_total", 1);
|
||||
}
|
||||
crate::progress_log::emit(
|
||||
"rrdp_fallback_rsync",
|
||||
serde_json::json!({
|
||||
"notify_uri": notification_uri,
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"rrdp_error": err.to_string(),
|
||||
}),
|
||||
);
|
||||
crate::progress_log::emit(
|
||||
"rrdp_failed_fallback_rsync",
|
||||
serde_json::json!({
|
||||
"notify_uri": notification_uri,
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"rrdp_error": err.to_string(),
|
||||
}),
|
||||
);
|
||||
let warnings = vec![
|
||||
Warning::new(format!("RRDP failed; falling back to rsync: {err}"))
|
||||
.with_rfc_refs(&[RfcRef("RFC 8182 §3.4.5")])
|
||||
.with_context(notification_uri),
|
||||
];
|
||||
let written = rsync_sync_into_current_store(
|
||||
store,
|
||||
rsync_base_uri,
|
||||
None,
|
||||
rsync_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)?;
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rsync_peer_aligned_profile_total", 1);
|
||||
}
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rsync_fallback_ok_total", 1);
|
||||
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
|
||||
}
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rsync,
|
||||
phase: RepoSyncPhase::RrdpFailedRsyncOk,
|
||||
objects_written: written,
|
||||
warnings,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
let written = rsync_sync_into_current_store(
|
||||
store,
|
||||
rsync_base_uri,
|
||||
None,
|
||||
rsync_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)?;
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rsync_peer_aligned_profile_total", 1);
|
||||
}
|
||||
crate::progress_log::emit(
|
||||
"repo_sync_rsync_direct",
|
||||
serde_json::json!({
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"objects_written": written,
|
||||
}),
|
||||
);
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rsync_direct_total", 1);
|
||||
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
|
||||
}
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rsync,
|
||||
phase: RepoSyncPhase::RsyncOnlyOk,
|
||||
objects_written: written,
|
||||
warnings: Vec::new(),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn sync_publication_point_replay(
|
||||
store: &RocksStore,
|
||||
replay_index: &ReplayArchiveIndex,
|
||||
rrdp_notification_uri: Option<&str>,
|
||||
rsync_base_uri: &str,
|
||||
http_fetcher: &dyn HttpFetcher,
|
||||
rsync_fetcher: &dyn RsyncFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<RepoSyncResult, RepoSyncError> {
|
||||
match resolve_replay_transport(replay_index, rrdp_notification_uri, rsync_base_uri)? {
|
||||
ReplayResolvedTransport::Rrdp(notification_uri) => {
|
||||
let written = try_rrdp_sync_with_retry(
|
||||
store,
|
||||
notification_uri,
|
||||
None,
|
||||
http_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)?;
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rrdp_ok_total", 1);
|
||||
t.record_count("repo_sync_rrdp_objects_written_total", written as u64);
|
||||
}
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rrdp,
|
||||
phase: RepoSyncPhase::ReplayRrdpOk,
|
||||
objects_written: written,
|
||||
warnings: Vec::new(),
|
||||
})
|
||||
}
|
||||
ReplayResolvedTransport::Rsync => {
|
||||
let written = rsync_sync_into_current_store(
|
||||
store,
|
||||
rsync_base_uri,
|
||||
None,
|
||||
rsync_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)?;
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rsync_direct_total", 1);
|
||||
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
|
||||
}
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rsync,
|
||||
phase: RepoSyncPhase::ReplayRsyncOk,
|
||||
objects_written: written,
|
||||
warnings: Vec::new(),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn sync_publication_point_replay_delta(
|
||||
store: &RocksStore,
|
||||
delta_index: &ReplayDeltaArchiveIndex,
|
||||
rrdp_notification_uri: Option<&str>,
|
||||
rsync_base_uri: &str,
|
||||
http_fetcher: &dyn HttpFetcher,
|
||||
rsync_fetcher: &dyn RsyncFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<RepoSyncResult, RepoSyncError> {
|
||||
match resolve_replay_delta_transport(store, delta_index, rrdp_notification_uri, rsync_base_uri)?
|
||||
{
|
||||
ReplayDeltaResolvedTransport::Rrdp(notification_uri) => {
|
||||
let written = try_rrdp_sync_with_retry(
|
||||
store,
|
||||
notification_uri,
|
||||
None,
|
||||
http_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)?;
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rrdp_ok_total", 1);
|
||||
t.record_count("repo_sync_rrdp_objects_written_total", written as u64);
|
||||
}
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rrdp,
|
||||
phase: RepoSyncPhase::ReplayRrdpOk,
|
||||
objects_written: written,
|
||||
warnings: Vec::new(),
|
||||
})
|
||||
}
|
||||
ReplayDeltaResolvedTransport::Rsync => {
|
||||
let written = rsync_sync_into_current_store(
|
||||
store,
|
||||
rsync_base_uri,
|
||||
None,
|
||||
rsync_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)?;
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("repo_sync_rsync_direct_total", 1);
|
||||
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
|
||||
}
|
||||
Ok(RepoSyncResult {
|
||||
source: RepoSyncSource::Rsync,
|
||||
phase: RepoSyncPhase::ReplayRsyncOk,
|
||||
objects_written: written,
|
||||
warnings: Vec::new(),
|
||||
})
|
||||
}
|
||||
ReplayDeltaResolvedTransport::Noop(source) => Ok(RepoSyncResult {
|
||||
source,
|
||||
phase: match source {
|
||||
RepoSyncSource::Rrdp => RepoSyncPhase::ReplayNoopRrdp,
|
||||
RepoSyncSource::Rsync => RepoSyncPhase::ReplayNoopRsync,
|
||||
},
|
||||
objects_written: 0,
|
||||
warnings: Vec::new(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum ReplayResolvedTransport<'a> {
|
||||
Rrdp(&'a str),
|
||||
Rsync,
|
||||
}
|
||||
|
||||
enum ReplayDeltaResolvedTransport<'a> {
|
||||
Rrdp(&'a str),
|
||||
Rsync,
|
||||
Noop(RepoSyncSource),
|
||||
}
|
||||
|
||||
fn resolve_replay_transport<'a>(
|
||||
replay_index: &'a ReplayArchiveIndex,
|
||||
rrdp_notification_uri: Option<&'a str>,
|
||||
rsync_base_uri: &str,
|
||||
) -> Result<ReplayResolvedTransport<'a>, RepoSyncError> {
|
||||
if let Some(notification_uri) = rrdp_notification_uri {
|
||||
let lock = replay_index.rrdp_lock(notification_uri).ok_or_else(|| {
|
||||
RepoSyncError::Replay(format!(
|
||||
"replay RRDP lock missing for notification URI: {notification_uri}"
|
||||
))
|
||||
})?;
|
||||
return Ok(match lock.transport {
|
||||
ReplayTransport::Rrdp => ReplayResolvedTransport::Rrdp(notification_uri),
|
||||
ReplayTransport::Rsync => ReplayResolvedTransport::Rsync,
|
||||
});
|
||||
}
|
||||
|
||||
replay_index
|
||||
.resolve_rsync_module_for_base_uri(rsync_base_uri)
|
||||
.map_err(|e| RepoSyncError::Replay(e.to_string()))?;
|
||||
Ok(ReplayResolvedTransport::Rsync)
|
||||
}
|
||||
|
||||
fn resolve_replay_delta_transport<'a>(
|
||||
store: &RocksStore,
|
||||
delta_index: &'a ReplayDeltaArchiveIndex,
|
||||
rrdp_notification_uri: Option<&'a str>,
|
||||
rsync_base_uri: &str,
|
||||
) -> Result<ReplayDeltaResolvedTransport<'a>, RepoSyncError> {
|
||||
if let Some(notification_uri) = rrdp_notification_uri {
|
||||
let repo = delta_index.rrdp_repo(notification_uri).ok_or_else(|| {
|
||||
RepoSyncError::Replay(format!(
|
||||
"delta replay RRDP entry missing for notification URI: {notification_uri}"
|
||||
))
|
||||
})?;
|
||||
validate_delta_replay_base_state_for_repo(store, notification_uri, &repo.transition.base)?;
|
||||
return match repo.transition.kind {
|
||||
ReplayDeltaRrdpKind::Delta => Ok(ReplayDeltaResolvedTransport::Rrdp(notification_uri)),
|
||||
ReplayDeltaRrdpKind::Unchanged => Ok(ReplayDeltaResolvedTransport::Noop(
|
||||
match repo.transition.target.transport {
|
||||
ReplayTransport::Rrdp => RepoSyncSource::Rrdp,
|
||||
ReplayTransport::Rsync => RepoSyncSource::Rsync,
|
||||
},
|
||||
)),
|
||||
ReplayDeltaRrdpKind::FallbackRsync => Ok(ReplayDeltaResolvedTransport::Rsync),
|
||||
ReplayDeltaRrdpKind::SessionReset => Err(RepoSyncError::Replay(format!(
|
||||
"delta replay kind session-reset requires fresh full replay for {notification_uri}"
|
||||
))),
|
||||
ReplayDeltaRrdpKind::Gap => Err(RepoSyncError::Replay(format!(
|
||||
"delta replay kind gap requires fresh full replay for {notification_uri}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
delta_index
|
||||
.resolve_rsync_module_for_base_uri(rsync_base_uri)
|
||||
.map_err(|e| RepoSyncError::Replay(e.to_string()))?;
|
||||
Ok(ReplayDeltaResolvedTransport::Rsync)
|
||||
}
|
||||
|
||||
fn validate_delta_replay_base_state_for_repo(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
base: &crate::replay::delta_archive::ReplayDeltaRrdpState,
|
||||
) -> Result<(), RepoSyncError> {
|
||||
match base.transport {
|
||||
ReplayTransport::Rrdp => {
|
||||
let state = load_rrdp_local_state(store, notification_uri)
|
||||
.map_err(RepoSyncError::Storage)?
|
||||
.ok_or_else(|| {
|
||||
RepoSyncError::Replay(format!(
|
||||
"delta replay base state missing for {notification_uri}: expected RRDP session={} serial={}",
|
||||
base.session.as_deref().unwrap_or("<none>"),
|
||||
base.serial
|
||||
.map(|v| v.to_string())
|
||||
.unwrap_or_else(|| "<none>".to_string())
|
||||
))
|
||||
})?;
|
||||
let expected_session = base.session.as_deref().unwrap_or("");
|
||||
let expected_serial = base.serial.unwrap_or_default();
|
||||
if state.session_id != expected_session || state.serial != expected_serial {
|
||||
return Err(RepoSyncError::Replay(format!(
|
||||
"delta replay base state mismatch for {notification_uri}: expected session={} serial={}, actual session={} serial={}",
|
||||
expected_session, expected_serial, state.session_id, state.serial
|
||||
)));
|
||||
}
|
||||
}
|
||||
ReplayTransport::Rsync => {}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn try_rrdp_sync(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
http_fetcher: &dyn HttpFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<usize, RrdpSyncError> {
|
||||
let notification_xml = {
|
||||
let _step = timing
|
||||
.as_ref()
|
||||
.map(|t| t.span_rrdp_repo_step(notification_uri, "fetch_notification"));
|
||||
let _total = timing
|
||||
.as_ref()
|
||||
.map(|t| t.span_phase("rrdp_fetch_notification_total"));
|
||||
let mut dl_span = download_log
|
||||
.map(|dl| dl.span_download(AuditDownloadKind::RrdpNotification, notification_uri));
|
||||
match http_fetcher.fetch(notification_uri) {
|
||||
Ok(v) => {
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("rrdp_notification_fetch_ok_total", 1);
|
||||
}
|
||||
if let Some(s) = dl_span.as_mut() {
|
||||
s.set_bytes(v.len() as u64);
|
||||
s.set_ok();
|
||||
}
|
||||
v
|
||||
}
|
||||
Err(e) => {
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("rrdp_notification_fetch_fail_total", 1);
|
||||
}
|
||||
if let Some(s) = dl_span.as_mut() {
|
||||
s.set_err(e.clone());
|
||||
}
|
||||
return Err(RrdpSyncError::Fetch(e));
|
||||
}
|
||||
}
|
||||
};
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count(
|
||||
"rrdp_notification_bytes_total",
|
||||
notification_xml.len() as u64,
|
||||
);
|
||||
}
|
||||
|
||||
sync_from_notification_with_timing_and_download_log(
|
||||
store,
|
||||
notification_uri,
|
||||
current_repo_index,
|
||||
¬ification_xml,
|
||||
http_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)
|
||||
}
|
||||
|
||||
fn is_retryable_http_fetch_error(msg: &str) -> bool {
|
||||
if msg.contains("http request failed:") || msg.contains("http read body failed:") {
|
||||
return true;
|
||||
}
|
||||
let Some(rest) = msg.strip_prefix("http status ") else {
|
||||
return false;
|
||||
};
|
||||
let code = rest
|
||||
.trim()
|
||||
.split_whitespace()
|
||||
.next()
|
||||
.and_then(|s| s.parse::<u16>().ok())
|
||||
.unwrap_or(0);
|
||||
code == 408 || code == 429 || (500..600).contains(&code)
|
||||
}
|
||||
|
||||
fn try_rrdp_sync_with_retry(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
http_fetcher: &dyn HttpFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<usize, RrdpSyncError> {
|
||||
let attempt = 1usize;
|
||||
crate::progress_log::emit(
|
||||
"rrdp_sync_attempt",
|
||||
serde_json::json!({
|
||||
"notify_uri": notification_uri,
|
||||
"attempt": attempt,
|
||||
}),
|
||||
);
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("rrdp_retry_attempt_total", 1);
|
||||
}
|
||||
|
||||
match try_rrdp_sync(
|
||||
store,
|
||||
notification_uri,
|
||||
current_repo_index,
|
||||
http_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
) {
|
||||
Ok(written) => {
|
||||
crate::progress_log::emit(
|
||||
"rrdp_sync_success",
|
||||
serde_json::json!({
|
||||
"notify_uri": notification_uri,
|
||||
"attempt": attempt,
|
||||
"objects_written": written,
|
||||
}),
|
||||
);
|
||||
Ok(written)
|
||||
}
|
||||
Err(err) => {
|
||||
let retryable = match &err {
|
||||
RrdpSyncError::Fetch(msg) => is_retryable_http_fetch_error(msg),
|
||||
_ => false,
|
||||
};
|
||||
crate::progress_log::emit(
|
||||
"rrdp_sync_failed",
|
||||
serde_json::json!({
|
||||
"notify_uri": notification_uri,
|
||||
"attempt": attempt,
|
||||
"retryable": retryable,
|
||||
"error": err.to_string(),
|
||||
}),
|
||||
);
|
||||
if let Some(t) = timing.as_ref() {
|
||||
match &err {
|
||||
RrdpSyncError::Fetch(_) => t.record_count("rrdp_failed_fetch_total", 1),
|
||||
RrdpSyncError::Rrdp(_) => t.record_count("rrdp_failed_protocol_total", 1),
|
||||
RrdpSyncError::Storage(_) => t.record_count("rrdp_failed_storage_total", 1),
|
||||
}
|
||||
}
|
||||
Err(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn run_rrdp_transport(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
http_fetcher: &dyn HttpFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<usize, RrdpSyncError> {
|
||||
try_rrdp_sync_with_retry(
|
||||
store,
|
||||
notification_uri,
|
||||
current_repo_index,
|
||||
http_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)
|
||||
}
|
||||
|
||||
fn rsync_sync_into_current_store(
|
||||
store: &RocksStore,
|
||||
rsync_base_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
rsync_fetcher: &dyn RsyncFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<usize, RepoSyncError> {
|
||||
let started = std::time::Instant::now();
|
||||
let sync_scope_uri = rsync_fetcher.dedup_key(rsync_base_uri);
|
||||
crate::progress_log::emit(
|
||||
"rsync_sync_start",
|
||||
serde_json::json!({
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"sync_scope_uri": &sync_scope_uri,
|
||||
}),
|
||||
);
|
||||
let _s = timing
|
||||
.as_ref()
|
||||
.map(|t| t.span_rrdp_repo_step(rsync_base_uri, "rsync_fetch_objects"));
|
||||
let _p = timing.as_ref().map(|t| t.span_phase("rsync_fetch_total"));
|
||||
let mut dl_span =
|
||||
download_log.map(|dl| dl.span_download(AuditDownloadKind::Rsync, rsync_base_uri));
|
||||
let mut new_set: HashSet<String> = HashSet::new();
|
||||
let mut fetched_objects: Vec<(String, Vec<u8>)> = Vec::new();
|
||||
let (object_count, bytes_total) =
|
||||
match rsync_fetcher.visit_objects(rsync_base_uri, &mut |uri, bytes| {
|
||||
new_set.insert(uri.clone());
|
||||
fetched_objects.push((uri, bytes));
|
||||
Ok(())
|
||||
}) {
|
||||
Ok(v) => {
|
||||
if let Some(s) = dl_span.as_mut() {
|
||||
s.set_objects(v.0 as u64, v.1);
|
||||
s.set_bytes(v.1);
|
||||
s.set_ok();
|
||||
}
|
||||
v
|
||||
}
|
||||
Err(e) => {
|
||||
if let Some(s) = dl_span.as_mut() {
|
||||
s.set_err(e.to_string());
|
||||
}
|
||||
return Err(e.into());
|
||||
}
|
||||
};
|
||||
crate::progress_log::emit(
|
||||
"rsync_sync_fetch_done",
|
||||
serde_json::json!({
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"sync_scope_uri": &sync_scope_uri,
|
||||
"object_count": object_count,
|
||||
"bytes_total": bytes_total,
|
||||
"duration_ms": started.elapsed().as_millis() as u64,
|
||||
}),
|
||||
);
|
||||
if let Some(t) = timing.as_ref() {
|
||||
t.record_count("rsync_objects_fetched_total", object_count as u64);
|
||||
t.record_count("rsync_objects_bytes_total", bytes_total);
|
||||
}
|
||||
drop(_p);
|
||||
|
||||
let existing_view = store
|
||||
.list_repository_view_entries_with_prefix(&sync_scope_uri)
|
||||
.map_err(|e| RepoSyncError::Storage(e.to_string()))?;
|
||||
|
||||
let _proj = timing
|
||||
.as_ref()
|
||||
.map(|t| t.span_phase("rsync_write_current_store_total"));
|
||||
let prepared_bytes =
|
||||
prepare_repo_bytes_batch_owned(fetched_objects).map_err(RepoSyncError::Storage)?;
|
||||
let mut repository_view_entries = Vec::new();
|
||||
for entry in existing_view {
|
||||
if !new_set.contains(&entry.rsync_uri) {
|
||||
repository_view_entries.push(build_repository_view_withdrawn_entry(
|
||||
&sync_scope_uri,
|
||||
&entry.rsync_uri,
|
||||
entry.current_hash,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
for uri in &new_set {
|
||||
let current_hash = prepared_bytes
|
||||
.uri_to_hash
|
||||
.get(uri)
|
||||
.cloned()
|
||||
.ok_or_else(|| {
|
||||
RepoSyncError::Storage(format!("missing raw_by_hash mapping for {uri}"))
|
||||
})?;
|
||||
repository_view_entries.push(build_repository_view_present_entry(
|
||||
&sync_scope_uri,
|
||||
uri,
|
||||
¤t_hash,
|
||||
));
|
||||
}
|
||||
|
||||
store
|
||||
.put_blob_bytes_batch(&prepared_bytes.blobs_to_write)
|
||||
.map_err(|e| RepoSyncError::Storage(e.to_string()))?;
|
||||
store
|
||||
.put_projection_batch(&repository_view_entries, &[], &[])
|
||||
.map_err(|e| RepoSyncError::Storage(e.to_string()))?;
|
||||
if let Some(index) = current_repo_index {
|
||||
index
|
||||
.write()
|
||||
.map_err(|_| RepoSyncError::Storage("current repo index lock poisoned".to_string()))?
|
||||
.apply_repository_view_entries(&repository_view_entries)
|
||||
.map_err(RepoSyncError::Storage)?;
|
||||
}
|
||||
|
||||
let total_duration_ms = started.elapsed().as_millis() as u64;
|
||||
crate::progress_log::emit(
|
||||
"rsync_sync_done",
|
||||
serde_json::json!({
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"sync_scope_uri": &sync_scope_uri,
|
||||
"object_count": object_count,
|
||||
"bytes_total": bytes_total,
|
||||
"duration_ms": total_duration_ms,
|
||||
}),
|
||||
);
|
||||
if (total_duration_ms as f64) / 1000.0 >= crate::progress_log::slow_threshold_secs() {
|
||||
crate::progress_log::emit(
|
||||
"rsync_sync_slow",
|
||||
serde_json::json!({
|
||||
"rsync_base_uri": rsync_base_uri,
|
||||
"sync_scope_uri": &sync_scope_uri,
|
||||
"object_count": object_count,
|
||||
"bytes_total": bytes_total,
|
||||
"duration_ms": total_duration_ms,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
Ok(object_count)
|
||||
}
|
||||
|
||||
pub(crate) fn run_rsync_transport(
|
||||
store: &RocksStore,
|
||||
rsync_base_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
rsync_fetcher: &dyn RsyncFetcher,
|
||||
timing: Option<&TimingHandle>,
|
||||
download_log: Option<&DownloadLogHandle>,
|
||||
) -> Result<usize, RepoSyncError> {
|
||||
rsync_sync_into_current_store(
|
||||
store,
|
||||
rsync_base_uri,
|
||||
current_repo_index,
|
||||
rsync_fetcher,
|
||||
timing,
|
||||
download_log,
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "repo/tests.rs"]
|
||||
mod tests;
|
||||
1332
crates/panda-rpki-validator/src/sync/repo/tests.rs
Normal file
1332
crates/panda-rpki-validator/src/sync/repo/tests.rs
Normal file
File diff suppressed because it is too large
Load Diff
1357
crates/panda-rpki-validator/src/sync/rrdp.rs
Normal file
1357
crates/panda-rpki-validator/src/sync/rrdp.rs
Normal file
File diff suppressed because it is too large
Load Diff
445
crates/panda-rpki-validator/src/sync/rrdp/snapshot_apply.rs
Normal file
445
crates/panda-rpki-validator/src/sync/rrdp/snapshot_apply.rs
Normal file
@ -0,0 +1,445 @@
|
||||
use base64::Engine;
|
||||
use quick_xml::Reader;
|
||||
use quick_xml::events::Event;
|
||||
use sha2::Digest;
|
||||
use std::io::{BufRead, Seek, SeekFrom, Write};
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::current_repo_index::CurrentRepoIndexHandle;
|
||||
use crate::storage::RocksStore;
|
||||
use crate::sync::store_projection::{
|
||||
build_repository_view_present_entry, build_repository_view_withdrawn_entry,
|
||||
build_rrdp_source_member_present_record, build_rrdp_source_member_withdrawn_record,
|
||||
build_rrdp_uri_owner_active_record, build_rrdp_uri_owner_withdrawn_record, compute_sha256_hex,
|
||||
current_rrdp_owner_is, ensure_rrdp_uri_can_be_owned_by, prepare_repo_bytes_batch,
|
||||
};
|
||||
|
||||
use super::{
|
||||
Fetcher, RRDP_SNAPSHOT_APPLY_BATCH_SIZE, RRDP_XMLNS, RrdpError, RrdpSyncError, parse_u64_str,
|
||||
strip_all_ascii_whitespace,
|
||||
};
|
||||
|
||||
#[cfg(test)]
|
||||
pub(super) fn apply_snapshot(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
snapshot_xml: &[u8],
|
||||
expected_session_id: Uuid,
|
||||
expected_serial: u64,
|
||||
) -> Result<usize, RrdpSyncError> {
|
||||
if snapshot_xml.iter().any(|&b| b > 0x7F) {
|
||||
return Err(RrdpError::NotAscii.into());
|
||||
}
|
||||
apply_snapshot_from_bufread(
|
||||
store,
|
||||
notification_uri,
|
||||
current_repo_index,
|
||||
std::io::Cursor::new(snapshot_xml),
|
||||
expected_session_id,
|
||||
expected_serial,
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn apply_snapshot_from_bufread<R: BufRead>(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
input: R,
|
||||
expected_session_id: Uuid,
|
||||
expected_serial: u64,
|
||||
) -> Result<usize, RrdpSyncError> {
|
||||
let previous_members: Vec<String> = store
|
||||
.list_current_rrdp_source_members(notification_uri)
|
||||
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?
|
||||
.into_iter()
|
||||
.map(|record| record.rsync_uri)
|
||||
.collect();
|
||||
let session_id = expected_session_id.to_string();
|
||||
let mut new_set: std::collections::HashSet<String> = std::collections::HashSet::new();
|
||||
let mut batch_published: Vec<(String, Vec<u8>)> =
|
||||
Vec::with_capacity(RRDP_SNAPSHOT_APPLY_BATCH_SIZE);
|
||||
let mut published_count = 0usize;
|
||||
|
||||
let mut reader = Reader::from_reader(input);
|
||||
reader.config_mut().trim_text(false);
|
||||
let mut buf = Vec::new();
|
||||
let mut root_seen = false;
|
||||
let mut in_publish = false;
|
||||
let mut publish_nested_depth = 0usize;
|
||||
let mut current_publish_uri: Option<String> = None;
|
||||
let mut current_publish_text = String::new();
|
||||
|
||||
loop {
|
||||
match reader.read_event_into(&mut buf) {
|
||||
Ok(Event::Start(e)) => {
|
||||
let local_name = e.local_name();
|
||||
let local_name = local_name.as_ref();
|
||||
if !root_seen {
|
||||
root_seen = true;
|
||||
if local_name != b"snapshot" {
|
||||
let got = String::from_utf8_lossy(local_name).to_string();
|
||||
return Err(RrdpError::UnexpectedRoot(got).into());
|
||||
}
|
||||
let mut xmlns = String::new();
|
||||
let mut version = String::new();
|
||||
let mut session_id_attr = String::new();
|
||||
let mut serial_attr = String::new();
|
||||
for attr in e.attributes().with_checks(false) {
|
||||
let attr = match attr {
|
||||
Ok(attr) => attr,
|
||||
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
|
||||
};
|
||||
let key = attr.key.as_ref();
|
||||
let value = attr
|
||||
.decode_and_unescape_value(reader.decoder())
|
||||
.map_err(|e| RrdpError::Xml(e.to_string()))?
|
||||
.into_owned();
|
||||
match key {
|
||||
b"xmlns" => xmlns = value,
|
||||
b"version" => version = value,
|
||||
b"session_id" => session_id_attr = value,
|
||||
b"serial" => serial_attr = value,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if xmlns != RRDP_XMLNS {
|
||||
return Err(RrdpError::InvalidNamespace(xmlns).into());
|
||||
}
|
||||
if version != "1" {
|
||||
return Err(RrdpError::InvalidVersion(version).into());
|
||||
}
|
||||
let got_session_id = Uuid::parse_str(&session_id_attr)
|
||||
.map_err(|_| RrdpError::InvalidSessionId(session_id_attr.clone()))?;
|
||||
if got_session_id != expected_session_id {
|
||||
return Err(RrdpError::SnapshotSessionIdMismatch {
|
||||
expected: expected_session_id.to_string(),
|
||||
got: got_session_id.to_string(),
|
||||
}
|
||||
.into());
|
||||
}
|
||||
let got_serial = parse_u64_str(&serial_attr)?;
|
||||
if got_serial != expected_serial {
|
||||
return Err(RrdpError::SnapshotSerialMismatch {
|
||||
expected: expected_serial,
|
||||
got: got_serial,
|
||||
}
|
||||
.into());
|
||||
}
|
||||
} else if in_publish {
|
||||
publish_nested_depth += 1;
|
||||
} else if local_name == b"publish" {
|
||||
let mut uri = None;
|
||||
for attr in e.attributes().with_checks(false) {
|
||||
let attr = match attr {
|
||||
Ok(attr) => attr,
|
||||
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
|
||||
};
|
||||
if attr.key.as_ref() == b"uri" {
|
||||
uri = Some(
|
||||
attr.decode_and_unescape_value(reader.decoder())
|
||||
.map_err(|e| RrdpError::Xml(e.to_string()))?
|
||||
.into_owned(),
|
||||
);
|
||||
}
|
||||
}
|
||||
let uri = uri.ok_or(RrdpError::PublishUriMissing)?;
|
||||
ensure_rrdp_uri_can_be_owned_by(store, notification_uri, &uri)
|
||||
.map_err(RrdpSyncError::Storage)?;
|
||||
in_publish = true;
|
||||
publish_nested_depth = 0;
|
||||
current_publish_uri = Some(uri);
|
||||
current_publish_text.clear();
|
||||
}
|
||||
}
|
||||
Ok(Event::Empty(e)) => {
|
||||
let local_name = e.local_name();
|
||||
let local_name = local_name.as_ref();
|
||||
if !root_seen {
|
||||
let got = String::from_utf8_lossy(local_name).to_string();
|
||||
return Err(RrdpError::UnexpectedRoot(got).into());
|
||||
}
|
||||
if local_name == b"publish" {
|
||||
let mut has_uri = false;
|
||||
for attr in e.attributes().with_checks(false) {
|
||||
let attr = match attr {
|
||||
Ok(attr) => attr,
|
||||
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
|
||||
};
|
||||
if attr.key.as_ref() == b"uri" {
|
||||
has_uri = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if !has_uri {
|
||||
return Err(RrdpError::PublishUriMissing.into());
|
||||
}
|
||||
return Err(RrdpError::PublishContentMissing.into());
|
||||
}
|
||||
}
|
||||
Ok(Event::Text(e)) => {
|
||||
if in_publish && publish_nested_depth == 0 {
|
||||
let text = reader
|
||||
.decoder()
|
||||
.decode(e.as_ref())
|
||||
.map_err(|e| RrdpError::Xml(e.to_string()))?;
|
||||
current_publish_text.push_str(&text);
|
||||
}
|
||||
}
|
||||
Ok(Event::CData(e)) => {
|
||||
if in_publish && publish_nested_depth == 0 {
|
||||
let text = reader
|
||||
.decoder()
|
||||
.decode(e.as_ref())
|
||||
.map_err(|e| RrdpError::Xml(e.to_string()))?;
|
||||
current_publish_text.push_str(&text);
|
||||
}
|
||||
}
|
||||
Ok(Event::End(e)) => {
|
||||
let local_name = e.local_name();
|
||||
let local_name = local_name.as_ref();
|
||||
if in_publish {
|
||||
if publish_nested_depth > 0 {
|
||||
publish_nested_depth -= 1;
|
||||
} else if local_name == b"publish" {
|
||||
let uri = current_publish_uri
|
||||
.take()
|
||||
.ok_or_else(|| RrdpError::Xml("publish uri missing in state".into()))?;
|
||||
let content_b64 = strip_all_ascii_whitespace(¤t_publish_text);
|
||||
current_publish_text.clear();
|
||||
if content_b64.is_empty() {
|
||||
return Err(RrdpError::PublishContentMissing.into());
|
||||
}
|
||||
let bytes = base64::engine::general_purpose::STANDARD
|
||||
.decode(content_b64.as_bytes())
|
||||
.map_err(|e| RrdpError::PublishBase64(e.to_string()))?;
|
||||
new_set.insert(uri.clone());
|
||||
batch_published.push((uri, bytes));
|
||||
published_count += 1;
|
||||
if batch_published.len() >= RRDP_SNAPSHOT_APPLY_BATCH_SIZE {
|
||||
flush_snapshot_publish_batch(
|
||||
store,
|
||||
notification_uri,
|
||||
current_repo_index,
|
||||
&session_id,
|
||||
expected_serial,
|
||||
&batch_published,
|
||||
)?;
|
||||
batch_published.clear();
|
||||
}
|
||||
in_publish = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Event::Eof) => break,
|
||||
Ok(Event::Decl(_) | Event::PI(_) | Event::Comment(_) | Event::DocType(_)) => {}
|
||||
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
|
||||
}
|
||||
buf.clear();
|
||||
}
|
||||
|
||||
if !root_seen {
|
||||
return Err(RrdpError::Xml("missing root element".to_string()).into());
|
||||
}
|
||||
if in_publish {
|
||||
return Err(RrdpError::PublishContentMissing.into());
|
||||
}
|
||||
if !batch_published.is_empty() {
|
||||
flush_snapshot_publish_batch(
|
||||
store,
|
||||
notification_uri,
|
||||
current_repo_index,
|
||||
&session_id,
|
||||
expected_serial,
|
||||
&batch_published,
|
||||
)?;
|
||||
batch_published.clear();
|
||||
}
|
||||
|
||||
let mut withdrawn: Vec<(String, Option<String>)> = Vec::new();
|
||||
for old_uri in &previous_members {
|
||||
if new_set.contains(old_uri) {
|
||||
continue;
|
||||
}
|
||||
let previous_hash = store
|
||||
.get_repository_view_entry(old_uri)
|
||||
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?
|
||||
.and_then(|entry| entry.current_hash)
|
||||
.or_else(|| {
|
||||
store
|
||||
.load_current_object_bytes_by_uri(old_uri)
|
||||
.ok()
|
||||
.flatten()
|
||||
.map(|bytes| compute_sha256_hex(&bytes))
|
||||
});
|
||||
withdrawn.push((old_uri.clone(), previous_hash));
|
||||
}
|
||||
|
||||
let mut repository_view_entries = Vec::with_capacity(withdrawn.len());
|
||||
let mut member_records = Vec::with_capacity(withdrawn.len());
|
||||
let mut owner_records = Vec::with_capacity(withdrawn.len());
|
||||
for (uri, previous_hash) in withdrawn {
|
||||
member_records.push(build_rrdp_source_member_withdrawn_record(
|
||||
notification_uri,
|
||||
&session_id,
|
||||
expected_serial,
|
||||
&uri,
|
||||
previous_hash.clone(),
|
||||
));
|
||||
if current_rrdp_owner_is(store, notification_uri, &uri).map_err(RrdpSyncError::Storage)? {
|
||||
repository_view_entries.push(build_repository_view_withdrawn_entry(
|
||||
notification_uri,
|
||||
&uri,
|
||||
previous_hash.clone(),
|
||||
));
|
||||
owner_records.push(build_rrdp_uri_owner_withdrawn_record(
|
||||
notification_uri,
|
||||
&session_id,
|
||||
expected_serial,
|
||||
&uri,
|
||||
previous_hash,
|
||||
));
|
||||
}
|
||||
}
|
||||
store
|
||||
.put_projection_batch(&repository_view_entries, &member_records, &owner_records)
|
||||
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?;
|
||||
if let Some(index) = current_repo_index {
|
||||
index
|
||||
.write()
|
||||
.map_err(|_| RrdpSyncError::Storage("current repo index lock poisoned".to_string()))?
|
||||
.apply_repository_view_entries(&repository_view_entries)
|
||||
.map_err(RrdpSyncError::Storage)?;
|
||||
}
|
||||
|
||||
Ok(published_count)
|
||||
}
|
||||
|
||||
fn flush_snapshot_publish_batch(
|
||||
store: &RocksStore,
|
||||
notification_uri: &str,
|
||||
current_repo_index: Option<&CurrentRepoIndexHandle>,
|
||||
session_id: &str,
|
||||
serial: u64,
|
||||
published: &[(String, Vec<u8>)],
|
||||
) -> Result<(), RrdpSyncError> {
|
||||
let prepared_bytes = prepare_repo_bytes_batch(published).map_err(RrdpSyncError::Storage)?;
|
||||
let mut repository_view_entries = Vec::with_capacity(published.len());
|
||||
let mut member_records = Vec::with_capacity(published.len());
|
||||
let mut owner_records = Vec::with_capacity(published.len());
|
||||
|
||||
for (uri, _bytes) in published {
|
||||
let current_hash = prepared_bytes
|
||||
.uri_to_hash
|
||||
.get(uri)
|
||||
.cloned()
|
||||
.ok_or_else(|| {
|
||||
RrdpSyncError::Storage(format!("missing raw_by_hash mapping for {uri}"))
|
||||
})?;
|
||||
repository_view_entries.push(build_repository_view_present_entry(
|
||||
notification_uri,
|
||||
uri,
|
||||
¤t_hash,
|
||||
));
|
||||
member_records.push(build_rrdp_source_member_present_record(
|
||||
notification_uri,
|
||||
session_id,
|
||||
serial,
|
||||
uri,
|
||||
¤t_hash,
|
||||
));
|
||||
owner_records.push(build_rrdp_uri_owner_active_record(
|
||||
notification_uri,
|
||||
session_id,
|
||||
serial,
|
||||
uri,
|
||||
¤t_hash,
|
||||
));
|
||||
}
|
||||
|
||||
store
|
||||
.put_blob_bytes_batch(&prepared_bytes.blobs_to_write)
|
||||
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?;
|
||||
store
|
||||
.put_projection_batch(&repository_view_entries, &member_records, &owner_records)
|
||||
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?;
|
||||
if let Some(index) = current_repo_index {
|
||||
index
|
||||
.write()
|
||||
.map_err(|_| RrdpSyncError::Storage("current repo index lock poisoned".to_string()))?
|
||||
.apply_repository_view_entries(&repository_view_entries)
|
||||
.map_err(RrdpSyncError::Storage)?;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
const SNAPSHOT_NON_ASCII_ERROR: &str = "snapshot body contains non-ASCII bytes";
|
||||
|
||||
struct SnapshotSpoolWriter<'a, W: Write> {
|
||||
inner: &'a mut W,
|
||||
hasher: sha2::Sha256,
|
||||
bytes: u64,
|
||||
}
|
||||
|
||||
impl<'a, W: Write> SnapshotSpoolWriter<'a, W> {
|
||||
fn new(inner: &'a mut W) -> Self {
|
||||
Self {
|
||||
inner,
|
||||
hasher: sha2::Sha256::new(),
|
||||
bytes: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn finalize_hash(self) -> [u8; 32] {
|
||||
let digest = self.hasher.finalize();
|
||||
let mut out = [0u8; 32];
|
||||
out.copy_from_slice(&digest);
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
impl<W: Write> Write for SnapshotSpoolWriter<'_, W> {
|
||||
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
|
||||
if buf.iter().any(|&b| b > 0x7F) {
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
SNAPSHOT_NON_ASCII_ERROR,
|
||||
));
|
||||
}
|
||||
let n = self.inner.write(buf)?;
|
||||
self.hasher.update(&buf[..n]);
|
||||
self.bytes += n as u64;
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
self.inner.flush()
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn fetch_snapshot_into_tempfile(
|
||||
fetcher: &dyn Fetcher,
|
||||
snapshot_uri: &str,
|
||||
expected_hash_sha256: &[u8; 32],
|
||||
) -> Result<(tempfile::NamedTempFile, u64), RrdpSyncError> {
|
||||
let mut tmp = tempfile::NamedTempFile::new()
|
||||
.map_err(|e| RrdpSyncError::Fetch(format!("tempfile create failed: {e}")))?;
|
||||
let mut spool = SnapshotSpoolWriter::new(tmp.as_file_mut());
|
||||
let bytes_written = match fetcher.fetch_to_writer(snapshot_uri, &mut spool) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(e) if e.contains(SNAPSHOT_NON_ASCII_ERROR) => return Err(RrdpError::NotAscii.into()),
|
||||
Err(e) => return Err(RrdpSyncError::Fetch(e)),
|
||||
};
|
||||
let computed = spool.finalize_hash();
|
||||
if computed.as_slice() != expected_hash_sha256.as_slice() {
|
||||
return Err(RrdpError::SnapshotHashMismatch.into());
|
||||
}
|
||||
tmp.as_file_mut()
|
||||
.flush()
|
||||
.map_err(|e| RrdpSyncError::Fetch(format!("tempfile flush failed: {e}")))?;
|
||||
tmp.as_file_mut()
|
||||
.seek(SeekFrom::Start(0))
|
||||
.map_err(|e| RrdpSyncError::Fetch(format!("tempfile rewind failed: {e}")))?;
|
||||
Ok((tmp, bytes_written))
|
||||
}
|
||||
1316
crates/panda-rpki-validator/src/sync/rrdp/tests.rs
Normal file
1316
crates/panda-rpki-validator/src/sync/rrdp/tests.rs
Normal file
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user