freeze initial panda-rpki staging
Some checks failed
ci / rust (push) Has been cancelled
ci / docker-runtime (push) Has been cancelled

This commit is contained in:
yuyr 2026-09-01 12:13:53 +08:00
commit 15c8de82ba
259 changed files with 98127 additions and 0 deletions

20
.dockerignore Normal file
View File

@ -0,0 +1,20 @@
.git
.git/**
.gitignore
target
artifacts
state
runs
logs
tmp
docker-out
provenance/generated
crates/panda-rpki-validator/tests
tests
specs
.env
.env.*
!.env.example
*.local
*.tmp
*.swp

27
.github/workflows/ci.yml vendored Normal file
View File

@ -0,0 +1,27 @@
name: ci
on:
push:
pull_request:
jobs:
rust:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.92.0
components: rustfmt, clippy
- run: cargo fmt --all --check
- run: cargo build --locked -p panda-rpki-validator
- run: cargo test --locked -p panda-rpki-validator
- run: cargo clippy --locked -p panda-rpki-validator --all-targets -- -D warnings
docker-runtime:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: docker buildx build --platform linux/amd64 --load -t panda-rpki-validator:ci -f docker/validator-runtime.Dockerfile .
- run: ./scripts/docker/verify_image.sh --image panda-rpki-validator:ci
- run: docker run --rm panda-rpki-validator:ci run --help

16
.gitignore vendored Normal file
View File

@ -0,0 +1,16 @@
/target/
/artifacts/
/state/
/runs/
/logs/
/tmp/
/docker-out/
/provenance/generated/
.env
.env.*
!.env.example
*.local
*.tmp
*.swp
__pycache__/
*.py[cod]

10
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,10 @@
# Contributing to panda-rpki
This repository is private staging for the `panda-rpki-validator` extraction.
External contributions are not enabled while ownership, license and contribution
governance are pending.
During M3M6, changes must preserve `docs/output-abi.md`, update the relevant
allowlist/provenance entry, and pass the locked Rust build and Docker smoke
checks. Do not add live RIR data, private state, credentials, internal hosts,
or copied history.

2633
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

4
Cargo.toml Normal file
View File

@ -0,0 +1,4 @@
[workspace]
members = ["crates/panda-rpki-validator"]
default-members = ["crates/panda-rpki-validator"]
resolver = "3"

100
README.md Normal file
View File

@ -0,0 +1,100 @@
# panda-rpki
`panda-rpki` is the staging workspace for the independently distributable RPKI
synchronization and validation component. The first component is
`panda-rpki-validator` (Cargo package and CLI); its Rust import path is
`panda_rpki_validator`.
This directory is currently a **private staging repository**. It has no copied
history from the existing private `rpki` repository. The normal synchronization
and validation path has now been extracted, together with its daemon and
Docker lifecycle wrapper. The first M5 canonical snapshot/delta baseline and
amd64/arm64 staging image checks are complete; remaining profile/performance
gates still block any public release. Do not use this staging build as a
production validator.
## Current milestone
M4/M5 contain the first functional extraction from source commit
`74cbebbd3334ac0063761c1a97a88ee000cc2a57`: normal RRDP/rsync synchronization,
RPKI validation, RocksDB state, CIR/CCR/report/CSV outputs, the run daemon, and
the snapshot/delta lifecycle wrapper. The `verification-only` mode is
intentionally deferred. The project license is intentionally **TBD**; no
external contributions or public release are accepted until ownership and
licensing are approved.
## Build and test the staging component
```bash
cargo build --locked -p panda-rpki-validator
cargo test --locked -p panda-rpki-validator
cargo fmt --all --check
cargo clippy --locked -p panda-rpki-validator --all-targets -- -D warnings
./scripts/docker/build_image.sh --arch amd64 --allow-dirty --no-save
```
The extracted test suite currently passes `735` tests with `1` ignored test in
the normal profile. The APNIC offline snapshot/delta profile has also passed
the M5 canonical comparator and a five-run pinned release smoke baseline; the
full cache/fallback/failure/profile matrix is still pending.
For a deterministic single-RIR run against the checked-in test repository,
build the binaries first and invoke the lifecycle wrapper with
`RPKI_EXTRA_ARGS=--disable-rrdp --rsync-local-dir ...`; the wrapper writes the
normal run ABI under `RUN_ROOT/runs/run_0001/`.
## Docker smoke
```bash
./scripts/docker/build_image.sh --arch amd64 --allow-dirty
./scripts/docker/verify_image.sh \
--image panda-rpki-validator:0.1.0-dirty-amd64
```
The runtime image contains both `panda-rpki-validator` and
`panda-rpki-validator-daemon`, the complete normal-run wrapper, and the
redistributable TAL/TA fixtures. It uses one persistent data root mounted at
`/var/lib/panda-rpki-validator`; `run`, `run-validator`, and `daemon` are
entrypoint subcommands. The wrapper preserves the output contract documented
in
[`docs/output-abi.md`](docs/output-abi.md).
For native or container A/B output checks, run the ABI verifier and canonical
comparator against two retained run directories:
```bash
tests/compat/verify_run_abi.sh /path/to/run_0001
tests/compat/compare_runs.py \
/path/to/original/runs/run_0001 \
/path/to/panda/runs/run_0001
```
The comparator has matched the original runtime for the APNIC offline
snapshot/delta baseline. Five serial release samples on that small single-RIR
profile stay within the current wall-time gate under a pinned CPU. A live APNIC
run also produced an identical decoded CCR state (MFT/VRP/VAP/TA/RK); only the
time-bearing `producedAt` byte differed in the raw DER. A concurrent all-RIR
(`all5`) run is retained as network/fallback evidence, but its source was not an
atomic snapshot (the original image timed out on the ARIN RRDP notification),
so its different CCR state is not a code-parity verdict. See the detailed
[`M5 live RIR comparison report`](../specs/develop/20260901/m5_live_rir_image_artifact_comparison_milestone_report.md).
The follow-up remote-231 serial all5 run (old image first, then this image;
one snapshot plus three warm deltas per image with cache/prefetch/parallel
flags) also completed 4/4 runs on each side and retained full artifacts. Its
live CCR/CIR and timing differences are documented separately and are not a
frozen-input parity result: [`remote-231 all5 cache/prefetch report`](../specs/develop/20260901_2/m5_live_all5_cache_prefetch_serial_4run_milestone_report.md).
The new image also completed a separate remote-231 all5 long sequence of one
snapshot plus ten deltas; the timing and cache counters are retained as live
health evidence, not as a frozen-input parity gate: [`one snapshot + ten delta
timing report`](../specs/develop/20260901_2/m5_new_image_all5_snapshot_10delta_timing_report.md).
Cache, RRDP/fallback, constraints, replay and failure-path profiles remain in
M5.
## Repository status
- `verification-only`: deferred to a separate backlog item.
- License and copyright owner: TBD; do not add a speculative `LICENSE` file.
- Staging Git remote: `https://git.nasp.fit/yuyr/panda-rpki.git` (private
staging target selected); public visibility, registry prefix, signing identity,
and release tags remain unselected until release governance is complete.
- Source provenance baseline: `provenance/source-baseline.toml`.

12
SECURITY.md Normal file
View File

@ -0,0 +1,12 @@
# Security policy
This project is not public yet and does not have a public vulnerability intake
address. Do not publish vulnerability details in an issue or commit. Report
urgent findings to the project owner through the private project channel and
include the affected commit, image tag/digest, reproduction steps, and whether
state or output data is exposed.
The public contact, response SLA, supported release branches, and disclosure
process will be added after ownership and license approval. Container images
must never contain credentials, production state, private registry settings,
or internal hostnames.

View File

@ -0,0 +1,49 @@
[package]
name = "panda-rpki-validator"
version = "0.1.0"
edition = "2024"
rust-version = "1.92"
description = "Panda RPKI synchronization and validation runtime"
publish = false
[lib]
name = "panda_rpki_validator"
path = "src/lib.rs"
[[bin]]
name = "panda-rpki-validator"
path = "src/main.rs"
[[bin]]
name = "panda-rpki-validator-daemon"
path = "src/bin/panda-rpki-validator-daemon.rs"
[features]
default = ["full"]
full = []
profile = ["dep:pprof", "dep:flate2"]
[dependencies]
asn1-rs = "0.7.1"
der-parser = { version = "10.0.0", features = ["serialize"] }
hex = "0.4.3"
base64 = "0.22.1"
sha2 = "0.10.8"
thiserror = "2.0.18"
time = "0.3.45"
ring = "0.17.14"
x509-parser = { version = "0.18.0", features = ["verify"] }
url = "2.5.8"
serde = { version = "1.0.218", features = ["derive"] }
serde_json = { version = "1.0.140", features = ["raw_value"] }
toml = "0.8.20"
rocksdb = { version = "0.22.0", default-features = false, features = ["lz4"] }
serde_cbor = "0.11.2"
memmap2 = "0.9.10"
roxmltree = "0.20.0"
quick-xml = "0.37.2"
uuid = { version = "1.7.0", features = ["v4"] }
reqwest = { version = "0.12.12", default-features = false, features = ["blocking", "rustls-tls", "gzip", "brotli", "deflate"] }
pprof = { version = "0.14.1", optional = true, features = ["flamegraph", "prost-codec"] }
flate2 = { version = "1.0.35", optional = true }
tempfile = "3.16.0"

View File

@ -0,0 +1 @@
pub mod timing;

View File

@ -0,0 +1,353 @@
use std::collections::HashMap;
use std::path::Path;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use serde::{Deserialize, Serialize};
#[derive(Clone)]
pub struct TimingHandle {
inner: Arc<Mutex<TimingCollector>>,
}
impl TimingHandle {
pub fn new(meta: TimingMeta) -> Self {
Self {
inner: Arc::new(Mutex::new(TimingCollector::new(meta))),
}
}
pub fn span_phase(&self, phase: &'static str) -> TimingSpanGuard<'_> {
TimingSpanGuard {
handle: self.clone(),
kind: TimingSpanKind::Phase(phase),
start: Instant::now(),
}
}
pub fn span_rrdp_repo<'a>(&self, repo_uri: &'a str) -> TimingSpanGuard<'a> {
TimingSpanGuard {
handle: self.clone(),
kind: TimingSpanKind::RrdpRepo(repo_uri),
start: Instant::now(),
}
}
pub fn span_rrdp_repo_step<'a>(
&self,
repo_uri: &'a str,
step: &'static str,
) -> TimingSpanGuard<'a> {
TimingSpanGuard {
handle: self.clone(),
kind: TimingSpanKind::RrdpRepoStep { repo_uri, step },
start: Instant::now(),
}
}
pub fn span_publication_point<'a>(&self, manifest_rsync_uri: &'a str) -> TimingSpanGuard<'a> {
TimingSpanGuard {
handle: self.clone(),
kind: TimingSpanKind::PublicationPoint(manifest_rsync_uri),
start: Instant::now(),
}
}
pub fn set_meta(&self, update: TimingMetaUpdate<'_>) {
let mut g = self.inner.lock().expect("timing lock");
if let Some(v) = update.tal_url {
g.meta.tal_url = Some(v.to_string());
}
if let Some(v) = update.db_path {
g.meta.db_path = Some(v.to_string());
}
}
pub fn record_count(&self, key: &'static str, inc: u64) {
let mut g = self.inner.lock().expect("timing lock");
g.counts
.entry(key)
.and_modify(|v| *v = v.saturating_add(inc))
.or_insert(inc);
}
pub fn counts_snapshot(&self) -> HashMap<String, u64> {
let g = self.inner.lock().expect("timing lock");
g.counts
.iter()
.map(|(key, value)| ((*key).to_string(), *value))
.collect()
}
pub fn report_snapshot(&self, top_n: usize) -> TimingReportV1 {
let g = self.inner.lock().expect("timing lock");
g.to_report(top_n)
}
/// Record a phase duration directly in nanoseconds.
///
/// This is useful when aggregating sub-phase timings locally (to reduce lock contention)
/// and then emitting a single record per publication point.
pub fn record_phase_nanos(&self, phase: &'static str, nanos: u64) {
let mut g = self.inner.lock().expect("timing lock");
g.phases.record(phase, nanos);
}
pub fn record_publication_point_nanos(&self, manifest_rsync_uri: &str, nanos: u64) {
let mut g = self.inner.lock().expect("timing lock");
g.publication_points.record(manifest_rsync_uri, nanos);
}
pub fn record_publication_point_step_nanos(
&self,
manifest_rsync_uri: &str,
step: &'static str,
nanos: u64,
) {
let mut g = self.inner.lock().expect("timing lock");
g.publication_point_steps
.record(&format!("{manifest_rsync_uri}::{step}"), nanos);
}
pub fn write_json(&self, path: &Path, top_n: usize) -> Result<(), String> {
let report = {
let g = self.inner.lock().expect("timing lock");
g.to_report(top_n)
};
let f = std::fs::File::create(path)
.map_err(|e| format!("create timing json failed: {}: {e}", path.display()))?;
serde_json::to_writer_pretty(f, &report)
.map_err(|e| format!("write timing json failed: {e}"))?;
Ok(())
}
fn record_duration(&self, kind: TimingSpanKind<'_>, duration: Duration) {
let nanos_u64 = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
let mut g = self.inner.lock().expect("timing lock");
match kind {
TimingSpanKind::Phase(name) => g.phases.record(name, nanos_u64),
TimingSpanKind::RrdpRepo(uri) => g.rrdp_repos.record(uri, nanos_u64),
TimingSpanKind::RrdpRepoStep { repo_uri, step } => g
.rrdp_repo_steps
.record(&format!("{repo_uri}::{step}"), nanos_u64),
TimingSpanKind::PublicationPoint(uri) => g.publication_points.record(uri, nanos_u64),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct TimingMeta {
pub recorded_at_utc_rfc3339: String,
pub validation_time_utc_rfc3339: String,
pub tal_url: Option<String>,
pub db_path: Option<String>,
}
#[derive(Clone, Debug, Default)]
pub struct TimingMetaUpdate<'a> {
pub tal_url: Option<&'a str>,
pub db_path: Option<&'a str>,
}
pub struct TimingSpanGuard<'a> {
handle: TimingHandle,
kind: TimingSpanKind<'a>,
start: Instant,
}
impl Drop for TimingSpanGuard<'_> {
fn drop(&mut self) {
self.handle
.record_duration(self.kind.clone(), self.start.elapsed());
}
}
#[derive(Clone, Debug)]
enum TimingSpanKind<'a> {
Phase(&'static str),
RrdpRepo(&'a str),
RrdpRepoStep {
repo_uri: &'a str,
step: &'static str,
},
PublicationPoint(&'a str),
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct DurationStats {
pub count: u64,
pub total_nanos: u64,
}
impl DurationStats {
fn record(&mut self, nanos: u64) {
self.count = self.count.saturating_add(1);
self.total_nanos = self.total_nanos.saturating_add(nanos);
}
}
#[derive(Clone, Debug, Default)]
struct DurationStatsMap {
map: HashMap<String, DurationStats>,
}
impl DurationStatsMap {
fn record(&mut self, key: &str, nanos: u64) {
self.map.entry(key.to_string()).or_default().record(nanos);
}
fn top(&self, n: usize) -> Vec<TopDurationEntry> {
let mut v = self
.map
.iter()
.map(|(k, s)| TopDurationEntry {
key: k.clone(),
count: s.count,
total_nanos: s.total_nanos,
})
.collect::<Vec<_>>();
v.sort_by(|a, b| b.total_nanos.cmp(&a.total_nanos));
v.truncate(n);
v
}
}
struct TimingCollector {
meta: TimingMeta,
counts: HashMap<&'static str, u64>,
phases: DurationStatsMap,
rrdp_repos: DurationStatsMap,
rrdp_repo_steps: DurationStatsMap,
publication_points: DurationStatsMap,
publication_point_steps: DurationStatsMap,
}
impl TimingCollector {
fn new(meta: TimingMeta) -> Self {
Self {
meta,
counts: HashMap::new(),
phases: DurationStatsMap::default(),
rrdp_repos: DurationStatsMap::default(),
rrdp_repo_steps: DurationStatsMap::default(),
publication_points: DurationStatsMap::default(),
publication_point_steps: DurationStatsMap::default(),
}
}
fn to_report(&self, top_n: usize) -> TimingReportV1 {
TimingReportV1 {
format_version: 1,
meta: self.meta.clone(),
counts: self
.counts
.iter()
.map(|(k, v)| ((*k).to_string(), *v))
.collect(),
phases: self
.phases
.map
.iter()
.map(|(k, s)| (k.clone(), s.clone()))
.collect(),
top_rrdp_repos: self.rrdp_repos.top(top_n),
top_rrdp_repo_steps: self.rrdp_repo_steps.top(top_n),
top_publication_points: self.publication_points.top(top_n),
top_publication_point_steps: self.publication_point_steps.top(top_n),
}
}
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct TimingReportV1 {
pub format_version: u64,
pub meta: TimingMeta,
pub counts: HashMap<String, u64>,
pub phases: HashMap<String, DurationStats>,
pub top_rrdp_repos: Vec<TopDurationEntry>,
pub top_rrdp_repo_steps: Vec<TopDurationEntry>,
pub top_publication_points: Vec<TopDurationEntry>,
pub top_publication_point_steps: Vec<TopDurationEntry>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct TopDurationEntry {
pub key: String,
pub count: u64,
pub total_nanos: u64,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn timing_handle_writes_json_with_phases_and_tops() {
let meta = TimingMeta {
recorded_at_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
validation_time_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
tal_url: Some("https://example.test/x.tal".to_string()),
db_path: Some("db".to_string()),
};
let h = TimingHandle::new(meta);
{
let _p = h.span_phase("tal_bootstrap");
}
{
let _r = h.span_rrdp_repo("https://rrdp.example.test/notification.xml");
}
{
let _s = h.span_rrdp_repo_step(
"https://rrdp.example.test/notification.xml",
"fetch_notification",
);
}
{
let _pp = h.span_publication_point("rsync://example.test/repo/manifest.mft");
}
h.record_count("vrps", 42);
h.record_publication_point_nanos("rsync://example.test/repo/manifest.mft", 1_000_000);
h.record_publication_point_step_nanos(
"rsync://example.test/repo/manifest.mft",
"fresh_snapshot_prepare",
1_000_000,
);
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("timing.json");
h.write_json(&path, 10).expect("write_json");
let rep: TimingReportV1 =
serde_json::from_slice(&std::fs::read(&path).expect("read timing.json"))
.expect("parse timing.json");
assert_eq!(rep.format_version, 1);
assert!(rep.phases.contains_key("tal_bootstrap"));
assert_eq!(rep.counts.get("vrps").copied(), Some(42));
assert!(
rep.top_rrdp_repos
.iter()
.any(|e| e.key.contains("rrdp.example.test")),
"expected repo in top list"
);
assert!(
rep.top_rrdp_repo_steps
.iter()
.any(|e| e.key.contains("fetch_notification")),
"expected repo step in top list"
);
assert!(
rep.top_publication_points
.iter()
.any(|e| e.key.contains("manifest.mft")),
"expected PP in top list"
);
assert!(
rep.top_publication_point_steps
.iter()
.any(|e| e.key.contains("fresh_snapshot_prepare")),
"expected PP step in top list"
);
}
}

View File

@ -0,0 +1,342 @@
use serde::Serialize;
use sha2::Digest;
use crate::policy::Policy;
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum AuditObjectKind {
Manifest,
Crl,
Certificate,
RouterCertificate,
Roa,
Aspa,
Other,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum AuditObjectResult {
Ok,
Skipped,
Error,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct ObjectAuditEntry {
pub rsync_uri: String,
pub sha256_hex: String,
pub kind: AuditObjectKind,
pub result: AuditObjectResult,
#[serde(skip_serializing_if = "Option::is_none")]
pub detail: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct AuditWarning {
pub message: String,
pub category: String,
pub rfc_refs: Vec<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub context: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct QueryAuditManifest {
pub schema_version: u32,
pub status: String,
pub events_path: String,
pub events_count: u64,
pub events_sha256: String,
pub writer_version: u32,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ValidationEventCounts {
#[serde(skip_serializing_if = "Option::is_none")]
pub objects: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub warnings: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub vrps: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aspas: Option<u64>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ValidationEvent {
pub schema_version: u32,
pub seq: u64,
pub event_type: String,
pub validation_time: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub pp_node_id: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub pp_manifest_uri: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub pp_rsync_base_uri: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub repo_sync_phase: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub repo_terminal_state: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub object_uri: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub sha256: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub object_type: Option<AuditObjectKind>,
#[serde(skip_serializing_if = "Option::is_none")]
pub result: Option<AuditObjectResult>,
#[serde(skip_serializing_if = "Option::is_none")]
pub reason: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub counts: Option<ValidationEventCounts>,
}
impl From<&crate::report::Warning> for AuditWarning {
fn from(w: &crate::report::Warning) -> Self {
Self {
message: w.message.clone(),
category: w.category.as_str().to_string(),
rfc_refs: w.rfc_refs.iter().map(|r| r.0.to_string()).collect(),
context: w.context.clone(),
}
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct PublicationPointAudit {
/// Monotonic node ID assigned by the traversal engine.
///
/// Present when running via the Stage2 tree engine; may be absent in ad-hoc runs.
#[serde(skip_serializing_if = "Option::is_none")]
pub node_id: Option<u64>,
/// Parent node ID in the traversal tree.
#[serde(skip_serializing_if = "Option::is_none")]
pub parent_node_id: Option<u64>,
/// Provenance metadata for non-root nodes (how this CA instance was discovered).
#[serde(skip_serializing_if = "Option::is_none")]
pub discovered_from: Option<DiscoveredFrom>,
pub rsync_base_uri: String,
pub manifest_rsync_uri: String,
pub publication_point_rsync_uri: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub rrdp_notification_uri: Option<String>,
pub source: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub repo_sync_source: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub repo_sync_phase: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub repo_sync_duration_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub repo_sync_error: Option<String>,
pub repo_terminal_state: String,
pub this_update_rfc3339_utc: String,
pub next_update_rfc3339_utc: String,
pub verified_at_rfc3339_utc: String,
pub warnings: Vec<AuditWarning>,
pub objects: Vec<ObjectAuditEntry>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct DiscoveredFrom {
pub parent_manifest_rsync_uri: String,
pub child_ca_certificate_rsync_uri: String,
pub child_ca_certificate_sha256_hex: String,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct TreeSummary {
pub instances_processed: usize,
pub instances_failed: usize,
pub warnings: Vec<AuditWarning>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct AuditRunMeta {
pub validation_time_rfc3339_utc: String,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum AuditDownloadKind {
RrdpNotification,
RrdpSnapshot,
RrdpDelta,
Rsync,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct AuditDownloadObjectsStat {
pub objects_count: u64,
pub objects_bytes_total: u64,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct AuditDownloadEvent {
pub kind: AuditDownloadKind,
pub uri: String,
pub started_at_rfc3339_utc: String,
pub finished_at_rfc3339_utc: String,
pub duration_ms: u64,
pub success: bool,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub bytes: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub objects: Option<AuditDownloadObjectsStat>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct AuditDownloadKindStats {
pub ok_total: u64,
pub fail_total: u64,
pub duration_ms_total: u64,
#[serde(skip_serializing_if = "Option::is_none")]
pub bytes_total: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub objects_count_total: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub objects_bytes_total: Option<u64>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct AuditDownloadStats {
pub events_total: u64,
/// Statistics keyed by serialized `AuditDownloadKind` string (e.g. "rrdp_snapshot").
pub by_kind: std::collections::BTreeMap<String, AuditDownloadKindStats>,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn publication_point_audit_serializes_object_audit_only() {
let audit = PublicationPointAudit {
objects: vec![ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/fresh.roa".to_string(),
sha256_hex: "11".repeat(32),
kind: AuditObjectKind::Roa,
result: AuditObjectResult::Ok,
detail: None,
}],
..PublicationPointAudit::default()
};
let value = serde_json::to_value(&audit).expect("serialize audit");
assert!(value.get("objects").is_some());
assert!(value.get("cir_fresh_objects").is_none());
assert!(value.get("cir_cached_objects").is_none());
}
#[test]
fn warning_audit_keeps_the_warning_category() {
let warning = crate::report::Warning::new("BER-compatible CMS accepted")
.with_category(crate::report::WarningCategory::BerCompatibleCmsEncoding);
let audit_warning = AuditWarning::from(&warning);
assert_eq!(audit_warning.category, "ber_compatible_cms_encoding");
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct AuditRepoSyncStateStat {
pub count: u64,
pub duration_ms_total: u64,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct AuditRepoSyncStats {
pub publication_points_total: u64,
pub by_phase: std::collections::BTreeMap<String, AuditRepoSyncStateStat>,
pub by_terminal_state: std::collections::BTreeMap<String, AuditRepoSyncStateStat>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct AuditReportV1 {
pub format_version: u32,
pub meta: AuditRunMeta,
pub policy: Policy,
pub tree: TreeSummary,
pub publication_points: Vec<PublicationPointAudit>,
pub vrps: Vec<VrpOutput>,
pub aspas: Vec<AspaOutput>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct AuditReportV2 {
pub format_version: u32,
pub meta: AuditRunMeta,
pub policy: Policy,
pub tree: TreeSummary,
pub publication_points: Vec<PublicationPointAudit>,
pub vrps: Vec<VrpOutput>,
pub aspas: Vec<AspaOutput>,
pub downloads: Vec<AuditDownloadEvent>,
pub download_stats: AuditDownloadStats,
pub repo_sync_stats: AuditRepoSyncStats,
#[serde(rename = "queryAudit", skip_serializing_if = "Option::is_none")]
pub query_audit: Option<QueryAuditManifest>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct VrpOutput {
pub asn: u32,
pub prefix: String,
pub max_length: u16,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct AspaOutput {
pub customer_as_id: u32,
pub provider_as_ids: Vec<u32>,
}
pub fn sha256_hex_from_32(bytes: &[u8; 32]) -> String {
hex::encode(bytes)
}
pub fn sha256_hex(bytes: &[u8]) -> String {
let digest = sha2::Sha256::digest(bytes);
hex::encode(digest)
}
pub fn format_roa_ip_prefix(p: &crate::data_model::roa::IpPrefix) -> String {
let addr = p.addr_bytes();
match p.afi {
crate::data_model::roa::RoaAfi::Ipv4 => {
format!(
"{}.{}.{}.{}{}",
addr[0],
addr[1],
addr[2],
addr[3],
format!("/{}", p.prefix_len)
)
}
crate::data_model::roa::RoaAfi::Ipv6 => {
let mut parts = Vec::with_capacity(8);
for i in 0..8 {
let hi = addr[i * 2] as u16;
let lo = addr[i * 2 + 1] as u16;
parts.push(format!("{:x}", (hi << 8) | lo));
}
format!("{}{}", parts.join(":"), format!("/{}", p.prefix_len))
}
}
}

View File

@ -0,0 +1,170 @@
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use crate::audit::{
AuditDownloadEvent, AuditDownloadKind, AuditDownloadKindStats, AuditDownloadObjectsStat,
AuditDownloadStats,
};
#[derive(Clone, Debug, Default)]
pub struct DownloadLogHandle {
inner: Arc<Mutex<Vec<AuditDownloadEvent>>>,
}
impl DownloadLogHandle {
pub fn new() -> Self {
Self::default()
}
pub fn record_event(&self, event: AuditDownloadEvent) {
self.inner.lock().expect("download log lock").push(event);
}
pub fn snapshot_events(&self) -> Vec<AuditDownloadEvent> {
self.inner.lock().expect("download log lock").clone()
}
pub fn stats_from_events(events: &[AuditDownloadEvent]) -> AuditDownloadStats {
let mut out = AuditDownloadStats {
events_total: events.len() as u64,
by_kind: BTreeMap::new(),
};
for e in events {
let kind_key = match e.kind {
AuditDownloadKind::RrdpNotification => "rrdp_notification",
AuditDownloadKind::RrdpSnapshot => "rrdp_snapshot",
AuditDownloadKind::RrdpDelta => "rrdp_delta",
AuditDownloadKind::Rsync => "rsync",
}
.to_string();
let st = out
.by_kind
.entry(kind_key)
.or_insert_with(|| AuditDownloadKindStats {
ok_total: 0,
fail_total: 0,
duration_ms_total: 0,
bytes_total: None,
objects_count_total: None,
objects_bytes_total: None,
});
if e.success {
st.ok_total = st.ok_total.saturating_add(1);
} else {
st.fail_total = st.fail_total.saturating_add(1);
}
st.duration_ms_total = st.duration_ms_total.saturating_add(e.duration_ms);
if let Some(b) = e.bytes {
st.bytes_total = Some(st.bytes_total.unwrap_or(0).saturating_add(b));
}
if let Some(objects) = &e.objects {
st.objects_count_total = Some(
st.objects_count_total
.unwrap_or(0)
.saturating_add(objects.objects_count),
);
st.objects_bytes_total = Some(
st.objects_bytes_total
.unwrap_or(0)
.saturating_add(objects.objects_bytes_total),
);
}
}
out
}
pub fn stats(&self) -> AuditDownloadStats {
let events = self.snapshot_events();
Self::stats_from_events(&events)
}
pub fn span_download<'a>(
&'a self,
kind: AuditDownloadKind,
uri: &'a str,
) -> DownloadSpanGuard<'a> {
DownloadSpanGuard {
handle: self,
kind,
uri,
start_instant: Instant::now(),
started_at: time::OffsetDateTime::now_utc(),
bytes: None,
objects: None,
error: None,
success: None,
}
}
}
pub struct DownloadSpanGuard<'a> {
handle: &'a DownloadLogHandle,
kind: AuditDownloadKind,
uri: &'a str,
start_instant: Instant,
started_at: time::OffsetDateTime,
bytes: Option<u64>,
objects: Option<AuditDownloadObjectsStat>,
error: Option<String>,
success: Option<bool>,
}
impl DownloadSpanGuard<'_> {
pub fn set_bytes(&mut self, bytes: u64) {
self.bytes = Some(bytes);
}
pub fn set_objects(&mut self, objects_count: u64, objects_bytes_total: u64) {
self.objects = Some(AuditDownloadObjectsStat {
objects_count,
objects_bytes_total,
});
}
pub fn set_ok(&mut self) {
self.success = Some(true);
}
pub fn set_err(&mut self, msg: impl Into<String>) {
self.success = Some(false);
self.error = Some(msg.into());
}
}
impl Drop for DownloadSpanGuard<'_> {
fn drop(&mut self) {
use time::format_description::well_known::Rfc3339;
let finished_at = time::OffsetDateTime::now_utc();
let dur = self.start_instant.elapsed();
let duration_ms = duration_to_ms(dur);
let started_at_rfc3339_utc = self
.started_at
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.unwrap_or_else(|_| "<format-error>".to_string());
let finished_at_rfc3339_utc = finished_at
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.unwrap_or_else(|_| "<format-error>".to_string());
let success = self.success.unwrap_or(false);
let event = AuditDownloadEvent {
kind: self.kind.clone(),
uri: self.uri.to_string(),
started_at_rfc3339_utc,
finished_at_rfc3339_utc,
duration_ms,
success,
error: if success { None } else { self.error.clone() },
bytes: self.bytes,
objects: self.objects.clone(),
};
self.handle.record_event(event);
}
}
fn duration_to_ms(d: Duration) -> u64 {
let ms = d.as_millis();
ms.min(u128::from(u64::MAX)) as u64
}

View File

@ -0,0 +1,6 @@
fn main() {
let code = panda_rpki_validator::daemon::main_entry();
if code != 0 {
std::process::exit(code);
}
}

View File

@ -0,0 +1,839 @@
use std::path::PathBuf;
use std::sync::Arc;
use rocksdb::{DB, Options, WriteBatch};
use crate::storage::{
RawByHashEntry, RocksDbMemoryDbSnapshot, RocksStore, StorageError, StorageResult,
memory_db_snapshot_for_column_families,
};
const RAW_BY_HASH_KEY_PREFIX: &str = "rawbyhash:";
const RAW_BLOB_KEY_PREFIX: &str = "rawblob:";
const REPO_BYTES_KEY_PREFIX: &str = "sha256:";
fn raw_by_hash_key(sha256_hex: &str) -> String {
format!("{RAW_BY_HASH_KEY_PREFIX}{sha256_hex}")
}
fn raw_blob_key(sha256_hex: &str) -> String {
format!("{RAW_BLOB_KEY_PREFIX}{sha256_hex}")
}
fn repo_bytes_key(sha256_hex: &str) -> String {
format!("{REPO_BYTES_KEY_PREFIX}{sha256_hex}")
}
fn validate_blob_sha256_hex(sha256_hex: &str) -> StorageResult<()> {
if sha256_hex.len() != 64 || !sha256_hex.as_bytes().iter().all(u8::is_ascii_hexdigit) {
return Err(StorageError::InvalidData {
entity: "raw_blob",
detail: format!("invalid sha256 hex: {sha256_hex}"),
});
}
Ok(())
}
fn validate_blob_bytes(bytes: &[u8]) -> StorageResult<()> {
if bytes.is_empty() {
return Err(StorageError::InvalidData {
entity: "raw_blob",
detail: "bytes must not be empty".to_string(),
});
}
Ok(())
}
pub trait RawObjectStore {
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>>;
fn get_raw_entries_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<RawByHashEntry>>>;
fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
self.get_raw_entry(sha256_hex)
.map(|entry| entry.map(|entry| entry.bytes))
}
fn get_blob_bytes_batch(&self, sha256_hexes: &[String]) -> StorageResult<Vec<Option<Vec<u8>>>> {
self.get_raw_entries_batch(sha256_hexes).map(|entries| {
entries
.into_iter()
.map(|entry| entry.map(|entry| entry.bytes))
.collect()
})
}
}
#[derive(Clone, Debug)]
pub struct ExternalRawStoreDb {
path: PathBuf,
db: Arc<DB>,
}
#[derive(Clone, Debug)]
pub struct ExternalRepoBytesDb {
path: PathBuf,
db: Arc<DB>,
read_only: bool,
secondary: bool,
}
impl ExternalRawStoreDb {
pub fn open(path: impl Into<PathBuf>) -> StorageResult<Self> {
let path = path.into();
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
}
let mut opts = Options::default();
opts.create_if_missing(true);
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
opts.set_max_open_files(512);
let db = DB::open(&opts, &path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(Self {
path,
db: Arc::new(db),
})
}
pub fn put_raw_entry(&self, entry: &RawByHashEntry) -> StorageResult<()> {
entry.validate_internal()?;
let key = raw_by_hash_key(&entry.sha256_hex);
let blob_key = raw_blob_key(&entry.sha256_hex);
let value = serde_cbor::to_vec(entry).map_err(|e| StorageError::Codec {
entity: "raw_by_hash",
detail: e.to_string(),
})?;
let blob_value = entry.bytes.clone();
self.db
.write({
let mut batch = WriteBatch::default();
batch.put(key.as_bytes(), value);
batch.put(blob_key.as_bytes(), blob_value);
batch
})
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn put_raw_entries_batch(&self, entries: &[RawByHashEntry]) -> StorageResult<()> {
if entries.is_empty() {
return Ok(());
}
let mut batch = WriteBatch::default();
for entry in entries {
entry.validate_internal()?;
let key = raw_by_hash_key(&entry.sha256_hex);
let blob_key = raw_blob_key(&entry.sha256_hex);
let value = serde_cbor::to_vec(entry).map_err(|e| StorageError::Codec {
entity: "raw_by_hash",
detail: e.to_string(),
})?;
batch.put(key.as_bytes(), value);
batch.put(blob_key.as_bytes(), entry.bytes.as_slice());
}
self.db
.write(batch)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn put_blob_bytes_batch(&self, blobs: &[(String, Vec<u8>)]) -> StorageResult<()> {
if blobs.is_empty() {
return Ok(());
}
let mut batch = WriteBatch::default();
for (sha256_hex, bytes) in blobs {
validate_blob_sha256_hex(sha256_hex)?;
validate_blob_bytes(bytes)?;
let blob_key = raw_blob_key(sha256_hex);
batch.put(blob_key.as_bytes(), bytes.as_slice());
}
self.db
.write(batch)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn delete_raw_entry(&self, sha256_hex: &str) -> StorageResult<()> {
let key = raw_by_hash_key(sha256_hex);
let blob_key = raw_blob_key(sha256_hex);
self.db
.write({
let mut batch = WriteBatch::default();
batch.delete(key.as_bytes());
batch.delete(blob_key.as_bytes());
batch
})
.map_err(|e| StorageError::RocksDb(e.to_string()))
}
pub fn path(&self) -> &PathBuf {
&self.path
}
pub(crate) fn memory_snapshot(&self, label: impl Into<String>) -> RocksDbMemoryDbSnapshot {
memory_db_snapshot_for_column_families(label, self.db.as_ref(), None)
}
}
impl ExternalRepoBytesDb {
pub fn open(path: impl Into<PathBuf>) -> StorageResult<Self> {
let path = path.into();
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
}
let mut opts = Options::default();
opts.create_if_missing(true);
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
opts.set_max_open_files(512);
let db = DB::open(&opts, &path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(Self {
path,
db: Arc::new(db),
read_only: false,
secondary: false,
})
}
pub fn open_read_only(path: impl Into<PathBuf>) -> StorageResult<Self> {
let path = path.into();
let mut opts = Options::default();
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
opts.set_max_open_files(512);
let db = DB::open_for_read_only(&opts, &path, false)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(Self {
path,
db: Arc::new(db),
read_only: true,
secondary: false,
})
}
/// Open the repo-bytes DB as a RocksDB secondary instance.
///
/// Unlike `open_read_only` (a frozen point-in-time view), a secondary
/// instance can follow the live primary via `try_catch_up_with_primary`,
/// which is required when the soak keeps writing new object bytes while
/// the query service is running.
pub fn open_as_secondary(
path: impl Into<PathBuf>,
secondary_path: impl Into<PathBuf>,
) -> StorageResult<Self> {
let path = path.into();
let secondary_path = secondary_path.into();
if let Some(parent) = secondary_path.parent() {
std::fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
}
let mut opts = Options::default();
opts.set_compression_type(rocksdb::DBCompressionType::Lz4);
opts.set_max_open_files(512);
let db = DB::open_as_secondary(&opts, &path, &secondary_path)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(Self {
path,
db: Arc::new(db),
read_only: true,
secondary: true,
})
}
/// Pull the secondary view up to the primary's current state. No-op for
/// non-secondary handles.
pub fn try_catch_up_with_primary(&self) -> StorageResult<()> {
if self.secondary {
self.db
.try_catch_up_with_primary()
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
}
Ok(())
}
pub(crate) fn is_secondary(&self) -> bool {
self.secondary
}
pub fn put_blob_bytes_batch(&self, blobs: &[(String, Vec<u8>)]) -> StorageResult<()> {
if blobs.is_empty() {
return Ok(());
}
if self.read_only {
return Err(StorageError::RocksDb(format!(
"repo-bytes DB is read-only: {}",
self.path.display()
)));
}
let mut batch = WriteBatch::default();
for (sha256_hex, bytes) in blobs {
validate_blob_sha256_hex(sha256_hex)?;
validate_blob_bytes(bytes)?;
let key = repo_bytes_key(sha256_hex);
batch.put(key.as_bytes(), bytes.as_slice());
}
self.db
.write(batch)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub(crate) fn is_read_only(&self) -> bool {
self.read_only
}
pub(crate) fn require_existing_blob_bytes_batch(
&self,
blobs: &[(String, Vec<u8>)],
) -> StorageResult<()> {
if blobs.is_empty() {
return Ok(());
}
let hashes = blobs
.iter()
.map(|(sha256_hex, _)| sha256_hex.clone())
.collect::<Vec<_>>();
let existing = self.get_blob_bytes_batch(&hashes)?;
for ((sha256_hex, expected_bytes), actual_bytes) in blobs.iter().zip(existing) {
match actual_bytes {
Some(actual_bytes) if actual_bytes == *expected_bytes => {}
Some(_) => {
return Err(StorageError::InvalidData {
entity: "read_only_repo_bytes",
detail: format!("existing bytes differ for SHA-256 {sha256_hex}"),
});
}
None => {
return Err(StorageError::InvalidData {
entity: "read_only_repo_bytes",
detail: format!("blob is missing for SHA-256 {sha256_hex}"),
});
}
}
}
Ok(())
}
pub fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
validate_blob_sha256_hex(sha256_hex)?;
let key = repo_bytes_key(sha256_hex);
self.db
.get(key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))
}
pub fn get_blob_bytes_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<Vec<u8>>>> {
if sha256_hexes.is_empty() {
return Ok(Vec::new());
}
let keys: Vec<String> = sha256_hexes
.iter()
.map(|hash| {
validate_blob_sha256_hex(hash)?;
Ok::<String, StorageError>(repo_bytes_key(hash))
})
.collect::<Result<_, _>>()?;
self.db
.multi_get(keys.iter().map(|key| key.as_bytes()))
.into_iter()
.map(|res| res.map_err(|e| StorageError::RocksDb(e.to_string())))
.collect()
}
pub fn path(&self) -> &PathBuf {
&self.path
}
pub(crate) fn memory_snapshot(&self, label: impl Into<String>) -> RocksDbMemoryDbSnapshot {
memory_db_snapshot_for_column_families(label, self.db.as_ref(), None)
}
}
impl RawObjectStore for RocksStore {
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
self.get_raw_by_hash_entry(sha256_hex)
}
fn get_raw_entries_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
self.get_raw_by_hash_entries_batch(sha256_hexes)
}
fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
RocksStore::get_blob_bytes(self, sha256_hex)
}
fn get_blob_bytes_batch(&self, sha256_hexes: &[String]) -> StorageResult<Vec<Option<Vec<u8>>>> {
RocksStore::get_blob_bytes_batch(self, sha256_hexes)
}
}
impl RawObjectStore for ExternalRawStoreDb {
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
let key = raw_by_hash_key(sha256_hex);
let Some(bytes) = self
.db
.get(key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let entry =
serde_cbor::from_slice::<RawByHashEntry>(&bytes).map_err(|e| StorageError::Codec {
entity: "raw_by_hash",
detail: e.to_string(),
})?;
entry.validate_internal()?;
Ok(Some(entry))
}
fn get_raw_entries_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
if sha256_hexes.is_empty() {
return Ok(Vec::new());
}
let keys: Vec<String> = sha256_hexes
.iter()
.map(|hash| raw_by_hash_key(hash))
.collect();
self.db
.multi_get(keys.iter().map(|key| key.as_bytes()))
.into_iter()
.map(|res| {
let maybe = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
match maybe {
Some(bytes) => {
let entry =
serde_cbor::from_slice::<RawByHashEntry>(&bytes).map_err(|e| {
StorageError::Codec {
entity: "raw_by_hash",
detail: e.to_string(),
}
})?;
entry.validate_internal()?;
Ok(Some(entry))
}
None => Ok(None),
}
})
.collect()
}
fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
let key = raw_blob_key(sha256_hex);
self.db
.get(key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))
}
fn get_blob_bytes_batch(&self, sha256_hexes: &[String]) -> StorageResult<Vec<Option<Vec<u8>>>> {
if sha256_hexes.is_empty() {
return Ok(Vec::new());
}
let keys: Vec<String> = sha256_hexes.iter().map(|hash| raw_blob_key(hash)).collect();
self.db
.multi_get(keys.iter().map(|key| key.as_bytes()))
.into_iter()
.map(|res| res.map_err(|e| StorageError::RocksDb(e.to_string())))
.collect()
}
}
#[cfg(test)]
mod tests {
use super::{ExternalRawStoreDb, ExternalRepoBytesDb, RawObjectStore};
use crate::storage::{RawByHashEntry, RocksStore, StorageError, StorageResult};
use std::collections::HashMap;
fn sha256_hex(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
hex::encode(Sha256::digest(bytes))
}
#[derive(Default)]
struct MockRawStore {
entries: HashMap<String, RawByHashEntry>,
}
impl RawObjectStore for MockRawStore {
fn get_raw_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
Ok(self.entries.get(sha256_hex).cloned())
}
fn get_raw_entries_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
Ok(sha256_hexes
.iter()
.map(|hash| self.entries.get(hash).cloned())
.collect())
}
}
#[test]
fn rocks_store_raw_object_store_reads_single_and_batch_entries() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let a = b"object-a".to_vec();
let b = b"object-b".to_vec();
let a_hash = sha256_hex(&a);
let b_hash = sha256_hex(&b);
store
.put_raw_by_hash_entry(&RawByHashEntry::from_bytes(a_hash.clone(), a.clone()))
.expect("put a");
store
.put_raw_by_hash_entry(&RawByHashEntry::from_bytes(b_hash.clone(), b.clone()))
.expect("put b");
let single = store
.get_raw_entry(&a_hash)
.expect("get single")
.expect("present");
assert_eq!(single.bytes, a);
let batch = store
.get_raw_entries_batch(&[a_hash.clone(), "00".repeat(32), b_hash.clone()])
.expect("get batch");
assert_eq!(batch.len(), 3);
assert_eq!(
batch[0].as_ref().map(|entry| entry.bytes.as_slice()),
Some(a.as_slice())
);
assert!(batch[1].is_none());
assert_eq!(
batch[2].as_ref().map(|entry| entry.bytes.as_slice()),
Some(b.as_slice())
);
}
#[test]
fn external_raw_store_db_roundtrips_entries() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let mut entry = RawByHashEntry::from_bytes(sha256_hex(b"blob"), b"blob".to_vec());
entry
.origin_uris
.push("rsync://example.test/repo/a.cer".to_string());
entry.object_type = Some("cer".to_string());
raw_store.put_raw_entry(&entry).expect("put raw entry");
let got = raw_store
.get_raw_entry(&entry.sha256_hex)
.expect("read raw entry")
.expect("entry exists");
assert_eq!(got, entry);
}
#[test]
fn external_raw_store_db_batch_writes_and_reads() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let a = RawByHashEntry::from_bytes(sha256_hex(b"a"), b"a".to_vec());
let b = RawByHashEntry::from_bytes(sha256_hex(b"b"), b"b".to_vec());
raw_store
.put_raw_entries_batch(&[a.clone(), b.clone()])
.expect("batch put");
let batch = raw_store
.get_raw_entries_batch(&[a.sha256_hex.clone(), b.sha256_hex.clone()])
.expect("batch get");
assert_eq!(batch.len(), 2);
assert_eq!(batch[0], Some(a));
assert_eq!(batch[1], Some(b));
}
#[test]
fn raw_object_store_default_blob_helpers_return_bytes_only() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store = ExternalRawStoreDb::open(td.path().join("nested/raw-store.db"))
.expect("open raw store");
let mut entry = RawByHashEntry::from_bytes(sha256_hex(b"blob"), b"blob".to_vec());
entry
.origin_uris
.push("rsync://example.test/repo/blob.roa".to_string());
raw_store.put_raw_entry(&entry).expect("put raw entry");
let single = raw_store
.get_blob_bytes(&entry.sha256_hex)
.expect("get blob bytes")
.expect("entry exists");
assert_eq!(single, b"blob".to_vec());
let batch = raw_store
.get_blob_bytes_batch(&[entry.sha256_hex.clone(), "00".repeat(32)])
.expect("get blob bytes batch");
assert_eq!(batch, vec![Some(b"blob".to_vec()), None]);
}
#[test]
fn raw_object_store_default_blob_helpers_work_for_custom_store() {
let mut store = MockRawStore::default();
let a = RawByHashEntry::from_bytes(sha256_hex(b"a"), b"a".to_vec());
let b = RawByHashEntry::from_bytes(sha256_hex(b"b"), b"b".to_vec());
store.entries.insert(a.sha256_hex.clone(), a.clone());
store.entries.insert(b.sha256_hex.clone(), b.clone());
let single = store
.get_blob_bytes(&a.sha256_hex)
.expect("single blob bytes")
.expect("present");
assert_eq!(single, b"a".to_vec());
let batch = store
.get_blob_bytes_batch(&[a.sha256_hex.clone(), "00".repeat(32), b.sha256_hex.clone()])
.expect("batch blob bytes");
assert_eq!(batch, vec![Some(b"a".to_vec()), None, Some(b"b".to_vec())]);
}
#[test]
fn rocks_store_blob_helpers_use_external_raw_store_fast_path() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open_with_external_raw_store(
&td.path().join("db"),
&td.path().join("raw-store.db"),
)
.expect("open store with external raw store");
let entry = RawByHashEntry::from_bytes(sha256_hex(b"blob-fast"), b"blob-fast".to_vec());
store.put_raw_by_hash_entry(&entry).expect("put");
let single = store
.get_blob_bytes(&entry.sha256_hex)
.expect("single blob bytes")
.expect("present");
assert_eq!(single, b"blob-fast".to_vec());
let batch = store
.get_blob_bytes_batch(&[entry.sha256_hex.clone(), "00".repeat(32)])
.expect("batch blob bytes");
assert_eq!(batch, vec![Some(b"blob-fast".to_vec()), None]);
}
#[test]
fn external_raw_store_db_delete_removes_entry() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let entry = RawByHashEntry::from_bytes(sha256_hex(b"gone"), b"gone".to_vec());
raw_store.put_raw_entry(&entry).expect("put");
assert!(
raw_store
.get_raw_entry(&entry.sha256_hex)
.unwrap()
.is_some()
);
raw_store
.delete_raw_entry(&entry.sha256_hex)
.expect("delete entry");
assert!(
raw_store
.get_raw_entry(&entry.sha256_hex)
.unwrap()
.is_none()
);
}
#[test]
fn put_blob_bytes_batch_round_trips_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let a = (sha256_hex(b"blob-a"), b"blob-a".to_vec());
let b = (sha256_hex(b"blob-b"), b"blob-b".to_vec());
raw_store
.put_blob_bytes_batch(&[a.clone(), b.clone()])
.expect("put blobs");
assert_eq!(
raw_store.get_blob_bytes(&a.0).expect("get blob a"),
Some(a.1.clone())
);
assert_eq!(
raw_store.get_blob_bytes(&b.0).expect("get blob b"),
Some(b.1.clone())
);
assert!(raw_store.get_raw_entry(&a.0).expect("get raw a").is_none());
assert!(raw_store.get_raw_entry(&b.0).expect("get raw b").is_none());
}
#[test]
fn put_blob_bytes_batch_rejects_invalid_inputs() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let err = raw_store
.put_blob_bytes_batch(&[("zz".repeat(32), b"blob".to_vec())])
.expect_err("invalid hash should fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
let err = raw_store
.put_blob_bytes_batch(&[(sha256_hex(b"blob"), Vec::new())])
.expect_err("empty bytes should fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn external_raw_store_db_rejects_invalid_entry_on_put() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let bad = RawByHashEntry {
sha256_hex: "11".repeat(32),
bytes: b"blob".to_vec(),
origin_uris: Vec::new(),
object_type: None,
encoding: None,
};
let err = raw_store
.put_raw_entry(&bad)
.expect_err("invalid hash should fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn external_raw_store_db_reports_codec_error_for_corrupt_value() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
raw_store
.db
.put(b"rawbyhash:deadbeef", b"not-cbor")
.expect("inject corrupt bytes");
let err = raw_store
.get_raw_entry("deadbeef")
.expect_err("corrupt value should fail");
assert!(matches!(
err,
StorageError::Codec {
entity: "raw_by_hash",
..
}
));
}
#[test]
fn external_raw_store_db_batch_returns_empty_for_empty_request() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store =
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store");
let entries = raw_store
.get_raw_entries_batch(&[])
.expect("empty batch succeeds");
assert!(entries.is_empty());
raw_store
.put_raw_entries_batch(&[])
.expect("empty put succeeds");
}
#[test]
fn external_repo_bytes_db_roundtrips_blob_bytes_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let repo_bytes =
ExternalRepoBytesDb::open(td.path().join("repo-bytes.db")).expect("open repo bytes");
let bytes = b"repo-bytes-object".to_vec();
let hash = sha256_hex(&bytes);
repo_bytes
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("put repo bytes");
assert_eq!(
repo_bytes.get_blob_bytes(&hash).expect("get repo bytes"),
Some(bytes.clone())
);
assert_eq!(
repo_bytes
.get_blob_bytes_batch(&[hash, "00".repeat(32)])
.expect("get repo bytes batch"),
vec![Some(bytes), None]
);
}
#[test]
fn external_repo_bytes_db_rejects_invalid_inputs() {
let td = tempfile::tempdir().expect("tempdir");
let repo_bytes =
ExternalRepoBytesDb::open(td.path().join("repo-bytes.db")).expect("open repo bytes");
assert!(
repo_bytes
.put_blob_bytes_batch(&[("not-a-valid-hash".to_string(), b"blob".to_vec())])
.is_err()
);
assert!(
repo_bytes
.put_blob_bytes_batch(&[(sha256_hex(b"blob"), Vec::new())])
.is_err()
);
assert!(repo_bytes.get_blob_bytes("not-a-valid-hash").is_err());
}
#[test]
fn external_repo_bytes_db_secondary_catches_up_with_live_primary() {
let td = tempfile::tempdir().expect("tempdir");
let primary_path = td.path().join("repo-bytes.db");
let secondary_path = td.path().join("repo-bytes.secondary");
let primary = ExternalRepoBytesDb::open(&primary_path).expect("open primary");
let bytes_a = b"repo-bytes-a".to_vec();
let hash_a = sha256_hex(&bytes_a);
primary
.put_blob_bytes_batch(&[(hash_a.clone(), bytes_a.clone())])
.expect("put a");
let secondary = ExternalRepoBytesDb::open_as_secondary(&primary_path, &secondary_path)
.expect("open secondary");
assert!(secondary.is_secondary());
// A secondary open does not necessarily see pre-existing data until it
// catches up with the primary's manifest.
secondary
.try_catch_up_with_primary()
.expect("initial catch up");
assert_eq!(
secondary
.get_blob_bytes(&hash_a)
.expect("get a via secondary"),
Some(bytes_a)
);
// Bytes written by the primary *after* the secondary open become
// visible after another catch-up (this is the live-soak scenario).
let bytes_b = b"repo-bytes-b".to_vec();
let hash_b = sha256_hex(&bytes_b);
primary
.put_blob_bytes_batch(&[(hash_b.clone(), bytes_b.clone())])
.expect("put b after secondary open");
secondary
.try_catch_up_with_primary()
.expect("second catch up");
assert_eq!(
secondary
.get_blob_bytes(&hash_b)
.expect("get b via secondary"),
Some(bytes_b)
);
}
}

View File

@ -0,0 +1,516 @@
use std::collections::BTreeMap;
use crate::ccr::build::{
build_aspa_payload_state, build_roa_payload_state, build_router_key_state_from_runtime,
build_trust_anchor_state,
};
use crate::ccr::encode::encode_manifest_state_payload_der;
use crate::ccr::hash::compute_state_hash;
use crate::ccr::manifest_location::select_manifest_signed_object_location_from_der;
use crate::ccr::model::{
CcrDigestAlgorithm, ManifestInstance, ManifestState, RpkiCanonicalCacheRepresentation,
};
use crate::data_model::common::BigUnsigned;
use crate::data_model::ta::TrustAnchor;
use crate::storage::VcirCcrManifestProjection;
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CcrManifestContribution {
pub manifest_rsync_uri: String,
pub hash: Vec<u8>,
pub size: u64,
pub aki: Vec<u8>,
pub manifest_number_be: Vec<u8>,
pub this_update: time::OffsetDateTime,
pub locations_der: Vec<Vec<u8>>,
pub subordinate_skis: Vec<Vec<u8>>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct CcrAccumulatorMemoryStats {
pub trust_anchor_count: u64,
pub manifest_count: u64,
pub estimated_heap_bytes: u64,
pub string_bytes: u64,
pub string_capacity_bytes: u64,
pub vec_payload_bytes: u64,
pub vec_capacity_bytes: u64,
pub locations_der_count: u64,
pub subordinate_ski_count: u64,
pub btree_key_capacity_bytes: u64,
pub btree_entry_shallow_bytes: u64,
}
impl CcrManifestContribution {
fn from_projection(projection: &VcirCcrManifestProjection) -> Result<Self, String> {
let this_update = projection
.manifest_this_update
.parse()
.map_err(|e| format!("parse projection manifest_this_update failed: {e}"))?;
Ok(Self {
manifest_rsync_uri: projection.manifest_rsync_uri.clone(),
hash: projection.manifest_sha256.clone(),
size: projection.manifest_size,
aki: projection.manifest_ee_aki.clone(),
manifest_number_be: projection.manifest_number_be.clone(),
this_update,
locations_der: vec![select_manifest_signed_object_location_from_der(
&projection.manifest_rsync_uri,
&projection.manifest_sia_locations_der,
)?],
subordinate_skis: projection.subordinate_skis.clone(),
})
}
fn to_manifest_instance(&self) -> ManifestInstance {
ManifestInstance {
hash: self.hash.clone(),
size: self.size,
aki: self.aki.clone(),
manifest_number: BigUnsigned {
bytes_be: self.manifest_number_be.clone(),
},
this_update: self.this_update,
locations: self.locations_der.clone(),
subordinates: self.subordinate_skis.clone(),
}
}
fn add_memory_stats(&self, stats: &mut CcrAccumulatorMemoryStats) {
stats.string_bytes += self.manifest_rsync_uri.len() as u64;
stats.string_capacity_bytes += self.manifest_rsync_uri.capacity() as u64;
stats.estimated_heap_bytes += self.manifest_rsync_uri.capacity() as u64;
add_vec_stats(&self.hash, stats);
add_vec_stats(&self.aki, stats);
add_vec_stats(&self.manifest_number_be, stats);
add_vec_of_vec_stats(&self.locations_der, stats);
add_vec_of_vec_stats(&self.subordinate_skis, stats);
stats.locations_der_count += self.locations_der.len() as u64;
stats.subordinate_ski_count += self.subordinate_skis.len() as u64;
}
}
fn add_vec_stats(value: &Vec<u8>, stats: &mut CcrAccumulatorMemoryStats) {
stats.vec_payload_bytes += value.len() as u64;
stats.vec_capacity_bytes += value.capacity() as u64;
stats.estimated_heap_bytes += value.capacity() as u64;
}
fn add_vec_of_vec_stats(values: &Vec<Vec<u8>>, stats: &mut CcrAccumulatorMemoryStats) {
let outer_capacity = values.capacity() * std::mem::size_of::<Vec<u8>>();
stats.vec_payload_bytes += (values.len() * std::mem::size_of::<Vec<u8>>()) as u64;
stats.vec_capacity_bytes += outer_capacity as u64;
stats.estimated_heap_bytes += outer_capacity as u64;
for value in values {
add_vec_stats(value, stats);
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CcrAccumulator {
trust_anchors: Vec<TrustAnchor>,
manifests_by_hash: BTreeMap<Vec<u8>, CcrManifestContribution>,
most_recent_update: time::OffsetDateTime,
}
impl CcrAccumulator {
pub fn new(trust_anchors: Vec<TrustAnchor>) -> Self {
Self {
trust_anchors,
manifests_by_hash: BTreeMap::new(),
most_recent_update: time::OffsetDateTime::UNIX_EPOCH,
}
}
pub fn append_manifest_projection(
&mut self,
projection: &VcirCcrManifestProjection,
) -> Result<(), String> {
let contribution = CcrManifestContribution::from_projection(projection)?;
match self.manifests_by_hash.get(contribution.hash.as_slice()) {
Some(existing) if existing != &contribution => {
return Err(format!(
"duplicate manifest hash with conflicting content for URI: {}",
contribution.manifest_rsync_uri
));
}
Some(_) => {}
None => {
self.manifests_by_hash
.insert(contribution.hash.clone(), contribution.clone());
}
}
if contribution.this_update > self.most_recent_update {
self.most_recent_update = contribution.this_update;
}
Ok(())
}
pub fn finish(
&self,
produced_at: time::OffsetDateTime,
vrps: &[Vrp],
aspas: &[AspaAttestation],
router_keys: &[RouterKeyPayload],
) -> Result<RpkiCanonicalCacheRepresentation, String> {
let manifest_instances = self
.manifests_by_hash
.values()
.map(CcrManifestContribution::to_manifest_instance)
.collect::<Vec<_>>();
let manifest_payload_der = encode_manifest_state_payload_der(&manifest_instances)
.map_err(|e| format!("manifest state encoding failed: {e}"))?;
let manifest_state = ManifestState {
mis: manifest_instances,
most_recent_update: self.most_recent_update,
hash: compute_state_hash(&manifest_payload_der),
};
let vrp_state = build_roa_payload_state(vrps).map_err(|e| e.to_string())?;
let aspa_state = build_aspa_payload_state(aspas).map_err(|e| e.to_string())?;
let ta_state = build_trust_anchor_state(&self.trust_anchors).map_err(|e| e.to_string())?;
let router_key_state =
build_router_key_state_from_runtime(router_keys).map_err(|e| e.to_string())?;
Ok(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at,
mfts: Some(manifest_state),
vrps: Some(vrp_state),
vaps: Some(aspa_state),
tas: Some(ta_state),
rks: Some(router_key_state),
})
}
pub fn manifest_count(&self) -> usize {
self.manifests_by_hash.len()
}
pub fn memory_stats(&self) -> CcrAccumulatorMemoryStats {
let mut stats = CcrAccumulatorMemoryStats {
trust_anchor_count: self.trust_anchors.len() as u64,
manifest_count: self.manifests_by_hash.len() as u64,
..CcrAccumulatorMemoryStats::default()
};
stats.estimated_heap_bytes +=
(self.trust_anchors.capacity() * std::mem::size_of::<TrustAnchor>()) as u64;
for trust_anchor in &self.trust_anchors {
add_vec_stats(&trust_anchor.tal.raw, &mut stats);
add_vec_of_string_stats(&trust_anchor.tal.comments, &mut stats);
stats.vec_payload_bytes +=
(trust_anchor.tal.ta_uris.len() * std::mem::size_of::<url::Url>()) as u64;
stats.vec_capacity_bytes +=
(trust_anchor.tal.ta_uris.capacity() * std::mem::size_of::<url::Url>()) as u64;
stats.estimated_heap_bytes +=
(trust_anchor.tal.ta_uris.capacity() * std::mem::size_of::<url::Url>()) as u64;
for uri in &trust_anchor.tal.ta_uris {
stats.string_bytes += uri.as_str().len() as u64;
stats.string_capacity_bytes += uri.as_str().len() as u64;
stats.estimated_heap_bytes += uri.as_str().len() as u64;
}
add_vec_stats(&trust_anchor.tal.subject_public_key_info_der, &mut stats);
add_vec_stats(&trust_anchor.ta_certificate.raw_der, &mut stats);
if let Some(uri) = &trust_anchor.resolved_ta_uri {
stats.string_bytes += uri.as_str().len() as u64;
stats.string_capacity_bytes += uri.as_str().len() as u64;
stats.estimated_heap_bytes += uri.as_str().len() as u64;
}
}
stats.btree_entry_shallow_bytes = (self.manifests_by_hash.len()
* (std::mem::size_of::<Vec<u8>>() + std::mem::size_of::<CcrManifestContribution>()))
as u64;
stats.estimated_heap_bytes += stats.btree_entry_shallow_bytes;
for (key, contribution) in &self.manifests_by_hash {
stats.btree_key_capacity_bytes += key.capacity() as u64;
stats.estimated_heap_bytes += key.capacity() as u64;
contribution.add_memory_stats(&mut stats);
}
stats
}
}
fn add_vec_of_string_stats(values: &Vec<String>, stats: &mut CcrAccumulatorMemoryStats) {
let outer_capacity = values.capacity() * std::mem::size_of::<String>();
stats.vec_payload_bytes += (values.len() * std::mem::size_of::<String>()) as u64;
stats.vec_capacity_bytes += outer_capacity as u64;
stats.estimated_heap_bytes += outer_capacity as u64;
for value in values {
stats.string_bytes += value.len() as u64;
stats.string_capacity_bytes += value.capacity() as u64;
stats.estimated_heap_bytes += value.capacity() as u64;
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ccr::export::build_ccr_from_run;
use crate::ccr::verify::verify_content_info;
use crate::data_model::manifest::ManifestObject;
use crate::data_model::oid::{OID_AD_RPKI_NOTIFY, OID_AD_SIGNED_OBJECT};
use crate::data_model::rc::{AccessDescription, SubjectInfoAccess};
use crate::data_model::roa::{IpPrefix, RoaAfi};
use crate::data_model::ta::TrustAnchor;
use crate::data_model::tal::Tal;
use crate::storage::{
PackTime, RawByHashEntry, RocksStore, ValidatedCaInstanceResult, ValidatedManifestMeta,
VcirArtifactKind, VcirArtifactRole, VcirArtifactValidationStatus, VcirAuditSummary,
VcirCcrManifestProjection, VcirChildEntry, VcirInstanceGate, VcirRelatedArtifact,
VcirSummary,
};
use sha2::Digest;
fn sample_trust_anchor() -> TrustAnchor {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let tal_bytes = std::fs::read(base.join("tests/fixtures/tal/apnic-rfc7730-https.tal"))
.expect("read tal");
let ta_der = std::fs::read(base.join("tests/fixtures/ta/apnic-ta.cer")).expect("read ta");
let tal = Tal::decode_bytes(&tal_bytes).expect("decode tal");
TrustAnchor::bind_der(tal, &ta_der, None).expect("bind ta")
}
fn sample_vcir_and_manifest(
store: &RocksStore,
) -> (
ValidatedCaInstanceResult,
Vec<Vrp>,
Vec<AspaAttestation>,
Vec<RouterKeyPayload>,
) {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let manifest_der = std::fs::read(
base.join(
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft",
),
)
.expect("read manifest");
let manifest = ManifestObject::decode_der(&manifest_der).expect("decode manifest");
let manifest_uri = match manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.subject_info_access
.as_ref()
.expect("manifest sia")
{
SubjectInfoAccess::Ee(ee_sia) => ee_sia
.access_descriptions
.iter()
.find(|ad| {
ad.access_method_oid == OID_AD_SIGNED_OBJECT
&& ad.access_location.starts_with("rsync://")
})
.expect("manifest rsync signedObject")
.access_location
.clone(),
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
};
let manifest_hash = hex::encode(sha2::Sha256::digest(&manifest_der));
let mut raw = RawByHashEntry::from_bytes(manifest_hash.clone(), manifest_der.clone());
raw.origin_uris.push(manifest_uri.clone());
raw.object_type = Some("mft".to_string());
raw.encoding = Some("der".to_string());
store.put_raw_by_hash_entry(&raw).expect("put raw");
let projection = VcirCcrManifestProjection {
manifest_rsync_uri: manifest_uri.clone(),
manifest_sha256: sha2::Sha256::digest(&manifest_der).to_vec(),
manifest_size: manifest_der.len() as u64,
manifest_ee_aki: manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.authority_key_identifier
.clone()
.expect("manifest aki"),
manifest_number_be: manifest.manifest.manifest_number.bytes_be.clone(),
manifest_this_update: PackTime::from_utc_offset_datetime(manifest.manifest.this_update),
manifest_sia_locations_der: match manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.subject_info_access
.as_ref()
.expect("manifest sia")
{
SubjectInfoAccess::Ee(ee_sia) => vec![
crate::ccr::manifest_location::select_manifest_signed_object_location(
&manifest_uri,
&ee_sia.access_descriptions,
)
.expect("select manifest signedObject"),
],
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
},
subordinate_skis: vec![vec![0x33; 20]],
};
let vcir = ValidatedCaInstanceResult {
manifest_rsync_uri: manifest_uri.clone(),
parent_manifest_rsync_uri: None,
tal_id: "apnic".to_string(),
ca_subject_name: "CN=test".to_string(),
ca_ski: "11".repeat(20),
issuer_ski: "22".repeat(20),
last_successful_validation_time: PackTime::from_utc_offset_datetime(
manifest.manifest.this_update,
),
current_manifest_rsync_uri: manifest_uri.clone(),
current_crl_rsync_uri: format!("{manifest_uri}.crl"),
validated_manifest_meta: ValidatedManifestMeta {
validated_manifest_number: manifest.manifest.manifest_number.bytes_be.clone(),
validated_manifest_this_update: PackTime::from_utc_offset_datetime(
manifest.manifest.this_update,
),
validated_manifest_next_update: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
},
ccr_manifest_projection: projection.clone(),
instance_gate: VcirInstanceGate {
manifest_next_update: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
current_crl_next_update: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
self_ca_not_after: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
instance_effective_until: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
},
child_entries: vec![VcirChildEntry {
child_manifest_rsync_uri: "rsync://example.test/repo/child.mft".to_string(),
child_cert_rsync_uri: "rsync://example.test/repo/child.cer".to_string(),
child_cert_hash: "aa".repeat(32),
child_ski: "33".repeat(20),
child_rsync_base_uri: "rsync://example.test/repo/".to_string(),
child_publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
child_rrdp_notification_uri: None,
child_effective_ip_resources: None,
child_effective_as_resources: None,
accepted_at_validation_time: PackTime::from_utc_offset_datetime(
manifest.manifest.this_update,
),
}],
local_outputs: Vec::new(),
related_artifacts: vec![VcirRelatedArtifact {
artifact_role: VcirArtifactRole::Manifest,
artifact_kind: VcirArtifactKind::Mft,
uri: Some(manifest_uri.clone()),
sha256: manifest_hash,
object_type: Some("mft".to_string()),
validation_status: VcirArtifactValidationStatus::Accepted,
reject_reason: None,
}],
summary: VcirSummary {
local_vrp_count: 0,
local_aspa_count: 0,
local_router_key_count: 0,
child_count: 1,
accepted_object_count: 1,
rejected_object_count: 0,
},
audit_summary: VcirAuditSummary {
failed_fetch_eligible: true,
last_failed_fetch_reason: None,
warning_count: 0,
audit_flags: Vec::new(),
},
};
let vrps = vec![Vrp {
asn: 64496,
prefix: IpPrefix {
afi: RoaAfi::Ipv4,
prefix_len: 8,
addr: [10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length: 8,
}];
let aspas = vec![AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64497],
}];
let router_keys = vec![RouterKeyPayload {
as_id: 64496,
ski: vec![0x11; 20],
spki_der: vec![0x30, 0x00],
source_object_uri: "rsync://example.test/repo/router.cer".to_string(),
source_object_hash: hex::encode([0x11; 32]),
source_ee_cert_hash: hex::encode([0x11; 32]),
item_effective_until: PackTime::from_utc_offset_datetime(
time::OffsetDateTime::now_utc() + time::Duration::hours(1),
),
}];
(vcir, vrps, aspas, router_keys)
}
#[test]
fn accumulator_finish_matches_builder_on_fresh_vcir_inputs() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let (vcir, vrps, aspas, router_keys) = sample_vcir_and_manifest(&store);
store.put_vcir(&vcir).expect("put vcir");
let trust_anchor = sample_trust_anchor();
let builder_ccr = build_ccr_from_run(
&store,
&[trust_anchor.clone()],
&vrps,
&aspas,
&router_keys,
time::OffsetDateTime::now_utc(),
)
.expect("build ccr from run");
let mut accumulator = CcrAccumulator::new(vec![trust_anchor]);
accumulator
.append_manifest_projection(&vcir.ccr_manifest_projection)
.expect("append manifest projection");
let accumulated_ccr = accumulator
.finish(time::OffsetDateTime::now_utc(), &vrps, &aspas, &router_keys)
.expect("finish accumulator");
assert_eq!(builder_ccr.mfts, accumulated_ccr.mfts);
assert_eq!(builder_ccr.vrps, accumulated_ccr.vrps);
assert_eq!(builder_ccr.vaps, accumulated_ccr.vaps);
assert_eq!(builder_ccr.tas, accumulated_ccr.tas);
assert_eq!(builder_ccr.rks, accumulated_ccr.rks);
let ci = crate::ccr::model::CcrContentInfo::new(accumulated_ccr);
verify_content_info(&ci).expect("verify accumulated ccr");
}
#[test]
fn accumulator_sanitizes_historical_projection_with_rpki_notify() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let (mut vcir, _, _, _) = sample_vcir_and_manifest(&store);
let notify_der =
crate::ccr::manifest_location::encode_access_description_der(&AccessDescription {
access_method_oid: OID_AD_RPKI_NOTIFY.to_string(),
access_location: "https://rrdp.example.test/notification.xml".to_string(),
})
.expect("encode rpkiNotify");
let expected_location = vcir.ccr_manifest_projection.manifest_sia_locations_der[0].clone();
vcir.ccr_manifest_projection
.manifest_sia_locations_der
.push(notify_der);
let mut accumulator = CcrAccumulator::new(Vec::new());
accumulator
.append_manifest_projection(&vcir.ccr_manifest_projection)
.expect("sanitize historical projection");
let contribution = accumulator
.manifests_by_hash
.values()
.next()
.expect("manifest contribution");
assert_eq!(contribution.locations_der, vec![expected_location]);
}
}

View File

@ -0,0 +1,975 @@
use std::collections::{BTreeMap, BTreeSet};
use serde::Serialize;
use sha2::Digest;
use crate::ccr::encode::{
encode_aspa_payload_state_payload_der, encode_manifest_state_payload_der,
encode_roa_payload_state_payload_der, encode_router_key_state_payload_der,
encode_trust_anchor_state_payload_der,
};
use crate::ccr::hash::compute_state_hash;
use crate::ccr::manifest_location::select_manifest_signed_object_location;
use crate::ccr::model::{
AspaPayloadSet, AspaPayloadState, ManifestInstance, ManifestState, RoaPayloadSet,
RoaPayloadState, RouterKey, RouterKeySet, RouterKeyState, TrustAnchorState,
};
use crate::data_model::manifest::ManifestObject;
use crate::data_model::rc::SubjectInfoAccess;
use crate::data_model::roa::RoaAfi;
use crate::data_model::router_cert::BgpsecRouterCertificate;
use crate::data_model::ta::TrustAnchor;
use crate::storage::{RocksStore, ValidatedCaInstanceResult, VcirArtifactRole};
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
#[derive(Debug, thiserror::Error)]
pub enum CcrBuildError {
#[error("trust anchor set must not be empty")]
EmptyTrustAnchors,
#[error("trust anchor certificate missing SubjectKeyIdentifier")]
MissingTrustAnchorSki,
#[error("ROA payload state encoding failed: {0}")]
RoaEncode(String),
#[error("ASPA payload state encoding failed: {0}")]
AspaEncode(String),
#[error("TrustAnchor state encoding failed: {0}")]
TrustAnchorEncode(String),
#[error("manifest artifact missing in VCIR: {0}")]
MissingManifestArtifact(String),
#[error("manifest raw bytes missing in store for {manifest_rsync_uri}: {sha256_hex}")]
MissingManifestRawBytes {
manifest_rsync_uri: String,
sha256_hex: String,
},
#[error("manifest raw bytes load failed for {manifest_rsync_uri}: {detail}")]
LoadManifestRawBytes {
manifest_rsync_uri: String,
detail: String,
},
#[error("manifest decode failed for {manifest_rsync_uri}: {detail}")]
ManifestDecode {
manifest_rsync_uri: String,
detail: String,
},
#[error("manifest EE certificate missing AuthorityKeyIdentifier: {0}")]
ManifestEeMissingAki(String),
#[error("manifest EE certificate missing Subject Information Access extension: {0}")]
ManifestEeMissingSia(String),
#[error("manifest EE certificate SIA is not the EE form: {0}")]
ManifestEeSiaWrongVariant(String),
#[error(
"manifest EE certificate SIA location selection failed for {manifest_rsync_uri}: {detail}"
)]
ManifestSiaLocationSelection {
manifest_rsync_uri: String,
detail: String,
},
#[error("manifest child SKI is not valid lowercase hex or not 20 bytes: {0}")]
InvalidChildSki(String),
#[error("manifest state encoding failed: {0}")]
ManifestEncode(String),
#[error("duplicate manifest hash with conflicting content for URI: {0}")]
DuplicateManifestHashConflict(String),
#[error("router key state encoding failed: {0}")]
RouterKeyEncode(String),
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct ManifestStateBuildBreakdown {
pub vcir_count: usize,
pub unique_manifest_count: usize,
pub find_manifest_artifact_ms: u64,
pub load_manifest_blob_ms: u64,
pub decode_manifest_der_ms: u64,
pub build_manifest_instance_ms: u64,
pub dedup_manifest_instance_ms: u64,
pub encode_manifest_state_ms: u64,
pub total_ms: u64,
}
pub fn build_roa_payload_state(vrps: &[Vrp]) -> Result<RoaPayloadState, CcrBuildError> {
let mut grouped: BTreeMap<u32, BTreeSet<RoaPayloadKey>> = BTreeMap::new();
for vrp in vrps {
grouped
.entry(vrp.asn)
.or_default()
.insert(RoaPayloadKey::from_vrp(vrp));
}
let mut rps = Vec::with_capacity(grouped.len());
for (asn, entries) in grouped {
let mut families: BTreeMap<u16, Vec<RoaPayloadKey>> = BTreeMap::new();
for entry in entries {
families.entry(entry.afi).or_default().push(entry);
}
let mut ip_addr_blocks = Vec::with_capacity(families.len());
for (afi, entries) in families {
ip_addr_blocks.push(encode_roa_ip_address_family(afi, &entries));
}
rps.push(RoaPayloadSet {
as_id: asn,
ip_addr_blocks,
});
}
let payload_der = encode_roa_payload_state_payload_der(&rps)
.map_err(|e| CcrBuildError::RoaEncode(e.to_string()))?;
Ok(RoaPayloadState {
rps,
hash: compute_state_hash(&payload_der),
})
}
pub fn build_aspa_payload_state(
attestations: &[AspaAttestation],
) -> Result<AspaPayloadState, CcrBuildError> {
let mut grouped: BTreeMap<u32, BTreeSet<u32>> = BTreeMap::new();
for attestation in attestations {
grouped
.entry(attestation.customer_as_id)
.or_default()
.extend(attestation.provider_as_ids.iter().copied());
}
let aps: Vec<AspaPayloadSet> = grouped
.into_iter()
.map(|(customer_as_id, providers)| AspaPayloadSet {
customer_as_id,
providers: providers.into_iter().collect(),
})
.collect();
let payload_der = encode_aspa_payload_state_payload_der(&aps)
.map_err(|e| CcrBuildError::AspaEncode(e.to_string()))?;
Ok(AspaPayloadState {
aps,
hash: compute_state_hash(&payload_der),
})
}
pub fn build_trust_anchor_state(
trust_anchors: &[TrustAnchor],
) -> Result<TrustAnchorState, CcrBuildError> {
if trust_anchors.is_empty() {
return Err(CcrBuildError::EmptyTrustAnchors);
}
let mut skis: BTreeSet<Vec<u8>> = BTreeSet::new();
for ta in trust_anchors {
let ski = ta
.ta_certificate
.rc_ca
.tbs
.extensions
.subject_key_identifier
.clone()
.ok_or(CcrBuildError::MissingTrustAnchorSki)?;
skis.insert(ski);
}
let skis: Vec<Vec<u8>> = skis.into_iter().collect();
let payload_der = encode_trust_anchor_state_payload_der(&skis)
.map_err(|e| CcrBuildError::TrustAnchorEncode(e.to_string()))?;
Ok(TrustAnchorState {
skis,
hash: compute_state_hash(&payload_der),
})
}
pub fn build_manifest_state_from_vcirs(
store: &RocksStore,
vcirs: &[ValidatedCaInstanceResult],
) -> Result<ManifestState, CcrBuildError> {
build_manifest_state_from_vcirs_with_breakdown(store, vcirs).map(|(state, _)| state)
}
pub fn build_manifest_state_from_vcirs_with_breakdown(
store: &RocksStore,
vcirs: &[ValidatedCaInstanceResult],
) -> Result<(ManifestState, ManifestStateBuildBreakdown), CcrBuildError> {
let total_started = std::time::Instant::now();
let mut breakdown = ManifestStateBuildBreakdown {
vcir_count: vcirs.len(),
..ManifestStateBuildBreakdown::default()
};
let mut find_manifest_artifact_duration = std::time::Duration::ZERO;
let mut load_manifest_blob_duration = std::time::Duration::ZERO;
let mut decode_manifest_der_duration = std::time::Duration::ZERO;
let mut build_manifest_instance_duration = std::time::Duration::ZERO;
let mut dedup_manifest_instance_duration = std::time::Duration::ZERO;
let mut mis_by_hash: BTreeMap<Vec<u8>, ManifestInstance> = BTreeMap::new();
let mut most_recent_update = time::OffsetDateTime::UNIX_EPOCH;
for vcir in vcirs {
let started = std::time::Instant::now();
let manifest_artifact = vcir
.related_artifacts
.iter()
.find(|artifact| {
artifact.artifact_role == VcirArtifactRole::Manifest
&& artifact.uri.as_deref() == Some(vcir.current_manifest_rsync_uri.as_str())
})
.ok_or_else(|| {
CcrBuildError::MissingManifestArtifact(vcir.current_manifest_rsync_uri.clone())
})?;
find_manifest_artifact_duration += started.elapsed();
let started = std::time::Instant::now();
let raw_bytes = store
.get_blob_bytes(&manifest_artifact.sha256)
.map_err(|e| CcrBuildError::LoadManifestRawBytes {
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
detail: e.to_string(),
})?
.ok_or_else(|| CcrBuildError::MissingManifestRawBytes {
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
sha256_hex: manifest_artifact.sha256.clone(),
})?;
load_manifest_blob_duration += started.elapsed();
let started = std::time::Instant::now();
let manifest =
ManifestObject::decode_der(&raw_bytes).map_err(|e| CcrBuildError::ManifestDecode {
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
detail: e.to_string(),
})?;
decode_manifest_der_duration += started.elapsed();
let started = std::time::Instant::now();
let ee = &manifest.signed_object.signed_data.certificates[0].resource_cert;
let aki = ee
.tbs
.extensions
.authority_key_identifier
.clone()
.ok_or_else(|| {
CcrBuildError::ManifestEeMissingAki(vcir.current_manifest_rsync_uri.clone())
})?;
let sia = ee
.tbs
.extensions
.subject_info_access
.as_ref()
.ok_or_else(|| {
CcrBuildError::ManifestEeMissingSia(vcir.current_manifest_rsync_uri.clone())
})?;
let locations = match sia {
SubjectInfoAccess::Ee(ee_sia) => vec![
select_manifest_signed_object_location(
&vcir.current_manifest_rsync_uri,
&ee_sia.access_descriptions,
)
.map_err(|detail| CcrBuildError::ManifestSiaLocationSelection {
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
detail,
})?,
],
SubjectInfoAccess::Ca(_) => {
return Err(CcrBuildError::ManifestEeSiaWrongVariant(
vcir.current_manifest_rsync_uri.clone(),
));
}
};
let subordinates = collect_subordinate_ski_bytes(vcir)?;
let this_update = vcir
.validated_manifest_meta
.validated_manifest_this_update
.parse()
.map_err(|e| CcrBuildError::ManifestDecode {
manifest_rsync_uri: vcir.current_manifest_rsync_uri.clone(),
detail: format!("invalid validated_manifest_this_update: {e}"),
})?;
if this_update > most_recent_update {
most_recent_update = this_update;
}
let instance = ManifestInstance {
hash: sha2::Sha256::digest(&raw_bytes).to_vec(),
size: raw_bytes.len() as u64,
aki,
manifest_number: crate::data_model::common::BigUnsigned {
bytes_be: vcir
.validated_manifest_meta
.validated_manifest_number
.clone(),
},
this_update,
locations,
subordinates,
};
build_manifest_instance_duration += started.elapsed();
let started = std::time::Instant::now();
match mis_by_hash.get(instance.hash.as_slice()) {
Some(existing) if existing != &instance => {
return Err(CcrBuildError::DuplicateManifestHashConflict(
vcir.current_manifest_rsync_uri.clone(),
));
}
Some(_) => {}
None => {
mis_by_hash.insert(instance.hash.clone(), instance);
}
}
dedup_manifest_instance_duration += started.elapsed();
}
let started = std::time::Instant::now();
let mis: Vec<ManifestInstance> = mis_by_hash.into_values().collect();
let payload_der = encode_manifest_state_payload_der(&mis)
.map_err(|e| CcrBuildError::ManifestEncode(e.to_string()))?;
let state = ManifestState {
mis,
most_recent_update,
hash: compute_state_hash(&payload_der),
};
breakdown.encode_manifest_state_ms = started.elapsed().as_millis() as u64;
breakdown.find_manifest_artifact_ms = find_manifest_artifact_duration.as_millis() as u64;
breakdown.load_manifest_blob_ms = load_manifest_blob_duration.as_millis() as u64;
breakdown.decode_manifest_der_ms = decode_manifest_der_duration.as_millis() as u64;
breakdown.build_manifest_instance_ms = build_manifest_instance_duration.as_millis() as u64;
breakdown.dedup_manifest_instance_ms = dedup_manifest_instance_duration.as_millis() as u64;
breakdown.unique_manifest_count = state.mis.len();
breakdown.total_ms = total_started.elapsed().as_millis() as u64;
Ok((state, breakdown))
}
fn collect_subordinate_ski_bytes(
vcir: &ValidatedCaInstanceResult,
) -> Result<Vec<Vec<u8>>, CcrBuildError> {
let mut skis = BTreeSet::new();
for child in &vcir.child_entries {
let bytes = hex::decode(&child.child_ski)
.map_err(|_| CcrBuildError::InvalidChildSki(child.child_ski.clone()))?;
if bytes.len() != 20 {
return Err(CcrBuildError::InvalidChildSki(child.child_ski.clone()));
}
skis.insert(bytes);
}
Ok(skis.into_iter().collect())
}
pub fn build_router_key_state(
router_certs: &[BgpsecRouterCertificate],
) -> Result<RouterKeyState, CcrBuildError> {
let mut grouped: BTreeMap<u32, BTreeSet<RouterKey>> = BTreeMap::new();
for cert in router_certs {
let key = RouterKey {
ski: cert.subject_key_identifier.clone(),
spki_der: cert.spki_der.clone(),
};
for asn in &cert.asns {
grouped.entry(*asn).or_default().insert(key.clone());
}
}
let rksets: Vec<RouterKeySet> = grouped
.into_iter()
.map(|(as_id, router_keys)| RouterKeySet {
as_id,
router_keys: router_keys.into_iter().collect(),
})
.collect();
let payload_der = encode_router_key_state_payload_der(&rksets)
.map_err(|e| CcrBuildError::RouterKeyEncode(e.to_string()))?;
Ok(RouterKeyState {
rksets,
hash: compute_state_hash(&payload_der),
})
}
pub fn build_router_key_state_from_runtime(
router_keys: &[RouterKeyPayload],
) -> Result<RouterKeyState, CcrBuildError> {
let mut grouped: BTreeMap<u32, BTreeSet<RouterKey>> = BTreeMap::new();
for router_key in router_keys {
grouped
.entry(router_key.as_id)
.or_default()
.insert(RouterKey {
ski: router_key.ski.clone(),
spki_der: router_key.spki_der.clone(),
});
}
let rksets: Vec<RouterKeySet> = grouped
.into_iter()
.map(|(as_id, router_keys)| RouterKeySet {
as_id,
router_keys: router_keys.into_iter().collect(),
})
.collect();
let payload_der = encode_router_key_state_payload_der(&rksets)
.map_err(|e| CcrBuildError::RouterKeyEncode(e.to_string()))?;
Ok(RouterKeyState {
rksets,
hash: compute_state_hash(&payload_der),
})
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
struct RoaPayloadKey {
afi: u16,
addr: Vec<u8>,
prefix_len: u8,
max_length: u8,
}
impl RoaPayloadKey {
fn from_vrp(vrp: &Vrp) -> Self {
let afi = match vrp.prefix.afi {
RoaAfi::Ipv4 => 1,
RoaAfi::Ipv6 => 2,
};
Self {
afi,
addr: vrp.prefix.addr.to_vec(),
prefix_len: vrp.prefix.prefix_len as u8,
max_length: vrp.max_length as u8,
}
}
}
fn encode_roa_ip_address_family(afi: u16, entries: &[RoaPayloadKey]) -> Vec<u8> {
let address_family = afi.to_be_bytes();
let addresses = entries
.iter()
.map(encode_roa_ip_address)
.collect::<Vec<_>>();
encode_sequence(&[
encode_octet_string(&address_family),
encode_sequence(&addresses),
])
}
fn encode_roa_ip_address(entry: &RoaPayloadKey) -> Vec<u8> {
let (unused_bits, content) = encode_prefix_bit_string(&entry.addr, entry.prefix_len);
let mut fields = vec![encode_bit_string(unused_bits, &content)];
if entry.max_length != entry.prefix_len {
fields.push(encode_integer_u8(entry.max_length));
}
encode_sequence(&fields)
}
fn encode_prefix_bit_string(addr: &[u8], prefix_len: u8) -> (u8, Vec<u8>) {
if prefix_len == 0 {
return (0, Vec::new());
}
let octets = ((prefix_len as usize) + 7) / 8;
let mut content = addr[..octets].to_vec();
let rem = prefix_len % 8;
let unused = if rem == 0 { 0 } else { 8 - rem };
if unused > 0 {
let mask = 0xFFu8 << unused;
let last = content.last_mut().expect("octets > 0");
*last &= mask;
}
(unused, content)
}
fn encode_integer_u8(v: u8) -> Vec<u8> {
encode_integer_bytes(vec![v])
}
fn encode_integer_bytes(mut bytes: Vec<u8>) -> Vec<u8> {
if bytes.is_empty() {
bytes.push(0);
}
if bytes[0] & 0x80 != 0 {
bytes.insert(0, 0);
}
encode_tlv(0x02, bytes)
}
fn encode_bit_string(unused_bits: u8, content: &[u8]) -> Vec<u8> {
let mut value = Vec::with_capacity(content.len() + 1);
value.push(unused_bits);
value.extend_from_slice(content);
encode_tlv(0x03, value)
}
fn encode_octet_string(bytes: &[u8]) -> Vec<u8> {
encode_tlv(0x04, bytes.to_vec())
}
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
let total_len: usize = elements.iter().map(Vec::len).sum();
let mut buf = Vec::with_capacity(total_len);
for element in elements {
buf.extend_from_slice(element);
}
encode_tlv(0x30, buf)
}
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
let mut out = Vec::with_capacity(1 + 9 + value.len());
out.push(tag);
encode_length(value.len(), &mut out);
out.extend_from_slice(&value);
out
}
fn encode_length(len: usize, out: &mut Vec<u8>) {
if len < 0x80 {
out.push(len as u8);
return;
}
let mut bytes = Vec::new();
let mut value = len;
while value > 0 {
bytes.push((value & 0xFF) as u8);
value >>= 8;
}
bytes.reverse();
out.push(0x80 | (bytes.len() as u8));
out.extend_from_slice(&bytes);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ccr::decode::decode_content_info;
use crate::ccr::encode::{
encode_aspa_payload_state, encode_content_info, encode_trust_anchor_state,
};
use crate::ccr::model::{CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation};
use crate::data_model::roa::{IpPrefix, RoaAfi};
use crate::data_model::ta::TrustAnchor;
use crate::data_model::tal::Tal;
fn sample_vrp_v4(asn: u32, a: [u8; 4], prefix_len: u16, max_length: u16) -> Vrp {
Vrp {
asn,
prefix: IpPrefix {
afi: RoaAfi::Ipv4,
prefix_len,
addr: [a[0], a[1], a[2], a[3], 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length,
}
}
fn sample_vrp_v6(asn: u32, a: [u8; 16], prefix_len: u16, max_length: u16) -> Vrp {
Vrp {
asn,
prefix: IpPrefix {
afi: RoaAfi::Ipv6,
prefix_len,
addr: a,
},
max_length,
}
}
fn decode_family_block(block: &[u8]) -> (u16, Vec<(u8, Vec<u8>, Option<u8>)>) {
let mut top = crate::data_model::common::DerReader::new(block);
let mut seq = top.take_sequence().expect("family seq");
assert!(top.is_empty());
let afi_bytes = seq.take_octet_string().expect("afi octet string");
let afi = u16::from_be_bytes([afi_bytes[0], afi_bytes[1]]);
let mut addrs = seq.take_sequence().expect("addresses seq");
let mut entries = Vec::new();
while !addrs.is_empty() {
let mut addr_seq = addrs.take_sequence().expect("roa ip addr seq");
let (unused_bits, content) = addr_seq.take_bit_string().expect("bit string");
let prefix_len = (content.len() * 8) as u8 - unused_bits;
let max_len = if addr_seq.is_empty() {
None
} else {
Some(addr_seq.take_uint_u64().expect("maxlen") as u8)
};
entries.push((prefix_len, content.to_vec(), max_len));
}
(afi, entries)
}
fn sample_trust_anchor(path_tal: &str, path_ta: &str) -> TrustAnchor {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let tal_bytes = std::fs::read(base.join(path_tal)).expect("read tal");
let ta_der = std::fs::read(base.join(path_ta)).expect("read ta");
let tal = Tal::decode_bytes(&tal_bytes).expect("decode tal");
TrustAnchor::bind_der(tal, &ta_der, None).expect("bind ta")
}
fn sample_router_cert(asns: Vec<u32>, ski_fill: u8, spki_fill: u8) -> BgpsecRouterCertificate {
let ta = sample_trust_anchor(
"tests/fixtures/tal/apnic-rfc7730-https.tal",
"tests/fixtures/ta/apnic-ta.cer",
);
BgpsecRouterCertificate {
raw_der: vec![0x01, 0x02],
resource_cert: ta.ta_certificate.rc_ca,
subject_key_identifier: vec![ski_fill; 20],
spki_der: vec![0x30, 0x03, 0x03, 0x01, spki_fill],
asns,
}
}
fn sample_manifest_vcir(manifest_der: &[u8], child_ski_hex: &str) -> ValidatedCaInstanceResult {
let manifest = ManifestObject::decode_der(manifest_der).expect("decode manifest fixture");
let manifest_uri = match manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.subject_info_access
.as_ref()
.expect("manifest sia")
{
SubjectInfoAccess::Ee(ee_sia) => ee_sia
.access_descriptions
.iter()
.find(|ad| {
ad.access_method_oid == crate::data_model::oid::OID_AD_SIGNED_OBJECT
&& ad.access_location.starts_with("rsync://")
})
.expect("manifest rsync signedObject")
.access_location
.clone(),
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
};
let hash = hex::encode(sha2::Sha256::digest(manifest_der));
let now = manifest.manifest.this_update;
let next = manifest.manifest.next_update;
let projection = crate::storage::VcirCcrManifestProjection {
manifest_rsync_uri: manifest_uri.clone(),
manifest_sha256: sha2::Sha256::digest(manifest_der).to_vec(),
manifest_size: manifest_der.len() as u64,
manifest_ee_aki: manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.authority_key_identifier
.clone()
.expect("manifest aki"),
manifest_number_be: manifest.manifest.manifest_number.bytes_be.clone(),
manifest_this_update: crate::storage::PackTime::from_utc_offset_datetime(now),
manifest_sia_locations_der: match manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.subject_info_access
.as_ref()
.expect("manifest sia")
{
SubjectInfoAccess::Ee(ee_sia) => vec![
select_manifest_signed_object_location(
&manifest_uri,
&ee_sia.access_descriptions,
)
.expect("select manifest signedObject"),
],
SubjectInfoAccess::Ca(_) => panic!("manifest ee sia should not be CA variant"),
},
subordinate_skis: vec![hex::decode(child_ski_hex).expect("decode child ski")],
};
crate::storage::ValidatedCaInstanceResult {
manifest_rsync_uri: manifest_uri.clone(),
parent_manifest_rsync_uri: None,
tal_id: "test-tal".to_string(),
ca_subject_name: "CN=test".to_string(),
ca_ski: "11".repeat(20),
issuer_ski: "22".repeat(20),
last_successful_validation_time: crate::storage::PackTime::from_utc_offset_datetime(
now,
),
current_manifest_rsync_uri: manifest_uri.clone(),
current_crl_rsync_uri: format!("{manifest_uri}.crl"),
validated_manifest_meta: crate::storage::ValidatedManifestMeta {
validated_manifest_number: manifest.manifest.manifest_number.bytes_be.clone(),
validated_manifest_this_update: crate::storage::PackTime::from_utc_offset_datetime(
now,
),
validated_manifest_next_update: crate::storage::PackTime::from_utc_offset_datetime(
next,
),
},
ccr_manifest_projection: projection,
instance_gate: crate::storage::VcirInstanceGate {
manifest_next_update: crate::storage::PackTime::from_utc_offset_datetime(next),
current_crl_next_update: crate::storage::PackTime::from_utc_offset_datetime(next),
self_ca_not_after: crate::storage::PackTime::from_utc_offset_datetime(next),
instance_effective_until: crate::storage::PackTime::from_utc_offset_datetime(next),
},
child_entries: vec![crate::storage::VcirChildEntry {
child_manifest_rsync_uri: format!("{manifest_uri}/child.mft"),
child_cert_rsync_uri: format!("{manifest_uri}/child.cer"),
child_cert_hash: "aa".repeat(32),
child_ski: child_ski_hex.to_string(),
child_rsync_base_uri: format!("{manifest_uri}/"),
child_publication_point_rsync_uri: format!("{manifest_uri}/"),
child_rrdp_notification_uri: None,
child_effective_ip_resources: None,
child_effective_as_resources: None,
accepted_at_validation_time: crate::storage::PackTime::from_utc_offset_datetime(
now,
),
}],
local_outputs: Vec::new(),
related_artifacts: vec![crate::storage::VcirRelatedArtifact {
artifact_role: crate::storage::VcirArtifactRole::Manifest,
artifact_kind: crate::storage::VcirArtifactKind::Mft,
uri: Some(manifest_uri.clone()),
sha256: hash,
object_type: Some("mft".to_string()),
validation_status: crate::storage::VcirArtifactValidationStatus::Accepted,
reject_reason: None,
}],
summary: crate::storage::VcirSummary {
local_vrp_count: 0,
local_aspa_count: 0,
local_router_key_count: 0,
child_count: 1,
accepted_object_count: 1,
rejected_object_count: 0,
},
audit_summary: crate::storage::VcirAuditSummary {
failed_fetch_eligible: true,
last_failed_fetch_reason: None,
warning_count: 0,
audit_flags: Vec::new(),
},
}
}
#[test]
fn build_manifest_state_from_vcirs_collects_current_manifests_and_hashes_payload() {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let manifest_a = std::fs::read(base.join("tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft")).expect("read manifest a");
let manifest_b = std::fs::read(base.join(
"tests/fixtures/repository/ca.rg.net/rpki/RGnet-OU/bW-_qXU9uNhGQz21NR2ansB8lr0.mft",
))
.expect("read manifest b");
let vcir_a = sample_manifest_vcir(&manifest_a, &"33".repeat(20));
let vcir_b = sample_manifest_vcir(&manifest_b, &"44".repeat(20));
let store_dir = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(store_dir.path()).expect("open rocksdb");
for (vcir, bytes) in [(&vcir_a, &manifest_a), (&vcir_b, &manifest_b)] {
let artifact = &vcir.related_artifacts[0];
let mut raw =
crate::storage::RawByHashEntry::from_bytes(artifact.sha256.clone(), bytes.to_vec());
raw.origin_uris
.push(vcir.current_manifest_rsync_uri.clone());
raw.object_type = Some("mft".to_string());
raw.encoding = Some("der".to_string());
store.put_raw_by_hash_entry(&raw).expect("put raw manifest");
}
let state = build_manifest_state_from_vcirs(&store, &[vcir_a.clone(), vcir_b.clone()])
.expect("build manifest state");
assert_eq!(state.mis.len(), 2);
assert!(state.mis[0].hash < state.mis[1].hash);
let expected_subordinates = [
hex::decode(vcir_a.child_entries[0].child_ski.clone()).unwrap(),
hex::decode(vcir_b.child_entries[0].child_ski.clone()).unwrap(),
];
let actual_subordinates = state
.mis
.iter()
.map(|mi| {
assert_eq!(mi.subordinates.len(), 1);
assert!(!mi.locations.is_empty());
mi.subordinates[0].clone()
})
.collect::<std::collections::BTreeSet<_>>();
let expected_subordinates = expected_subordinates
.into_iter()
.collect::<std::collections::BTreeSet<_>>();
assert_eq!(actual_subordinates, expected_subordinates);
let payload_der =
encode_manifest_state_payload_der(&state.mis).expect("encode mis payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
let max_time = [
vcir_a
.validated_manifest_meta
.validated_manifest_this_update
.parse()
.unwrap(),
vcir_b
.validated_manifest_meta
.validated_manifest_this_update
.parse()
.unwrap(),
]
.into_iter()
.max()
.unwrap();
assert_eq!(state.most_recent_update, max_time);
}
#[test]
fn build_manifest_state_from_vcirs_empty_uses_epoch() {
let store_dir = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(store_dir.path()).expect("open rocksdb");
let state = build_manifest_state_from_vcirs(&store, &[]).expect("empty manifest state");
assert!(state.mis.is_empty());
assert_eq!(state.most_recent_update, time::OffsetDateTime::UNIX_EPOCH);
let payload_der =
encode_manifest_state_payload_der(&state.mis).expect("encode mis payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
}
#[test]
fn build_router_key_state_groups_dedupes_and_sorts() {
let certs = vec![
sample_router_cert(vec![64497], 0x21, 0xA1),
sample_router_cert(vec![64496, 64497], 0x11, 0xA0),
sample_router_cert(vec![64496], 0x11, 0xA0),
sample_router_cert(vec![64496], 0x11, 0xA2),
];
let state = build_router_key_state(&certs).expect("build router key state");
assert_eq!(state.rksets.len(), 2);
assert_eq!(state.rksets[0].as_id, 64496);
assert_eq!(state.rksets[1].as_id, 64497);
assert_eq!(state.rksets[0].router_keys.len(), 2);
assert_eq!(state.rksets[1].router_keys.len(), 2);
assert!(state.rksets[0].router_keys[0].ski <= state.rksets[0].router_keys[1].ski);
let payload_der =
encode_router_key_state_payload_der(&state.rksets).expect("encode rk payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
}
#[test]
fn build_router_key_state_empty_is_valid_and_hashes_empty_sequence() {
let state = build_router_key_state(&[]).expect("empty router key state");
assert!(state.rksets.is_empty());
let payload_der =
encode_router_key_state_payload_der(&state.rksets).expect("encode rk payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
}
#[test]
fn build_roa_payload_state_groups_dedupes_and_sorts() {
let vrps = vec![
sample_vrp_v4(64497, [10, 1, 0, 0], 16, 16),
sample_vrp_v4(64496, [10, 0, 0, 0], 8, 8),
sample_vrp_v4(64496, [10, 0, 0, 0], 8, 8),
sample_vrp_v4(64496, [10, 1, 0, 0], 16, 24),
sample_vrp_v6(
64496,
[0x20, 0x01, 0x0d, 0xb8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
32,
48,
),
];
let state = build_roa_payload_state(&vrps).expect("build roa state");
assert_eq!(state.rps.len(), 2);
assert_eq!(state.rps[0].as_id, 64496);
assert_eq!(state.rps[1].as_id, 64497);
assert_eq!(state.rps[0].ip_addr_blocks.len(), 2);
let (afi4, entries4) = decode_family_block(&state.rps[0].ip_addr_blocks[0]);
let (afi6, entries6) = decode_family_block(&state.rps[0].ip_addr_blocks[1]);
assert_eq!(afi4, 1);
assert_eq!(afi6, 2);
assert_eq!(entries4.len(), 2);
assert_eq!(entries4[0], (8, vec![10], None));
assert_eq!(entries4[1], (16, vec![10, 1], Some(24)));
assert_eq!(entries6.len(), 1);
assert_eq!(entries6[0], (32, vec![0x20, 0x01, 0x0d, 0xb8], Some(48)));
let payload_der = encode_roa_payload_state_payload_der(&state.rps).expect("encode payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
}
#[test]
fn build_roa_payload_state_empty_is_valid_and_hashes_empty_sequence() {
let state = build_roa_payload_state(&[]).expect("empty roa state");
assert!(state.rps.is_empty());
let payload_der = encode_roa_payload_state_payload_der(&state.rps).expect("encode payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
}
#[test]
fn build_aspa_payload_state_merges_and_sorts() {
let aspas = vec![
AspaAttestation {
customer_as_id: 64497,
provider_as_ids: vec![65002],
},
AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![65003, 65001],
},
AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![65002, 65001],
},
];
let state = build_aspa_payload_state(&aspas).expect("build aspa state");
assert_eq!(state.aps.len(), 2);
assert_eq!(state.aps[0].customer_as_id, 64496);
assert_eq!(state.aps[0].providers, vec![65001, 65002, 65003]);
assert_eq!(state.aps[1].customer_as_id, 64497);
let encoded = encode_aspa_payload_state(&state).expect("encode aspa state");
let decoded = decode_content_info(
&encode_content_info(&CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at: time::OffsetDateTime::now_utc(),
mfts: None,
vrps: None,
vaps: Some(state.clone()),
tas: None,
rks: None,
}))
.expect("encode ccr"),
)
.expect("decode ccr");
assert_eq!(decoded.content.vaps, Some(state));
assert!(!encoded.is_empty());
}
#[test]
fn build_trust_anchor_state_collects_sorted_unique_skis() {
let apnic = sample_trust_anchor(
"tests/fixtures/tal/apnic-rfc7730-https.tal",
"tests/fixtures/ta/apnic-ta.cer",
);
let arin = sample_trust_anchor(
"tests/fixtures/tal/arin.tal",
"tests/fixtures/ta/arin-ta.cer",
);
let state = build_trust_anchor_state(&[apnic.clone(), arin.clone(), apnic])
.expect("build ta state");
assert_eq!(state.skis.len(), 2);
assert!(state.skis[0] < state.skis[1]);
let payload_der =
encode_trust_anchor_state_payload_der(&state.skis).expect("encode ta payload");
assert!(crate::ccr::verify_state_hash(&state.hash, &payload_der));
let encoded = encode_trust_anchor_state(&state).expect("encode ta state");
assert!(!encoded.is_empty());
}
#[test]
fn build_trust_anchor_state_rejects_empty_and_missing_ski() {
let err = build_trust_anchor_state(&[]).expect_err("empty TAs must fail");
assert!(err.to_string().contains("trust anchor set"), "{err}");
let mut ta = sample_trust_anchor(
"tests/fixtures/tal/apnic-rfc7730-https.tal",
"tests/fixtures/ta/apnic-ta.cer",
);
ta.ta_certificate
.rc_ca
.tbs
.extensions
.subject_key_identifier = None;
let err = build_trust_anchor_state(&[ta]).expect_err("missing ski must fail");
assert!(err.to_string().contains("SubjectKeyIdentifier"), "{err}");
}
}

View File

@ -0,0 +1,235 @@
use std::collections::BTreeSet;
use std::io::Write;
use std::path::Path;
use crate::ccr::{CcrContentInfo, extract_vrp_rows};
use crate::validation::objects::{AspaAttestation, Vrp};
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct VrpCompareRow {
pub asn: String,
pub ip_prefix: String,
pub max_length: String,
pub trust_anchor: String,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct VapCompareRow {
pub customer_asn: String,
pub providers: String,
pub trust_anchor: String,
}
fn normalize_asn(asn: u32) -> String {
format!("AS{asn}")
}
pub fn canonical_vrp_prefix(prefix: &crate::data_model::roa::IpPrefix) -> String {
let mut addr = prefix.addr_bytes().to_vec();
let total_bits = match prefix.afi {
crate::data_model::roa::RoaAfi::Ipv4 => 32usize,
crate::data_model::roa::RoaAfi::Ipv6 => 128usize,
};
let keep = usize::from(prefix.prefix_len);
for bit in keep..total_bits {
let byte = bit / 8;
let offset = 7 - (bit % 8);
addr[byte] &= !(1u8 << offset);
}
match prefix.afi {
crate::data_model::roa::RoaAfi::Ipv4 => {
let ipv4 = std::net::Ipv4Addr::new(addr[0], addr[1], addr[2], addr[3]);
format!("{ipv4}/{}", prefix.prefix_len)
}
crate::data_model::roa::RoaAfi::Ipv6 => {
let mut bytes = [0u8; 16];
bytes.copy_from_slice(&addr[..16]);
let ipv6 = std::net::Ipv6Addr::from(bytes);
format!("{ipv6}/{}", prefix.prefix_len)
}
}
}
pub fn build_vrp_compare_rows(vrps: &[Vrp]) -> BTreeSet<VrpCompareRow> {
vrps.iter()
.map(|vrp| VrpCompareRow {
asn: normalize_asn(vrp.asn),
ip_prefix: canonical_vrp_prefix(&vrp.prefix),
max_length: vrp.max_length.to_string(),
trust_anchor: "unknown".to_string(),
})
.collect()
}
pub fn build_vap_compare_rows(aspas: &[AspaAttestation]) -> BTreeSet<VapCompareRow> {
aspas
.iter()
.map(|aspa| {
let mut providers = aspa.provider_as_ids.iter().copied().collect::<Vec<_>>();
providers.sort_unstable();
providers.dedup();
VapCompareRow {
customer_asn: normalize_asn(aspa.customer_as_id),
providers: providers
.into_iter()
.map(normalize_asn)
.collect::<Vec<_>>()
.join(";"),
trust_anchor: "unknown".to_string(),
}
})
.collect()
}
pub fn decode_ccr_compare_views(
content_info: &CcrContentInfo,
) -> Result<(BTreeSet<VrpCompareRow>, BTreeSet<VapCompareRow>), String> {
let vrps = extract_vrp_rows(content_info)
.map_err(|e| format!("extract vrp rows from ccr failed: {e}"))?
.into_iter()
.map(|(asn, prefix, max_length)| VrpCompareRow {
asn: normalize_asn(asn),
ip_prefix: prefix,
max_length: max_length.to_string(),
trust_anchor: "unknown".to_string(),
})
.collect::<BTreeSet<_>>();
let vaps = content_info
.content
.vaps
.as_ref()
.map(|state| {
state
.aps
.iter()
.map(|vap| VapCompareRow {
customer_asn: normalize_asn(vap.customer_as_id),
providers: vap
.providers
.iter()
.copied()
.map(normalize_asn)
.collect::<Vec<_>>()
.join(";"),
trust_anchor: "unknown".to_string(),
})
.collect::<BTreeSet<_>>()
})
.unwrap_or_default();
Ok((vrps, vaps))
}
pub fn write_vrp_csv(path: &Path, rows: &BTreeSet<VrpCompareRow>) -> Result<(), String> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
}
let mut file = std::io::BufWriter::new(
std::fs::File::create(path)
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?,
);
writeln!(file, "ASN,IP Prefix,Max Length,Trust Anchor").map_err(|e| e.to_string())?;
for row in rows {
writeln!(
file,
"{},{},{},{}",
row.asn, row.ip_prefix, row.max_length, row.trust_anchor
)
.map_err(|e| e.to_string())?;
}
Ok(())
}
pub fn write_vap_csv(path: &Path, rows: &BTreeSet<VapCompareRow>) -> Result<(), String> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
}
let mut file = std::io::BufWriter::new(
std::fs::File::create(path)
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?,
);
writeln!(file, "Customer ASN,Providers,Trust Anchor").map_err(|e| e.to_string())?;
for row in rows {
writeln!(
file,
"{},{},{}",
row.customer_asn, row.providers, row.trust_anchor
)
.map_err(|e| e.to_string())?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ccr::{
CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation,
build_aspa_payload_state, build_roa_payload_state,
};
use crate::data_model::roa::{IpPrefix, RoaAfi};
#[test]
fn build_vap_compare_rows_sorts_and_dedups_providers() {
let rows = build_vap_compare_rows(&[AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64498, 64497, 64498],
}]);
let row = rows.iter().next().expect("one row");
assert_eq!(row.customer_asn, "AS64496");
assert_eq!(row.providers, "AS64497;AS64498");
assert_eq!(row.trust_anchor, "unknown");
}
#[test]
fn decode_ccr_compare_views_extracts_vrps_and_vaps() {
let vrps = build_roa_payload_state(&[Vrp {
asn: 64496,
prefix: IpPrefix {
afi: RoaAfi::Ipv4,
prefix_len: 24,
addr: [192, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length: 24,
}])
.expect("build vrps");
let vaps = build_aspa_payload_state(&[AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64497],
}])
.expect("build vaps");
let content = CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at: time::OffsetDateTime::now_utc(),
mfts: None,
vrps: Some(vrps),
vaps: Some(vaps),
tas: None,
rks: None,
});
let (vrp_rows, vap_rows) =
decode_ccr_compare_views(&content).expect("decode compare views");
assert_eq!(vrp_rows.len(), 1);
assert_eq!(vap_rows.len(), 1);
assert_eq!(vap_rows.iter().next().unwrap().providers, "AS64497");
}
#[test]
fn build_vrp_compare_rows_canonicalizes_ipv6_prefix_text() {
let rows = build_vrp_compare_rows(&[Vrp {
asn: 64496,
prefix: IpPrefix {
afi: RoaAfi::Ipv6,
prefix_len: 32,
addr: [0x20, 0x01, 0x0d, 0xb8, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
},
max_length: 48,
}]);
let row = rows.iter().next().expect("row");
assert_eq!(row.ip_prefix, "2001:db8::/32");
}
}

View File

@ -0,0 +1,541 @@
use crate::ccr::model::{
AspaPayloadSet, AspaPayloadState, CCR_VERSION_V0, CcrContentInfo, CcrDigestAlgorithm,
ManifestInstance, ManifestState, RoaPayloadSet, RoaPayloadState, RouterKey, RouterKeySet,
RouterKeyState, RpkiCanonicalCacheRepresentation, TrustAnchorState,
};
use crate::data_model::common::{BigUnsigned, DerReader};
use crate::data_model::oid::{OID_CT_RPKI_CCR, OID_CT_RPKI_CCR_RAW, OID_SHA256, OID_SHA256_RAW};
use der_parser::der::parse_der_oid;
#[derive(Debug, thiserror::Error)]
pub enum CcrDecodeError {
#[error("DER parse error: {0}")]
Parse(String),
#[error("unexpected contentType OID: expected {expected}, got {actual}")]
UnexpectedContentType {
expected: &'static str,
actual: String,
},
#[error("unexpected digest algorithm OID: expected {expected}, got {actual}")]
UnexpectedDigestAlgorithm {
expected: &'static str,
actual: String,
},
#[error("CCR model validation failed after decode: {0}")]
Validate(String),
}
pub fn decode_content_info(der: &[u8]) -> Result<CcrContentInfo, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ContentInfo".into(),
));
}
let content_type_raw = seq.take_tag(0x06).map_err(CcrDecodeError::Parse)?;
if content_type_raw != OID_CT_RPKI_CCR_RAW {
return Err(CcrDecodeError::UnexpectedContentType {
expected: OID_CT_RPKI_CCR,
actual: oid_string(content_type_raw)?,
});
}
let inner = seq.take_tag(0xA0).map_err(CcrDecodeError::Parse)?;
if !seq.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing fields in ContentInfo".into(),
));
}
let content = decode_ccr(inner)?;
let ci = CcrContentInfo::new(content);
ci.validate().map_err(CcrDecodeError::Validate)?;
Ok(ci)
}
pub fn decode_ccr(der: &[u8]) -> Result<RpkiCanonicalCacheRepresentation, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse("trailing bytes after CCR".into()));
}
let version = if !seq.is_empty() && seq.peek_tag().map_err(CcrDecodeError::Parse)? == 0xA0 {
let explicit = seq.take_tag(0xA0).map_err(CcrDecodeError::Parse)?;
let mut inner = DerReader::new(explicit);
let version = inner.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
if !inner.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes inside CCR version EXPLICIT".into(),
));
}
version
} else {
CCR_VERSION_V0
};
let hash_alg = decode_digest_algorithm(seq.take_sequence().map_err(CcrDecodeError::Parse)?)?;
let produced_at = parse_generalized_time(seq.take_tag(0x18).map_err(CcrDecodeError::Parse)?)?;
let mut mfts = None;
let mut vrps = None;
let mut vaps = None;
let mut tas = None;
let mut rks = None;
while !seq.is_empty() {
let tag = seq.peek_tag().map_err(CcrDecodeError::Parse)?;
let (tag_read, value) = seq.take_any().map_err(CcrDecodeError::Parse)?;
debug_assert_eq!(tag, tag_read);
match tag {
0xA1 => mfts = Some(decode_manifest_state(value)?),
0xA2 => vrps = Some(decode_roa_payload_state(value)?),
0xA3 => vaps = Some(decode_aspa_payload_state(value)?),
0xA4 => tas = Some(decode_trust_anchor_state(value)?),
0xA5 => rks = Some(decode_router_key_state(value)?),
_ => {
return Err(CcrDecodeError::Parse(format!(
"unexpected CCR field tag 0x{tag:02X}"
)));
}
}
}
let ccr = RpkiCanonicalCacheRepresentation {
version,
hash_alg,
produced_at,
mfts,
vrps,
vaps,
tas,
rks,
};
ccr.validate().map_err(CcrDecodeError::Validate)?;
Ok(ccr)
}
fn decode_manifest_state(explicit_der: &[u8]) -> Result<ManifestState, CcrDecodeError> {
let mut outer = DerReader::new(explicit_der);
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
if !outer.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ManifestState".into(),
));
}
let mis_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut mis_reader = DerReader::new(mis_der);
let mut mis = Vec::new();
while !mis_reader.is_empty() {
let (_tag, full, _value) = mis_reader.take_any_full().map_err(CcrDecodeError::Parse)?;
mis.push(decode_manifest_instance(full)?);
}
let most_recent_update =
parse_generalized_time(seq.take_tag(0x18).map_err(CcrDecodeError::Parse)?)?;
let hash = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
if !seq.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing fields in ManifestState".into(),
));
}
Ok(ManifestState {
mis,
most_recent_update,
hash,
})
}
fn decode_manifest_instance(der: &[u8]) -> Result<ManifestInstance, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ManifestInstance".into(),
));
}
let hash = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
let size = seq.take_uint_u64().map_err(CcrDecodeError::Parse)?;
let aki = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
let manifest_number = decode_big_unsigned(seq.take_tag(0x02).map_err(CcrDecodeError::Parse)?)?;
let this_update = parse_generalized_time(seq.take_tag(0x18).map_err(CcrDecodeError::Parse)?)?;
let locations_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut locations_reader = DerReader::new(locations_der);
let mut locations = Vec::new();
while !locations_reader.is_empty() {
let (_tag, full, _value) = locations_reader
.take_any_full()
.map_err(CcrDecodeError::Parse)?;
locations.push(full.to_vec());
}
let subordinates = if !seq.is_empty() {
let subordinate_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(subordinate_der);
let mut out = Vec::new();
while !reader.is_empty() {
out.push(
reader
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec(),
);
}
out
} else {
Vec::new()
};
Ok(ManifestInstance {
hash,
size,
aki,
manifest_number,
this_update,
locations,
subordinates,
})
}
fn decode_roa_payload_state(explicit_der: &[u8]) -> Result<RoaPayloadState, CcrDecodeError> {
let mut outer = DerReader::new(explicit_der);
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
if !outer.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ROAPayloadState".into(),
));
}
let payload_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(payload_der);
let mut rps = Vec::new();
while !reader.is_empty() {
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
rps.push(decode_roa_payload_set(full)?);
}
let hash = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
Ok(RoaPayloadState { rps, hash })
}
fn decode_roa_payload_set(der: &[u8]) -> Result<RoaPayloadSet, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ROAPayloadSet".into(),
));
}
let as_id = seq.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
let blocks_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(blocks_der);
let mut ip_addr_blocks = Vec::new();
while !reader.is_empty() {
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
ip_addr_blocks.push(full.to_vec());
}
Ok(RoaPayloadSet {
as_id,
ip_addr_blocks,
})
}
fn decode_aspa_payload_state(explicit_der: &[u8]) -> Result<AspaPayloadState, CcrDecodeError> {
let mut outer = DerReader::new(explicit_der);
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
if !outer.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ASPAPayloadState".into(),
));
}
let payload_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(payload_der);
let mut aps = Vec::new();
while !reader.is_empty() {
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
aps.push(decode_aspa_payload_set(full)?);
}
let hash = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
Ok(AspaPayloadState { aps, hash })
}
fn decode_aspa_payload_set(der: &[u8]) -> Result<AspaPayloadSet, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after ASPAPayloadSet".into(),
));
}
let customer_as_id = seq.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
let providers_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(providers_der);
let mut providers = Vec::new();
while !reader.is_empty() {
providers.push(reader.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32);
}
Ok(AspaPayloadSet {
customer_as_id,
providers,
})
}
fn decode_trust_anchor_state(explicit_der: &[u8]) -> Result<TrustAnchorState, CcrDecodeError> {
let mut outer = DerReader::new(explicit_der);
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
if !outer.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after TrustAnchorState".into(),
));
}
let skis_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(skis_der);
let mut skis = Vec::new();
while !reader.is_empty() {
skis.push(
reader
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec(),
);
}
let hash = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
Ok(TrustAnchorState { skis, hash })
}
fn decode_router_key_state(explicit_der: &[u8]) -> Result<RouterKeyState, CcrDecodeError> {
let mut outer = DerReader::new(explicit_der);
let mut seq = outer.take_sequence().map_err(CcrDecodeError::Parse)?;
if !outer.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after RouterKeyState".into(),
));
}
let sets_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(sets_der);
let mut rksets = Vec::new();
while !reader.is_empty() {
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
rksets.push(decode_router_key_set(full)?);
}
let hash = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
Ok(RouterKeyState { rksets, hash })
}
fn decode_router_key_set(der: &[u8]) -> Result<RouterKeySet, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after RouterKeySet".into(),
));
}
let as_id = seq.take_uint_u64().map_err(CcrDecodeError::Parse)? as u32;
let keys_der = seq.take_tag(0x30).map_err(CcrDecodeError::Parse)?;
let mut reader = DerReader::new(keys_der);
let mut router_keys = Vec::new();
while !reader.is_empty() {
let (_tag, full, _value) = reader.take_any_full().map_err(CcrDecodeError::Parse)?;
router_keys.push(decode_router_key(full)?);
}
Ok(RouterKeySet { as_id, router_keys })
}
fn decode_router_key(der: &[u8]) -> Result<RouterKey, CcrDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrDecodeError::Parse)?;
if !top.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing bytes after RouterKey".into(),
));
}
let ski = seq
.take_octet_string()
.map_err(CcrDecodeError::Parse)?
.to_vec();
let (_tag, full, _value) = seq.take_any_full().map_err(CcrDecodeError::Parse)?;
if !seq.is_empty() {
return Err(CcrDecodeError::Parse("trailing fields in RouterKey".into()));
}
Ok(RouterKey {
ski,
spki_der: full.to_vec(),
})
}
fn decode_digest_algorithm(mut seq: DerReader<'_>) -> Result<CcrDigestAlgorithm, CcrDecodeError> {
let oid_raw = seq.take_tag(0x06).map_err(CcrDecodeError::Parse)?;
if oid_raw != OID_SHA256_RAW {
return Err(CcrDecodeError::UnexpectedDigestAlgorithm {
expected: OID_SHA256,
actual: oid_string(oid_raw)?,
});
}
if !seq.is_empty() {
let tag = seq.peek_tag().map_err(CcrDecodeError::Parse)?;
if tag == 0x05 {
let null = seq.take_tag(0x05).map_err(CcrDecodeError::Parse)?;
if !null.is_empty() {
return Err(CcrDecodeError::Parse(
"AlgorithmIdentifier NULL parameters must be empty".into(),
));
}
}
}
if !seq.is_empty() {
return Err(CcrDecodeError::Parse(
"trailing fields in DigestAlgorithmIdentifier".into(),
));
}
Ok(CcrDigestAlgorithm::Sha256)
}
fn oid_string(raw_body: &[u8]) -> Result<String, CcrDecodeError> {
let der = {
let mut out = Vec::with_capacity(raw_body.len() + 2);
out.push(0x06);
if raw_body.len() < 0x80 {
out.push(raw_body.len() as u8);
} else {
return Err(CcrDecodeError::Parse("OID too long".into()));
}
out.extend_from_slice(raw_body);
out
};
let (_rem, oid) = parse_der_oid(&der).map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
let oid = oid
.as_oid_val()
.map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
Ok(oid.to_string())
}
fn parse_generalized_time(bytes: &[u8]) -> Result<time::OffsetDateTime, CcrDecodeError> {
let s = std::str::from_utf8(bytes).map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
if s.len() != 15 || !s.ends_with('Z') {
return Err(CcrDecodeError::Parse(
"GeneralizedTime must be YYYYMMDDHHMMSSZ".into(),
));
}
let parse = |range: std::ops::Range<usize>| -> Result<u32, CcrDecodeError> {
s[range]
.parse::<u32>()
.map_err(|e| CcrDecodeError::Parse(e.to_string()))
};
let year = parse(0..4)? as i32;
let month = parse(4..6)? as u8;
let day = parse(6..8)? as u8;
let hour = parse(8..10)? as u8;
let minute = parse(10..12)? as u8;
let second = parse(12..14)? as u8;
let month = time::Month::try_from(month).map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
let date = time::Date::from_calendar_date(year, month, day)
.map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
let timev = time::Time::from_hms(hour, minute, second)
.map_err(|e| CcrDecodeError::Parse(e.to_string()))?;
Ok(time::PrimitiveDateTime::new(date, timev).assume_utc())
}
fn decode_big_unsigned(bytes: &[u8]) -> Result<BigUnsigned, CcrDecodeError> {
if bytes.is_empty() {
return Err(CcrDecodeError::Parse("INTEGER has empty content".into()));
}
if bytes[0] & 0x80 != 0 {
return Err(CcrDecodeError::Parse("INTEGER must be non-negative".into()));
}
if bytes.len() > 1 && bytes[0] == 0x00 && (bytes[1] & 0x80) == 0 {
return Err(CcrDecodeError::Parse(
"INTEGER not minimally encoded".into(),
));
}
let bytes_be = if bytes.len() > 1 && bytes[0] == 0x00 {
bytes[1..].to_vec()
} else {
bytes.to_vec()
};
Ok(BigUnsigned { bytes_be })
}
#[cfg(all(test, feature = "full"))]
mod tests {
use super::*;
use crate::ccr::encode::{encode_content_info, encode_manifest_state_payload_der};
use crate::ccr::hash::compute_state_hash;
use crate::ccr::manifest_location::encode_access_description_der;
use crate::ccr::model::{
CcrContentInfo, CcrDigestAlgorithm, ManifestState, RpkiCanonicalCacheRepresentation,
};
use crate::data_model::oid::{OID_AD_RPKI_NOTIFY, OID_AD_SIGNED_OBJECT};
use crate::data_model::rc::AccessDescription;
#[test]
fn generic_decoder_preserves_multiple_manifest_locations() {
let signed_object = encode_access_description_der(&AccessDescription {
access_method_oid: OID_AD_SIGNED_OBJECT.to_string(),
access_location: "rsync://example.test/repository/current.mft".to_string(),
})
.expect("encode signedObject");
let rpki_notify = encode_access_description_der(&AccessDescription {
access_method_oid: OID_AD_RPKI_NOTIFY.to_string(),
access_location: "https://rrdp.example.test/notification.xml".to_string(),
})
.expect("encode rpkiNotify");
let manifest = ManifestInstance {
hash: vec![0x11; 32],
size: 1024,
aki: vec![0x22; 20],
manifest_number: BigUnsigned { bytes_be: vec![1] },
this_update: time::OffsetDateTime::parse(
"2026-07-20T00:00:00Z",
&time::format_description::well_known::Rfc3339,
)
.expect("parse time"),
locations: vec![signed_object, rpki_notify],
subordinates: Vec::new(),
};
let manifest_payload = encode_manifest_state_payload_der(&[manifest.clone()])
.expect("encode manifest payload");
let content = CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at: time::OffsetDateTime::parse(
"2026-07-20T00:00:00Z",
&time::format_description::well_known::Rfc3339,
)
.expect("parse time"),
mfts: Some(ManifestState {
mis: vec![manifest],
most_recent_update: time::OffsetDateTime::parse(
"2026-07-20T00:00:00Z",
&time::format_description::well_known::Rfc3339,
)
.expect("parse time"),
hash: compute_state_hash(&manifest_payload),
}),
vrps: None,
vaps: None,
tas: None,
rks: None,
});
let encoded = encode_content_info(&content).expect("encode CCR");
let decoded = decode_content_info(&encoded).expect("decode CCR");
assert_eq!(decoded.content.mfts.unwrap().mis[0].locations.len(), 2);
}
}

View File

@ -0,0 +1,112 @@
use crate::ccr::decode::{CcrDecodeError, decode_content_info};
use serde_json::json;
#[derive(Debug, thiserror::Error)]
pub enum CcrDumpError {
#[error("CCR decode failed: {0}")]
Decode(#[from] CcrDecodeError),
#[error("format time failed: {0}")]
FormatTime(String),
}
pub fn dump_content_info_json_value(der: &[u8]) -> Result<serde_json::Value, CcrDumpError> {
let content_info = decode_content_info(der)?;
dump_content_info_json(&content_info)
}
pub fn dump_content_info_json(
content_info: &crate::ccr::model::CcrContentInfo,
) -> Result<serde_json::Value, CcrDumpError> {
let produced_at = content_info
.content
.produced_at
.to_offset(time::UtcOffset::UTC)
.format(&time::format_description::well_known::Rfc3339)
.map_err(|e| CcrDumpError::FormatTime(e.to_string()))?;
let mfts = content_info.content.mfts.as_ref().map(|state| {
json!({
"present": true,
"manifest_instances": state.mis.len(),
"most_recent_update_rfc3339_utc": state.most_recent_update.to_offset(time::UtcOffset::UTC).format(&time::format_description::well_known::Rfc3339).unwrap(),
"hash_hex": hex::encode(&state.hash),
})
}).unwrap_or_else(|| json!({"present": false}));
let vrps_total = content_info
.content
.vrps
.as_ref()
.map(|state| {
state
.rps
.iter()
.map(|set| set.ip_addr_blocks.len())
.sum::<usize>()
})
.unwrap_or(0);
let vrps = content_info
.content
.vrps
.as_ref()
.map(|state| {
json!({
"present": true,
"payload_sets": state.rps.len(),
"hash_hex": hex::encode(&state.hash),
"ip_addr_block_count": vrps_total,
})
})
.unwrap_or_else(|| json!({"present": false}));
let vaps = content_info
.content
.vaps
.as_ref()
.map(|state| {
json!({
"present": true,
"payload_sets": state.aps.len(),
"hash_hex": hex::encode(&state.hash),
"provider_count": state.aps.iter().map(|set| set.providers.len()).sum::<usize>(),
})
})
.unwrap_or_else(|| json!({"present": false}));
let tas = content_info
.content
.tas
.as_ref()
.map(|state| {
json!({
"present": true,
"ski_count": state.skis.len(),
"hash_hex": hex::encode(&state.hash),
})
})
.unwrap_or_else(|| json!({"present": false}));
let rks = content_info.content.rks.as_ref().map(|state| {
json!({
"present": true,
"router_key_sets": state.rksets.len(),
"router_key_count": state.rksets.iter().map(|set| set.router_keys.len()).sum::<usize>(),
"hash_hex": hex::encode(&state.hash),
})
}).unwrap_or_else(|| json!({"present": false}));
Ok(json!({
"content_type_oid": content_info.content_type_oid,
"version": content_info.content.version,
"hash_alg": content_info.content.hash_alg.oid(),
"produced_at_rfc3339_utc": produced_at,
"state_aspects": {
"mfts": mfts,
"vrps": vrps,
"vaps": vaps,
"tas": tas,
"rks": rks,
}
}))
}

View File

@ -0,0 +1,303 @@
use crate::ccr::model::{
AspaPayloadSet, AspaPayloadState, CCR_VERSION_V0, CcrContentInfo, CcrDigestAlgorithm,
ManifestInstance, ManifestState, RoaPayloadSet, RoaPayloadState, RouterKey, RouterKeySet,
RouterKeyState, RpkiCanonicalCacheRepresentation, TrustAnchorState,
};
use crate::data_model::common::BigUnsigned;
use crate::data_model::oid::{OID_CT_RPKI_CCR_RAW, OID_SHA256_RAW};
#[derive(Debug, thiserror::Error)]
pub enum CcrEncodeError {
#[error("CCR model validation failed: {0}")]
Validate(String),
#[error("GeneralizedTime formatting failed: {0}")]
ProducedAtFormat(String),
}
pub fn encode_content_info(content_info: &CcrContentInfo) -> Result<Vec<u8>, CcrEncodeError> {
content_info.validate().map_err(CcrEncodeError::Validate)?;
let content_der = encode_ccr(&content_info.content)?;
Ok(encode_sequence(&[
encode_oid(OID_CT_RPKI_CCR_RAW),
encode_explicit(0, &content_der),
]))
}
pub fn encode_ccr(ccr: &RpkiCanonicalCacheRepresentation) -> Result<Vec<u8>, CcrEncodeError> {
ccr.validate().map_err(CcrEncodeError::Validate)?;
let mut fields = Vec::new();
if ccr.version != CCR_VERSION_V0 {
fields.push(encode_explicit(0, &encode_integer_u32(ccr.version)));
}
fields.push(encode_digest_algorithm(&ccr.hash_alg));
fields.push(encode_generalized_time(ccr.produced_at)?);
if let Some(mfts) = &ccr.mfts {
fields.push(encode_explicit(1, &encode_manifest_state(mfts)?));
}
if let Some(vrps) = &ccr.vrps {
fields.push(encode_explicit(2, &encode_roa_payload_state(vrps)?));
}
if let Some(vaps) = &ccr.vaps {
fields.push(encode_explicit(3, &encode_aspa_payload_state(vaps)?));
}
if let Some(tas) = &ccr.tas {
fields.push(encode_explicit(4, &encode_trust_anchor_state(tas)?));
}
if let Some(rks) = &ccr.rks {
fields.push(encode_explicit(5, &encode_router_key_state(rks)?));
}
Ok(encode_sequence(&fields))
}
pub fn encode_manifest_state(state: &ManifestState) -> Result<Vec<u8>, CcrEncodeError> {
state.validate().map_err(CcrEncodeError::Validate)?;
let mis = encode_manifest_state_payload_der(&state.mis)?;
Ok(encode_sequence(&[
mis,
encode_generalized_time(state.most_recent_update)?,
encode_octet_string(&state.hash),
]))
}
pub fn encode_manifest_state_payload_der(
instances: &[ManifestInstance],
) -> Result<Vec<u8>, CcrEncodeError> {
Ok(encode_sequence(
&instances
.iter()
.map(encode_manifest_instance)
.collect::<Result<Vec<_>, _>>()?,
))
}
fn encode_manifest_instance(instance: &ManifestInstance) -> Result<Vec<u8>, CcrEncodeError> {
instance.validate().map_err(CcrEncodeError::Validate)?;
let mut fields = vec![
encode_octet_string(&instance.hash),
encode_integer_u64(instance.size),
encode_octet_string(&instance.aki),
encode_integer_bigunsigned(&instance.manifest_number),
encode_generalized_time(instance.this_update)?,
encode_sequence(&instance.locations),
];
if !instance.subordinates.is_empty() {
fields.push(encode_sequence(
&instance
.subordinates
.iter()
.map(|ski| encode_octet_string(ski))
.collect::<Vec<_>>(),
));
}
Ok(encode_sequence(&fields))
}
pub fn encode_roa_payload_state(state: &RoaPayloadState) -> Result<Vec<u8>, CcrEncodeError> {
state.validate().map_err(CcrEncodeError::Validate)?;
let rps = encode_roa_payload_state_payload_der(&state.rps)?;
Ok(encode_sequence(&[rps, encode_octet_string(&state.hash)]))
}
pub fn encode_roa_payload_state_payload_der(
sets: &[RoaPayloadSet],
) -> Result<Vec<u8>, CcrEncodeError> {
Ok(encode_sequence(
&sets
.iter()
.map(encode_roa_payload_set)
.collect::<Result<Vec<_>, _>>()?,
))
}
fn encode_roa_payload_set(set: &RoaPayloadSet) -> Result<Vec<u8>, CcrEncodeError> {
set.validate().map_err(CcrEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_integer_u32(set.as_id),
encode_sequence(&set.ip_addr_blocks),
]))
}
pub fn encode_aspa_payload_state(state: &AspaPayloadState) -> Result<Vec<u8>, CcrEncodeError> {
state.validate().map_err(CcrEncodeError::Validate)?;
let aps = encode_aspa_payload_state_payload_der(&state.aps)?;
Ok(encode_sequence(&[aps, encode_octet_string(&state.hash)]))
}
pub fn encode_aspa_payload_state_payload_der(
sets: &[AspaPayloadSet],
) -> Result<Vec<u8>, CcrEncodeError> {
Ok(encode_sequence(
&sets
.iter()
.map(encode_aspa_payload_set)
.collect::<Result<Vec<_>, _>>()?,
))
}
fn encode_aspa_payload_set(set: &AspaPayloadSet) -> Result<Vec<u8>, CcrEncodeError> {
set.validate().map_err(CcrEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_integer_u32(set.customer_as_id),
encode_sequence(
&set.providers
.iter()
.map(|provider| encode_integer_u32(*provider))
.collect::<Vec<_>>(),
),
]))
}
pub fn encode_trust_anchor_state(state: &TrustAnchorState) -> Result<Vec<u8>, CcrEncodeError> {
state.validate().map_err(CcrEncodeError::Validate)?;
let skis = encode_trust_anchor_state_payload_der(&state.skis)?;
Ok(encode_sequence(&[skis, encode_octet_string(&state.hash)]))
}
pub fn encode_trust_anchor_state_payload_der(skis: &[Vec<u8>]) -> Result<Vec<u8>, CcrEncodeError> {
Ok(encode_sequence(
&skis
.iter()
.map(|ski| encode_octet_string(ski))
.collect::<Vec<_>>(),
))
}
pub fn encode_router_key_state(state: &RouterKeyState) -> Result<Vec<u8>, CcrEncodeError> {
state.validate().map_err(CcrEncodeError::Validate)?;
let rksets = encode_router_key_state_payload_der(&state.rksets)?;
Ok(encode_sequence(&[rksets, encode_octet_string(&state.hash)]))
}
pub fn encode_router_key_state_payload_der(
sets: &[RouterKeySet],
) -> Result<Vec<u8>, CcrEncodeError> {
Ok(encode_sequence(
&sets
.iter()
.map(encode_router_key_set)
.collect::<Result<Vec<_>, _>>()?,
))
}
fn encode_router_key_set(set: &RouterKeySet) -> Result<Vec<u8>, CcrEncodeError> {
set.validate().map_err(CcrEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_integer_u32(set.as_id),
encode_sequence(
&set.router_keys
.iter()
.map(encode_router_key)
.collect::<Result<Vec<_>, _>>()?,
),
]))
}
fn encode_router_key(key: &RouterKey) -> Result<Vec<u8>, CcrEncodeError> {
key.validate().map_err(CcrEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_octet_string(&key.ski),
key.spki_der.clone(),
]))
}
fn encode_digest_algorithm(alg: &CcrDigestAlgorithm) -> Vec<u8> {
match alg {
CcrDigestAlgorithm::Sha256 => encode_sequence(&[encode_oid(OID_SHA256_RAW)]),
}
}
fn encode_generalized_time(t: time::OffsetDateTime) -> Result<Vec<u8>, CcrEncodeError> {
let t = t.to_offset(time::UtcOffset::UTC);
let s = format!(
"{:04}{:02}{:02}{:02}{:02}{:02}Z",
t.year(),
u8::from(t.month()),
t.day(),
t.hour(),
t.minute(),
t.second()
);
Ok(encode_tlv(0x18, s.into_bytes()))
}
fn encode_integer_u32(v: u32) -> Vec<u8> {
encode_integer_bytes(unsigned_integer_bytes(v as u64))
}
fn encode_integer_u64(v: u64) -> Vec<u8> {
encode_integer_bytes(unsigned_integer_bytes(v))
}
fn encode_integer_bigunsigned(v: &BigUnsigned) -> Vec<u8> {
encode_integer_bytes(v.bytes_be.clone())
}
fn encode_integer_bytes(mut bytes: Vec<u8>) -> Vec<u8> {
if bytes.is_empty() {
bytes.push(0);
}
if bytes[0] & 0x80 != 0 {
bytes.insert(0, 0);
}
encode_tlv(0x02, bytes)
}
fn unsigned_integer_bytes(v: u64) -> Vec<u8> {
if v == 0 {
return vec![0];
}
let mut out = Vec::new();
let mut n = v;
while n > 0 {
out.push((n & 0xFF) as u8);
n >>= 8;
}
out.reverse();
out
}
fn encode_oid(raw_body: &[u8]) -> Vec<u8> {
encode_tlv(0x06, raw_body.to_vec())
}
fn encode_octet_string(bytes: &[u8]) -> Vec<u8> {
encode_tlv(0x04, bytes.to_vec())
}
fn encode_explicit(tag_number: u8, inner_der: &[u8]) -> Vec<u8> {
encode_tlv(0xA0 + tag_number, inner_der.to_vec())
}
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
let total_len: usize = elements.iter().map(Vec::len).sum();
let mut buf = Vec::with_capacity(total_len);
for element in elements {
buf.extend_from_slice(element);
}
encode_tlv(0x30, buf)
}
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
let mut out = Vec::with_capacity(1 + 9 + value.len());
out.push(tag);
encode_length(value.len(), &mut out);
out.extend_from_slice(&value);
out
}
fn encode_length(len: usize, out: &mut Vec<u8>) {
if len < 0x80 {
out.push(len as u8);
return;
}
let mut bytes = Vec::new();
let mut value = len;
while value > 0 {
bytes.push((value & 0xFF) as u8);
value >>= 8;
}
bytes.reverse();
out.push(0x80 | (bytes.len() as u8));
out.extend_from_slice(&bytes);
}

View File

@ -0,0 +1,343 @@
use crate::ccr::build::{
CcrBuildError, ManifestStateBuildBreakdown, build_aspa_payload_state,
build_manifest_state_from_vcirs_with_breakdown, build_roa_payload_state,
build_router_key_state_from_runtime, build_trust_anchor_state,
};
use crate::ccr::encode::{CcrEncodeError, encode_content_info};
use crate::ccr::model::{CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation};
use crate::data_model::ta::TrustAnchor;
use crate::storage::RocksStore;
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
use serde::Serialize;
use std::path::Path;
#[derive(Debug, thiserror::Error)]
pub enum CcrExportError {
#[error("list VCIRs failed: {0}")]
ListVcirs(String),
#[error("build CCR state failed: {0}")]
Build(#[from] CcrBuildError),
#[error("encode CCR failed: {0}")]
Encode(#[from] CcrEncodeError),
#[error("write CCR file failed: {0}: {1}")]
Write(String, String),
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct CcrBuildBreakdown {
pub vcir_count: usize,
pub list_vcirs_ms: u64,
pub manifest_state_ms: u64,
pub manifest_state_breakdown: ManifestStateBuildBreakdown,
pub roa_payload_state_ms: u64,
pub aspa_payload_state_ms: u64,
pub trust_anchor_state_ms: u64,
pub router_key_state_ms: u64,
pub total_ms: u64,
}
pub fn build_ccr_from_run(
store: &RocksStore,
trust_anchors: &[TrustAnchor],
vrps: &[Vrp],
aspas: &[AspaAttestation],
router_keys: &[RouterKeyPayload],
produced_at: time::OffsetDateTime,
) -> Result<RpkiCanonicalCacheRepresentation, CcrExportError> {
build_ccr_from_run_with_breakdown(store, trust_anchors, vrps, aspas, router_keys, produced_at)
.map(|(ccr, _)| ccr)
}
pub fn build_ccr_from_run_with_breakdown(
store: &RocksStore,
trust_anchors: &[TrustAnchor],
vrps: &[Vrp],
aspas: &[AspaAttestation],
router_keys: &[RouterKeyPayload],
produced_at: time::OffsetDateTime,
) -> Result<(RpkiCanonicalCacheRepresentation, CcrBuildBreakdown), CcrExportError> {
let total_started = std::time::Instant::now();
let mut breakdown = CcrBuildBreakdown::default();
let started = std::time::Instant::now();
let vcirs = store
.list_vcirs()
.map_err(|e| CcrExportError::ListVcirs(e.to_string()))?;
breakdown.list_vcirs_ms = started.elapsed().as_millis() as u64;
breakdown.vcir_count = vcirs.len();
let started = std::time::Instant::now();
let (mfts, manifest_state_breakdown) =
build_manifest_state_from_vcirs_with_breakdown(store, &vcirs)?;
breakdown.manifest_state_ms = started.elapsed().as_millis() as u64;
breakdown.manifest_state_breakdown = manifest_state_breakdown;
let started = std::time::Instant::now();
let vrps = build_roa_payload_state(vrps)?;
breakdown.roa_payload_state_ms = started.elapsed().as_millis() as u64;
let started = std::time::Instant::now();
let vaps = build_aspa_payload_state(aspas)?;
breakdown.aspa_payload_state_ms = started.elapsed().as_millis() as u64;
let started = std::time::Instant::now();
let tas = build_trust_anchor_state(trust_anchors)?;
breakdown.trust_anchor_state_ms = started.elapsed().as_millis() as u64;
let started = std::time::Instant::now();
let rks = build_router_key_state_from_runtime(router_keys)?;
breakdown.router_key_state_ms = started.elapsed().as_millis() as u64;
let ccr = RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at,
mfts: Some(mfts),
vrps: Some(vrps),
vaps: Some(vaps),
tas: Some(tas),
rks: Some(rks),
};
breakdown.total_ms = total_started.elapsed().as_millis() as u64;
Ok((ccr, breakdown))
}
pub fn write_ccr_file(
path: &Path,
ccr: &RpkiCanonicalCacheRepresentation,
) -> Result<(), CcrExportError> {
let der = encode_content_info(&CcrContentInfo::new(ccr.clone()))?;
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| CcrExportError::Write(path.display().to_string(), e.to_string()))?;
}
std::fs::write(path, der)
.map_err(|e| CcrExportError::Write(path.display().to_string(), e.to_string()))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ccr::decode::decode_content_info;
use crate::data_model::manifest::ManifestObject;
use crate::data_model::oid::OID_AD_SIGNED_OBJECT;
use crate::data_model::rc::SubjectInfoAccess;
use crate::data_model::roa::{IpPrefix, RoaAfi};
use crate::data_model::ta::TrustAnchor;
use crate::data_model::tal::Tal;
use crate::storage::{
PackTime, RawByHashEntry, RocksStore, ValidatedCaInstanceResult, ValidatedManifestMeta,
VcirArtifactKind, VcirArtifactRole, VcirArtifactValidationStatus, VcirAuditSummary,
VcirCcrManifestProjection, VcirChildEntry, VcirInstanceGate, VcirRelatedArtifact,
VcirSummary,
};
use crate::validation::objects::{AspaAttestation, RouterKeyPayload, Vrp};
use sha2::Digest;
fn sample_trust_anchor() -> TrustAnchor {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let tal_bytes = std::fs::read(base.join("tests/fixtures/tal/apnic-rfc7730-https.tal"))
.expect("read tal");
let ta_der = std::fs::read(base.join("tests/fixtures/ta/apnic-ta.cer")).expect("read ta");
let tal = Tal::decode_bytes(&tal_bytes).expect("decode tal");
TrustAnchor::bind_der(tal, &ta_der, None).expect("bind ta")
}
fn sample_vcir_and_raw(store: &RocksStore) -> ValidatedCaInstanceResult {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let manifest_der = std::fs::read(base.join("tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft")).expect("read manifest");
let manifest = ManifestObject::decode_der(&manifest_der).expect("decode manifest");
let manifest_uri = match manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.subject_info_access
.as_ref()
.expect("manifest sia")
{
SubjectInfoAccess::Ee(ee_sia) => ee_sia
.access_descriptions
.iter()
.find(|ad| {
ad.access_method_oid == OID_AD_SIGNED_OBJECT
&& ad.access_location.starts_with("rsync://")
})
.expect("manifest rsync signedObject")
.access_location
.clone(),
SubjectInfoAccess::Ca(_) => panic!("manifest EE SIA should not be CA variant"),
};
let hash = hex::encode(sha2::Sha256::digest(&manifest_der));
let mut raw = RawByHashEntry::from_bytes(hash.clone(), manifest_der.clone());
raw.origin_uris.push(manifest_uri.clone());
raw.object_type = Some("mft".to_string());
raw.encoding = Some("der".to_string());
store.put_raw_by_hash_entry(&raw).expect("put raw");
let projection = VcirCcrManifestProjection {
manifest_rsync_uri: manifest_uri.clone(),
manifest_sha256: sha2::Sha256::digest(&manifest_der).to_vec(),
manifest_size: manifest_der.len() as u64,
manifest_ee_aki: manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.authority_key_identifier
.clone()
.expect("manifest aki"),
manifest_number_be: manifest.manifest.manifest_number.bytes_be.clone(),
manifest_this_update: PackTime::from_utc_offset_datetime(manifest.manifest.this_update),
manifest_sia_locations_der: match manifest.signed_object.signed_data.certificates[0]
.resource_cert
.tbs
.extensions
.subject_info_access
.as_ref()
.expect("manifest sia")
{
SubjectInfoAccess::Ee(ee_sia) => vec![
crate::ccr::manifest_location::select_manifest_signed_object_location(
&manifest_uri,
&ee_sia.access_descriptions,
)
.expect("select manifest signedObject"),
],
SubjectInfoAccess::Ca(_) => panic!("manifest EE SIA should not be CA variant"),
},
subordinate_skis: vec![vec![0x33; 20]],
};
ValidatedCaInstanceResult {
manifest_rsync_uri: manifest_uri.clone(),
parent_manifest_rsync_uri: None,
tal_id: "apnic".to_string(),
ca_subject_name: "CN=test".to_string(),
ca_ski: "11".repeat(20),
issuer_ski: "22".repeat(20),
last_successful_validation_time: PackTime::from_utc_offset_datetime(
manifest.manifest.this_update,
),
current_manifest_rsync_uri: manifest_uri.clone(),
current_crl_rsync_uri: "rsync://example.test/repo/current.crl".to_string(),
validated_manifest_meta: ValidatedManifestMeta {
validated_manifest_number: manifest.manifest.manifest_number.bytes_be.clone(),
validated_manifest_this_update: PackTime::from_utc_offset_datetime(
manifest.manifest.this_update,
),
validated_manifest_next_update: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
},
ccr_manifest_projection: projection,
instance_gate: VcirInstanceGate {
manifest_next_update: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
current_crl_next_update: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
self_ca_not_after: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
instance_effective_until: PackTime::from_utc_offset_datetime(
manifest.manifest.next_update,
),
},
child_entries: vec![VcirChildEntry {
child_manifest_rsync_uri: "rsync://example.test/repo/child.mft".to_string(),
child_cert_rsync_uri: "rsync://example.test/repo/child.cer".to_string(),
child_cert_hash: "aa".repeat(32),
child_ski: "33".repeat(20),
child_rsync_base_uri: "rsync://example.test/repo/".to_string(),
child_publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
child_rrdp_notification_uri: None,
child_effective_ip_resources: None,
child_effective_as_resources: None,
accepted_at_validation_time: PackTime::from_utc_offset_datetime(
manifest.manifest.this_update,
),
}],
local_outputs: Vec::new(),
related_artifacts: vec![VcirRelatedArtifact {
artifact_role: VcirArtifactRole::Manifest,
artifact_kind: VcirArtifactKind::Mft,
uri: Some(manifest_uri),
sha256: hash,
object_type: Some("mft".to_string()),
validation_status: VcirArtifactValidationStatus::Accepted,
reject_reason: None,
}],
summary: VcirSummary {
local_vrp_count: 0,
local_aspa_count: 0,
local_router_key_count: 0,
child_count: 1,
accepted_object_count: 1,
rejected_object_count: 0,
},
audit_summary: VcirAuditSummary {
failed_fetch_eligible: true,
last_failed_fetch_reason: None,
warning_count: 0,
audit_flags: Vec::new(),
},
}
}
#[test]
fn build_and_write_ccr_from_run_exports_der_content_info() {
let td = tempfile::tempdir().expect("tempdir");
let db_path = td.path().join("db");
let store = RocksStore::open(&db_path).expect("open rocksdb");
let vcir = sample_vcir_and_raw(&store);
store.put_vcir(&vcir).expect("put vcir");
let trust_anchor = sample_trust_anchor();
let vrps = vec![Vrp {
asn: 64496,
prefix: IpPrefix {
afi: RoaAfi::Ipv4,
prefix_len: 8,
addr: [10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length: 8,
}];
let aspas = vec![AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64497],
}];
let router_keys = vec![RouterKeyPayload {
as_id: 64496,
ski: vec![0x11; 20],
spki_der: vec![0x30, 0x00],
source_object_uri: "rsync://example.test/repo/router.cer".to_string(),
source_object_hash: hex::encode([0x11; 32]),
source_ee_cert_hash: hex::encode([0x11; 32]),
item_effective_until: PackTime::from_utc_offset_datetime(
time::OffsetDateTime::now_utc() + time::Duration::hours(1),
),
}];
let ccr = build_ccr_from_run(
&store,
&[trust_anchor],
&vrps,
&aspas,
&router_keys,
time::OffsetDateTime::now_utc(),
)
.expect("build ccr");
assert!(ccr.mfts.is_some());
assert!(ccr.vrps.is_some());
assert!(ccr.vaps.is_some());
assert!(ccr.tas.is_some());
assert!(ccr.rks.is_some());
assert_eq!(ccr.rks.as_ref().unwrap().rksets.len(), 1);
let out = td.path().join("out/example.ccr");
write_ccr_file(&out, &ccr).expect("write ccr");
let der = std::fs::read(&out).expect("read ccr file");
let decoded = decode_content_info(&der).expect("decode ccr");
assert_eq!(decoded.content.version, 0);
assert!(decoded.content.mfts.is_some());
}
}

View File

@ -0,0 +1,9 @@
use sha2::Digest;
pub fn compute_state_hash(payload_der: &[u8]) -> Vec<u8> {
sha2::Sha256::digest(payload_der).to_vec()
}
pub fn verify_state_hash(expected: &[u8], payload_der: &[u8]) -> bool {
compute_state_hash(payload_der).as_slice() == expected
}

View File

@ -0,0 +1,424 @@
use crate::data_model::common::DerReader;
use crate::data_model::oid::OID_AD_SIGNED_OBJECT;
use crate::data_model::rc::AccessDescription;
pub(crate) fn select_manifest_signed_object_location(
manifest_rsync_uri: &str,
access_descriptions: &[AccessDescription],
) -> Result<Vec<u8>, String> {
let matching = access_descriptions
.iter()
.filter(|access_description| {
access_description.access_method_oid == OID_AD_SIGNED_OBJECT
&& access_description.access_location == manifest_rsync_uri
})
.collect::<Vec<_>>();
let access_description = expect_single_matching_location(
manifest_rsync_uri,
matching.len(),
"parsed Manifest EE SIA",
)?;
encode_access_description_der(matching[access_description])
}
pub(crate) fn select_manifest_signed_object_location_from_der(
manifest_rsync_uri: &str,
locations_der: &[Vec<u8>],
) -> Result<Vec<u8>, String> {
let matching = locations_der
.iter()
.filter_map(
|location_der| match decode_access_description_der(location_der) {
Ok(access_description)
if access_description.access_method_oid == OID_AD_SIGNED_OBJECT
&& access_description.access_location == manifest_rsync_uri =>
{
Some(Ok(location_der.clone()))
}
Ok(_) => None,
Err(detail) => Some(Err(detail)),
},
)
.collect::<Result<Vec<_>, _>>()?;
let index = expect_single_matching_location(
manifest_rsync_uri,
matching.len(),
"persisted VCIR CCR projection",
)?;
Ok(matching[index].clone())
}
pub(crate) fn encode_access_description_der(
access_description: &AccessDescription,
) -> Result<Vec<u8>, String> {
let oid = encode_oid_der(&access_description.access_method_oid)?;
let uri = encode_tlv(0x86, access_description.access_location.as_bytes().to_vec());
Ok(encode_sequence(&[oid, uri]))
}
fn expect_single_matching_location(
manifest_rsync_uri: &str,
matching_count: usize,
source: &str,
) -> Result<usize, String> {
if matching_count == 1 {
return Ok(0);
}
Err(format!(
"{source} contains {matching_count} id-ad-signedObject locations matching manifest URI {manifest_rsync_uri}; expected exactly one"
))
}
fn decode_access_description_der(der: &[u8]) -> Result<AccessDescription, String> {
let mut top = DerReader::new(der);
let mut sequence = top.take_sequence()?;
if !top.is_empty() {
return Err("trailing bytes after AccessDescription".to_string());
}
let access_method_oid = decode_oid_der(sequence.take_tag(0x06)?)?;
let access_location = std::str::from_utf8(sequence.take_tag(0x86)?)
.map_err(|error| format!("AccessDescription URI is not UTF-8: {error}"))?
.to_string();
if !sequence.is_empty() {
return Err("trailing fields in AccessDescription".to_string());
}
Ok(AccessDescription {
access_method_oid,
access_location,
})
}
fn decode_oid_der(value: &[u8]) -> Result<String, String> {
let mut offset = 0usize;
let first = decode_base128(value, &mut offset)?;
let (first_arc, second_arc) = match first {
0..=39 => (0, first),
40..=79 => (1, first - 40),
value => (2, value - 80),
};
let mut arcs = vec![first_arc, second_arc];
while offset < value.len() {
arcs.push(decode_base128(value, &mut offset)?);
}
Ok(arcs
.into_iter()
.map(|arc| arc.to_string())
.collect::<Vec<_>>()
.join("."))
}
fn decode_base128(value: &[u8], offset: &mut usize) -> Result<u64, String> {
let first = *value
.get(*offset)
.ok_or_else(|| "truncated OBJECT IDENTIFIER".to_string())?;
if first == 0x80 {
return Err("non-minimal OBJECT IDENTIFIER base-128 encoding".to_string());
}
let mut out = 0u64;
loop {
let byte = *value
.get(*offset)
.ok_or_else(|| "truncated OBJECT IDENTIFIER".to_string())?;
*offset += 1;
out = out
.checked_shl(7)
.ok_or_else(|| "OBJECT IDENTIFIER arc overflows u64".to_string())?
.checked_add((byte & 0x7f) as u64)
.ok_or_else(|| "OBJECT IDENTIFIER arc overflows u64".to_string())?;
if byte & 0x80 == 0 {
return Ok(out);
}
}
}
fn encode_oid_der(oid: &str) -> Result<Vec<u8>, String> {
let arcs = oid
.split('.')
.map(|part| {
part.parse::<u64>()
.map_err(|_| format!("unsupported accessMethod OID: {oid}"))
})
.collect::<Result<Vec<_>, _>>()?;
if arcs.len() < 2 || arcs[0] > 2 || (arcs[0] < 2 && arcs[1] >= 40) {
return Err(format!("unsupported accessMethod OID: {oid}"));
}
let mut body = Vec::new();
encode_base128(
arcs[0]
.checked_mul(40)
.and_then(|value| value.checked_add(arcs[1]))
.ok_or_else(|| format!("unsupported accessMethod OID: {oid}"))?,
&mut body,
);
for arc in &arcs[2..] {
encode_base128(*arc, &mut body);
}
Ok(encode_tlv(0x06, body))
}
fn encode_base128(mut value: u64, out: &mut Vec<u8>) {
let mut encoded = vec![(value & 0x7f) as u8];
value >>= 7;
while value > 0 {
encoded.push(((value & 0x7f) as u8) | 0x80);
value >>= 7;
}
encoded.reverse();
out.extend_from_slice(&encoded);
}
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
let total_len = elements.iter().map(Vec::len).sum();
let mut value = Vec::with_capacity(total_len);
for element in elements {
value.extend_from_slice(element);
}
encode_tlv(0x30, value)
}
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
let mut out = Vec::with_capacity(1 + 9 + value.len());
out.push(tag);
encode_length(value.len(), &mut out);
out.extend_from_slice(&value);
out
}
fn encode_length(len: usize, out: &mut Vec<u8>) {
if len < 0x80 {
out.push(len as u8);
return;
}
let mut bytes = Vec::new();
let mut value = len;
while value > 0 {
bytes.push((value & 0xff) as u8);
value >>= 8;
}
bytes.reverse();
out.push(0x80 | bytes.len() as u8);
out.extend_from_slice(&bytes);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::data_model::oid::OID_AD_RPKI_NOTIFY;
const MANIFEST_URI: &str = "rsync://example.test/repo/manifest.mft";
fn access_description(access_method_oid: &str, access_location: &str) -> AccessDescription {
AccessDescription {
access_method_oid: access_method_oid.to_string(),
access_location: access_location.to_string(),
}
}
#[test]
fn selects_matching_signed_object_and_excludes_rpki_notify() {
let signed_object = access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI);
let selected = select_manifest_signed_object_location(
MANIFEST_URI,
&[
signed_object.clone(),
access_description(
OID_AD_RPKI_NOTIFY,
"https://rrdp.example.test/notification.xml",
),
],
)
.expect("select signed object");
assert_eq!(
selected,
encode_access_description_der(&signed_object).unwrap()
);
}
#[test]
fn selects_only_the_signed_object_matching_manifest_uri() {
let expected = access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI);
let selected = select_manifest_signed_object_location(
MANIFEST_URI,
&[
access_description(
OID_AD_SIGNED_OBJECT,
"https://backup.example.test/manifest.mft",
),
expected.clone(),
],
)
.expect("select matching signed object");
assert_eq!(selected, encode_access_description_der(&expected).unwrap());
}
#[test]
fn rejects_missing_or_duplicate_matching_signed_object() {
let missing = select_manifest_signed_object_location(
MANIFEST_URI,
&[access_description(
OID_AD_RPKI_NOTIFY,
"https://rrdp.example.test/notification.xml",
)],
)
.expect_err("missing signed object must fail");
assert!(missing.contains("contains 0"), "{missing}");
let duplicate = select_manifest_signed_object_location(
MANIFEST_URI,
&[
access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI),
access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI),
],
)
.expect_err("duplicate signed object must fail");
assert!(duplicate.contains("contains 2"), "{duplicate}");
}
#[test]
fn selects_matching_signed_object_from_historical_projection() {
let signed_object = access_description(OID_AD_SIGNED_OBJECT, MANIFEST_URI);
let signed_object_der = encode_access_description_der(&signed_object).unwrap();
let notify_der = encode_access_description_der(&access_description(
OID_AD_RPKI_NOTIFY,
"https://rrdp.example.test/notification.xml",
))
.unwrap();
let selected = select_manifest_signed_object_location_from_der(
MANIFEST_URI,
&[notify_der, signed_object_der.clone()],
)
.expect("select historical signed object");
assert_eq!(selected, signed_object_der);
}
#[test]
fn rejects_malformed_historical_projection() {
let error = select_manifest_signed_object_location_from_der(
MANIFEST_URI,
&[vec![0x30, 0x01, 0x06]],
)
.expect_err("malformed historical projection must fail");
assert!(error.contains("truncated DER"), "{error}");
}
#[test]
fn access_description_der_codec_covers_long_and_invalid_forms() {
let long = access_description(
"2.999.200.1",
&format!("rsync://example.test/repo/{}", "x".repeat(160)),
);
let encoded = encode_access_description_der(&long).expect("encode long access description");
assert_eq!(
decode_access_description_der(&encoded).expect("decode long access description"),
long
);
let mut trailing = encoded.clone();
trailing.push(0);
assert!(
decode_access_description_der(&trailing)
.expect_err("trailing bytes must fail")
.contains("trailing bytes")
);
let oid = encode_oid_der(OID_AD_SIGNED_OBJECT).expect("encode signedObject OID");
let uri = encode_tlv(0x86, MANIFEST_URI.as_bytes().to_vec());
assert!(
decode_access_description_der(&encode_sequence(&[
oid.clone(),
uri.clone(),
encode_tlv(0x05, Vec::new()),
]))
.expect_err("trailing field must fail")
.contains("trailing fields")
);
assert!(
decode_access_description_der(&encode_sequence(&[oid, encode_tlv(0x86, vec![0xff]),]))
.expect_err("non-UTF8 URI must fail")
.contains("not UTF-8")
);
assert!(
decode_access_description_der(&encode_sequence(&[
encode_tlv(0x06, vec![0x80, 0x00]),
uri.clone(),
]))
.expect_err("non-minimal OID must fail")
.contains("non-minimal")
);
assert!(
decode_access_description_der(&encode_sequence(&[encode_tlv(0x06, vec![0x81]), uri,]))
.expect_err("truncated OID must fail")
.contains("truncated OBJECT IDENTIFIER")
);
assert!(
encode_access_description_der(&access_description("3.1", MANIFEST_URI))
.expect_err("invalid OID must fail")
.contains("unsupported accessMethod OID")
);
}
#[test]
fn access_description_der_codec_reports_malformed_selector_inputs() {
let malformed_sequence = decode_access_description_der(&[0x31, 0x00])
.expect_err("non-sequence AccessDescription must fail");
assert!(!malformed_sequence.is_empty());
let missing_method = decode_access_description_der(&encode_sequence(&[encode_tlv(
0x86,
MANIFEST_URI.as_bytes().to_vec(),
)]))
.expect_err("missing accessMethod must fail");
assert!(!missing_method.is_empty());
let oid = encode_oid_der(OID_AD_SIGNED_OBJECT).expect("encode signedObject OID");
let missing_location = decode_access_description_der(&encode_sequence(&[oid.clone()]))
.expect_err("missing accessLocation must fail");
assert!(!missing_location.is_empty());
let wrong_location_tag = decode_access_description_der(&encode_sequence(&[
oid.clone(),
encode_tlv(0x04, MANIFEST_URI.as_bytes().to_vec()),
]))
.expect_err("wrong accessLocation tag must fail");
assert!(!wrong_location_tag.is_empty());
let empty_oid = decode_access_description_der(&encode_sequence(&[
encode_tlv(0x06, Vec::new()),
encode_tlv(0x86, MANIFEST_URI.as_bytes().to_vec()),
]))
.expect_err("empty OID must fail");
assert!(!empty_oid.is_empty());
let first_arc_zero = decode_access_description_der(&encode_sequence(&[
encode_tlv(0x06, vec![0x01, 0x02]),
encode_tlv(0x86, MANIFEST_URI.as_bytes().to_vec()),
]))
.expect("decode first OID arc zero");
assert_eq!(first_arc_zero.access_method_oid, "0.1.2");
assert_eq!(first_arc_zero.access_location, MANIFEST_URI);
let non_numeric =
encode_access_description_der(&access_description("no.such.oid", MANIFEST_URI))
.expect_err("non-numeric OID must fail");
assert!(!non_numeric.is_empty());
let too_short = encode_access_description_der(&access_description("1", MANIFEST_URI))
.expect_err("OID without second arc must fail");
assert!(!too_short.is_empty());
let invalid_second_arc =
encode_access_description_der(&access_description("1.40", MANIFEST_URI))
.expect_err("invalid second OID arc must fail");
assert!(!invalid_second_arc.is_empty());
let overflowing_first_subidentifier = encode_access_description_der(&access_description(
"2.18446744073709551615",
MANIFEST_URI,
))
.expect_err("overflowing first OID subidentifier must fail");
assert!(!overflowing_first_subidentifier.is_empty());
}
}

View File

@ -0,0 +1,56 @@
#[cfg(feature = "full")]
pub mod accumulator;
#[cfg(feature = "full")]
pub mod build;
#[cfg(feature = "full")]
pub mod compare_view;
pub mod decode;
pub mod dump;
pub mod encode;
#[cfg(feature = "full")]
pub mod export;
pub mod hash;
#[cfg(feature = "full")]
pub(crate) mod manifest_location;
pub mod model;
pub mod state_digest;
#[cfg(feature = "full")]
pub mod verify;
#[cfg(feature = "full")]
pub use accumulator::{CcrAccumulator, CcrManifestContribution};
#[cfg(feature = "full")]
pub use build::{
CcrBuildError, ManifestStateBuildBreakdown, build_aspa_payload_state,
build_manifest_state_from_vcirs, build_manifest_state_from_vcirs_with_breakdown,
build_roa_payload_state, build_router_key_state_from_runtime, build_trust_anchor_state,
};
#[cfg(feature = "full")]
pub use compare_view::{
VapCompareRow, VrpCompareRow, build_vap_compare_rows, build_vrp_compare_rows,
canonical_vrp_prefix, decode_ccr_compare_views, write_vap_csv, write_vrp_csv,
};
pub use decode::{CcrDecodeError, decode_content_info};
pub use dump::{CcrDumpError, dump_content_info_json, dump_content_info_json_value};
pub use encode::{CcrEncodeError, encode_content_info};
#[cfg(feature = "full")]
pub use export::{
CcrBuildBreakdown, CcrExportError, build_ccr_from_run, build_ccr_from_run_with_breakdown,
write_ccr_file,
};
pub use hash::{compute_state_hash, verify_state_hash};
pub use model::{
AspaPayloadSet, AspaPayloadState, CcrContentInfo, CcrDigestAlgorithm, ManifestInstance,
ManifestState, RoaPayloadSet, RoaPayloadState, RouterKey, RouterKeySet, RouterKeyState,
RpkiCanonicalCacheRepresentation, TrustAnchorState,
};
pub use state_digest::{
CcrStateDigestComparison, CcrStateDigestError, CcrStateDigestStateComparison,
CcrStateDigestSummary, compare_state_digests, decode_state_digest_summary,
};
#[cfg(feature = "full")]
pub use verify::{
CcrVerifyError, CcrVerifySummary, extract_vrp_rows, verify_against_report_json_path,
verify_against_vcir_store, verify_against_vcir_store_path, verify_content_info,
verify_content_info_bytes,
};

View File

@ -0,0 +1,398 @@
use crate::data_model::common::{BigUnsigned, der_take_tlv};
use crate::data_model::oid::{OID_CT_RPKI_CCR, OID_SHA256};
pub const CCR_VERSION_V0: u32 = 0;
pub const DIGEST_LEN_SHA256: usize = 32;
pub const KEY_IDENTIFIER_LEN_SHA1: usize = 20;
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum CcrDigestAlgorithm {
Sha256,
}
impl CcrDigestAlgorithm {
pub fn oid(&self) -> &'static str {
match self {
Self::Sha256 => OID_SHA256,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CcrContentInfo {
pub content_type_oid: String,
pub content: RpkiCanonicalCacheRepresentation,
}
impl CcrContentInfo {
pub fn new(content: RpkiCanonicalCacheRepresentation) -> Self {
Self {
content_type_oid: OID_CT_RPKI_CCR.to_string(),
content,
}
}
pub fn validate(&self) -> Result<(), String> {
if self.content_type_oid != OID_CT_RPKI_CCR {
return Err(format!(
"contentType must be {OID_CT_RPKI_CCR}, got {}",
self.content_type_oid
));
}
self.content.validate()
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RpkiCanonicalCacheRepresentation {
pub version: u32,
pub hash_alg: CcrDigestAlgorithm,
pub produced_at: time::OffsetDateTime,
pub mfts: Option<ManifestState>,
pub vrps: Option<RoaPayloadState>,
pub vaps: Option<AspaPayloadState>,
pub tas: Option<TrustAnchorState>,
pub rks: Option<RouterKeyState>,
}
impl RpkiCanonicalCacheRepresentation {
pub fn validate(&self) -> Result<(), String> {
if self.version != CCR_VERSION_V0 {
return Err(format!("CCR version must be 0, got {}", self.version));
}
if !matches!(self.hash_alg, CcrDigestAlgorithm::Sha256) {
return Err("CCR hashAlg must be SHA-256".into());
}
if self.mfts.is_none()
&& self.vrps.is_none()
&& self.vaps.is_none()
&& self.tas.is_none()
&& self.rks.is_none()
{
return Err("at least one of mfts/vrps/vaps/tas/rks must be present".into());
}
if let Some(mfts) = &self.mfts {
mfts.validate()?;
}
if let Some(vrps) = &self.vrps {
vrps.validate()?;
}
if let Some(vaps) = &self.vaps {
vaps.validate()?;
}
if let Some(tas) = &self.tas {
tas.validate()?;
}
if let Some(rks) = &self.rks {
rks.validate()?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ManifestState {
pub mis: Vec<ManifestInstance>,
pub most_recent_update: time::OffsetDateTime,
pub hash: Vec<u8>,
}
impl ManifestState {
pub fn validate(&self) -> Result<(), String> {
validate_sha256_digest("ManifestState.hash", &self.hash)?;
validate_sorted_unique_by(
&self.mis,
|item| item.hash.as_slice(),
"ManifestState.mis must be sorted by hash and unique",
)?;
for instance in &self.mis {
instance.validate()?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ManifestInstance {
pub hash: Vec<u8>,
pub size: u64,
pub aki: Vec<u8>,
pub manifest_number: BigUnsigned,
pub this_update: time::OffsetDateTime,
pub locations: Vec<Vec<u8>>,
pub subordinates: Vec<Vec<u8>>,
}
impl ManifestInstance {
pub fn validate(&self) -> Result<(), String> {
validate_sha256_digest("ManifestInstance.hash", &self.hash)?;
if self.size < 1000 {
return Err(format!(
"ManifestInstance.size must be >= 1000, got {}",
self.size
));
}
validate_key_identifier("ManifestInstance.aki", &self.aki)?;
validate_big_unsigned_bytes(
"ManifestInstance.manifest_number",
&self.manifest_number.bytes_be,
)?;
if self.locations.is_empty() {
return Err(
"ManifestInstance.locations must contain at least one AccessDescription".into(),
);
}
for location in &self.locations {
validate_full_der_with_tag("ManifestInstance.locations[]", location, Some(0x30))?;
}
if !self.subordinates.is_empty() {
validate_sorted_unique_bytes(
&self.subordinates,
KEY_IDENTIFIER_LEN_SHA1,
"ManifestInstance.subordinates must be sorted/unique 20-byte SKIs",
)?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaPayloadState {
pub rps: Vec<RoaPayloadSet>,
pub hash: Vec<u8>,
}
impl RoaPayloadState {
pub fn validate(&self) -> Result<(), String> {
validate_sha256_digest("ROAPayloadState.hash", &self.hash)?;
validate_sorted_unique_by(
&self.rps,
|item| &item.as_id,
"ROAPayloadState.rps must be sorted by asID and unique",
)?;
for set in &self.rps {
set.validate()?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaPayloadSet {
pub as_id: u32,
pub ip_addr_blocks: Vec<Vec<u8>>,
}
impl RoaPayloadSet {
pub fn validate(&self) -> Result<(), String> {
if self.ip_addr_blocks.is_empty() || self.ip_addr_blocks.len() > 2 {
return Err(format!(
"ROAPayloadSet.ip_addr_blocks must contain 1..=2 entries, got {}",
self.ip_addr_blocks.len()
));
}
for block in &self.ip_addr_blocks {
validate_full_der_with_tag("ROAPayloadSet.ip_addr_blocks[]", block, Some(0x30))?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AspaPayloadState {
pub aps: Vec<AspaPayloadSet>,
pub hash: Vec<u8>,
}
impl AspaPayloadState {
pub fn validate(&self) -> Result<(), String> {
validate_sha256_digest("ASPAPayloadState.hash", &self.hash)?;
validate_sorted_unique_by(
&self.aps,
|item| &item.customer_as_id,
"ASPAPayloadState.aps must be sorted by customerASID and unique",
)?;
for set in &self.aps {
set.validate()?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AspaPayloadSet {
pub customer_as_id: u32,
pub providers: Vec<u32>,
}
impl AspaPayloadSet {
pub fn validate(&self) -> Result<(), String> {
if self.providers.is_empty() {
return Err("ASPAPayloadSet.providers must be non-empty".into());
}
validate_sorted_unique_by(
&self.providers,
|provider| provider,
"ASPAPayloadSet.providers must be sorted ascending and unique",
)
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TrustAnchorState {
pub skis: Vec<Vec<u8>>,
pub hash: Vec<u8>,
}
impl TrustAnchorState {
pub fn validate(&self) -> Result<(), String> {
if self.skis.is_empty() {
return Err("TrustAnchorState.skis must be non-empty".into());
}
validate_sha256_digest("TrustAnchorState.hash", &self.hash)?;
validate_sorted_unique_bytes(
&self.skis,
KEY_IDENTIFIER_LEN_SHA1,
"TrustAnchorState.skis must be sorted/unique 20-byte SKIs",
)
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RouterKeyState {
pub rksets: Vec<RouterKeySet>,
pub hash: Vec<u8>,
}
impl RouterKeyState {
pub fn validate(&self) -> Result<(), String> {
validate_sha256_digest("RouterKeyState.hash", &self.hash)?;
validate_sorted_unique_by(
&self.rksets,
|item| &item.as_id,
"RouterKeyState.rksets must be sorted by asID and unique",
)?;
for rkset in &self.rksets {
rkset.validate()?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RouterKeySet {
pub as_id: u32,
pub router_keys: Vec<RouterKey>,
}
impl RouterKeySet {
pub fn validate(&self) -> Result<(), String> {
if self.router_keys.is_empty() {
return Err("RouterKeySet.router_keys must be non-empty".into());
}
validate_sorted_unique_by(
&self.router_keys,
|key| key,
"RouterKeySet.router_keys must be sorted by SKI and unique by (SKI, SPKI DER)",
)?;
for key in &self.router_keys {
key.validate()?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct RouterKey {
pub ski: Vec<u8>,
pub spki_der: Vec<u8>,
}
impl RouterKey {
pub fn validate(&self) -> Result<(), String> {
validate_key_identifier("RouterKey.ski", &self.ski)?;
validate_full_der_with_tag("RouterKey.spki_der", &self.spki_der, Some(0x30))
}
}
fn validate_sha256_digest(field: &str, bytes: &[u8]) -> Result<(), String> {
if bytes.len() != DIGEST_LEN_SHA256 {
return Err(format!(
"{field} must be {DIGEST_LEN_SHA256} bytes, got {}",
bytes.len()
));
}
Ok(())
}
fn validate_key_identifier(field: &str, bytes: &[u8]) -> Result<(), String> {
if bytes.len() != KEY_IDENTIFIER_LEN_SHA1 {
return Err(format!(
"{field} must be {KEY_IDENTIFIER_LEN_SHA1} bytes, got {}",
bytes.len()
));
}
Ok(())
}
fn validate_big_unsigned_bytes(field: &str, bytes: &[u8]) -> Result<(), String> {
if bytes.is_empty() {
return Err(format!("{field} must not be empty"));
}
if bytes.len() > 1 && bytes[0] == 0x00 {
return Err(format!(
"{field} must be minimally encoded as an unsigned integer"
));
}
Ok(())
}
fn validate_sorted_unique_by<T, K: Ord + ?Sized>(
values: &[T],
key_fn: impl Fn(&T) -> &K,
message: &str,
) -> Result<(), String> {
for window in values.windows(2) {
if key_fn(&window[0]) >= key_fn(&window[1]) {
return Err(message.to_string());
}
}
Ok(())
}
fn validate_sorted_unique_bytes(
values: &[Vec<u8>],
expected_len: usize,
message: &str,
) -> Result<(), String> {
for value in values {
if value.len() != expected_len {
return Err(message.to_string());
}
}
for window in values.windows(2) {
if window[0] >= window[1] {
return Err(message.to_string());
}
}
Ok(())
}
fn validate_full_der_with_tag(
field: &str,
der: &[u8],
expected_tag: Option<u8>,
) -> Result<(), String> {
let (tag, _value, rem) = der_take_tlv(der).map_err(|e| format!("{field}: {e}"))?;
if !rem.is_empty() {
return Err(format!("{field}: trailing bytes after DER object"));
}
if let Some(expected_tag) = expected_tag {
if tag != expected_tag {
return Err(format!(
"{field}: unexpected tag 0x{tag:02X}, expected 0x{expected_tag:02X}"
));
}
}
Ok(())
}

View File

@ -0,0 +1,332 @@
use crate::data_model::common::DerReader;
use crate::data_model::oid::{OID_CT_RPKI_CCR_RAW, OID_SHA256_RAW};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CcrStateDigestSummary {
pub version: u32,
pub hash_alg_oid: String,
pub mfts: Option<Vec<u8>>,
pub vrps: Option<Vec<u8>>,
pub vaps: Option<Vec<u8>>,
pub tas: Option<Vec<u8>>,
pub rks: Option<Vec<u8>>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CcrStateDigestComparison {
pub ours: CcrStateDigestSummary,
pub peer: CcrStateDigestSummary,
pub states: Vec<CcrStateDigestStateComparison>,
}
impl CcrStateDigestComparison {
pub fn matches(&self) -> bool {
self.ours.version == self.peer.version
&& self.ours.hash_alg_oid == self.peer.hash_alg_oid
&& self.states.iter().all(|state| state.matches)
}
pub fn mismatched_state_names(&self) -> Vec<&'static str> {
self.states
.iter()
.filter(|state| !state.matches)
.map(|state| state.name)
.collect()
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CcrStateDigestStateComparison {
pub name: &'static str,
pub ours_present: bool,
pub peer_present: bool,
pub ours_hash_hex: Option<String>,
pub peer_hash_hex: Option<String>,
pub matches: bool,
}
#[derive(Debug, thiserror::Error)]
pub enum CcrStateDigestError {
#[error("DER parse error: {0}")]
Parse(String),
#[error("unexpected contentType OID")]
UnexpectedContentType,
#[error("unexpected digest algorithm OID")]
UnexpectedDigestAlgorithm,
#[error("unexpected CCR field tag 0x{0:02X}")]
UnexpectedCcrField(u8),
}
pub fn decode_state_digest_summary(
der: &[u8],
) -> Result<CcrStateDigestSummary, CcrStateDigestError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrStateDigestError::Parse)?;
if !top.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing bytes after ContentInfo".into(),
));
}
let content_type_raw = seq.take_tag(0x06).map_err(CcrStateDigestError::Parse)?;
if content_type_raw != OID_CT_RPKI_CCR_RAW {
return Err(CcrStateDigestError::UnexpectedContentType);
}
let inner = seq.take_tag(0xA0).map_err(CcrStateDigestError::Parse)?;
if !seq.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing fields in ContentInfo".into(),
));
}
decode_ccr_state_digest_summary(inner)
}
pub fn compare_state_digests(
ours_der: &[u8],
peer_der: &[u8],
) -> Result<CcrStateDigestComparison, CcrStateDigestError> {
let ours = decode_state_digest_summary(ours_der)?;
let peer = decode_state_digest_summary(peer_der)?;
let states = vec![
compare_state("mfts", &ours.mfts, &peer.mfts),
compare_state("vrps", &ours.vrps, &peer.vrps),
compare_state("vaps", &ours.vaps, &peer.vaps),
compare_state("tas", &ours.tas, &peer.tas),
compare_state("rks", &ours.rks, &peer.rks),
];
Ok(CcrStateDigestComparison { ours, peer, states })
}
fn decode_ccr_state_digest_summary(
der: &[u8],
) -> Result<CcrStateDigestSummary, CcrStateDigestError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CcrStateDigestError::Parse)?;
if !top.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing bytes after CCR".into(),
));
}
let version = if !seq.is_empty() && seq.peek_tag().map_err(CcrStateDigestError::Parse)? == 0xA0
{
let explicit = seq.take_tag(0xA0).map_err(CcrStateDigestError::Parse)?;
let mut inner = DerReader::new(explicit);
let version = inner.take_uint_u64().map_err(CcrStateDigestError::Parse)? as u32;
if !inner.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing bytes inside CCR version EXPLICIT".into(),
));
}
version
} else {
crate::ccr::model::CCR_VERSION_V0
};
let hash_alg_oid =
decode_digest_algorithm_oid(seq.take_sequence().map_err(CcrStateDigestError::Parse)?)?;
let _produced_at = seq.take_tag(0x18).map_err(CcrStateDigestError::Parse)?;
let mut mfts = None;
let mut vrps = None;
let mut vaps = None;
let mut tas = None;
let mut rks = None;
while !seq.is_empty() {
let (tag, value) = seq.take_any().map_err(CcrStateDigestError::Parse)?;
match tag {
0xA1 => mfts = Some(read_state_hash(value, StateShape::Manifest)?),
0xA2 => vrps = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
0xA3 => vaps = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
0xA4 => tas = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
0xA5 => rks = Some(read_state_hash(value, StateShape::PayloadThenHash)?),
other => return Err(CcrStateDigestError::UnexpectedCcrField(other)),
}
}
Ok(CcrStateDigestSummary {
version,
hash_alg_oid,
mfts,
vrps,
vaps,
tas,
rks,
})
}
#[derive(Clone, Copy)]
enum StateShape {
Manifest,
PayloadThenHash,
}
fn read_state_hash(explicit_der: &[u8], shape: StateShape) -> Result<Vec<u8>, CcrStateDigestError> {
let mut outer = DerReader::new(explicit_der);
let mut seq = outer.take_sequence().map_err(CcrStateDigestError::Parse)?;
if !outer.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing bytes after CCR state".into(),
));
}
match shape {
StateShape::Manifest => {
seq.skip_any().map_err(CcrStateDigestError::Parse)?;
seq.take_tag(0x18).map_err(CcrStateDigestError::Parse)?;
}
StateShape::PayloadThenHash => {
seq.skip_any().map_err(CcrStateDigestError::Parse)?;
}
}
let hash = seq
.take_octet_string()
.map_err(CcrStateDigestError::Parse)?
.to_vec();
if hash.len() != crate::ccr::model::DIGEST_LEN_SHA256 {
return Err(CcrStateDigestError::Parse(format!(
"state hash must be {} bytes, got {}",
crate::ccr::model::DIGEST_LEN_SHA256,
hash.len()
)));
}
if !seq.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing fields after CCR state hash".into(),
));
}
Ok(hash)
}
fn decode_digest_algorithm_oid(mut seq: DerReader<'_>) -> Result<String, CcrStateDigestError> {
let oid_raw = seq.take_tag(0x06).map_err(CcrStateDigestError::Parse)?;
if oid_raw != OID_SHA256_RAW {
return Err(CcrStateDigestError::UnexpectedDigestAlgorithm);
}
if !seq.is_empty() {
let tag = seq.peek_tag().map_err(CcrStateDigestError::Parse)?;
if tag == 0x05 {
let null = seq.take_tag(0x05).map_err(CcrStateDigestError::Parse)?;
if !null.is_empty() {
return Err(CcrStateDigestError::Parse(
"AlgorithmIdentifier NULL parameters must be empty".into(),
));
}
}
}
if !seq.is_empty() {
return Err(CcrStateDigestError::Parse(
"trailing fields in DigestAlgorithmIdentifier".into(),
));
}
Ok(crate::data_model::oid::OID_SHA256.to_string())
}
fn compare_state(
name: &'static str,
ours: &Option<Vec<u8>>,
peer: &Option<Vec<u8>>,
) -> CcrStateDigestStateComparison {
CcrStateDigestStateComparison {
name,
ours_present: ours.is_some(),
peer_present: peer.is_some(),
ours_hash_hex: ours.as_ref().map(hex::encode),
peer_hash_hex: peer.as_ref().map(hex::encode),
matches: ours == peer,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ccr::{
CcrContentInfo, CcrDigestAlgorithm, RpkiCanonicalCacheRepresentation,
build_aspa_payload_state, build_roa_payload_state, encode_content_info,
};
use crate::data_model::roa::{IpPrefix, RoaAfi};
use crate::validation::objects::{AspaAttestation, Vrp};
fn sample_content(produced_at: time::OffsetDateTime) -> CcrContentInfo {
let vrps = build_roa_payload_state(&[Vrp {
asn: 64496,
prefix: IpPrefix {
afi: RoaAfi::Ipv4,
prefix_len: 24,
addr: [192, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length: 24,
}])
.expect("build vrps");
let vaps = build_aspa_payload_state(&[AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64497],
}])
.expect("build vaps");
CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at,
mfts: None,
vrps: Some(vrps),
vaps: Some(vaps),
tas: None,
rks: None,
})
}
#[test]
fn decode_state_digest_summary_extracts_hashes_without_full_model() {
let content = sample_content(time::OffsetDateTime::UNIX_EPOCH);
let der = encode_content_info(&content).expect("encode");
let summary = decode_state_digest_summary(&der).expect("summary");
assert_eq!(summary.version, 0);
assert_eq!(summary.hash_alg_oid, crate::data_model::oid::OID_SHA256);
assert_eq!(
summary.vrps.as_deref(),
content
.content
.vrps
.as_ref()
.map(|state| state.hash.as_slice())
);
assert_eq!(
summary.vaps.as_deref(),
content
.content
.vaps
.as_ref()
.map(|state| state.hash.as_slice())
);
assert!(summary.mfts.is_none());
assert!(summary.tas.is_none());
assert!(summary.rks.is_none());
}
#[test]
fn compare_state_digests_ignores_produced_at() {
let first = encode_content_info(&sample_content(time::OffsetDateTime::UNIX_EPOCH))
.expect("encode first");
let second = encode_content_info(&sample_content(
time::OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(60),
))
.expect("encode second");
let comparison = compare_state_digests(&first, &second).expect("compare");
assert!(comparison.matches());
assert!(comparison.mismatched_state_names().is_empty());
}
#[test]
fn compare_state_digests_reports_mismatched_state() {
let first = encode_content_info(&sample_content(time::OffsetDateTime::UNIX_EPOCH))
.expect("encode first");
let mut second_content =
sample_content(time::OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(60));
second_content.content.vaps.as_mut().expect("vaps").hash[0] ^= 0xFF;
let second = encode_content_info(&second_content).expect("encode second");
let comparison = compare_state_digests(&first, &second).expect("compare");
assert!(!comparison.matches());
assert_eq!(comparison.mismatched_state_names(), vec!["vaps"]);
}
}

View File

@ -0,0 +1,704 @@
use crate::ccr::decode::{CcrDecodeError, decode_content_info};
use crate::ccr::encode::{
encode_aspa_payload_state_payload_der, encode_manifest_state_payload_der,
encode_roa_payload_state_payload_der, encode_router_key_state_payload_der,
encode_trust_anchor_state_payload_der,
};
use crate::ccr::hash::verify_state_hash;
use crate::ccr::model::{CcrContentInfo, RouterKeyState, TrustAnchorState};
use crate::storage::{RocksStore, VcirArtifactRole};
use serde::Serialize;
use std::collections::BTreeSet;
use std::path::Path;
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct CcrVerifySummary {
pub content_type_oid: String,
pub version: u32,
pub produced_at_rfc3339_utc: String,
pub state_hashes_ok: bool,
pub manifest_instances: usize,
pub roa_payload_sets: usize,
pub roa_vrp_count: usize,
pub aspa_payload_sets: usize,
pub trust_anchor_ski_count: usize,
pub router_key_sets: usize,
pub router_key_count: usize,
}
#[derive(Debug, thiserror::Error)]
pub enum CcrVerifyError {
#[error("CCR decode failed: {0}")]
Decode(#[from] CcrDecodeError),
#[error("ManifestState hash mismatch")]
ManifestHashMismatch,
#[error("ROAPayloadState hash mismatch")]
RoaHashMismatch,
#[error("ASPAPayloadState hash mismatch")]
AspaHashMismatch,
#[error("TrustAnchorState hash mismatch")]
TrustAnchorHashMismatch,
#[error("RouterKeyState hash mismatch")]
RouterKeyHashMismatch,
#[error("read report json failed: {0}: {1}")]
ReportRead(String, String),
#[error("parse report json failed: {0}")]
ReportParse(String),
#[error("VRP set mismatch: only_in_ccr={only_in_ccr} only_in_report={only_in_report}")]
ReportVrpMismatch {
only_in_ccr: usize,
only_in_report: usize,
},
#[error("ASPA set mismatch: only_in_ccr={only_in_ccr} only_in_report={only_in_report}")]
ReportAspaMismatch {
only_in_ccr: usize,
only_in_report: usize,
},
#[error("open RocksDB failed: {0}")]
OpenStore(String),
#[error("list VCIRs failed: {0}")]
ListVcirs(String),
#[error("VCIR manifest set mismatch: only_in_ccr={only_in_ccr} only_in_vcir={only_in_vcir}")]
VcirManifestMismatch {
only_in_ccr: usize,
only_in_vcir: usize,
},
}
pub fn verify_content_info_bytes(der: &[u8]) -> Result<CcrVerifySummary, CcrVerifyError> {
let content_info = decode_content_info(der)?;
verify_content_info(&content_info)
}
pub fn verify_content_info(
content_info: &CcrContentInfo,
) -> Result<CcrVerifySummary, CcrVerifyError> {
content_info.validate().map_err(CcrDecodeError::Validate)?;
let state_hashes_ok = true;
let mut manifest_instances = 0usize;
let mut roa_payload_sets = 0usize;
let mut roa_vrp_count = 0usize;
let mut aspa_payload_sets = 0usize;
let mut trust_anchor_ski_count = 0usize;
let mut router_key_sets = 0usize;
let mut router_key_count = 0usize;
if let Some(mfts) = &content_info.content.mfts {
let payload_der = encode_manifest_state_payload_der(&mfts.mis)
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
if !verify_state_hash(&mfts.hash, &payload_der) {
return Err(CcrVerifyError::ManifestHashMismatch);
}
manifest_instances = mfts.mis.len();
}
if let Some(vrps) = &content_info.content.vrps {
let payload_der = encode_roa_payload_state_payload_der(&vrps.rps)
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
if !verify_state_hash(&vrps.hash, &payload_der) {
return Err(CcrVerifyError::RoaHashMismatch);
}
roa_payload_sets = vrps.rps.len();
roa_vrp_count = vrps
.rps
.iter()
.map(|set| count_roa_block_entries(&set.ip_addr_blocks))
.sum();
}
if let Some(vaps) = &content_info.content.vaps {
let payload_der = encode_aspa_payload_state_payload_der(&vaps.aps)
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
if !verify_state_hash(&vaps.hash, &payload_der) {
return Err(CcrVerifyError::AspaHashMismatch);
}
aspa_payload_sets = vaps.aps.len();
}
if let Some(tas) = &content_info.content.tas {
verify_trust_anchor_state_hash(tas)?;
trust_anchor_ski_count = tas.skis.len();
}
if let Some(rks) = &content_info.content.rks {
verify_router_key_state_hash(rks)?;
router_key_sets = rks.rksets.len();
router_key_count = rks.rksets.iter().map(|set| set.router_keys.len()).sum();
}
let produced_at_rfc3339_utc = content_info
.content
.produced_at
.to_offset(time::UtcOffset::UTC)
.format(&time::format_description::well_known::Rfc3339)
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
Ok(CcrVerifySummary {
content_type_oid: content_info.content_type_oid.clone(),
version: content_info.content.version,
produced_at_rfc3339_utc,
state_hashes_ok,
manifest_instances,
roa_payload_sets,
roa_vrp_count,
aspa_payload_sets,
trust_anchor_ski_count,
router_key_sets,
router_key_count,
})
}
pub fn verify_against_report_json_path(
content_info: &CcrContentInfo,
report_json_path: &Path,
) -> Result<(), CcrVerifyError> {
let bytes = std::fs::read(report_json_path).map_err(|e| {
CcrVerifyError::ReportRead(report_json_path.display().to_string(), e.to_string())
})?;
let json: serde_json::Value =
serde_json::from_slice(&bytes).map_err(|e| CcrVerifyError::ReportParse(e.to_string()))?;
let report_vrps = report_vrp_keys(&json)?;
let ccr_vrps = extract_vrp_rows(content_info)?;
let only_in_ccr = ccr_vrps.difference(&report_vrps).count();
let only_in_report = report_vrps.difference(&ccr_vrps).count();
if only_in_ccr != 0 || only_in_report != 0 {
return Err(CcrVerifyError::ReportVrpMismatch {
only_in_ccr,
only_in_report,
});
}
let report_aspas = report_aspa_keys(&json)?;
let ccr_aspas = ccr_aspa_keys(content_info)?;
let only_in_ccr = ccr_aspas.difference(&report_aspas).count();
let only_in_report = report_aspas.difference(&ccr_aspas).count();
if only_in_ccr != 0 || only_in_report != 0 {
return Err(CcrVerifyError::ReportAspaMismatch {
only_in_ccr,
only_in_report,
});
}
Ok(())
}
pub fn verify_against_vcir_store_path(
content_info: &CcrContentInfo,
db_path: &Path,
) -> Result<(), CcrVerifyError> {
let store = RocksStore::open(db_path).map_err(|e| CcrVerifyError::OpenStore(e.to_string()))?;
verify_against_vcir_store(content_info, &store)
}
pub fn verify_against_vcir_store(
content_info: &CcrContentInfo,
store: &RocksStore,
) -> Result<(), CcrVerifyError> {
let Some(mfts) = &content_info.content.mfts else {
return Ok(());
};
let vcirs = store
.list_vcirs()
.map_err(|e| CcrVerifyError::ListVcirs(e.to_string()))?;
let mut vcir_hashes = BTreeSet::new();
for vcir in vcirs {
if let Some(artifact) = vcir.related_artifacts.iter().find(|artifact| {
artifact.artifact_role == VcirArtifactRole::Manifest
&& artifact.uri.as_deref() == Some(vcir.current_manifest_rsync_uri.as_str())
}) {
if let Ok(bytes) = hex::decode(&artifact.sha256) {
vcir_hashes.insert(bytes);
}
}
}
let ccr_hashes = mfts
.mis
.iter()
.map(|mi| mi.hash.clone())
.collect::<BTreeSet<_>>();
let only_in_ccr = ccr_hashes.difference(&vcir_hashes).count();
let only_in_vcir = vcir_hashes.difference(&ccr_hashes).count();
if only_in_ccr != 0 || only_in_vcir != 0 {
return Err(CcrVerifyError::VcirManifestMismatch {
only_in_ccr,
only_in_vcir,
});
}
Ok(())
}
fn verify_trust_anchor_state_hash(state: &TrustAnchorState) -> Result<(), CcrVerifyError> {
let payload_der = encode_trust_anchor_state_payload_der(&state.skis)
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
if !verify_state_hash(&state.hash, &payload_der) {
return Err(CcrVerifyError::TrustAnchorHashMismatch);
}
Ok(())
}
fn verify_router_key_state_hash(state: &RouterKeyState) -> Result<(), CcrVerifyError> {
let payload_der = encode_router_key_state_payload_der(&state.rksets)
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Validate(e.to_string())))?;
if !verify_state_hash(&state.hash, &payload_der) {
return Err(CcrVerifyError::RouterKeyHashMismatch);
}
Ok(())
}
fn report_vrp_keys(
json: &serde_json::Value,
) -> Result<BTreeSet<(u32, String, u16)>, CcrVerifyError> {
let mut out = BTreeSet::new();
let Some(items) = json.get("vrps").and_then(|v| v.as_array()) else {
return Ok(out);
};
for item in items {
let asn = item
.get("asn")
.and_then(|v| v.as_u64())
.ok_or_else(|| CcrVerifyError::ReportParse("vrps[].asn missing".into()))?
as u32;
let prefix = item
.get("prefix")
.and_then(|v| v.as_str())
.ok_or_else(|| CcrVerifyError::ReportParse("vrps[].prefix missing".into()))?
.to_string();
let max_length = item
.get("max_length")
.and_then(|v| v.as_u64())
.ok_or_else(|| CcrVerifyError::ReportParse("vrps[].max_length missing".into()))?
as u16;
out.insert((asn, prefix, max_length));
}
Ok(out)
}
fn report_aspa_keys(json: &serde_json::Value) -> Result<BTreeSet<(u32, Vec<u32>)>, CcrVerifyError> {
let mut out = BTreeSet::new();
let Some(items) = json.get("aspas").and_then(|v| v.as_array()) else {
return Ok(out);
};
for item in items {
let customer = item
.get("customer_as_id")
.and_then(|v| v.as_u64())
.ok_or_else(|| CcrVerifyError::ReportParse("aspas[].customer_as_id missing".into()))?
as u32;
let mut providers = item
.get("provider_as_ids")
.and_then(|v| v.as_array())
.ok_or_else(|| CcrVerifyError::ReportParse("aspas[].provider_as_ids missing".into()))?
.iter()
.map(|v| {
v.as_u64()
.ok_or_else(|| CcrVerifyError::ReportParse("provider_as_ids[] invalid".into()))
.map(|v| v as u32)
})
.collect::<Result<Vec<_>, _>>()?;
providers.sort_unstable();
providers.dedup();
out.insert((customer, providers));
}
Ok(out)
}
pub fn extract_vrp_rows(
content_info: &CcrContentInfo,
) -> Result<BTreeSet<(u32, String, u16)>, CcrVerifyError> {
let mut out = BTreeSet::new();
let Some(vrps) = &content_info.content.vrps else {
return Ok(out);
};
for set in &vrps.rps {
for block in &set.ip_addr_blocks {
let (afi, entries) = decode_roa_family_block(block)?;
for (prefix_len, addr_bytes, max_len) in entries {
let prefix = format_prefix(afi, &addr_bytes, prefix_len)?;
out.insert((set.as_id, prefix, max_len.unwrap_or(prefix_len as u16)));
}
}
}
Ok(out)
}
fn ccr_aspa_keys(
content_info: &CcrContentInfo,
) -> Result<BTreeSet<(u32, Vec<u32>)>, CcrVerifyError> {
let mut out = BTreeSet::new();
let Some(vaps) = &content_info.content.vaps else {
return Ok(out);
};
for set in &vaps.aps {
out.insert((set.customer_as_id, set.providers.clone()));
}
Ok(out)
}
fn decode_roa_family_block(
block: &[u8],
) -> Result<(u16, Vec<(u8, Vec<u8>, Option<u16>)>), CcrVerifyError> {
let mut top = crate::data_model::common::DerReader::new(block);
let mut seq = top
.take_sequence()
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
if !top.is_empty() {
return Err(CcrVerifyError::Decode(CcrDecodeError::Parse(
"trailing bytes after ROAIPAddressFamily".into(),
)));
}
let afi_bytes = seq
.take_octet_string()
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
let afi = u16::from_be_bytes([afi_bytes[0], afi_bytes[1]]);
let mut addrs = seq
.take_sequence()
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
let mut entries = Vec::new();
while !addrs.is_empty() {
let mut addr_seq = addrs
.take_sequence()
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
let (unused_bits, content) = addr_seq
.take_bit_string()
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?;
let prefix_len = (content.len() * 8) as u8 - unused_bits;
let max_len = if addr_seq.is_empty() {
None
} else {
Some(
addr_seq
.take_uint_u64()
.map_err(|e| CcrVerifyError::Decode(CcrDecodeError::Parse(e)))?
as u16,
)
};
entries.push((prefix_len, content.to_vec(), max_len));
}
Ok((afi, entries))
}
fn format_prefix(afi: u16, addr_bytes: &[u8], prefix_len: u8) -> Result<String, CcrVerifyError> {
match afi {
1 => {
let mut full = [0u8; 4];
full[..addr_bytes.len()].copy_from_slice(addr_bytes);
Ok(format!("{}/{prefix_len}", std::net::Ipv4Addr::from(full)))
}
2 => {
let mut full = [0u8; 16];
full[..addr_bytes.len()].copy_from_slice(addr_bytes);
Ok(format!("{}/{prefix_len}", std::net::Ipv6Addr::from(full)))
}
other => Err(CcrVerifyError::Decode(CcrDecodeError::Parse(format!(
"unsupported AFI {other}"
)))),
}
}
fn count_roa_block_entries(blocks: &[Vec<u8>]) -> usize {
blocks
.iter()
.map(|block| {
decode_roa_family_block(block)
.map(|(_, entries)| entries.len())
.unwrap_or(0)
})
.sum()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ccr::build::{build_aspa_payload_state, build_roa_payload_state};
use crate::ccr::encode::{
encode_manifest_state_payload_der, encode_roa_payload_state_payload_der,
encode_router_key_state_payload_der, encode_trust_anchor_state_payload_der,
};
use crate::ccr::model::{
CcrDigestAlgorithm, ManifestInstance, ManifestState, RouterKey, RouterKeySet,
RpkiCanonicalCacheRepresentation,
};
use crate::data_model::common::BigUnsigned;
use crate::data_model::roa::{IpPrefix, RoaAfi};
use crate::storage::{
PackTime, ValidatedCaInstanceResult, ValidatedManifestMeta, VcirArtifactKind,
VcirArtifactRole, VcirArtifactValidationStatus, VcirAuditSummary,
VcirCcrManifestProjection, VcirChildEntry, VcirInstanceGate, VcirRelatedArtifact,
VcirSummary,
};
use crate::validation::objects::{AspaAttestation, Vrp};
fn sample_time() -> time::OffsetDateTime {
time::OffsetDateTime::parse(
"2026-03-24T00:00:00Z",
&time::format_description::well_known::Rfc3339,
)
.expect("time")
}
fn sample_content_info() -> CcrContentInfo {
let mis = vec![ManifestInstance {
hash: vec![0x10; 32],
size: 2048,
aki: vec![0x20; 20],
manifest_number: BigUnsigned { bytes_be: vec![1] },
this_update: sample_time(),
locations: vec![vec![0x30, 0x00]],
subordinates: vec![vec![0x30; 20]],
}];
let mfts = ManifestState {
most_recent_update: sample_time(),
hash: crate::ccr::compute_state_hash(&encode_manifest_state_payload_der(&mis).unwrap()),
mis,
};
let vrps = build_roa_payload_state(&[Vrp {
asn: 64496,
prefix: IpPrefix {
afi: RoaAfi::Ipv4,
prefix_len: 0,
addr: [0; 16],
},
max_length: 0,
}])
.expect("build roa state");
let vaps = build_aspa_payload_state(&[AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64497],
}])
.expect("build aspa state");
let skis = vec![vec![0x11; 20]];
let tas = TrustAnchorState {
hash: crate::ccr::compute_state_hash(
&encode_trust_anchor_state_payload_der(&skis).unwrap(),
),
skis,
};
let rksets = vec![RouterKeySet {
as_id: 64496,
router_keys: vec![RouterKey {
ski: vec![0x22; 20],
spki_der: vec![0x30, 0x00],
}],
}];
let rks = RouterKeyState {
hash: crate::ccr::compute_state_hash(
&encode_router_key_state_payload_der(&rksets).unwrap(),
),
rksets,
};
CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at: sample_time(),
mfts: Some(mfts),
vrps: Some(vrps),
vaps: Some(vaps),
tas: Some(tas),
rks: Some(rks),
})
}
fn sample_ccr_manifest_projection(manifest_rsync_uri: &str) -> VcirCcrManifestProjection {
VcirCcrManifestProjection {
manifest_rsync_uri: manifest_rsync_uri.to_string(),
manifest_sha256: vec![0x44; 32],
manifest_size: 2048,
manifest_ee_aki: vec![0x55; 20],
manifest_number_be: vec![1],
manifest_this_update: PackTime::from_utc_offset_datetime(sample_time()),
manifest_sia_locations_der: vec![vec![
0x30, 0x11, 0x06, 0x08, 0x2b, 0x06, 0x01, 0x05, 0x05, 0x07, 0x30, 0x05, 0x86, 0x05,
b'r', b's', b'y', b'n', b'c',
]],
subordinate_skis: vec![vec![0x33; 20]],
}
}
#[test]
fn verify_detects_each_state_hash_mismatch() {
let mut ci = sample_content_info();
ci.content.vrps.as_mut().unwrap().hash[0] ^= 0x01;
assert!(matches!(
verify_content_info(&ci),
Err(CcrVerifyError::RoaHashMismatch)
));
let mut ci = sample_content_info();
ci.content.vaps.as_mut().unwrap().hash[0] ^= 0x01;
assert!(matches!(
verify_content_info(&ci),
Err(CcrVerifyError::AspaHashMismatch)
));
let mut ci = sample_content_info();
ci.content.tas.as_mut().unwrap().hash[0] ^= 0x01;
assert!(matches!(
verify_content_info(&ci),
Err(CcrVerifyError::TrustAnchorHashMismatch)
));
let mut ci = sample_content_info();
ci.content.rks.as_mut().unwrap().hash[0] ^= 0x01;
assert!(matches!(
verify_content_info(&ci),
Err(CcrVerifyError::RouterKeyHashMismatch)
));
}
#[test]
fn verify_against_report_json_accepts_matching_report_and_rejects_parse_errors() {
let td = tempfile::tempdir().expect("tempdir");
let report = serde_json::json!({
"vrps": [{"asn": 64496, "prefix": "0.0.0.0/0", "max_length": 0}],
"aspas": [{"customer_as_id": 64496, "provider_as_ids": [64497]}]
});
let report_path = td.path().join("report.json");
std::fs::write(&report_path, serde_json::to_vec(&report).unwrap()).unwrap();
verify_against_report_json_path(&sample_content_info(), &report_path)
.expect("matching report");
let bad_path = td.path().join("bad.json");
std::fs::write(&bad_path, b"not-json").unwrap();
assert!(matches!(
verify_against_report_json_path(&sample_content_info(), &bad_path),
Err(CcrVerifyError::ReportParse(_))
));
}
#[test]
fn verify_against_report_json_rejects_missing_fields_and_aspa_mismatch() {
let td = tempfile::tempdir().expect("tempdir");
let missing =
serde_json::json!({"vrps":[{"prefix":"0.0.0.0/0","max_length":0}],"aspas":[]});
let missing_path = td.path().join("missing.json");
std::fs::write(&missing_path, serde_json::to_vec(&missing).unwrap()).unwrap();
assert!(matches!(
verify_against_report_json_path(&sample_content_info(), &missing_path),
Err(CcrVerifyError::ReportParse(_))
));
let mismatch = serde_json::json!({
"vrps": [{"asn": 64496, "prefix": "0.0.0.0/0", "max_length": 0}],
"aspas": [{"customer_as_id": 64496, "provider_as_ids": [65000]}]
});
let mismatch_path = td.path().join("mismatch.json");
std::fs::write(&mismatch_path, serde_json::to_vec(&mismatch).unwrap()).unwrap();
assert!(matches!(
verify_against_report_json_path(&sample_content_info(), &mismatch_path),
Err(CcrVerifyError::ReportAspaMismatch { .. })
));
}
#[test]
fn verify_against_vcir_store_rejects_mismatched_manifest_hashes() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open db");
let vcir = ValidatedCaInstanceResult {
manifest_rsync_uri: "rsync://example.test/current.mft".to_string(),
parent_manifest_rsync_uri: None,
tal_id: "apnic".to_string(),
ca_subject_name: "CN=test".to_string(),
ca_ski: "11".repeat(20),
issuer_ski: "22".repeat(20),
last_successful_validation_time: PackTime::from_utc_offset_datetime(sample_time()),
current_manifest_rsync_uri: "rsync://example.test/current.mft".to_string(),
current_crl_rsync_uri: "rsync://example.test/current.crl".to_string(),
validated_manifest_meta: ValidatedManifestMeta {
validated_manifest_number: vec![1],
validated_manifest_this_update: PackTime::from_utc_offset_datetime(sample_time()),
validated_manifest_next_update: PackTime::from_utc_offset_datetime(sample_time()),
},
ccr_manifest_projection: sample_ccr_manifest_projection(
"rsync://example.test/current.mft",
),
instance_gate: VcirInstanceGate {
manifest_next_update: PackTime::from_utc_offset_datetime(sample_time()),
current_crl_next_update: PackTime::from_utc_offset_datetime(sample_time()),
self_ca_not_after: PackTime::from_utc_offset_datetime(sample_time()),
instance_effective_until: PackTime::from_utc_offset_datetime(sample_time()),
},
child_entries: vec![VcirChildEntry {
child_manifest_rsync_uri: "rsync://example.test/child.mft".to_string(),
child_cert_rsync_uri: "rsync://example.test/child.cer".to_string(),
child_cert_hash: "aa".repeat(32),
child_ski: "33".repeat(20),
child_rsync_base_uri: "rsync://example.test/".to_string(),
child_publication_point_rsync_uri: "rsync://example.test/".to_string(),
child_rrdp_notification_uri: None,
child_effective_ip_resources: None,
child_effective_as_resources: None,
accepted_at_validation_time: PackTime::from_utc_offset_datetime(sample_time()),
}],
local_outputs: Vec::new(),
related_artifacts: vec![VcirRelatedArtifact {
artifact_role: VcirArtifactRole::Manifest,
artifact_kind: VcirArtifactKind::Mft,
uri: Some("rsync://example.test/current.mft".to_string()),
sha256: "ff".repeat(32),
object_type: Some("mft".to_string()),
validation_status: VcirArtifactValidationStatus::Accepted,
reject_reason: None,
}],
summary: VcirSummary {
local_vrp_count: 0,
local_aspa_count: 0,
local_router_key_count: 0,
child_count: 1,
accepted_object_count: 1,
rejected_object_count: 0,
},
audit_summary: VcirAuditSummary {
failed_fetch_eligible: true,
last_failed_fetch_reason: None,
warning_count: 0,
audit_flags: Vec::new(),
},
};
store.put_vcir(&vcir).unwrap();
assert!(matches!(
verify_against_vcir_store(&sample_content_info(), &store),
Err(CcrVerifyError::VcirManifestMismatch { .. })
));
}
#[test]
fn verify_vrp_helpers_reject_bad_afi_and_count_invalid_block_as_zero() {
let block = vec![0x30, 0x08, 0x04, 0x02, 0x00, 0x63, 0x30, 0x02, 0x03, 0x00];
let ci = CcrContentInfo::new(RpkiCanonicalCacheRepresentation {
version: 0,
hash_alg: CcrDigestAlgorithm::Sha256,
produced_at: sample_time(),
mfts: None,
vrps: Some(crate::ccr::model::RoaPayloadState {
rps: vec![crate::ccr::model::RoaPayloadSet {
as_id: 64496,
ip_addr_blocks: vec![block.clone()],
}],
hash: crate::ccr::compute_state_hash(
&encode_roa_payload_state_payload_der(&[crate::ccr::model::RoaPayloadSet {
as_id: 64496,
ip_addr_blocks: vec![block],
}])
.unwrap(),
),
}),
vaps: None,
tas: None,
rks: None,
});
assert!(matches!(
extract_vrp_rows(&ci),
Err(CcrVerifyError::Decode(_))
));
let bad_count = count_roa_block_entries(&[vec![0x04, 0x00]]);
assert_eq!(bad_count, 0);
}
}

View File

@ -0,0 +1,406 @@
use std::collections::HashMap;
use crate::audit::{AuditObjectResult, ObjectAuditEntry};
use crate::cir::export::CirExportError;
use crate::cir::model::{CirObject, CirRejectedObject};
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum CirInputSection {
Fresh,
Cached,
}
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct CirInputSnapshot {
pub fresh_validated_objects: Vec<CirObject>,
pub cached_validated_objects: Vec<CirObject>,
pub fresh_rejected_objects: Vec<CirRejectedObject>,
pub cached_rejected_objects: Vec<CirRejectedObject>,
}
#[derive(Clone, Debug, Default)]
pub struct CirInputAccumulator {
fresh_objects: HashMap<String, [u8; 32]>,
cached_objects: HashMap<String, [u8; 32]>,
fresh_rejects: HashMap<String, Option<String>>,
cached_rejects: HashMap<String, Option<String>>,
}
impl CirInputAccumulator {
pub fn submit_audit_entries(
&mut self,
section: CirInputSection,
entries: &[ObjectAuditEntry],
) -> Result<(), CirExportError> {
for entry in entries {
if !matches!(
entry.result,
AuditObjectResult::Ok | AuditObjectResult::Error
) {
continue;
}
self.insert_object(section, &entry.rsync_uri, &entry.sha256_hex)?;
if entry.result == AuditObjectResult::Error {
self.insert_reject(section, &entry.rsync_uri, entry.detail.clone());
}
}
Ok(())
}
pub fn submit_snapshot(&mut self, snapshot: CirInputSnapshot) -> Result<(), CirExportError> {
for object in snapshot.fresh_validated_objects {
self.insert_object_bytes(CirInputSection::Fresh, &object.rsync_uri, object.sha256)?;
}
for object in snapshot.cached_validated_objects {
self.insert_object_bytes(CirInputSection::Cached, &object.rsync_uri, object.sha256)?;
}
for rejected in snapshot.fresh_rejected_objects {
self.insert_reject(
CirInputSection::Fresh,
&rejected.object_uri,
rejected.reason,
);
}
for rejected in snapshot.cached_rejected_objects {
self.insert_reject(
CirInputSection::Cached,
&rejected.object_uri,
rejected.reason,
);
}
Ok(())
}
pub fn insert_object(
&mut self,
section: CirInputSection,
rsync_uri: &str,
sha256_hex: &str,
) -> Result<(), CirExportError> {
if !rsync_uri.starts_with("rsync://") || !is_sha256_hex(sha256_hex) {
return Ok(());
}
let mut digest = [0u8; 32];
hex::decode_to_slice(sha256_hex, &mut digest).expect("validated sha256 hex");
self.insert_object_digest(section, rsync_uri, digest)
}
fn insert_object_bytes(
&mut self,
section: CirInputSection,
rsync_uri: &str,
sha256: Vec<u8>,
) -> Result<(), CirExportError> {
if !rsync_uri.starts_with("rsync://") {
return Ok(());
}
let Ok(digest) = <[u8; 32]>::try_from(sha256) else {
return Ok(());
};
self.insert_object_digest(section, rsync_uri, digest)
}
fn insert_object_digest(
&mut self,
section: CirInputSection,
rsync_uri: &str,
digest: [u8; 32],
) -> Result<(), CirExportError> {
let objects = match section {
CirInputSection::Fresh => &mut self.fresh_objects,
CirInputSection::Cached => &mut self.cached_objects,
};
if let Some(existing) = objects.get(rsync_uri) {
if existing != &digest {
return Err(CirExportError::ConflictingObjectHash {
rsync_uri: rsync_uri.to_string(),
first: hex::encode(existing),
second: hex::encode(digest),
});
}
return Ok(());
}
objects.insert(rsync_uri.to_string(), digest);
Ok(())
}
pub fn insert_reject(
&mut self,
section: CirInputSection,
rsync_uri: &str,
reason: Option<String>,
) {
if !rsync_uri.starts_with("rsync://") {
return;
}
let rejects = match section {
CirInputSection::Fresh => &mut self.fresh_rejects,
CirInputSection::Cached => &mut self.cached_rejects,
};
rejects.entry(rsync_uri.to_string()).or_insert(reason);
}
/// Merge another accumulator into this one. Object digests conflict-check
/// exactly like repeated `insert_object` calls; rejects keep the first
/// reason seen, matching sequential submission order when `other` holds
/// later publication points than `self`.
pub fn merge(&mut self, other: CirInputAccumulator) -> Result<(), CirExportError> {
for (rsync_uri, digest) in other.fresh_objects {
self.insert_object_digest(CirInputSection::Fresh, &rsync_uri, digest)?;
}
for (rsync_uri, digest) in other.cached_objects {
self.insert_object_digest(CirInputSection::Cached, &rsync_uri, digest)?;
}
for (rsync_uri, reason) in other.fresh_rejects {
self.insert_reject(CirInputSection::Fresh, &rsync_uri, reason);
}
for (rsync_uri, reason) in other.cached_rejects {
self.insert_reject(CirInputSection::Cached, &rsync_uri, reason);
}
Ok(())
}
pub fn finalize(self) -> CirInputSnapshot {
CirInputSnapshot {
fresh_validated_objects: finalize_objects(self.fresh_objects),
cached_validated_objects: finalize_objects(self.cached_objects),
fresh_rejected_objects: finalize_rejects(self.fresh_rejects),
cached_rejected_objects: finalize_rejects(self.cached_rejects),
}
}
}
fn finalize_objects(objects: HashMap<String, [u8; 32]>) -> Vec<CirObject> {
let mut objects = objects
.into_iter()
.map(|(rsync_uri, sha256)| CirObject {
rsync_uri,
sha256: sha256.to_vec(),
})
.collect::<Vec<_>>();
objects.sort_by(|a, b| a.rsync_uri.cmp(&b.rsync_uri));
objects
}
fn finalize_rejects(rejects: HashMap<String, Option<String>>) -> Vec<CirRejectedObject> {
let mut rejects = rejects
.into_iter()
.map(|(object_uri, reason)| CirRejectedObject { object_uri, reason })
.collect::<Vec<_>>();
rejects.sort_by(|a, b| a.object_uri.cmp(&b.object_uri));
rejects
}
fn is_sha256_hex(value: &str) -> bool {
value.len() == 64 && value.as_bytes().iter().all(u8::is_ascii_hexdigit)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::audit::{AuditObjectKind, AuditObjectResult};
fn entry(uri: &str, hash_byte: u8, result: AuditObjectResult) -> ObjectAuditEntry {
let is_error = result == AuditObjectResult::Error;
ObjectAuditEntry {
rsync_uri: uri.to_string(),
sha256_hex: format!("{}", hex::encode([hash_byte; 32])),
kind: AuditObjectKind::Roa,
result,
detail: is_error.then(|| "invalid".to_string()),
}
}
#[test]
fn accumulator_finalizes_sorted_fresh_and_cached_sections() {
let mut acc = CirInputAccumulator::default();
acc.submit_audit_entries(
CirInputSection::Fresh,
&[
entry("rsync://example.net/z.roa", 0x22, AuditObjectResult::Ok),
entry("rsync://example.net/a.roa", 0x11, AuditObjectResult::Error),
],
)
.unwrap();
acc.submit_audit_entries(
CirInputSection::Cached,
&[entry(
"rsync://example.net/c.roa",
0x33,
AuditObjectResult::Ok,
)],
)
.unwrap();
let snapshot = acc.finalize();
assert_eq!(
snapshot.fresh_validated_objects[0].rsync_uri,
"rsync://example.net/a.roa"
);
assert_eq!(
snapshot.fresh_validated_objects[1].rsync_uri,
"rsync://example.net/z.roa"
);
assert_eq!(
snapshot.cached_validated_objects[0].rsync_uri,
"rsync://example.net/c.roa"
);
assert_eq!(
snapshot.fresh_rejected_objects[0].object_uri,
"rsync://example.net/a.roa"
);
assert!(snapshot.cached_rejected_objects.is_empty());
}
#[test]
fn accumulator_rejects_conflicting_hashes_within_same_section() {
let mut acc = CirInputAccumulator::default();
acc.submit_audit_entries(
CirInputSection::Fresh,
&[entry(
"rsync://example.net/a.roa",
0x11,
AuditObjectResult::Ok,
)],
)
.unwrap();
let err = acc
.submit_audit_entries(
CirInputSection::Fresh,
&[entry(
"rsync://example.net/a.roa",
0x22,
AuditObjectResult::Ok,
)],
)
.unwrap_err();
assert!(matches!(err, CirExportError::ConflictingObjectHash { .. }));
}
#[test]
fn accumulator_merges_finalized_snapshots() {
let snapshot = CirInputSnapshot {
fresh_validated_objects: vec![CirObject {
rsync_uri: "rsync://example.net/fresh.roa".to_string(),
sha256: vec![0x11; 32],
}],
cached_validated_objects: vec![CirObject {
rsync_uri: "rsync://example.net/cached.roa".to_string(),
sha256: vec![0x22; 32],
}],
fresh_rejected_objects: vec![CirRejectedObject {
object_uri: "rsync://example.net/fresh.roa".to_string(),
reason: Some("fresh invalid".to_string()),
}],
cached_rejected_objects: Vec::new(),
};
let mut acc = CirInputAccumulator::default();
acc.submit_snapshot(snapshot).unwrap();
let merged = acc.finalize();
assert_eq!(merged.fresh_validated_objects.len(), 1);
assert_eq!(merged.cached_validated_objects.len(), 1);
assert_eq!(merged.fresh_rejected_objects.len(), 1);
}
#[test]
fn accumulator_merge_matches_sequential_submission() {
let mut sequential = CirInputAccumulator::default();
sequential
.submit_audit_entries(
CirInputSection::Fresh,
&[
entry("rsync://example.net/a.roa", 0x11, AuditObjectResult::Ok),
entry("rsync://example.net/b.roa", 0x22, AuditObjectResult::Error),
],
)
.unwrap();
sequential
.submit_audit_entries(
CirInputSection::Cached,
&[entry(
"rsync://example.net/c.roa",
0x33,
AuditObjectResult::Ok,
)],
)
.unwrap();
let mut first = CirInputAccumulator::default();
first
.submit_audit_entries(
CirInputSection::Fresh,
&[entry(
"rsync://example.net/a.roa",
0x11,
AuditObjectResult::Ok,
)],
)
.unwrap();
let mut second = CirInputAccumulator::default();
second
.submit_audit_entries(
CirInputSection::Fresh,
&[entry(
"rsync://example.net/b.roa",
0x22,
AuditObjectResult::Error,
)],
)
.unwrap();
second
.submit_audit_entries(
CirInputSection::Cached,
&[entry(
"rsync://example.net/c.roa",
0x33,
AuditObjectResult::Ok,
)],
)
.unwrap();
first.merge(second).unwrap();
assert_eq!(sequential.finalize(), first.finalize());
}
#[test]
fn accumulator_merge_detects_conflicting_hashes_across_shards() {
let mut first = CirInputAccumulator::default();
first
.insert_object(
CirInputSection::Fresh,
"rsync://example.net/a.roa",
&"11".repeat(32),
)
.unwrap();
let mut second = CirInputAccumulator::default();
second
.insert_object(
CirInputSection::Fresh,
"rsync://example.net/a.roa",
&"22".repeat(32),
)
.unwrap();
let err = first.merge(second).unwrap_err();
assert!(matches!(err, CirExportError::ConflictingObjectHash { .. }));
}
#[test]
fn accumulator_merge_keeps_first_reject_reason() {
let mut first = CirInputAccumulator::default();
first.insert_reject(
CirInputSection::Fresh,
"rsync://example.net/a.roa",
Some("first".to_string()),
);
let mut second = CirInputAccumulator::default();
second.insert_reject(
CirInputSection::Fresh,
"rsync://example.net/a.roa",
Some("second".to_string()),
);
first.merge(second).unwrap();
let snapshot = first.finalize();
assert_eq!(
snapshot.fresh_rejected_objects[0].reason,
Some("first".to_string())
);
}
}

View File

@ -0,0 +1,286 @@
use crate::cir::model::{
CIR_VERSION_V4, CanonicalInputRepresentation, CirHashAlgorithm, CirObject, CirRejectedObject,
CirTrustAnchor,
};
use crate::data_model::common::DerReader;
use crate::data_model::oid::{OID_SHA256, OID_SHA256_RAW};
use der_parser::der::parse_der_oid;
#[derive(Debug, thiserror::Error)]
pub enum CirDecodeError {
#[error("DER parse error: {0}")]
Parse(String),
#[error("unexpected CIR version: expected {expected}, got {actual}")]
UnexpectedVersion { expected: u32, actual: u32 },
#[error("unexpected digest algorithm OID: expected {expected}, got {actual}")]
UnexpectedDigestAlgorithm {
expected: &'static str,
actual: String,
},
#[error("CIR model validation failed after decode: {0}")]
Validate(String),
}
pub fn decode_cir(der: &[u8]) -> Result<CanonicalInputRepresentation, CirDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
if !top.is_empty() {
return Err(CirDecodeError::Parse("trailing bytes after CIR".into()));
}
let version = seq.take_uint_u64().map_err(CirDecodeError::Parse)? as u32;
if version != CIR_VERSION_V4 {
return Err(CirDecodeError::UnexpectedVersion {
expected: CIR_VERSION_V4,
actual: version,
});
}
let hash_alg = decode_hash_alg(seq.take_tag(0x06).map_err(CirDecodeError::Parse)?)?;
let validation_time =
parse_generalized_time(seq.take_tag(0x18).map_err(CirDecodeError::Parse)?)?;
let fresh_objects_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
let mut fresh_objects_reader = DerReader::new(fresh_objects_der);
let mut fresh_validated_objects = Vec::new();
while !fresh_objects_reader.is_empty() {
let (_tag, full, _value) = fresh_objects_reader
.take_any_full()
.map_err(CirDecodeError::Parse)?;
fresh_validated_objects.push(decode_object(full)?);
}
let cached_objects_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
let mut cached_objects_reader = DerReader::new(cached_objects_der);
let mut cached_validated_objects = Vec::new();
while !cached_objects_reader.is_empty() {
let (_tag, full, _value) = cached_objects_reader
.take_any_full()
.map_err(CirDecodeError::Parse)?;
cached_validated_objects.push(decode_object(full)?);
}
let trust_anchors_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
let mut trust_anchors_reader = DerReader::new(trust_anchors_der);
let mut trust_anchors = Vec::new();
while !trust_anchors_reader.is_empty() {
let (_tag, full, _value) = trust_anchors_reader
.take_any_full()
.map_err(CirDecodeError::Parse)?;
trust_anchors.push(decode_trust_anchor(full)?);
}
let object_list_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let fresh_object_list_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let cached_object_list_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let reject_list_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let fresh_reject_list_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let cached_reject_list_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let fresh_rejected_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
let mut fresh_rejected_reader = DerReader::new(fresh_rejected_der);
let mut fresh_rejected_objects = Vec::new();
while !fresh_rejected_reader.is_empty() {
let (_tag, full, _value) = fresh_rejected_reader
.take_any_full()
.map_err(CirDecodeError::Parse)?;
fresh_rejected_objects.push(decode_rejected_object(full)?);
}
let cached_rejected_der = seq.take_tag(0x30).map_err(CirDecodeError::Parse)?;
let mut cached_rejected_reader = DerReader::new(cached_rejected_der);
let mut cached_rejected_objects = Vec::new();
while !cached_rejected_reader.is_empty() {
let (_tag, full, _value) = cached_rejected_reader
.take_any_full()
.map_err(CirDecodeError::Parse)?;
cached_rejected_objects.push(decode_rejected_object(full)?);
}
if !seq.is_empty() {
return Err(CirDecodeError::Parse("trailing fields in CIR".into()));
}
let cir = CanonicalInputRepresentation {
version,
hash_alg,
validation_time,
fresh_validated_objects,
cached_validated_objects,
trust_anchors,
object_list_sha256,
fresh_object_list_sha256,
cached_object_list_sha256,
reject_list_sha256,
fresh_reject_list_sha256,
cached_reject_list_sha256,
fresh_rejected_objects,
cached_rejected_objects,
};
cir.validate().map_err(CirDecodeError::Validate)?;
Ok(cir)
}
fn decode_hash_alg(raw_body: &[u8]) -> Result<CirHashAlgorithm, CirDecodeError> {
if raw_body != OID_SHA256_RAW {
return Err(CirDecodeError::UnexpectedDigestAlgorithm {
expected: OID_SHA256,
actual: oid_string(raw_body)?,
});
}
Ok(CirHashAlgorithm::Sha256)
}
fn decode_object(der: &[u8]) -> Result<CirObject, CirDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
if !top.is_empty() {
return Err(CirDecodeError::Parse(
"trailing bytes after CirObject".into(),
));
}
let rsync_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
.to_string();
let sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
if !seq.is_empty() {
return Err(CirDecodeError::Parse("trailing fields in CirObject".into()));
}
Ok(CirObject { rsync_uri, sha256 })
}
fn decode_trust_anchor(der: &[u8]) -> Result<CirTrustAnchor, CirDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
if !top.is_empty() {
return Err(CirDecodeError::Parse(
"trailing bytes after CirTrustAnchor".into(),
));
}
let ta_rsync_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
.to_string();
let tal_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
.to_string();
let tal_bytes = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let ta_certificate_der = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
let ta_certificate_sha256 = seq
.take_octet_string()
.map_err(CirDecodeError::Parse)?
.to_vec();
if !seq.is_empty() {
return Err(CirDecodeError::Parse(
"trailing fields in CirTrustAnchor".into(),
));
}
Ok(CirTrustAnchor {
ta_rsync_uri,
tal_uri,
tal_bytes,
ta_certificate_der,
ta_certificate_sha256,
})
}
fn decode_rejected_object(der: &[u8]) -> Result<CirRejectedObject, CirDecodeError> {
let mut top = DerReader::new(der);
let mut seq = top.take_sequence().map_err(CirDecodeError::Parse)?;
if !top.is_empty() {
return Err(CirDecodeError::Parse(
"trailing bytes after CirRejectedObject".into(),
));
}
let object_uri = std::str::from_utf8(seq.take_tag(0x16).map_err(CirDecodeError::Parse)?)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
.to_string();
let reason = if seq.is_empty() {
None
} else {
Some(
std::str::from_utf8(seq.take_octet_string().map_err(CirDecodeError::Parse)?)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?
.to_string(),
)
};
if !seq.is_empty() {
return Err(CirDecodeError::Parse(
"trailing fields in CirRejectedObject".into(),
));
}
Ok(CirRejectedObject { object_uri, reason })
}
fn oid_string(raw_body: &[u8]) -> Result<String, CirDecodeError> {
let der = {
let mut out = Vec::with_capacity(raw_body.len() + 2);
out.push(0x06);
if raw_body.len() < 0x80 {
out.push(raw_body.len() as u8);
} else {
return Err(CirDecodeError::Parse("OID too long".into()));
}
out.extend_from_slice(raw_body);
out
};
let (_rem, oid) = parse_der_oid(&der).map_err(|e| CirDecodeError::Parse(e.to_string()))?;
let oid = oid
.as_oid_val()
.map_err(|e| CirDecodeError::Parse(e.to_string()))?;
Ok(oid.to_string())
}
fn parse_generalized_time(bytes: &[u8]) -> Result<time::OffsetDateTime, CirDecodeError> {
let s = std::str::from_utf8(bytes).map_err(|e| CirDecodeError::Parse(e.to_string()))?;
if s.len() != 15 || !s.ends_with('Z') {
return Err(CirDecodeError::Parse(
"GeneralizedTime must be YYYYMMDDHHMMSSZ".into(),
));
}
let parse = |range: std::ops::Range<usize>| -> Result<u32, CirDecodeError> {
s[range]
.parse::<u32>()
.map_err(|e| CirDecodeError::Parse(e.to_string()))
};
let year = parse(0..4)? as i32;
let month = parse(4..6)? as u8;
let day = parse(6..8)? as u8;
let hour = parse(8..10)? as u8;
let minute = parse(10..12)? as u8;
let second = parse(12..14)? as u8;
let month = time::Month::try_from(month).map_err(|e| CirDecodeError::Parse(e.to_string()))?;
let date = time::Date::from_calendar_date(year, month, day)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?;
let timev = time::Time::from_hms(hour, minute, second)
.map_err(|e| CirDecodeError::Parse(e.to_string()))?;
Ok(time::PrimitiveDateTime::new(date, timev).assume_utc())
}

View File

@ -0,0 +1,179 @@
use crate::cir::model::{
CIR_VERSION_V4, CanonicalInputRepresentation, CirHashAlgorithm, CirObject, CirRejectedObject,
CirTrustAnchor,
};
use crate::data_model::oid::OID_SHA256_RAW;
#[derive(Debug, thiserror::Error)]
pub enum CirEncodeError {
#[error("CIR model validation failed: {0}")]
Validate(String),
}
pub fn encode_cir(cir: &CanonicalInputRepresentation) -> Result<Vec<u8>, CirEncodeError> {
cir.validate().map_err(CirEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_integer_u32(CIR_VERSION_V4),
encode_oid(match cir.hash_alg {
CirHashAlgorithm::Sha256 => OID_SHA256_RAW,
}),
encode_generalized_time(cir.validation_time),
encode_sequence(
&cir.fresh_validated_objects
.iter()
.map(encode_object)
.collect::<Result<Vec<_>, _>>()?,
),
encode_sequence(
&cir.cached_validated_objects
.iter()
.map(encode_object)
.collect::<Result<Vec<_>, _>>()?,
),
encode_sequence(
&cir.trust_anchors
.iter()
.map(encode_trust_anchor)
.collect::<Result<Vec<_>, _>>()?,
),
encode_octet_string(&cir.object_list_sha256),
encode_octet_string(&cir.fresh_object_list_sha256),
encode_octet_string(&cir.cached_object_list_sha256),
encode_octet_string(&cir.reject_list_sha256),
encode_octet_string(&cir.fresh_reject_list_sha256),
encode_octet_string(&cir.cached_reject_list_sha256),
encode_sequence(
&cir.fresh_rejected_objects
.iter()
.map(encode_rejected_object)
.collect::<Result<Vec<_>, _>>()?,
),
encode_sequence(
&cir.cached_rejected_objects
.iter()
.map(encode_rejected_object)
.collect::<Result<Vec<_>, _>>()?,
),
]))
}
fn encode_object(object: &CirObject) -> Result<Vec<u8>, CirEncodeError> {
object.validate().map_err(CirEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_ia5_string(object.rsync_uri.as_bytes()),
encode_octet_string(&object.sha256),
]))
}
fn encode_trust_anchor(trust_anchor: &CirTrustAnchor) -> Result<Vec<u8>, CirEncodeError> {
trust_anchor.validate().map_err(CirEncodeError::Validate)?;
Ok(encode_sequence(&[
encode_ia5_string(trust_anchor.ta_rsync_uri.as_bytes()),
encode_ia5_string(trust_anchor.tal_uri.as_bytes()),
encode_octet_string(&trust_anchor.tal_bytes),
encode_octet_string(&trust_anchor.ta_certificate_der),
encode_octet_string(&trust_anchor.ta_certificate_sha256),
]))
}
fn encode_rejected_object(item: &CirRejectedObject) -> Result<Vec<u8>, CirEncodeError> {
item.validate().map_err(CirEncodeError::Validate)?;
let mut fields = vec![encode_ia5_string(item.object_uri.as_bytes())];
if let Some(reason) = &item.reason {
fields.push(encode_octet_string(reason.as_bytes()));
}
Ok(encode_sequence(&fields))
}
fn encode_generalized_time(t: time::OffsetDateTime) -> Vec<u8> {
let t = t.to_offset(time::UtcOffset::UTC);
let s = format!(
"{:04}{:02}{:02}{:02}{:02}{:02}Z",
t.year(),
u8::from(t.month()),
t.day(),
t.hour(),
t.minute(),
t.second()
);
encode_tlv(0x18, s.into_bytes())
}
fn encode_integer_u32(v: u32) -> Vec<u8> {
encode_integer_bytes(unsigned_integer_bytes(v as u64))
}
fn encode_integer_bytes(mut bytes: Vec<u8>) -> Vec<u8> {
if bytes.is_empty() {
bytes.push(0);
}
if bytes[0] & 0x80 != 0 {
bytes.insert(0, 0);
}
encode_tlv(0x02, bytes)
}
fn unsigned_integer_bytes(v: u64) -> Vec<u8> {
if v == 0 {
return vec![0];
}
let mut out = Vec::new();
let mut n = v;
while n > 0 {
out.push((n & 0xFF) as u8);
n >>= 8;
}
out.reverse();
out
}
fn encode_oid(raw_body: &[u8]) -> Vec<u8> {
encode_tlv(0x06, raw_body.to_vec())
}
fn encode_ia5_string(bytes: &[u8]) -> Vec<u8> {
encode_tlv(0x16, bytes.to_vec())
}
fn encode_octet_string(bytes: &[u8]) -> Vec<u8> {
encode_tlv(0x04, bytes.to_vec())
}
fn encode_sequence(elements: &[Vec<u8>]) -> Vec<u8> {
let mut body = Vec::new();
for element in elements {
body.extend_from_slice(element);
}
encode_tlv(0x30, body)
}
fn encode_tlv(tag: u8, value: Vec<u8>) -> Vec<u8> {
let mut out = Vec::with_capacity(1 + encoded_len_len(value.len()) + value.len());
out.push(tag);
encode_len_into(value.len(), &mut out);
out.extend_from_slice(&value);
out
}
fn encoded_len_len(len: usize) -> usize {
if len < 0x80 {
1
} else {
1 + len.to_be_bytes().iter().skip_while(|&&b| b == 0).count()
}
}
fn encode_len_into(len: usize, out: &mut Vec<u8>) {
if len < 0x80 {
out.push(len as u8);
return;
}
let bytes = len.to_be_bytes();
let first_non_zero = bytes
.iter()
.position(|&b| b != 0)
.unwrap_or(bytes.len() - 1);
let len_bytes = &bytes[first_non_zero..];
out.push(0x80 | (len_bytes.len() as u8));
out.extend_from_slice(len_bytes);
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,867 @@
use std::collections::BTreeSet;
use std::fs;
use std::path::{Path, PathBuf};
use crate::blob_store::{ExternalRawStoreDb, ExternalRepoBytesDb, RawObjectStore};
use crate::cir::model::CanonicalInputRepresentation;
#[derive(Debug, thiserror::Error)]
pub enum CirMaterializeError {
#[error("invalid rsync URI: {0}")]
InvalidRsyncUri(String),
#[error("rsync URI must reference a file object, got directory-like URI: {0}")]
DirectoryLikeRsyncUri(String),
#[error("create mirror root failed: {path}: {detail}")]
CreateMirrorRoot { path: String, detail: String },
#[error("remove mirror root failed: {path}: {detail}")]
RemoveMirrorRoot { path: String, detail: String },
#[error("create parent directory failed: {path}: {detail}")]
CreateParent { path: String, detail: String },
#[error("remove existing target failed: {path}: {detail}")]
RemoveExistingTarget { path: String, detail: String },
#[error("static object not found for sha256={sha256_hex}")]
MissingStaticObject { sha256_hex: String },
#[error("link target failed: {src} -> {dst}: {detail}")]
Link {
src: String,
dst: String,
detail: String,
},
#[error("copy target failed: {src} -> {dst}: {detail}")]
Copy {
src: String,
dst: String,
detail: String,
},
#[error("mirror tree mismatch after materialize: {0}")]
TreeMismatch(String),
#[error("open raw store failed: {path}: {detail}")]
OpenRawStore { path: String, detail: String },
#[error("raw object not found for sha256={sha256_hex}")]
MissingRawStoreObject { sha256_hex: String },
#[error("read raw store failed for sha256={sha256_hex}: {detail}")]
ReadRawStore { sha256_hex: String, detail: String },
#[error("open repo bytes store failed: {path}: {detail}")]
OpenRepoBytesStore { path: String, detail: String },
#[error("repo bytes object not found for sha256={sha256_hex}")]
MissingRepoBytesObject { sha256_hex: String },
#[error("read repo bytes store failed for sha256={sha256_hex}: {detail}")]
ReadRepoBytesStore { sha256_hex: String, detail: String },
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirMaterializeSummary {
pub object_count: usize,
pub trust_anchor_count: usize,
pub materialized_file_count: usize,
pub linked_files: usize,
pub copied_files: usize,
}
pub fn materialize_cir(
cir: &CanonicalInputRepresentation,
static_root: &Path,
mirror_root: &Path,
clean_rebuild: bool,
) -> Result<CirMaterializeSummary, CirMaterializeError> {
cir.validate().map_err(CirMaterializeError::TreeMismatch)?;
prepare_mirror_root(mirror_root, clean_rebuild)?;
let mut linked_files = 0usize;
let mut copied_files = 0usize;
for object in cir.validated_objects() {
let sha256_hex = hex::encode(&object.sha256);
let source = resolve_static_pool_file(static_root, &sha256_hex)?;
let relative = mirror_relative_path_for_rsync_uri(&object.rsync_uri)?;
let target = mirror_root.join(&relative);
if let Some(parent) = target.parent() {
fs::create_dir_all(parent).map_err(|e| CirMaterializeError::CreateParent {
path: parent.display().to_string(),
detail: e.to_string(),
})?;
}
if target.exists() {
fs::remove_file(&target).map_err(|e| CirMaterializeError::RemoveExistingTarget {
path: target.display().to_string(),
detail: e.to_string(),
})?;
}
match fs::hard_link(&source, &target) {
Ok(()) => linked_files += 1,
Err(link_err) => {
fs::copy(&source, &target).map_err(|copy_err| CirMaterializeError::Copy {
src: source.display().to_string(),
dst: target.display().to_string(),
detail: format!("{copy_err}; original link error: {link_err}"),
})?;
copied_files += 1;
}
}
}
for trust_anchor in &cir.trust_anchors {
write_bytes_to_mirror_uri(
mirror_root,
&trust_anchor.ta_rsync_uri,
&trust_anchor.ta_certificate_der,
"cir trust anchor",
)?;
copied_files += 1;
}
let actual = collect_materialized_uris(mirror_root)?;
let expected = expected_materialized_uris(cir);
if actual != expected {
return Err(CirMaterializeError::TreeMismatch(format!(
"expected {} files, got {} files",
expected.len(),
actual.len()
)));
}
Ok(CirMaterializeSummary {
object_count: cir.validated_object_count(),
trust_anchor_count: cir.trust_anchors.len(),
materialized_file_count: expected.len(),
linked_files,
copied_files,
})
}
pub fn materialize_cir_from_raw_store(
cir: &CanonicalInputRepresentation,
raw_store_db: &Path,
mirror_root: &Path,
clean_rebuild: bool,
) -> Result<CirMaterializeSummary, CirMaterializeError> {
cir.validate().map_err(CirMaterializeError::TreeMismatch)?;
prepare_mirror_root(mirror_root, clean_rebuild)?;
let raw_store =
ExternalRawStoreDb::open(raw_store_db).map_err(|e| CirMaterializeError::OpenRawStore {
path: raw_store_db.display().to_string(),
detail: e.to_string(),
})?;
let mut copied_files = 0usize;
for object in cir.validated_objects() {
let sha256_hex = hex::encode(&object.sha256);
let bytes = raw_store
.get_blob_bytes(&sha256_hex)
.map_err(|e| CirMaterializeError::ReadRawStore {
sha256_hex: sha256_hex.clone(),
detail: e.to_string(),
})?
.ok_or_else(|| CirMaterializeError::MissingRawStoreObject {
sha256_hex: sha256_hex.clone(),
})?;
write_bytes_to_mirror_uri(
mirror_root,
&object.rsync_uri,
&bytes,
&raw_store_db.display().to_string(),
)?;
copied_files += 1;
}
for trust_anchor in &cir.trust_anchors {
write_bytes_to_mirror_uri(
mirror_root,
&trust_anchor.ta_rsync_uri,
&trust_anchor.ta_certificate_der,
"cir trust anchor",
)?;
copied_files += 1;
}
let actual = collect_materialized_uris(mirror_root)?;
let expected = expected_materialized_uris(cir);
if actual != expected {
return Err(CirMaterializeError::TreeMismatch(format!(
"expected {} files, got {} files",
expected.len(),
actual.len()
)));
}
Ok(CirMaterializeSummary {
object_count: cir.validated_object_count(),
trust_anchor_count: cir.trust_anchors.len(),
materialized_file_count: expected.len(),
linked_files: 0,
copied_files,
})
}
pub fn materialize_cir_from_repo_bytes(
cir: &CanonicalInputRepresentation,
repo_bytes_db: &Path,
mirror_root: &Path,
clean_rebuild: bool,
) -> Result<CirMaterializeSummary, CirMaterializeError> {
cir.validate().map_err(CirMaterializeError::TreeMismatch)?;
prepare_mirror_root(mirror_root, clean_rebuild)?;
let repo_bytes = ExternalRepoBytesDb::open(repo_bytes_db).map_err(|e| {
CirMaterializeError::OpenRepoBytesStore {
path: repo_bytes_db.display().to_string(),
detail: e.to_string(),
}
})?;
let mut copied_files = 0usize;
for object in cir.validated_objects() {
let sha256_hex = hex::encode(&object.sha256);
let bytes = repo_bytes
.get_blob_bytes(&sha256_hex)
.map_err(|e| CirMaterializeError::ReadRepoBytesStore {
sha256_hex: sha256_hex.clone(),
detail: e.to_string(),
})?
.ok_or_else(|| CirMaterializeError::MissingRepoBytesObject {
sha256_hex: sha256_hex.clone(),
})?;
write_bytes_to_mirror_uri(
mirror_root,
&object.rsync_uri,
&bytes,
&repo_bytes_db.display().to_string(),
)?;
copied_files += 1;
}
for trust_anchor in &cir.trust_anchors {
write_bytes_to_mirror_uri(
mirror_root,
&trust_anchor.ta_rsync_uri,
&trust_anchor.ta_certificate_der,
"cir trust anchor",
)?;
copied_files += 1;
}
let actual = collect_materialized_uris(mirror_root)?;
let expected = expected_materialized_uris(cir);
if actual != expected {
return Err(CirMaterializeError::TreeMismatch(format!(
"expected {} files, got {} files",
expected.len(),
actual.len()
)));
}
Ok(CirMaterializeSummary {
object_count: cir.validated_object_count(),
trust_anchor_count: cir.trust_anchors.len(),
materialized_file_count: expected.len(),
linked_files: 0,
copied_files,
})
}
fn prepare_mirror_root(mirror_root: &Path, clean_rebuild: bool) -> Result<(), CirMaterializeError> {
if clean_rebuild && mirror_root.exists() {
fs::remove_dir_all(mirror_root).map_err(|e| CirMaterializeError::RemoveMirrorRoot {
path: mirror_root.display().to_string(),
detail: e.to_string(),
})?;
}
fs::create_dir_all(mirror_root).map_err(|e| CirMaterializeError::CreateMirrorRoot {
path: mirror_root.display().to_string(),
detail: e.to_string(),
})
}
fn write_bytes_to_mirror_uri(
mirror_root: &Path,
rsync_uri: &str,
bytes: &[u8],
src_label: &str,
) -> Result<(), CirMaterializeError> {
let relative = mirror_relative_path_for_rsync_uri(rsync_uri)?;
let target = mirror_root.join(&relative);
if let Some(parent) = target.parent() {
fs::create_dir_all(parent).map_err(|e| CirMaterializeError::CreateParent {
path: parent.display().to_string(),
detail: e.to_string(),
})?;
}
if target.exists() {
fs::remove_file(&target).map_err(|e| CirMaterializeError::RemoveExistingTarget {
path: target.display().to_string(),
detail: e.to_string(),
})?;
}
fs::write(&target, bytes).map_err(|e| CirMaterializeError::Copy {
src: src_label.to_string(),
dst: target.display().to_string(),
detail: e.to_string(),
})
}
fn expected_materialized_uris(cir: &CanonicalInputRepresentation) -> BTreeSet<String> {
cir.validated_objects()
.map(|item| item.rsync_uri.clone())
.chain(
cir.trust_anchors
.iter()
.map(|item| item.ta_rsync_uri.clone()),
)
.collect()
}
pub fn mirror_relative_path_for_rsync_uri(rsync_uri: &str) -> Result<PathBuf, CirMaterializeError> {
let url = url::Url::parse(rsync_uri)
.map_err(|_| CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()))?;
if url.scheme() != "rsync" {
return Err(CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()));
}
let host = url
.host_str()
.ok_or_else(|| CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()))?;
let segments = url
.path_segments()
.ok_or_else(|| CirMaterializeError::InvalidRsyncUri(rsync_uri.to_string()))?
.collect::<Vec<_>>();
if segments.is_empty() || segments.last().copied().unwrap_or_default().is_empty() {
return Err(CirMaterializeError::DirectoryLikeRsyncUri(
rsync_uri.to_string(),
));
}
let mut path = PathBuf::from(host);
for segment in segments {
if !segment.is_empty() {
path.push(segment);
}
}
Ok(path)
}
pub fn resolve_static_pool_file(
static_root: &Path,
sha256_hex: &str,
) -> Result<PathBuf, CirMaterializeError> {
if sha256_hex.len() != 64 || !sha256_hex.as_bytes().iter().all(u8::is_ascii_hexdigit) {
return Err(CirMaterializeError::MissingStaticObject {
sha256_hex: sha256_hex.to_string(),
});
}
let prefix1 = &sha256_hex[0..2];
let prefix2 = &sha256_hex[2..4];
let entries =
fs::read_dir(static_root).map_err(|_| CirMaterializeError::MissingStaticObject {
sha256_hex: sha256_hex.to_string(),
})?;
let mut dates = entries
.filter_map(Result::ok)
.filter(|entry| entry.path().is_dir())
.map(|entry| entry.path())
.collect::<Vec<_>>();
dates.sort();
for date_dir in dates {
let candidate = date_dir.join(prefix1).join(prefix2).join(sha256_hex);
if candidate.is_file() {
return Ok(candidate);
}
}
Err(CirMaterializeError::MissingStaticObject {
sha256_hex: sha256_hex.to_string(),
})
}
fn collect_materialized_uris(mirror_root: &Path) -> Result<BTreeSet<String>, CirMaterializeError> {
let mut out = BTreeSet::new();
let mut stack = vec![mirror_root.to_path_buf()];
while let Some(path) = stack.pop() {
for entry in fs::read_dir(&path).map_err(|e| CirMaterializeError::CreateMirrorRoot {
path: path.display().to_string(),
detail: e.to_string(),
})? {
let entry = entry.map_err(|e| CirMaterializeError::CreateMirrorRoot {
path: path.display().to_string(),
detail: e.to_string(),
})?;
let path = entry.path();
if path.is_dir() {
stack.push(path);
} else {
let rel = path
.strip_prefix(mirror_root)
.expect("materialized path under mirror root")
.to_string_lossy()
.replace('\\', "/");
let uri = format!("rsync://{rel}");
out.insert(uri);
}
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::{
CirMaterializeError, materialize_cir, materialize_cir_from_raw_store,
materialize_cir_from_repo_bytes, mirror_relative_path_for_rsync_uri,
resolve_static_pool_file,
};
use crate::blob_store::{ExternalRawStoreDb, ExternalRepoBytesDb};
use crate::cir::model::{CanonicalInputRepresentation, CirObject, CirTrustAnchor, sha256};
use sha2::Digest;
use std::path::{Path, PathBuf};
fn sample_time() -> time::OffsetDateTime {
time::OffsetDateTime::parse(
"2026-04-07T12:34:56Z",
&time::format_description::well_known::Rfc3339,
)
.unwrap()
}
fn sample_trust_anchor() -> CirTrustAnchor {
let ta_rsync_uri = "rsync://example.net/repo/ta.cer";
let ta_certificate_der = b"ta-der".to_vec();
CirTrustAnchor {
ta_rsync_uri: ta_rsync_uri.to_string(),
tal_uri: "https://tal.example.net/root.tal".to_string(),
tal_bytes: format!("{ta_rsync_uri}\n\nAQID\n").into_bytes(),
ta_certificate_sha256: sha256(&ta_certificate_der),
ta_certificate_der,
}
}
fn sample_cir() -> CanonicalInputRepresentation {
CanonicalInputRepresentation::new_v4(
sample_time(),
vec![
CirObject {
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
sha256: hex::decode(
"1111111111111111111111111111111111111111111111111111111111111111",
)
.unwrap(),
},
CirObject {
rsync_uri: "rsync://example.net/repo/nested/b.roa".to_string(),
sha256: hex::decode(
"2222222222222222222222222222222222222222222222222222222222222222",
)
.unwrap(),
},
],
Vec::new(),
vec![sample_trust_anchor()],
Vec::new(),
Vec::new(),
)
}
fn cir_with_real_hashes(a: &[u8], b: &[u8]) -> CanonicalInputRepresentation {
CanonicalInputRepresentation::new_v4(
sample_time(),
vec![
CirObject {
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
sha256: sha2::Sha256::digest(a).to_vec(),
},
CirObject {
rsync_uri: "rsync://example.net/repo/nested/b.roa".to_string(),
sha256: sha2::Sha256::digest(b).to_vec(),
},
],
Vec::new(),
vec![sample_trust_anchor()],
Vec::new(),
Vec::new(),
)
}
#[test]
fn mirror_relative_path_for_rsync_uri_maps_host_and_path() {
let path =
mirror_relative_path_for_rsync_uri("rsync://example.net/repo/nested/b.roa").unwrap();
assert_eq!(
path,
PathBuf::from("example.net")
.join("repo")
.join("nested")
.join("b.roa")
);
}
#[test]
fn resolve_static_pool_file_finds_hash_across_dates() {
let td = tempfile::tempdir().unwrap();
let path = td.path().join("20260407").join("11").join("11");
std::fs::create_dir_all(&path).unwrap();
let file = path.join("1111111111111111111111111111111111111111111111111111111111111111");
std::fs::write(&file, b"x").unwrap();
let resolved = resolve_static_pool_file(
td.path(),
"1111111111111111111111111111111111111111111111111111111111111111",
)
.unwrap();
assert_eq!(resolved, file);
}
#[test]
fn resolve_static_pool_file_rejects_invalid_hash_and_missing_hash() {
let td = tempfile::tempdir().unwrap();
let err = resolve_static_pool_file(td.path(), "not-a-hash")
.expect_err("invalid hash should fail");
assert!(matches!(
err,
CirMaterializeError::MissingStaticObject { .. }
));
let err = resolve_static_pool_file(
td.path(),
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
)
.expect_err("missing hash should fail");
assert!(matches!(
err,
CirMaterializeError::MissingStaticObject { .. }
));
}
#[test]
fn mirror_relative_path_rejects_non_rsync_and_directory_like_uris() {
let err = mirror_relative_path_for_rsync_uri("https://example.net/repo/a.roa")
.expect_err("non-rsync uri must fail");
assert!(matches!(err, CirMaterializeError::InvalidRsyncUri(_)));
let err = mirror_relative_path_for_rsync_uri("rsync://example.net/repo/")
.expect_err("directory-like uri must fail");
assert!(matches!(err, CirMaterializeError::DirectoryLikeRsyncUri(_)));
}
#[test]
fn materialize_clean_rebuild_creates_exact_tree_and_removes_stale_files() {
let td = tempfile::tempdir().unwrap();
let static_root = td.path().join("static");
let mirror_root = td.path().join("mirror");
write_static(
&static_root,
"20260407",
"1111111111111111111111111111111111111111111111111111111111111111",
b"a",
);
write_static(
&static_root,
"20260407",
"2222222222222222222222222222222222222222222222222222222222222222",
b"b",
);
std::fs::create_dir_all(mirror_root.join("stale")).unwrap();
std::fs::write(mirror_root.join("stale/old.txt"), b"old").unwrap();
let summary = materialize_cir(&sample_cir(), &static_root, &mirror_root, true).unwrap();
assert_eq!(summary.object_count, 2);
assert_eq!(summary.trust_anchor_count, 1);
assert_eq!(summary.materialized_file_count, 3);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
b"a"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/nested/b.roa")).unwrap(),
b"b"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
b"ta-der"
);
assert!(!mirror_root.join("stale/old.txt").exists());
}
#[test]
fn materialize_fails_when_static_object_missing() {
let td = tempfile::tempdir().unwrap();
let err = materialize_cir(&sample_cir(), td.path(), &td.path().join("mirror"), true)
.expect_err("missing static object must fail");
assert!(matches!(
err,
CirMaterializeError::MissingStaticObject { .. }
));
}
#[test]
fn materialize_without_clean_rebuild_detects_stale_extra_files() {
let td = tempfile::tempdir().unwrap();
let static_root = td.path().join("static");
let mirror_root = td.path().join("mirror");
write_static(
&static_root,
"20260407",
"1111111111111111111111111111111111111111111111111111111111111111",
b"a",
);
write_static(
&static_root,
"20260407",
"2222222222222222222222222222222222222222222222222222222222222222",
b"b",
);
std::fs::create_dir_all(mirror_root.join("extra")).unwrap();
std::fs::write(mirror_root.join("extra/stale.txt"), b"stale").unwrap();
let err = materialize_cir(&sample_cir(), &static_root, &mirror_root, false)
.expect_err("stale extra files should fail exact tree check");
assert!(matches!(err, CirMaterializeError::TreeMismatch(_)));
}
#[test]
fn materialize_from_raw_store_creates_expected_tree() {
let td = tempfile::tempdir().unwrap();
let raw_store_path = td.path().join("raw-store.db");
let mirror_root = td.path().join("mirror");
let a = b"a".to_vec();
let b = b"b".to_vec();
let cir = cir_with_real_hashes(&a, &b);
{
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
raw_store
.put_blob_bytes_batch(&[
(hex::encode(&cir.fresh_validated_objects[0].sha256), a),
(hex::encode(&cir.fresh_validated_objects[1].sha256), b),
])
.unwrap();
}
let summary =
materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, true).unwrap();
assert_eq!(summary.object_count, 2);
assert_eq!(summary.trust_anchor_count, 1);
assert_eq!(summary.materialized_file_count, 3);
assert_eq!(summary.linked_files, 0);
assert_eq!(summary.copied_files, 3);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
b"a"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/nested/b.roa")).unwrap(),
b"b"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
b"ta-der"
);
}
#[test]
fn materialize_from_raw_store_fails_when_object_missing() {
let td = tempfile::tempdir().unwrap();
let raw_store_path = td.path().join("raw-store.db");
let mirror_root = td.path().join("mirror");
let cir = cir_with_real_hashes(b"a", b"b");
{
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
raw_store
.put_blob_bytes_batch(&[(
hex::encode(&cir.fresh_validated_objects[0].sha256),
b"a".to_vec(),
)])
.unwrap();
}
let err = materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, true)
.expect_err("missing second object should fail");
assert!(matches!(
err,
CirMaterializeError::MissingRawStoreObject { .. }
));
}
#[test]
fn materialize_from_raw_store_detects_stale_tree_when_not_clean_rebuild() {
let td = tempfile::tempdir().unwrap();
let raw_store_path = td.path().join("raw-store.db");
let mirror_root = td.path().join("mirror");
let cir = cir_with_real_hashes(b"a", b"b");
{
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
raw_store
.put_blob_bytes_batch(&[
(
hex::encode(&cir.fresh_validated_objects[0].sha256),
b"a".to_vec(),
),
(
hex::encode(&cir.fresh_validated_objects[1].sha256),
b"b".to_vec(),
),
])
.unwrap();
}
std::fs::create_dir_all(mirror_root.join("extra")).unwrap();
std::fs::write(mirror_root.join("extra/stale.txt"), b"stale").unwrap();
let err = materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, false)
.expect_err("stale file should fail exact tree check");
assert!(matches!(err, CirMaterializeError::TreeMismatch(_)));
}
#[test]
fn materialize_from_raw_store_overwrites_existing_targets() {
let td = tempfile::tempdir().unwrap();
let raw_store_path = td.path().join("raw-store.db");
let mirror_root = td.path().join("mirror");
let a = b"new-a".to_vec();
let b = b"new-b".to_vec();
let cir = cir_with_real_hashes(&a, &b);
{
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
raw_store
.put_blob_bytes_batch(&[
(
hex::encode(&cir.fresh_validated_objects[0].sha256),
a.clone(),
),
(
hex::encode(&cir.fresh_validated_objects[1].sha256),
b.clone(),
),
])
.unwrap();
}
let target = mirror_root.join("example.net/repo/a.cer");
std::fs::create_dir_all(target.parent().unwrap()).unwrap();
std::fs::write(&target, b"old").unwrap();
let summary =
materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, false).unwrap();
assert_eq!(summary.copied_files, 3);
assert_eq!(std::fs::read(&target).unwrap(), a);
}
#[test]
fn materialize_from_raw_store_ignores_corrupt_raw_entry_when_blob_exists() {
let td = tempfile::tempdir().unwrap();
let raw_store_path = td.path().join("raw-store.db");
let mirror_root = td.path().join("mirror");
let cir = CanonicalInputRepresentation::new_v4(
sample_time(),
vec![CirObject {
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
sha256: hex::decode(
"1111111111111111111111111111111111111111111111111111111111111111",
)
.unwrap(),
}],
Vec::new(),
vec![sample_trust_anchor()],
Vec::new(),
Vec::new(),
);
{
let raw_store = ExternalRawStoreDb::open(&raw_store_path).unwrap();
raw_store
.put_blob_bytes_batch(&[(
"1111111111111111111111111111111111111111111111111111111111111111".to_string(),
b"blob-a".to_vec(),
)])
.unwrap();
}
{
let db = rocksdb::DB::open_default(&raw_store_path).unwrap();
db.put(
b"rawbyhash:1111111111111111111111111111111111111111111111111111111111111111",
b"bad-cbor",
)
.unwrap();
}
let summary =
materialize_cir_from_raw_store(&cir, &raw_store_path, &mirror_root, true).unwrap();
assert_eq!(summary.object_count, 1);
assert_eq!(summary.materialized_file_count, 2);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
b"blob-a"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
b"ta-der"
);
}
#[test]
fn materialize_from_repo_bytes_creates_expected_tree() {
let td = tempfile::tempdir().unwrap();
let repo_bytes_db = td.path().join("repo-bytes.db");
let mirror_root = td.path().join("mirror");
let a = b"a".to_vec();
let b = b"b".to_vec();
let cir = cir_with_real_hashes(&a, &b);
{
let repo_bytes = ExternalRepoBytesDb::open(&repo_bytes_db).unwrap();
repo_bytes
.put_blob_bytes_batch(&[
(hex::encode(&cir.fresh_validated_objects[0].sha256), a),
(hex::encode(&cir.fresh_validated_objects[1].sha256), b),
])
.unwrap();
}
let summary =
materialize_cir_from_repo_bytes(&cir, &repo_bytes_db, &mirror_root, true).unwrap();
assert_eq!(summary.object_count, 2);
assert_eq!(summary.trust_anchor_count, 1);
assert_eq!(summary.materialized_file_count, 3);
assert_eq!(summary.linked_files, 0);
assert_eq!(summary.copied_files, 3);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/a.cer")).unwrap(),
b"a"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/nested/b.roa")).unwrap(),
b"b"
);
assert_eq!(
std::fs::read(mirror_root.join("example.net/repo/ta.cer")).unwrap(),
b"ta-der"
);
}
fn write_static(root: &Path, date: &str, hash: &str, bytes: &[u8]) {
let path = root.join(date).join(&hash[0..2]).join(&hash[2..4]);
std::fs::create_dir_all(&path).unwrap();
std::fs::write(path.join(hash), bytes).unwrap();
}
}

View File

@ -0,0 +1,441 @@
pub mod accumulator;
pub mod decode;
pub mod encode;
#[cfg(feature = "full")]
pub mod export;
#[cfg(feature = "full")]
pub mod materialize;
pub mod model;
pub mod sequence;
#[cfg(feature = "full")]
pub mod static_pool;
pub use accumulator::{CirInputAccumulator, CirInputSection, CirInputSnapshot};
pub use decode::{CirDecodeError, decode_cir};
pub use encode::{CirEncodeError, encode_cir};
#[cfg(feature = "full")]
pub use export::{
CirExportError, CirExportSummary, CirTrustAnchorBinding, build_cir_from_input_snapshot_multi,
build_cir_from_run, build_cir_from_run_multi, export_cir_from_input_snapshot_multi,
export_cir_from_run, export_cir_from_run_multi, write_cir_file,
};
#[cfg(feature = "full")]
pub use materialize::{
CirMaterializeError, CirMaterializeSummary, materialize_cir, materialize_cir_from_raw_store,
materialize_cir_from_repo_bytes, mirror_relative_path_for_rsync_uri, resolve_static_pool_file,
};
pub use model::{
CIR_VERSION_V1, CIR_VERSION_V3, CIR_VERSION_V4, CanonicalInputRepresentation, CirHashAlgorithm,
CirObject, CirRejectedObject, CirTrustAnchor, compute_reject_list_sha256, sha256,
};
pub use sequence::{CirSequenceManifest, CirSequenceStep, CirSequenceStepKind};
#[cfg(feature = "full")]
pub use static_pool::{
CirStaticPoolError, CirStaticPoolExportSummary, CirStaticPoolWriteResult,
export_hashes_from_store, static_pool_path, static_pool_relative_path,
write_bytes_to_static_pool, write_raw_entry_to_static_pool,
};
#[cfg(test)]
mod tests {
use super::{
CIR_VERSION_V4, CanonicalInputRepresentation, CirObject, CirRejectedObject, CirTrustAnchor,
decode_cir, encode_cir,
};
fn sample_time() -> time::OffsetDateTime {
time::OffsetDateTime::parse(
"2026-04-07T12:34:56Z",
&time::format_description::well_known::Rfc3339,
)
.expect("valid rfc3339")
}
fn sample_trust_anchor(ta_rsync_uri: &str, tal_uri: &str, ta_der: &[u8]) -> CirTrustAnchor {
CirTrustAnchor {
ta_rsync_uri: ta_rsync_uri.to_string(),
tal_uri: tal_uri.to_string(),
tal_bytes: format!("{ta_rsync_uri}\n\nAQID\n").into_bytes(),
ta_certificate_der: ta_der.to_vec(),
ta_certificate_sha256: super::sha256(ta_der),
}
}
fn sample_cir() -> CanonicalInputRepresentation {
let rejected_objects = vec![
CirRejectedObject {
object_uri: "rsync://example.net/repo/rejected-a.roa".to_string(),
reason: Some("invalid roa".to_string()),
},
CirRejectedObject {
object_uri: "rsync://example.net/repo/rejected-b.asa".to_string(),
reason: None,
},
];
CanonicalInputRepresentation::new_v4(
sample_time(),
vec![
CirObject {
rsync_uri: "rsync://example.net/repo/a.cer".to_string(),
sha256: vec![0x11; 32],
},
CirObject {
rsync_uri: "rsync://example.net/repo/b.roa".to_string(),
sha256: vec![0x22; 32],
},
CirObject {
rsync_uri: "rsync://example.net/repo/rejected-a.roa".to_string(),
sha256: vec![0x33; 32],
},
CirObject {
rsync_uri: "rsync://example.net/repo/rejected-b.asa".to_string(),
sha256: vec![0x44; 32],
},
],
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der",
)],
rejected_objects,
Vec::new(),
)
}
fn test_encode_tlv(tag: u8, value: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(8 + value.len());
out.push(tag);
if value.len() < 0x80 {
out.push(value.len() as u8);
} else {
let len = value.len();
let bytes = len.to_be_bytes();
let first_non_zero = bytes
.iter()
.position(|&b| b != 0)
.unwrap_or(bytes.len() - 1);
let len_bytes = &bytes[first_non_zero..];
out.push(0x80 | len_bytes.len() as u8);
out.extend_from_slice(len_bytes);
}
out.extend_from_slice(value);
out
}
#[test]
fn cir_roundtrip_full_succeeds() {
let cir = sample_cir();
let der = encode_cir(&cir).expect("encode cir");
let decoded = decode_cir(&der).expect("decode cir");
assert_eq!(decoded, cir);
}
#[test]
fn cir_roundtrip_minimal_succeeds() {
let cir = CanonicalInputRepresentation::new_v4(
sample_time(),
Vec::new(),
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/minimal-ta.cer",
"https://tal.example.net/minimal.tal",
b"minimal-ta-der",
)],
Vec::new(),
Vec::new(),
);
let der = encode_cir(&cir).expect("encode minimal cir");
let decoded = decode_cir(&der).expect("decode minimal cir");
assert_eq!(decoded, cir);
}
#[test]
fn cir_model_rejects_unsorted_duplicate_objects() {
let cir = CanonicalInputRepresentation::new_v4(
sample_time(),
vec![
CirObject {
rsync_uri: "rsync://example.net/repo/z.roa".to_string(),
sha256: vec![0x11; 32],
},
CirObject {
rsync_uri: "rsync://example.net/repo/a.roa".to_string(),
sha256: vec![0x22; 32],
},
],
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der",
)],
Vec::new(),
Vec::new(),
);
let err = encode_cir(&cir).expect_err("unsorted objects must fail");
assert!(
err.to_string().contains("CIR.freshValidatedObjects"),
"{err}"
);
}
#[test]
fn cir_model_rejects_duplicate_tals() {
let cir = CanonicalInputRepresentation::new_v4(
sample_time(),
Vec::new(),
Vec::new(),
vec![
sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der-a",
),
sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der-b",
),
],
Vec::new(),
Vec::new(),
);
let err = encode_cir(&cir).expect_err("duplicate trust_anchors must fail");
assert!(err.to_string().contains("CIR.trustAnchors"), "{err}");
}
#[test]
fn cir_decode_rejects_wrong_version() {
let mut der = encode_cir(&sample_cir()).expect("encode cir");
let pos = der
.windows(3)
.position(|window| window == [0x02, 0x01, CIR_VERSION_V4 as u8])
.or_else(|| {
der.windows(3)
.position(|window| window == [0x02, 0x01, CIR_VERSION_V4 as u8])
})
.expect("find version integer");
der[pos + 2] = 2;
let err = decode_cir(&der).expect_err("wrong version must fail");
assert!(err.to_string().contains("unexpected CIR version"), "{err}");
}
#[test]
fn cir_decode_rejects_wrong_hash_oid() {
let mut der = encode_cir(&sample_cir()).expect("encode cir");
let sha256_bytes = crate::data_model::oid::OID_SHA256_RAW;
let idx = der
.windows(sha256_bytes.len())
.position(|window| window == sha256_bytes)
.expect("find sha256 oid");
der[idx + sha256_bytes.len() - 1] ^= 0x01;
let err = decode_cir(&der).expect_err("wrong oid must fail");
assert!(
err.to_string().contains(crate::data_model::oid::OID_SHA256),
"{err}"
);
}
#[test]
fn cir_decode_rejects_bad_generalized_time() {
let mut der = encode_cir(&sample_cir()).expect("encode cir");
let pos = der
.windows(15)
.position(|window| window == b"20260407123456Z")
.expect("find generalized time");
der[pos + 14] = b'X';
let err = decode_cir(&der).expect_err("bad time must fail");
assert!(err.to_string().contains("GeneralizedTime"), "{err}");
}
#[test]
fn cir_model_rejects_non_rsync_object_uri_and_empty_tals() {
let bad_object = CanonicalInputRepresentation::new_v4(
sample_time(),
vec![CirObject {
rsync_uri: "https://example.net/repo/a.roa".to_string(),
sha256: vec![0x11; 32],
}],
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der",
)],
Vec::new(),
Vec::new(),
);
let err = encode_cir(&bad_object).expect_err("non-rsync object uri must fail");
assert!(err.to_string().contains("rsync://"), "{err}");
let no_tals = CanonicalInputRepresentation::new_v4(
sample_time(),
Vec::new(),
Vec::new(),
Vec::new(),
Vec::new(),
Vec::new(),
);
let err = encode_cir(&no_tals).expect_err("empty trust_anchors must fail");
assert!(
err.to_string()
.contains("CIR.trustAnchors must be non-empty"),
"{err}"
);
}
#[test]
fn cir_model_rejects_non_utc_time_bad_hash_len_and_non_http_tal_uri() {
let bad_time = CanonicalInputRepresentation::new_v4(
sample_time().to_offset(time::UtcOffset::from_hms(8, 0, 0).unwrap()),
Vec::new(),
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der",
)],
Vec::new(),
Vec::new(),
);
let bad_time = CanonicalInputRepresentation {
validation_time: sample_time().to_offset(time::UtcOffset::from_hms(8, 0, 0).unwrap()),
..bad_time
};
let err = encode_cir(&bad_time).expect_err("non-utc validation time must fail");
assert!(err.to_string().contains("UTC"), "{err}");
let bad_hash = CanonicalInputRepresentation::new_v4(
sample_time(),
vec![CirObject {
rsync_uri: "rsync://example.net/repo/a.roa".to_string(),
sha256: vec![0x11; 31],
}],
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"https://tal.example.net/root.tal",
b"ta-der",
)],
Vec::new(),
Vec::new(),
);
let err = encode_cir(&bad_hash).expect_err("bad digest len must fail");
assert!(err.to_string().contains("32 bytes"), "{err}");
let bad_tal_uri = CanonicalInputRepresentation::new_v4(
sample_time(),
Vec::new(),
Vec::new(),
vec![sample_trust_anchor(
"rsync://example.net/repo/ta.cer",
"ftp://tal.example.net/root.tal",
b"ta-der",
)],
Vec::new(),
Vec::new(),
);
let err = encode_cir(&bad_tal_uri).expect_err("bad tal uri must fail");
assert!(err.to_string().contains("http:// or https://"), "{err}");
}
#[test]
fn cir_decode_rejects_trailing_bytes_and_trailing_fields() {
let cir = sample_cir();
let mut der = encode_cir(&cir).expect("encode cir");
der.push(0);
let err = decode_cir(&der).expect_err("trailing bytes after cir must fail");
assert!(
err.to_string().contains("trailing bytes after CIR"),
"{err}"
);
let object = test_encode_tlv(
0x30,
&[
test_encode_tlv(0x16, b"rsync://example.net/repo/a.roa"),
test_encode_tlv(0x04, &[0x11; 32]),
test_encode_tlv(0x02, &[0x01]),
]
.concat(),
);
let trust_anchor = test_encode_tlv(
0x30,
&[
test_encode_tlv(0x16, b"rsync://example.net/repo/ta.cer"),
test_encode_tlv(0x16, b"https://tal.example.net/root.tal"),
test_encode_tlv(0x04, b"rsync://example.net/repo/ta.cer\n\nAQID\n"),
test_encode_tlv(0x04, b"ta-der"),
test_encode_tlv(0x04, &super::sha256(b"ta-der")),
]
.concat(),
);
let bad = test_encode_tlv(
0x30,
&[
test_encode_tlv(0x02, &[CIR_VERSION_V4 as u8]),
test_encode_tlv(0x06, crate::data_model::oid::OID_SHA256_RAW),
test_encode_tlv(0x18, b"20260407123456Z"),
test_encode_tlv(0x30, &object),
test_encode_tlv(0x30, &[]),
test_encode_tlv(0x30, &trust_anchor),
test_encode_tlv(0x04, &[0x33; 32]),
test_encode_tlv(0x04, &[0x33; 32]),
test_encode_tlv(0x04, &[0x33; 32]),
test_encode_tlv(0x04, &[0x33; 32]),
test_encode_tlv(0x04, &[0x33; 32]),
test_encode_tlv(0x04, &[0x33; 32]),
test_encode_tlv(0x30, &[]),
test_encode_tlv(0x30, &[]),
]
.concat(),
);
let err = decode_cir(&bad).expect_err("trailing field in object must fail");
assert!(
err.to_string().contains("trailing fields in CirObject"),
"{err}"
);
}
#[test]
fn cir_decode_rejects_invalid_object_and_tal_shapes() {
let cir = sample_cir();
let mut der = encode_cir(&cir).expect("encode cir");
let rsync_text = b"rsync://example.net/repo/a.cer";
let idx = der
.windows(rsync_text.len())
.position(|window| window == rsync_text)
.expect("find object uri");
der[idx] = 0xFF;
let err = decode_cir(&der).expect_err("invalid utf8 object uri must fail");
assert!(err.to_string().contains("utf-8"), "{err}");
let mut der = encode_cir(&cir).expect("encode cir");
let tal_text = b"https://tal.example.net/root.tal";
let idx = der
.windows(tal_text.len())
.position(|window| window == tal_text)
.expect("find tal uri");
der[idx] = 0xFF;
let err = decode_cir(&der).expect_err("invalid utf8 tal uri must fail");
assert!(err.to_string().contains("utf-8"), "{err}");
}
#[test]
fn cir_model_rejects_unsorted_rejected_objects_and_bad_digest() {
let mut cir = sample_cir();
cir.fresh_rejected_objects.swap(0, 1);
let err = encode_cir(&cir).expect_err("unsorted rejected objects must fail");
assert!(
err.to_string().contains("CIR.freshRejectedObjects"),
"{err}"
);
let mut cir = sample_cir();
cir.reject_list_sha256 = vec![0x55; 32];
let err = encode_cir(&cir).expect_err("bad reject list digest must fail");
assert!(err.to_string().contains("rejectListSha256"), "{err}");
}
}

View File

@ -0,0 +1,441 @@
use crate::data_model::oid::OID_SHA256;
pub const CIR_VERSION_V1: u32 = 1;
pub const CIR_VERSION_V2: u32 = 2;
pub const CIR_VERSION_V3: u32 = 3;
pub const CIR_VERSION_V4: u32 = 4;
pub const DIGEST_LEN_SHA256: usize = 32;
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum CirHashAlgorithm {
Sha256,
}
impl CirHashAlgorithm {
pub fn oid(&self) -> &'static str {
match self {
Self::Sha256 => OID_SHA256,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CanonicalInputRepresentation {
pub version: u32,
pub hash_alg: CirHashAlgorithm,
pub validation_time: time::OffsetDateTime,
pub fresh_validated_objects: Vec<CirObject>,
pub cached_validated_objects: Vec<CirObject>,
pub trust_anchors: Vec<CirTrustAnchor>,
pub object_list_sha256: Vec<u8>,
pub fresh_object_list_sha256: Vec<u8>,
pub cached_object_list_sha256: Vec<u8>,
pub reject_list_sha256: Vec<u8>,
pub fresh_reject_list_sha256: Vec<u8>,
pub cached_reject_list_sha256: Vec<u8>,
pub fresh_rejected_objects: Vec<CirRejectedObject>,
pub cached_rejected_objects: Vec<CirRejectedObject>,
}
impl CanonicalInputRepresentation {
pub fn new_v4(
validation_time: time::OffsetDateTime,
fresh_validated_objects: Vec<CirObject>,
cached_validated_objects: Vec<CirObject>,
trust_anchors: Vec<CirTrustAnchor>,
fresh_rejected_objects: Vec<CirRejectedObject>,
cached_rejected_objects: Vec<CirRejectedObject>,
) -> Self {
let fresh_object_list_sha256 = compute_object_list_sha256(fresh_validated_objects.iter());
let cached_object_list_sha256 = compute_object_list_sha256(cached_validated_objects.iter());
let object_list_sha256 = compute_sectioned_object_list_sha256(
fresh_validated_objects
.iter()
.map(|object| ("fresh", object))
.chain(
cached_validated_objects
.iter()
.map(|object| ("cached", object)),
),
);
let fresh_reject_list_sha256 = compute_reject_list_sha256(
fresh_rejected_objects
.iter()
.map(|item| item.object_uri.as_str()),
);
let cached_reject_list_sha256 = compute_reject_list_sha256(
cached_rejected_objects
.iter()
.map(|item| item.object_uri.as_str()),
);
let mut all_rejected_uris = fresh_rejected_objects
.iter()
.chain(cached_rejected_objects.iter())
.map(|item| item.object_uri.as_str())
.collect::<Vec<_>>();
all_rejected_uris.sort_unstable();
all_rejected_uris.dedup();
let reject_list_sha256 = compute_reject_list_sha256(all_rejected_uris.iter().copied());
Self {
version: CIR_VERSION_V4,
hash_alg: CirHashAlgorithm::Sha256,
validation_time: validation_time.to_offset(time::UtcOffset::UTC),
fresh_validated_objects,
cached_validated_objects,
trust_anchors,
object_list_sha256,
fresh_object_list_sha256,
cached_object_list_sha256,
reject_list_sha256,
fresh_reject_list_sha256,
cached_reject_list_sha256,
fresh_rejected_objects,
cached_rejected_objects,
}
}
pub fn validate(&self) -> Result<(), String> {
if self.version != CIR_VERSION_V4 {
return Err(format!(
"CIR version must be {CIR_VERSION_V4}, got {}",
self.version
));
}
if !matches!(self.hash_alg, CirHashAlgorithm::Sha256) {
return Err("CIR hashAlg must be SHA-256".into());
}
if self.validation_time.offset() != time::UtcOffset::UTC {
return Err("CIR validationTime must be UTC".into());
}
validate_sorted_unique_strings(
self.fresh_validated_objects
.iter()
.map(|item| item.rsync_uri.as_str()),
"CIR.freshValidatedObjects must be sorted by rsyncUri and unique",
)?;
validate_sorted_unique_strings(
self.cached_validated_objects
.iter()
.map(|item| item.rsync_uri.as_str()),
"CIR.cachedValidatedObjects must be sorted by rsyncUri and unique",
)?;
validate_sorted_unique_strings(
self.trust_anchors
.iter()
.map(|item| item.ta_rsync_uri.as_str()),
"CIR.trustAnchors must be sorted by taRsyncUri and unique",
)?;
validate_sorted_unique_strings(
self.fresh_rejected_objects
.iter()
.map(|item| item.object_uri.as_str()),
"CIR.freshRejectedObjects must be sorted by objectUri and unique",
)?;
validate_sorted_unique_strings(
self.cached_rejected_objects
.iter()
.map(|item| item.object_uri.as_str()),
"CIR.cachedRejectedObjects must be sorted by objectUri and unique",
)?;
let object_uris = self
.fresh_validated_objects
.iter()
.chain(self.cached_validated_objects.iter())
.map(|item| item.rsync_uri.as_str())
.collect::<std::collections::BTreeSet<_>>();
for trust_anchor in &self.trust_anchors {
if object_uris.contains(trust_anchor.ta_rsync_uri.as_str()) {
return Err(format!(
"CIR.objects must not include trust anchor URI {}",
trust_anchor.ta_rsync_uri
));
}
}
let fresh_object_uris = self
.fresh_validated_objects
.iter()
.map(|item| item.rsync_uri.as_str())
.collect::<std::collections::BTreeSet<_>>();
for item in &self.fresh_rejected_objects {
if !fresh_object_uris.contains(item.object_uri.as_str()) {
return Err(format!(
"CIR.freshRejectedObjects URI must exist in freshValidatedObjects: {}",
item.object_uri
));
}
}
let cached_object_uris = self
.cached_validated_objects
.iter()
.map(|item| item.rsync_uri.as_str())
.collect::<std::collections::BTreeSet<_>>();
for item in &self.cached_rejected_objects {
if !cached_object_uris.contains(item.object_uri.as_str()) {
return Err(format!(
"CIR.cachedRejectedObjects URI must exist in cachedValidatedObjects: {}",
item.object_uri
));
}
}
if self.trust_anchors.is_empty() {
return Err("CIR.trustAnchors must be non-empty".into());
}
for (label, digest) in [
("CIR.objectListSha256", &self.object_list_sha256),
("CIR.freshObjectListSha256", &self.fresh_object_list_sha256),
(
"CIR.cachedObjectListSha256",
&self.cached_object_list_sha256,
),
("CIR.rejectListSha256", &self.reject_list_sha256),
("CIR.freshRejectListSha256", &self.fresh_reject_list_sha256),
(
"CIR.cachedRejectListSha256",
&self.cached_reject_list_sha256,
),
] {
if digest.len() != DIGEST_LEN_SHA256 {
return Err(format!(
"{label} must be {DIGEST_LEN_SHA256} bytes, got {}",
digest.len()
));
}
}
for object in self.validated_objects() {
object.validate()?;
}
for trust_anchor in &self.trust_anchors {
trust_anchor.validate()?;
}
for item in self.rejected_objects() {
item.validate()?;
}
let expected_fresh_object_digest =
compute_object_list_sha256(self.fresh_validated_objects.iter());
if self.fresh_object_list_sha256 != expected_fresh_object_digest {
return Err("CIR.freshObjectListSha256 does not match freshValidatedObjects".into());
}
let expected_cached_object_digest =
compute_object_list_sha256(self.cached_validated_objects.iter());
if self.cached_object_list_sha256 != expected_cached_object_digest {
return Err("CIR.cachedObjectListSha256 does not match cachedValidatedObjects".into());
}
let expected_object_digest = compute_sectioned_object_list_sha256(
self.fresh_validated_objects
.iter()
.map(|object| ("fresh", object))
.chain(
self.cached_validated_objects
.iter()
.map(|object| ("cached", object)),
),
);
if self.object_list_sha256 != expected_object_digest {
return Err("CIR.objectListSha256 does not match fresh/cached objects".into());
}
let expected_fresh_reject_digest = compute_reject_list_sha256(
self.fresh_rejected_objects
.iter()
.map(|item| item.object_uri.as_str()),
);
if self.fresh_reject_list_sha256 != expected_fresh_reject_digest {
return Err("CIR.freshRejectListSha256 does not match freshRejectedObjects".into());
}
let expected_cached_reject_digest = compute_reject_list_sha256(
self.cached_rejected_objects
.iter()
.map(|item| item.object_uri.as_str()),
);
if self.cached_reject_list_sha256 != expected_cached_reject_digest {
return Err("CIR.cachedRejectListSha256 does not match cachedRejectedObjects".into());
}
let mut all_rejected_uris = self
.rejected_objects()
.map(|item| item.object_uri.as_str())
.collect::<Vec<_>>();
all_rejected_uris.sort_unstable();
all_rejected_uris.dedup();
let expected_digest = compute_reject_list_sha256(all_rejected_uris.iter().copied());
if self.reject_list_sha256 != expected_digest {
return Err("CIR.rejectListSha256 does not match fresh/cached rejectedObjects".into());
}
Ok(())
}
pub fn validated_objects(&self) -> impl Iterator<Item = &CirObject> {
self.fresh_validated_objects
.iter()
.chain(self.cached_validated_objects.iter())
}
pub fn rejected_objects(&self) -> impl Iterator<Item = &CirRejectedObject> {
self.fresh_rejected_objects
.iter()
.chain(self.cached_rejected_objects.iter())
}
pub fn validated_object_count(&self) -> usize {
self.fresh_validated_objects.len() + self.cached_validated_objects.len()
}
pub fn rejected_object_count(&self) -> usize {
self.fresh_rejected_objects.len() + self.cached_rejected_objects.len()
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirObject {
pub rsync_uri: String,
pub sha256: Vec<u8>,
}
impl CirObject {
pub fn validate(&self) -> Result<(), String> {
if !self.rsync_uri.starts_with("rsync://") {
return Err(format!(
"CirObject.rsync_uri must start with rsync://, got {}",
self.rsync_uri
));
}
if self.sha256.len() != DIGEST_LEN_SHA256 {
return Err(format!(
"CirObject.sha256 must be {DIGEST_LEN_SHA256} bytes, got {}",
self.sha256.len()
));
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirTrustAnchor {
pub ta_rsync_uri: String,
pub tal_uri: String,
pub tal_bytes: Vec<u8>,
pub ta_certificate_der: Vec<u8>,
pub ta_certificate_sha256: Vec<u8>,
}
impl CirTrustAnchor {
pub fn validate(&self) -> Result<(), String> {
if !self.ta_rsync_uri.starts_with("rsync://") {
return Err(format!(
"CirTrustAnchor.ta_rsync_uri must start with rsync://, got {}",
self.ta_rsync_uri
));
}
if !(self.tal_uri.starts_with("https://") || self.tal_uri.starts_with("http://")) {
return Err(format!(
"CirTrustAnchor.tal_uri must start with http:// or https://, got {}",
self.tal_uri
));
}
if self.tal_bytes.is_empty() {
return Err("CirTrustAnchor.tal_bytes must be non-empty".into());
}
let tal = crate::data_model::tal::Tal::decode_bytes(&self.tal_bytes)
.map_err(|e| format!("CirTrustAnchor.tal_bytes must decode as TAL: {e}"))?;
if !tal
.ta_uris
.iter()
.any(|uri| uri.as_str() == self.ta_rsync_uri)
{
return Err(format!(
"CirTrustAnchor.ta_rsync_uri must be listed in TAL bytes: {}",
self.ta_rsync_uri
));
}
if self.ta_certificate_der.is_empty() {
return Err("CirTrustAnchor.ta_certificate_der must be non-empty".into());
}
if self.ta_certificate_sha256.len() != DIGEST_LEN_SHA256 {
return Err(format!(
"CirTrustAnchor.ta_certificate_sha256 must be {DIGEST_LEN_SHA256} bytes, got {}",
self.ta_certificate_sha256.len()
));
}
let expected = sha256(&self.ta_certificate_der);
if self.ta_certificate_sha256 != expected {
return Err("CirTrustAnchor.ta_certificate_sha256 does not match DER bytes".into());
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirRejectedObject {
pub object_uri: String,
pub reason: Option<String>,
}
impl CirRejectedObject {
pub fn validate(&self) -> Result<(), String> {
if !self.object_uri.starts_with("rsync://") {
return Err(format!(
"CirRejectedObject.object_uri must start with rsync://, got {}",
self.object_uri
));
}
Ok(())
}
}
pub fn compute_reject_list_sha256<'a>(uris: impl IntoIterator<Item = &'a str>) -> Vec<u8> {
let mut body = Vec::new();
for uri in uris {
let bytes = uri.as_bytes();
body.extend_from_slice(&(bytes.len() as u32).to_be_bytes());
body.extend_from_slice(bytes);
}
sha256(&body)
}
pub fn compute_object_list_sha256<'a>(objects: impl IntoIterator<Item = &'a CirObject>) -> Vec<u8> {
let mut body = Vec::new();
for object in objects {
append_digest_string(&mut body, &object.rsync_uri);
body.extend_from_slice(&object.sha256);
}
sha256(&body)
}
pub fn compute_sectioned_object_list_sha256<'a>(
objects: impl IntoIterator<Item = (&'a str, &'a CirObject)>,
) -> Vec<u8> {
let mut body = Vec::new();
for (section, object) in objects {
append_digest_string(&mut body, section);
append_digest_string(&mut body, &object.rsync_uri);
body.extend_from_slice(&object.sha256);
}
sha256(&body)
}
pub fn sha256(bytes: &[u8]) -> Vec<u8> {
use sha2::Digest;
sha2::Sha256::digest(bytes).to_vec()
}
fn append_digest_string(body: &mut Vec<u8>, value: &str) {
let bytes = value.as_bytes();
body.extend_from_slice(&(bytes.len() as u32).to_be_bytes());
body.extend_from_slice(bytes);
}
fn validate_sorted_unique_strings<'a>(
items: impl IntoIterator<Item = &'a str>,
message: &str,
) -> Result<(), String> {
let mut prev: Option<&'a str> = None;
for key in items {
if let Some(prev_key) = prev
&& key <= prev_key
{
return Err(message.into());
}
prev = Some(key);
}
Ok(())
}

View File

@ -0,0 +1,203 @@
use serde::{Deserialize, Serialize};
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum CirSequenceStepKind {
Full,
Delta,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CirSequenceStep {
pub step_id: String,
pub kind: CirSequenceStepKind,
pub validation_time: String,
pub cir_path: String,
pub ccr_path: String,
pub report_path: String,
#[serde(default)]
pub timing_path: Option<String>,
#[serde(default)]
pub stdout_log_path: Option<String>,
#[serde(default)]
pub stderr_log_path: Option<String>,
#[serde(default)]
pub artifact_prefix: Option<String>,
pub previous_step_id: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CirSequenceManifest {
pub version: u32,
pub repo_bytes_db_path: String,
pub steps: Vec<CirSequenceStep>,
}
impl CirSequenceManifest {
pub fn validate(&self) -> Result<(), String> {
if self.version == 0 {
return Err("sequence.version must be positive".to_string());
}
if self.repo_bytes_db_path.trim().is_empty() {
return Err("sequence.repo_bytes_db_path must not be empty".to_string());
}
if self.steps.is_empty() {
return Err("sequence.steps must not be empty".to_string());
}
let mut previous_ids = std::collections::BTreeSet::new();
for (idx, step) in self.steps.iter().enumerate() {
if step.step_id.trim().is_empty() {
return Err(format!("sequence.steps[{idx}].step_id must not be empty"));
}
if !previous_ids.insert(step.step_id.clone()) {
return Err(format!("sequence.steps[{idx}].step_id must be unique"));
}
if step.validation_time.trim().is_empty() {
return Err(format!(
"sequence.steps[{idx}].validation_time must not be empty"
));
}
if let Some(timing_path) = &step.timing_path
&& timing_path.trim().is_empty()
{
return Err(format!(
"sequence.steps[{idx}].timing_path must not be empty when set"
));
}
if let Some(stdout_log_path) = &step.stdout_log_path
&& stdout_log_path.trim().is_empty()
{
return Err(format!(
"sequence.steps[{idx}].stdout_log_path must not be empty when set"
));
}
if let Some(stderr_log_path) = &step.stderr_log_path
&& stderr_log_path.trim().is_empty()
{
return Err(format!(
"sequence.steps[{idx}].stderr_log_path must not be empty when set"
));
}
if let Some(artifact_prefix) = &step.artifact_prefix
&& artifact_prefix.trim().is_empty()
{
return Err(format!(
"sequence.steps[{idx}].artifact_prefix must not be empty when set"
));
}
if step.cir_path.trim().is_empty()
|| step.ccr_path.trim().is_empty()
|| step.report_path.trim().is_empty()
{
return Err(format!(
"sequence.steps[{idx}] output paths must not be empty"
));
}
match step.kind {
CirSequenceStepKind::Full => {
if idx != 0 {
return Err("full step must be the first step".to_string());
}
if step.previous_step_id.is_some() {
return Err("full step must not reference previous_step_id".to_string());
}
}
CirSequenceStepKind::Delta => {
if idx == 0 {
return Err("delta step cannot be the first step".to_string());
}
let previous = step.previous_step_id.as_ref().ok_or_else(|| {
format!("sequence.steps[{idx}] delta step must set previous_step_id")
})?;
if !previous_ids.contains(previous) {
return Err(format!(
"sequence.steps[{idx}] previous_step_id must reference an earlier step"
));
}
}
}
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::{CirSequenceManifest, CirSequenceStep, CirSequenceStepKind};
fn sample_manifest() -> CirSequenceManifest {
CirSequenceManifest {
version: 1,
repo_bytes_db_path: "repo-bytes.db".to_string(),
steps: vec![
CirSequenceStep {
step_id: "full".to_string(),
kind: CirSequenceStepKind::Full,
validation_time: "2026-04-09T00:00:00Z".to_string(),
cir_path: "full/input.cir".to_string(),
ccr_path: "full/result.ccr".to_string(),
report_path: "full/report.json".to_string(),
timing_path: Some("full/timing.json".to_string()),
stdout_log_path: Some("full/stdout.log".to_string()),
stderr_log_path: Some("full/stderr.log".to_string()),
artifact_prefix: Some("2026-04-09T00:00:00Z-test".to_string()),
previous_step_id: None,
},
CirSequenceStep {
step_id: "delta-001".to_string(),
kind: CirSequenceStepKind::Delta,
validation_time: "2026-04-09T00:10:00Z".to_string(),
cir_path: "delta-001/input.cir".to_string(),
ccr_path: "delta-001/result.ccr".to_string(),
report_path: "delta-001/report.json".to_string(),
timing_path: Some("delta-001/timing.json".to_string()),
stdout_log_path: Some("delta-001/stdout.log".to_string()),
stderr_log_path: Some("delta-001/stderr.log".to_string()),
artifact_prefix: Some("2026-04-09T00:10:00Z-test".to_string()),
previous_step_id: Some("full".to_string()),
},
],
}
}
#[test]
fn sequence_manifest_validate_accepts_minimal_chain() {
sample_manifest().validate().expect("valid sequence");
}
#[test]
fn sequence_manifest_validate_rejects_bad_order_and_duplicates() {
let mut bad = sample_manifest();
bad.steps.swap(0, 1);
let err = bad.validate().expect_err("full must be first");
assert!(
err.contains("delta step cannot be the first step")
|| err.contains("full step must be the first step")
);
let mut dup = sample_manifest();
dup.steps[1].step_id = "full".to_string();
let err = dup.validate().expect_err("duplicate id must fail");
assert!(err.contains("must be unique"));
}
#[test]
fn sequence_manifest_validate_rejects_missing_previous_reference() {
let mut bad = sample_manifest();
bad.steps[1].previous_step_id = Some("missing".to_string());
let err = bad.validate().expect_err("missing previous must fail");
assert!(err.contains("previous_step_id"));
}
#[test]
fn sequence_manifest_validate_rejects_empty_repo_bytes_backend() {
let mut manifest = sample_manifest();
manifest.repo_bytes_db_path = " ".to_string();
let err = manifest
.validate()
.expect_err("empty repo bytes backend must fail");
assert!(err.contains("repo_bytes_db_path"));
}
}

View File

@ -0,0 +1,376 @@
use std::collections::BTreeSet;
use std::fs::{self, OpenOptions};
use std::io::Write;
use std::path::{Path, PathBuf};
use crate::storage::{RawByHashEntry, RocksStore};
#[derive(Debug, thiserror::Error)]
pub enum CirStaticPoolError {
#[error("invalid sha256 hex: {0}")]
InvalidSha256Hex(String),
#[error("raw bytes are empty for sha256={sha256_hex}")]
EmptyBytes { sha256_hex: String },
#[error("raw bytes do not match sha256 hex: {sha256_hex}")]
HashMismatch { sha256_hex: String },
#[error("create directory failed: {path}: {detail}")]
CreateDir { path: String, detail: String },
#[error("create temp file failed: {path}: {detail}")]
CreateTemp { path: String, detail: String },
#[error("write temp file failed: {path}: {detail}")]
WriteTemp { path: String, detail: String },
#[error("sync temp file failed: {path}: {detail}")]
SyncTemp { path: String, detail: String },
#[error("publish temp file failed: {temp_path} -> {final_path}: {detail}")]
Publish {
temp_path: String,
final_path: String,
detail: String,
},
#[error("remove temp file failed: {path}: {detail}")]
RemoveTemp { path: String, detail: String },
#[error("raw_by_hash entry missing for sha256={sha256_hex}")]
MissingRawByHash { sha256_hex: String },
#[error("storage error: {0}")]
Storage(String),
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirStaticPoolWriteResult {
pub final_path: PathBuf,
pub written: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirStaticPoolExportSummary {
pub unique_hashes: usize,
pub written_files: usize,
pub reused_files: usize,
}
pub fn static_pool_relative_path(
capture_date_utc: time::Date,
sha256_hex: &str,
) -> Result<PathBuf, CirStaticPoolError> {
validate_sha256_hex(sha256_hex)?;
let date = format_utc_date(capture_date_utc);
Ok(PathBuf::from(date)
.join(&sha256_hex[0..2])
.join(&sha256_hex[2..4])
.join(sha256_hex))
}
pub fn static_pool_path(
static_root: &Path,
capture_date_utc: time::Date,
sha256_hex: &str,
) -> Result<PathBuf, CirStaticPoolError> {
Ok(static_root.join(static_pool_relative_path(capture_date_utc, sha256_hex)?))
}
pub fn write_bytes_to_static_pool(
static_root: &Path,
capture_date_utc: time::Date,
sha256_hex_value: &str,
bytes: &[u8],
) -> Result<CirStaticPoolWriteResult, CirStaticPoolError> {
validate_sha256_hex(sha256_hex_value)?;
if bytes.is_empty() {
return Err(CirStaticPoolError::EmptyBytes {
sha256_hex: sha256_hex_value.to_string(),
});
}
let computed = compute_sha256_hex(bytes);
if computed != sha256_hex_value.to_ascii_lowercase() {
return Err(CirStaticPoolError::HashMismatch {
sha256_hex: sha256_hex_value.to_string(),
});
}
let final_path = static_pool_path(static_root, capture_date_utc, sha256_hex_value)?;
if final_path.exists() {
return Ok(CirStaticPoolWriteResult {
final_path,
written: false,
});
}
let parent = final_path.parent().expect("static pool file has parent");
fs::create_dir_all(parent).map_err(|e| CirStaticPoolError::CreateDir {
path: parent.display().to_string(),
detail: e.to_string(),
})?;
let temp_path = parent.join(format!("{sha256_hex_value}.tmp.{}", uuid::Uuid::new_v4()));
let mut file = OpenOptions::new()
.create_new(true)
.write(true)
.open(&temp_path)
.map_err(|e| CirStaticPoolError::CreateTemp {
path: temp_path.display().to_string(),
detail: e.to_string(),
})?;
file.write_all(bytes)
.map_err(|e| CirStaticPoolError::WriteTemp {
path: temp_path.display().to_string(),
detail: e.to_string(),
})?;
file.sync_all().map_err(|e| CirStaticPoolError::SyncTemp {
path: temp_path.display().to_string(),
detail: e.to_string(),
})?;
drop(file);
match fs::hard_link(&temp_path, &final_path) {
Ok(()) => {
fs::remove_file(&temp_path).map_err(|e| CirStaticPoolError::RemoveTemp {
path: temp_path.display().to_string(),
detail: e.to_string(),
})?;
Ok(CirStaticPoolWriteResult {
final_path,
written: true,
})
}
Err(e) if final_path.exists() => {
fs::remove_file(&temp_path).map_err(|remove_err| CirStaticPoolError::RemoveTemp {
path: temp_path.display().to_string(),
detail: remove_err.to_string(),
})?;
let _ = e;
Ok(CirStaticPoolWriteResult {
final_path,
written: false,
})
}
Err(e) => Err(CirStaticPoolError::Publish {
temp_path: temp_path.display().to_string(),
final_path: final_path.display().to_string(),
detail: e.to_string(),
}),
}
}
pub fn write_raw_entry_to_static_pool(
static_root: &Path,
capture_date_utc: time::Date,
entry: &RawByHashEntry,
) -> Result<CirStaticPoolWriteResult, CirStaticPoolError> {
write_bytes_to_static_pool(
static_root,
capture_date_utc,
&entry.sha256_hex,
&entry.bytes,
)
}
pub fn export_hashes_from_store(
store: &RocksStore,
static_root: &Path,
capture_date_utc: time::Date,
sha256_hexes: &[String],
) -> Result<CirStaticPoolExportSummary, CirStaticPoolError> {
let unique: BTreeSet<String> = sha256_hexes
.iter()
.map(|item| item.to_ascii_lowercase())
.collect();
let mut written_files = 0usize;
let mut reused_files = 0usize;
for sha256_hex in &unique {
let bytes = store
.get_blob_bytes(sha256_hex)
.map_err(|e| CirStaticPoolError::Storage(e.to_string()))?
.ok_or_else(|| CirStaticPoolError::MissingRawByHash {
sha256_hex: sha256_hex.clone(),
})?;
let entry = RawByHashEntry::from_bytes(sha256_hex.clone(), bytes);
let result = write_raw_entry_to_static_pool(static_root, capture_date_utc, &entry)?;
if result.written {
written_files += 1;
} else {
reused_files += 1;
}
}
Ok(CirStaticPoolExportSummary {
unique_hashes: unique.len(),
written_files,
reused_files,
})
}
fn format_utc_date(date: time::Date) -> String {
format!(
"{:04}{:02}{:02}",
date.year(),
u8::from(date.month()),
date.day()
)
}
fn validate_sha256_hex(sha256_hex: &str) -> Result<(), CirStaticPoolError> {
if sha256_hex.len() != 64 || !sha256_hex.as_bytes().iter().all(u8::is_ascii_hexdigit) {
return Err(CirStaticPoolError::InvalidSha256Hex(sha256_hex.to_string()));
}
Ok(())
}
fn compute_sha256_hex(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
hex::encode(Sha256::digest(bytes))
}
#[cfg(test)]
mod tests {
use super::{
CirStaticPoolError, compute_sha256_hex, export_hashes_from_store,
static_pool_relative_path, write_bytes_to_static_pool,
};
use crate::storage::{RawByHashEntry, RepositoryViewEntry, RepositoryViewState, RocksStore};
use std::fs;
fn sample_date() -> time::Date {
time::Date::from_calendar_date(2026, time::Month::April, 7).unwrap()
}
#[test]
fn static_pool_relative_path_uses_date_and_hash_prefixes() {
let path = static_pool_relative_path(
sample_date(),
"abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
)
.expect("relative path");
assert_eq!(
path,
std::path::PathBuf::from("20260407")
.join("ab")
.join("cd")
.join("abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789")
);
}
#[test]
fn write_bytes_to_static_pool_is_idempotent_and_leaves_no_temp_files() {
let td = tempfile::tempdir().expect("tempdir");
let bytes = b"static-pool-object";
let sha = compute_sha256_hex(bytes);
let first =
write_bytes_to_static_pool(td.path(), sample_date(), &sha, bytes).expect("first write");
let second = write_bytes_to_static_pool(td.path(), sample_date(), &sha, bytes)
.expect("second write");
assert!(first.written);
assert!(!second.written);
assert_eq!(fs::read(&first.final_path).expect("read final"), bytes);
let all_files: Vec<_> = walk_files(td.path());
assert_eq!(all_files.len(), 1);
assert!(
!all_files[0]
.file_name()
.and_then(|name| name.to_str())
.unwrap_or_default()
.contains(".tmp.")
);
}
#[test]
fn write_bytes_to_static_pool_rejects_bad_hash_and_empty_bytes() {
let td = tempfile::tempdir().expect("tempdir");
let err = write_bytes_to_static_pool(td.path(), sample_date(), "not-a-hash", b"x")
.expect_err("bad hash must fail");
assert!(matches!(err, CirStaticPoolError::InvalidSha256Hex(_)));
let err = write_bytes_to_static_pool(
td.path(),
sample_date(),
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
b"",
)
.expect_err("empty bytes must fail");
assert!(matches!(err, CirStaticPoolError::EmptyBytes { .. }));
}
#[test]
fn export_hashes_from_store_writes_unique_entries_and_fails_when_missing() {
let td = tempfile::tempdir().expect("tempdir");
let store_dir = td.path().join("db");
let static_root = td.path().join("static");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let bytes = b"store-object".to_vec();
let sha = compute_sha256_hex(&bytes);
let mut entry = RawByHashEntry::from_bytes(sha.clone(), bytes.clone());
entry
.origin_uris
.push("rsync://example.test/repo/object.cer".to_string());
store.put_raw_by_hash_entry(&entry).expect("put raw entry");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/object.cer".to_string(),
current_hash: Some(sha.clone()),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("cer".to_string()),
state: RepositoryViewState::Present,
})
.expect("put repository view");
let summary = export_hashes_from_store(
&store,
&static_root,
sample_date(),
&[sha.clone(), sha.clone()],
)
.expect("export hashes");
assert_eq!(summary.unique_hashes, 1);
assert_eq!(summary.written_files, 1);
assert_eq!(summary.reused_files, 0);
let summary = export_hashes_from_store(&store, &static_root, sample_date(), &[sha.clone()])
.expect("re-export hashes");
assert_eq!(summary.unique_hashes, 1);
assert_eq!(summary.written_files, 0);
assert_eq!(summary.reused_files, 1);
let err = export_hashes_from_store(
&store,
&static_root,
sample_date(),
&[String::from(
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
)],
)
.expect_err("missing raw_by_hash must fail");
assert!(matches!(err, CirStaticPoolError::MissingRawByHash { .. }));
}
fn walk_files(root: &std::path::Path) -> Vec<std::path::PathBuf> {
let mut out = Vec::new();
let mut stack = vec![root.to_path_buf()];
while let Some(path) = stack.pop() {
for entry in fs::read_dir(path).expect("read_dir") {
let entry = entry.expect("dir entry");
let path = entry.path();
if path.is_dir() {
stack.push(path);
} else {
out.push(path);
}
}
}
out.sort();
out
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,419 @@
use std::io::BufWriter;
use std::path::Path;
use serde::Serialize;
use serde::ser::SerializeSeq;
use sha2::Digest;
use crate::audit::{
AspaOutput, AuditRunMeta, AuditWarning, QueryAuditManifest, ValidationEvent,
ValidationEventCounts, VrpOutput,
};
use crate::ccr::canonical_vrp_prefix;
use super::{PostValidationShared, RunStageTiming};
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) enum ReportJsonFormat {
Pretty,
Compact,
}
pub(super) fn write_json<T: Serialize>(
path: &Path,
report: &T,
format: ReportJsonFormat,
) -> Result<(), String> {
let f = std::fs::File::create(path)
.map_err(|e| format!("create report file failed: {}: {e}", path.display()))?;
let writer = BufWriter::new(f);
match format {
ReportJsonFormat::Pretty => serde_json::to_writer_pretty(writer, report),
ReportJsonFormat::Compact => serde_json::to_writer(writer, report),
}
.map_err(|e| format!("write report json failed: {e}"))?;
Ok(())
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct ReportJsonWriteTiming {
pub(super) build_ms: u64,
pub(super) write_ms: u64,
}
#[derive(Serialize)]
struct BorrowedAuditReportV2<'a> {
format_version: u32,
meta: AuditRunMeta,
policy: &'a crate::policy::Policy,
tree: BorrowedTreeSummary<'a>,
publication_points: &'a [crate::audit::PublicationPointAudit],
vrps: VrpReportSequence<'a>,
aspas: AspaReportSequence<'a>,
downloads: &'a [crate::audit::AuditDownloadEvent],
download_stats: &'a crate::audit::AuditDownloadStats,
repo_sync_stats: crate::audit::AuditRepoSyncStats,
#[serde(rename = "queryAudit", skip_serializing_if = "Option::is_none")]
query_audit: Option<QueryAuditManifest>,
}
#[derive(Serialize)]
struct BorrowedTreeSummary<'a> {
instances_processed: usize,
instances_failed: usize,
warnings: WarningReportSequence<'a>,
}
struct WarningReportSequence<'a>(&'a [crate::report::Warning]);
impl Serialize for WarningReportSequence<'_> {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let mut seq = serializer.serialize_seq(Some(self.0.len()))?;
for warning in self.0 {
seq.serialize_element(&AuditWarning::from(warning))?;
}
seq.end()
}
}
struct VrpReportSequence<'a>(&'a [crate::validation::objects::Vrp]);
impl Serialize for VrpReportSequence<'_> {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let mut seq = serializer.serialize_seq(Some(self.0.len()))?;
for vrp in self.0 {
seq.serialize_element(&VrpOutput {
asn: vrp.asn,
prefix: crate::audit::format_roa_ip_prefix(&vrp.prefix),
max_length: vrp.max_length,
})?;
}
seq.end()
}
}
struct AspaReportSequence<'a>(&'a [crate::validation::objects::AspaAttestation]);
impl Serialize for AspaReportSequence<'_> {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let mut seq = serializer.serialize_seq(Some(self.0.len()))?;
for aspa in self.0 {
seq.serialize_element(&AspaOutput {
customer_as_id: aspa.customer_as_id,
provider_as_ids: aspa.provider_as_ids.clone(),
})?;
}
seq.end()
}
}
pub(super) fn write_report_json_from_shared(
path: &Path,
policy: &crate::policy::Policy,
validation_time: time::OffsetDateTime,
shared: &PostValidationShared,
format: ReportJsonFormat,
) -> Result<ReportJsonWriteTiming, String> {
use time::format_description::well_known::Rfc3339;
let build_started = std::time::Instant::now();
let validation_time_rfc3339_utc = validation_time
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.expect("format validation_time");
let repo_sync_stats = super::build_repo_sync_stats(shared.publication_points.as_ref());
let query_audit = write_validation_events_sidecar(path, &validation_time_rfc3339_utc, shared)?;
let report = BorrowedAuditReportV2 {
format_version: 2,
meta: AuditRunMeta {
validation_time_rfc3339_utc,
},
policy,
tree: BorrowedTreeSummary {
instances_processed: shared.instances_processed,
instances_failed: shared.instances_failed,
warnings: WarningReportSequence(shared.tree_warnings.as_ref()),
},
publication_points: shared.publication_points.as_ref(),
vrps: VrpReportSequence(shared.vrps.as_ref()),
aspas: AspaReportSequence(shared.aspas.as_ref()),
downloads: shared.downloads.as_ref(),
download_stats: &shared.download_stats,
repo_sync_stats,
query_audit: Some(query_audit),
};
let build_ms = build_started.elapsed().as_millis() as u64;
let write_started = std::time::Instant::now();
write_json(path, &report, format)?;
Ok(ReportJsonWriteTiming {
build_ms,
write_ms: write_started.elapsed().as_millis() as u64,
})
}
fn write_validation_events_sidecar(
report_path: &Path,
validation_time: &str,
shared: &PostValidationShared,
) -> Result<QueryAuditManifest, String> {
let events_path = report_path.with_file_name("validation-events.jsonl");
if let Some(parent) = events_path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| format!("create validation events parent failed: {e}"))?;
}
let mut writer = BufWriter::new(std::fs::File::create(&events_path).map_err(|e| {
format!(
"create validation events failed: {}: {e}",
events_path.display()
)
})?);
let mut seq = 0u64;
let mut hasher = sha2::Sha256::new();
emit_validation_events(validation_time, shared, &mut seq, &mut |event| {
let mut line = serde_json::to_vec(&event)
.map_err(|e| format!("serialize validation event failed: {e}"))?;
line.push(b'\n');
std::io::Write::write_all(&mut writer, &line)
.map_err(|e| format!("write validation event failed: {e}"))?;
hasher.update(&line);
Ok(())
})?;
std::io::Write::flush(&mut writer)
.map_err(|e| format!("flush validation events failed: {e}"))?;
let events_count = seq;
let events_sha256 = hex::encode(hasher.finalize());
Ok(QueryAuditManifest {
schema_version: 1,
status: "complete".to_string(),
events_path: events_path
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("validation-events.jsonl")
.to_string(),
events_count,
events_sha256,
writer_version: 1,
error: None,
})
}
fn emit_validation_events(
validation_time: &str,
shared: &PostValidationShared,
seq: &mut u64,
emit: &mut impl FnMut(ValidationEvent) -> Result<(), String>,
) -> Result<(), String> {
emit(next_event(seq, "run_summary", validation_time, |event| {
event.counts = Some(ValidationEventCounts {
objects: Some(
shared
.publication_points
.iter()
.map(|pp| pp.objects.len() as u64)
.sum(),
),
warnings: Some(
(shared.tree_warnings.len()
+ shared
.publication_points
.iter()
.map(|pp| pp.warnings.len())
.sum::<usize>()) as u64,
),
vrps: Some(shared.vrps.len() as u64),
aspas: Some(shared.aspas.len() as u64),
});
}))?;
for pp in shared.publication_points.iter() {
emit(next_event(
seq,
"publication_point",
validation_time,
|event| {
event.pp_node_id = pp.node_id;
event.pp_manifest_uri = Some(pp.manifest_rsync_uri.clone());
event.pp_rsync_base_uri = Some(pp.rsync_base_uri.clone());
event.repo_sync_phase = pp.repo_sync_phase.clone();
event.repo_terminal_state = Some(pp.repo_terminal_state.clone());
event.counts = Some(ValidationEventCounts {
objects: Some(pp.objects.len() as u64),
warnings: Some(pp.warnings.len() as u64),
vrps: None,
aspas: None,
});
},
))?;
for object in &pp.objects {
emit(next_event(seq, "object", validation_time, |event| {
event.pp_node_id = pp.node_id;
event.pp_manifest_uri = Some(pp.manifest_rsync_uri.clone());
event.object_uri = Some(object.rsync_uri.clone());
event.sha256 = Some(object.sha256_hex.clone());
event.object_type = Some(object.kind.clone());
event.result = Some(object.result.clone());
event.reason = object.detail.clone();
}))?;
}
for warning in &pp.warnings {
emit(next_event(seq, "warning", validation_time, |event| {
event.pp_node_id = pp.node_id;
event.pp_manifest_uri = Some(pp.manifest_rsync_uri.clone());
event.reason = Some(warning.message.clone());
}))?;
}
}
Ok(())
}
fn next_event(
seq: &mut u64,
event_type: &str,
validation_time: &str,
fill: impl FnOnce(&mut ValidationEvent),
) -> ValidationEvent {
*seq += 1;
let mut event = ValidationEvent {
schema_version: 1,
seq: *seq,
event_type: event_type.to_string(),
validation_time: validation_time.to_string(),
pp_node_id: None,
pp_manifest_uri: None,
pp_rsync_base_uri: None,
repo_sync_phase: None,
repo_terminal_state: None,
object_uri: None,
sha256: None,
object_type: None,
result: None,
reason: None,
counts: None,
};
fill(&mut event);
event
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct CompareViewTaskOutput {
pub(super) build_ms: Option<u64>,
pub(super) write_ms: Option<u64>,
}
pub(super) fn run_compare_view_task(
shared: &PostValidationShared,
vrps_csv_out_path: Option<&Path>,
vaps_csv_out_path: Option<&Path>,
trust_anchor: &str,
) -> Result<CompareViewTaskOutput, String> {
let mut build_ms = None;
let mut write_ms = None;
if let (Some(vrps_path), Some(vaps_path)) = (vrps_csv_out_path, vaps_csv_out_path) {
let started = std::time::Instant::now();
build_ms = Some(0);
write_direct_vrp_csv(vrps_path, shared.vrps.as_ref(), trust_anchor)?;
write_direct_vap_csv(vaps_path, shared.aspas.as_ref(), trust_anchor)?;
write_ms = Some(started.elapsed().as_millis() as u64);
eprintln!(
"wrote compare views: vrps={} vaps={}",
vrps_path.display(),
vaps_path.display()
);
}
Ok(CompareViewTaskOutput { build_ms, write_ms })
}
fn write_direct_vrp_csv(
path: &Path,
vrps: &[crate::validation::objects::Vrp],
trust_anchor: &str,
) -> Result<(), String> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
}
let file = std::fs::File::create(path)
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?;
let mut writer = BufWriter::new(file);
use std::io::Write;
let trust_anchor = trust_anchor.to_ascii_lowercase();
writeln!(writer, "ASN,IP Prefix,Max Length,Trust Anchor").map_err(|e| e.to_string())?;
for vrp in vrps {
writeln!(
writer,
"AS{},{},{},{}",
vrp.asn,
canonical_vrp_prefix(&vrp.prefix),
vrp.max_length,
trust_anchor
)
.map_err(|e| e.to_string())?;
}
Ok(())
}
fn write_direct_vap_csv(
path: &Path,
aspas: &[crate::validation::objects::AspaAttestation],
trust_anchor: &str,
) -> Result<(), String> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| format!("create parent dirs failed: {}: {e}", parent.display()))?;
}
let file = std::fs::File::create(path)
.map_err(|e| format!("create file failed: {}: {e}", path.display()))?;
let mut writer = BufWriter::new(file);
use std::io::Write;
let trust_anchor = trust_anchor.to_ascii_lowercase();
writeln!(writer, "Customer ASN,Providers,Trust Anchor").map_err(|e| e.to_string())?;
for aspa in aspas {
let mut providers = aspa.provider_as_ids.clone();
providers.sort_unstable();
providers.dedup();
let providers = providers
.into_iter()
.map(|asn| format!("AS{asn}"))
.collect::<Vec<_>>()
.join(";");
writeln!(
writer,
"AS{},{},{}",
aspa.customer_as_id, providers, trust_anchor
)
.map_err(|e| e.to_string())?;
}
Ok(())
}
pub(super) fn write_stage_timing(
report_json_path: Option<&Path>,
stage_timing: &RunStageTiming,
) -> Result<(), String> {
if let Some(path) = report_json_path
&& let Some(parent) = path.parent()
{
let stage_timing_path = parent.join("stage-timing.json");
std::fs::write(
&stage_timing_path,
serde_json::to_vec_pretty(stage_timing).map_err(|e| e.to_string())?,
)
.map_err(|e| {
format!(
"write stage timing failed: {}: {e}",
stage_timing_path.display()
)
})?;
eprintln!("analysis: wrote {}", stage_timing_path.display());
}
Ok(())
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,127 @@
//! Normal-run validation contract.
//!
//! This is deliberately limited to the normal synchronization/validation
//! path. Offline revalidation of a previous run is not part of this package's
//! first public contract.
use std::path::Path;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use time::format_description::well_known::Rfc3339;
use crate::fetch::rsync_system::RsyncScopePolicy;
use crate::policy::Policy;
pub const VALIDATION_CONTRACT_SCHEMA_VERSION: u32 = 1;
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ValidationCacheContract {
pub publication_point: bool,
pub roa: bool,
pub child_certificate: bool,
pub transport_prefetch: bool,
#[serde(default)]
pub crypto_signature: bool,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ValidationContract {
pub schema_version: u32,
pub validation_time: String,
pub binary_sha256: String,
pub policy: Policy,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub ta_constraints_fingerprint: Option<String>,
pub max_ca_depth: usize,
pub max_instances: Option<usize>,
pub cache: ValidationCacheContract,
#[serde(default)]
pub rsync_scope_policy: RsyncScopePolicy,
}
impl ValidationContract {
pub fn for_current_binary(
validation_time: time::OffsetDateTime,
policy: Policy,
max_ca_depth: usize,
max_instances: Option<usize>,
cache: ValidationCacheContract,
rsync_scope_policy: RsyncScopePolicy,
) -> Result<Self, String> {
Ok(Self {
schema_version: VALIDATION_CONTRACT_SCHEMA_VERSION,
validation_time: format_validation_time(validation_time)?,
binary_sha256: current_binary_sha256()?,
policy,
ta_constraints_fingerprint: None,
max_ca_depth,
max_instances,
cache,
rsync_scope_policy,
})
}
pub fn validate(&self) -> Result<(), String> {
if self.schema_version != VALIDATION_CONTRACT_SCHEMA_VERSION {
return Err(format!(
"unsupported validation contract schemaVersion: {}",
self.schema_version
));
}
time::OffsetDateTime::parse(&self.validation_time, &Rfc3339)
.map_err(|error| format!("invalid validation contract validationTime: {error}"))?;
validate_sha256_hex("validation contract binarySha256", &self.binary_sha256)?;
if let Some(fingerprint) = self.ta_constraints_fingerprint.as_deref() {
validate_sha256_hex("validation contract taConstraintsFingerprint", fingerprint)?;
}
if self.max_ca_depth == 0 {
return Err("validation contract maxCaDepth must be greater than zero".to_string());
}
Ok(())
}
}
pub fn write_validation_contract(path: &Path, contract: &ValidationContract) -> Result<(), String> {
contract.validate()?;
let bytes = serde_json::to_vec_pretty(contract)
.map_err(|error| format!("encode validation contract failed: {error}"))?;
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|error| format!("create validation contract directory failed: {error}"))?;
}
std::fs::write(path, bytes).map_err(|error| {
format!(
"write validation contract failed: {}: {error}",
path.display()
)
})
}
pub fn format_validation_time(value: time::OffsetDateTime) -> Result<String, String> {
value
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.map_err(|error| format!("format validation time failed: {error}"))
}
fn current_binary_sha256() -> Result<String, String> {
let path = std::env::current_exe()
.map_err(|error| format!("resolve current executable failed: {error}"))?;
let bytes = std::fs::read(&path).map_err(|error| {
format!(
"read current executable failed: {}: {error}",
path.display()
)
})?;
Ok(hex::encode(Sha256::digest(bytes)))
}
pub(crate) fn validate_sha256_hex(label: &str, value: &str) -> Result<(), String> {
if value.len() != 64 || !value.bytes().all(|byte| byte.is_ascii_hexdigit()) {
return Err(format!("{label} must be a 64-character SHA-256 hex string"));
}
Ok(())
}

View File

@ -0,0 +1,876 @@
//! Crypto signature verification cache (feature #126).
//!
//! Caches the context-free cryptographic fact "this signature over these exact bytes verifies
//! under this public key" so that later runs can skip the RSA verification entirely. Only
//! positive conclusions are stored (no negative caching). The cache never replaces any other
//! validation semantics: revocation (CRL), time-window, and policy checks all run unchanged on
//! the hit path — the hook wraps exactly the crypto verify call and nothing else.
//!
//! Modes (CLI):
//!
//! - `--crypto-signature-cache-observe-only` (M1): compute keys, maintain the persistent set,
//! record hit/miss/timing statistics, always run the real verification.
//! - `--enable-crypto-signature-cache` (M2): on a cache hit, skip the real verification and
//! return success; on a miss, run the real verification and store the positive conclusion.
//! Statistics are recorded the same way; `verify_skipped`/`verify_executed` report what
//! actually happened. Both flags may be combined (reuse + statistics).
//!
//! The five choke points (see the development plan for the full caller inventory):
//!
//! - `CmsSignedObject`: `data_model::signed_object` CMS verify (ROA / ASPA / manifest on the
//! main validation path; GBR objects are not signature-verified by the tree runner).
//! - `EeCertFast`: `validation::cert_path::verify_ee_cert_signature_fast` (ring, TBS + signature
//! bytes from the signed-object embedded EE certificate; ROA / ASPA / manifest EE paths).
//! - `EeCertX509`: `validation::cert_path::verify_ee_cert_signature` (x509-parser; router
//! certificates and the `validate_ee_cert_path` family).
//! - `ChildCaCert`: `validation::ca_path::verify_child_signature` (x509-parser; subordinate CA
//! certificates).
//! - `Crl`: `data_model::crl::verify_signature_with_issuer_spki` (x509-parser; issuer CRLs).
//!
//! Trust-anchor self-signature verification is deliberately excluded (a handful of calls per
//! run).
//!
//! Cache key (M1 decision: verify-input bytes, not full object bytes — the key material is in
//! hand inside each choke point, while the repo-bytes content hash is only available at the
//! upper processing layer):
//!
//! ```text
//! key = SHA-256( "rpki-crypto-sig-cache-key-v1" | point-tag | "sha256WithRSAEncryption"
//! | SHA-256(verify input bytes) | SHA-256(signature bytes)
//! | SHA-256(verifier SPKI DER) )
//! ```
//!
//! The signature bytes are part of the key so that a different signature over the same input
//! under the same key can never inherit a positive verification fact.
//!
//! Storage:
//!
//! - Single small file next to the work DB (`<work-db-name>.crypto-sig-cache`), never inside
//! the work DB, not touched by periodic snapshot reset (#091/#092): the reset paths only
//! remove/replace the work DB directory and the PP/child-cert cache artifacts, not siblings
//! of their own choosing.
//! - Binary format: 8-byte magic `RPKICSC2` | u64 record-count | records of
//! key[32] | verified_at_unix_secs(i64) | crypto_impl_version(u32) | reserved(u32).
//! A missing, corrupt, or magic-mismatched file (including the M1 `RPKICSC1` seen-set
//! format) is silently rebuilt empty. Writes are atomic (write-then-rename).
//! - The cache is persisted once at the end of the validation stage (same point as the M1
//! seen-set). If a run fails mid-validation, entries added during that run are lost;
//! previously persisted entries are unaffected.
//! - Size cap: approximate上限 `capacity` entries (default 2,000,000; measured M1 APNIC
//! ~148k keys/run, all5 estimated ~750k). When an insert would exceed the cap, the whole
//! cache is cleared and rebuilt from scratch (full-rebuild eviction — the simplest policy;
//! concurrent inserts may briefly overshoot the cap by a few entries). Each eviction is
//! counted in `evictions_total`.
//! - Concurrency: entries live in 64 shards keyed by the first key byte, so parallel
//! validation workers rarely contend; statistics use relaxed atomics.
use std::collections::{BTreeMap, HashMap};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex, RwLock};
use serde::Serialize;
use sha2::Digest;
const CACHE_FILE_MAGIC: &[u8; 8] = b"RPKICSC2";
const KEY_DOMAIN: &[u8] = b"rpki-crypto-sig-cache-key-v1";
const ALG_ID: &[u8] = b"sha256WithRSAEncryption";
const SHARD_COUNT: usize = 64;
const RECORD_BYTES: usize = 32 + 8 + 4 + 4;
/// The schema version reported in summaries. The on-disk magic already carries the format
/// version; this constant is the semantic version of the key/value definitions.
pub const CACHE_SCHEMA_VERSION: u32 = 2;
/// Default maximum number of cached positive conclusions (see module docs).
pub const DEFAULT_CAPACITY: usize = 2_000_000;
/// Crypto implementation identifier stored in every value. Bump when the verification
/// backend changes so old entries can be invalidated wholesale.
/// `1` = ring 0.17 `RSA_PKCS1_2048_8192_SHA256` (direct ring calls and x509-parser's
/// `verify` feature, which is backed by ring in this crate).
pub const CRYPTO_IMPL_VERSION: u32 = 1;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum CryptoSigVerifyPoint {
CmsSignedObject,
EeCertFast,
EeCertX509,
ChildCaCert,
Crl,
}
impl CryptoSigVerifyPoint {
pub const ALL: [Self; 5] = [
Self::CmsSignedObject,
Self::EeCertFast,
Self::EeCertX509,
Self::ChildCaCert,
Self::Crl,
];
fn tag(self) -> u8 {
match self {
Self::CmsSignedObject => 1,
Self::EeCertFast => 2,
Self::EeCertX509 => 3,
Self::ChildCaCert => 4,
Self::Crl => 5,
}
}
pub fn name(self) -> &'static str {
match self {
Self::CmsSignedObject => "cms_signed_object",
Self::EeCertFast => "ee_cert_fast",
Self::EeCertX509 => "ee_cert_x509",
Self::ChildCaCert => "child_ca_cert",
Self::Crl => "crl",
}
}
fn from_index(index: usize) -> Self {
Self::ALL[index]
}
}
/// Compute the cache key for one verification fact.
///
/// `input_bytes` is the exact byte string fed to the verifier (CMS signedAttrs DER for
/// signature, TBSCertificate DER, TBSCertList DER), `signature_bytes` the signature value,
/// and `verifier_spki_der` the DER of the verifying SubjectPublicKeyInfo.
pub fn compute_cache_key(
point: CryptoSigVerifyPoint,
input_bytes: &[u8],
signature_bytes: &[u8],
verifier_spki_der: &[u8],
) -> [u8; 32] {
let mut h = sha2::Sha256::new();
h.update(KEY_DOMAIN);
h.update([point.tag()]);
h.update(ALG_ID);
for part in [input_bytes, signature_bytes, verifier_spki_der] {
let part_hash = sha2::Sha256::digest(part);
h.update(part_hash);
}
h.finalize().into()
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct CacheValue {
pub verified_at_unix_secs: i64,
pub crypto_impl_version: u32,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct CryptoSigVerifyPointStats {
pub calls: u64,
/// Key already present in the cache (positive conclusion available).
pub would_hit: u64,
/// Key not present; after a successful verification it is inserted.
pub new_keys: u64,
/// Real cryptographic verification executed (always equals `calls` in observe-only
/// mode; equals `new_keys` plus hit-but-observe calls otherwise).
pub verify_executed: u64,
/// Real verification skipped because of a cache hit (only > 0 when reuse is enabled).
pub verify_skipped: u64,
pub verify_nanos_on_would_hit: u64,
pub verify_nanos_on_new_key: u64,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct CryptoSigCacheSummary {
pub schema_version: u32,
pub cache_file: String,
pub reuse_enabled: bool,
pub capacity: u64,
pub entries_loaded: u64,
pub entries_total: u64,
pub evictions_total: u64,
pub per_point: BTreeMap<String, CryptoSigVerifyPointStats>,
}
#[derive(Default)]
struct PointCounters {
calls: AtomicU64,
would_hit: AtomicU64,
new_keys: AtomicU64,
verify_executed: AtomicU64,
verify_skipped: AtomicU64,
verify_nanos_on_would_hit: AtomicU64,
verify_nanos_on_new_key: AtomicU64,
}
fn shard_index(key: &[u8; 32]) -> usize {
(key[0] as usize) % SHARD_COUNT
}
/// The persistent positive-conclusion signature cache plus per-run statistics.
///
/// See the module docs for storage format, eviction, and failure semantics.
pub struct CryptoSigCache {
cache_file: PathBuf,
reuse_enabled: bool,
capacity: usize,
entries_loaded: u64,
shards: Vec<Mutex<HashMap<[u8; 32], CacheValue>>>,
len: AtomicUsize,
evictions: AtomicU64,
eviction_lock: Mutex<()>,
points: [PointCounters; 5],
}
impl CryptoSigCache {
/// Load the cache from `cache_file`; start empty when the file is missing, corrupt, or
/// has an unknown schema header. `reuse_enabled` selects the M2 hit-skips-verify
/// behavior; when false the cache is observe-only (M1 semantics).
pub fn load_or_rebuild(cache_file: PathBuf, reuse_enabled: bool) -> Self {
Self::load_or_rebuild_with_capacity(cache_file, reuse_enabled, DEFAULT_CAPACITY)
}
pub fn load_or_rebuild_with_capacity(
cache_file: PathBuf,
reuse_enabled: bool,
capacity: usize,
) -> Self {
let entries = load_cache_file(&cache_file).unwrap_or_default();
let mut shards = Vec::with_capacity(SHARD_COUNT);
for _ in 0..SHARD_COUNT {
shards.push(Mutex::new(HashMap::new()));
}
let mut len = 0usize;
for (key, value) in entries {
// Respect the cap even when loading a foreign/oversized file: keep the first
// `capacity` records in file order.
if len >= capacity {
break;
}
shards[shard_index(&key)]
.lock()
.expect("crypto sig shard lock")
.insert(key, value);
len += 1;
}
Self {
cache_file,
reuse_enabled,
capacity,
entries_loaded: len as u64,
shards,
len: AtomicUsize::new(len),
evictions: AtomicU64::new(0),
eviction_lock: Mutex::new(()),
points: Default::default(),
}
}
pub fn reuse_enabled(&self) -> bool {
self.reuse_enabled
}
fn lookup(&self, key: &[u8; 32]) -> bool {
self.shards[shard_index(key)]
.lock()
.expect("crypto sig shard lock")
.contains_key(key)
}
#[cfg(test)]
pub fn contains(&self, key: &[u8; 32]) -> bool {
self.lookup(key)
}
/// Insert a positive conclusion (used by the verify path and by tests).
pub fn insert_positive(&self, key: [u8; 32]) {
let value = CacheValue {
verified_at_unix_secs: time::OffsetDateTime::now_utc().unix_timestamp(),
crypto_impl_version: CRYPTO_IMPL_VERSION,
};
if self.len.load(Ordering::Relaxed) >= self.capacity {
self.evict_all();
}
let mut shard = self.shards[shard_index(&key)]
.lock()
.expect("crypto sig shard lock");
if shard.insert(key, value).is_none() {
self.len.fetch_add(1, Ordering::Relaxed);
}
}
/// Full-rebuild eviction: clear every shard. See the module docs for the semantics.
fn evict_all(&self) {
let _guard = self.eviction_lock.lock().expect("crypto sig eviction lock");
if self.len.load(Ordering::Relaxed) < self.capacity {
// Another thread already evicted.
return;
}
for shard in &self.shards {
shard.lock().expect("crypto sig shard lock").clear();
}
self.len.store(0, Ordering::Relaxed);
self.evictions.fetch_add(1, Ordering::Relaxed);
}
/// Cache-aware verification: observe-only or reuse depending on `reuse_enabled`.
/// The `verify` closure must perform exactly the real cryptographic verification.
pub fn verify<E>(
&self,
point: CryptoSigVerifyPoint,
input_bytes: &[u8],
signature_bytes: &[u8],
verifier_spki_der: &[u8],
verify: impl FnOnce() -> Result<(), E>,
) -> Result<(), E> {
let key = compute_cache_key(point, input_bytes, signature_bytes, verifier_spki_der);
let hit = self.lookup(&key);
let counters = &self.points[(point.tag() - 1) as usize];
counters.calls.fetch_add(1, Ordering::Relaxed);
if hit {
counters.would_hit.fetch_add(1, Ordering::Relaxed);
if self.reuse_enabled {
counters.verify_skipped.fetch_add(1, Ordering::Relaxed);
return Ok(());
}
} else {
counters.new_keys.fetch_add(1, Ordering::Relaxed);
}
counters.verify_executed.fetch_add(1, Ordering::Relaxed);
let started = std::time::Instant::now();
let result = verify();
let nanos = started.elapsed().as_nanos().min(u128::from(u64::MAX)) as u64;
if hit {
counters
.verify_nanos_on_would_hit
.fetch_add(nanos, Ordering::Relaxed);
} else {
counters
.verify_nanos_on_new_key
.fetch_add(nanos, Ordering::Relaxed);
if result.is_ok() {
self.insert_positive(key);
}
}
result
}
/// Persist the cache atomically (write-then-rename).
pub fn persist(&self) -> Result<(), String> {
persist_cache_file(&self.cache_file, &self.shards)
}
pub fn summary(&self) -> CryptoSigCacheSummary {
let mut per_point = BTreeMap::new();
for (index, counters) in self.points.iter().enumerate() {
let stats = CryptoSigVerifyPointStats {
calls: counters.calls.load(Ordering::Relaxed),
would_hit: counters.would_hit.load(Ordering::Relaxed),
new_keys: counters.new_keys.load(Ordering::Relaxed),
verify_executed: counters.verify_executed.load(Ordering::Relaxed),
verify_skipped: counters.verify_skipped.load(Ordering::Relaxed),
verify_nanos_on_would_hit: counters
.verify_nanos_on_would_hit
.load(Ordering::Relaxed),
verify_nanos_on_new_key: counters.verify_nanos_on_new_key.load(Ordering::Relaxed),
};
if stats.calls > 0 {
per_point.insert(
CryptoSigVerifyPoint::from_index(index).name().to_string(),
stats,
);
}
}
CryptoSigCacheSummary {
schema_version: CACHE_SCHEMA_VERSION,
cache_file: self.cache_file.to_string_lossy().into_owned(),
reuse_enabled: self.reuse_enabled,
capacity: self.capacity as u64,
entries_loaded: self.entries_loaded,
entries_total: self.len.load(Ordering::Relaxed) as u64,
evictions_total: self.evictions.load(Ordering::Relaxed),
per_point,
}
}
}
/// Default cache file location: a sibling of the work DB directory, mirroring
/// `default_pp_cache_index_dir` (`<work-db-name>.crypto-sig-cache`).
pub fn default_cache_file_path(db_path: &Path) -> PathBuf {
let file_name = db_path
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("work-db");
db_path.with_file_name(format!("{file_name}.crypto-sig-cache"))
}
fn load_cache_file(path: &Path) -> Option<HashMap<[u8; 32], CacheValue>> {
let bytes = std::fs::read(path).ok()?;
let header_len = CACHE_FILE_MAGIC.len() + 8;
if bytes.len() < header_len || &bytes[..CACHE_FILE_MAGIC.len()] != CACHE_FILE_MAGIC {
return None;
}
let mut count_bytes = [0u8; 8];
count_bytes.copy_from_slice(&bytes[CACHE_FILE_MAGIC.len()..header_len]);
let count = u64::from_le_bytes(count_bytes) as usize;
let body = &bytes[header_len..];
if body.len() != count.saturating_mul(RECORD_BYTES) {
return None;
}
let mut entries = HashMap::with_capacity(count);
for chunk in body.chunks_exact(RECORD_BYTES) {
let mut key = [0u8; 32];
key.copy_from_slice(&chunk[..32]);
let mut secs_bytes = [0u8; 8];
secs_bytes.copy_from_slice(&chunk[32..40]);
let mut version_bytes = [0u8; 4];
version_bytes.copy_from_slice(&chunk[40..44]);
entries.insert(
key,
CacheValue {
verified_at_unix_secs: i64::from_le_bytes(secs_bytes),
crypto_impl_version: u32::from_le_bytes(version_bytes),
},
);
}
Some(entries)
}
fn persist_cache_file(
path: &Path,
shards: &[Mutex<HashMap<[u8; 32], CacheValue>>],
) -> Result<(), String> {
let mut bytes = Vec::new();
bytes.extend_from_slice(CACHE_FILE_MAGIC);
// Placeholder for the record count; patched after collecting.
bytes.extend_from_slice(&[0u8; 8]);
let mut count = 0u64;
for shard in shards {
let map = shard.lock().expect("crypto sig shard lock");
for (key, value) in map.iter() {
bytes.extend_from_slice(key);
bytes.extend_from_slice(&value.verified_at_unix_secs.to_le_bytes());
bytes.extend_from_slice(&value.crypto_impl_version.to_le_bytes());
bytes.extend_from_slice(&[0u8; 4]);
count += 1;
}
}
bytes[CACHE_FILE_MAGIC.len()..CACHE_FILE_MAGIC.len() + 8].copy_from_slice(&count.to_le_bytes());
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| format!("create cache-file dir failed: {}: {e}", parent.display()))?;
}
let tmp_path = path.with_extension("tmp");
std::fs::write(&tmp_path, &bytes)
.map_err(|e| format!("write cache file failed: {}: {e}", tmp_path.display()))?;
std::fs::rename(&tmp_path, path).map_err(|e| {
format!(
"rename cache file failed: {} -> {}: {e}",
tmp_path.display(),
path.display()
)
})?;
Ok(())
}
static GLOBAL_CACHE: RwLock<Option<Arc<CryptoSigCache>>> = RwLock::new(None);
/// Install the process-global cache (called once from the CLI run path when
/// `--crypto-signature-cache-observe-only` and/or `--enable-crypto-signature-cache` is set).
pub fn install_global(cache: Arc<CryptoSigCache>) {
*GLOBAL_CACHE.write().expect("crypto sig cache lock") = Some(cache);
}
/// Remove the process-global cache. Used by tests and by the CLI after the run finished.
pub fn clear_global() {
*GLOBAL_CACHE.write().expect("crypto sig cache lock") = None;
}
pub fn global_cache() -> Option<Arc<CryptoSigCache>> {
GLOBAL_CACHE.read().expect("crypto sig cache lock").clone()
}
/// Choke-point hook: cache-aware wrapper around a real signature verification.
///
/// With no cache installed this is a single read-lock check plus the verification itself;
/// verification semantics never change either way (a hit only skips the crypto primitive,
/// never the surrounding revocation/time/policy checks).
pub fn verify_with_cache<E>(
point: CryptoSigVerifyPoint,
input_bytes: &[u8],
signature_bytes: &[u8],
verifier_spki_der: &[u8],
verify: impl FnOnce() -> Result<(), E>,
) -> Result<(), E> {
let Some(cache) = global_cache() else {
return verify();
};
cache.verify(
point,
input_bytes,
signature_bytes,
verifier_spki_der,
verify,
)
}
#[cfg(test)]
pub(crate) static GLOBAL_TEST_LOCK: Mutex<()> = Mutex::new(());
#[cfg(test)]
mod tests {
use super::*;
fn temp_cache(reuse_enabled: bool) -> (tempfile::TempDir, CryptoSigCache) {
let dir = tempfile::tempdir().expect("tempdir");
let cache = CryptoSigCache::load_or_rebuild(
dir.path().join("work-db.crypto-sig-cache"),
reuse_enabled,
);
(dir, cache)
}
#[test]
fn cache_key_is_deterministic_and_domain_separated() {
let a = compute_cache_key(
CryptoSigVerifyPoint::CmsSignedObject,
b"input",
b"sig",
b"spki",
);
let b = compute_cache_key(
CryptoSigVerifyPoint::CmsSignedObject,
b"input",
b"sig",
b"spki",
);
assert_eq!(a, b);
// Different verify point.
assert_ne!(
a,
compute_cache_key(CryptoSigVerifyPoint::Crl, b"input", b"sig", b"spki")
);
// Different input bytes.
assert_ne!(
a,
compute_cache_key(
CryptoSigVerifyPoint::CmsSignedObject,
b"input2",
b"sig",
b"spki"
)
);
// Different signature bytes over the same input must not share a key.
assert_ne!(
a,
compute_cache_key(
CryptoSigVerifyPoint::CmsSignedObject,
b"input",
b"sig2",
b"spki"
)
);
// Different verifier key.
assert_ne!(
a,
compute_cache_key(
CryptoSigVerifyPoint::CmsSignedObject,
b"input",
b"sig",
b"spki2"
)
);
}
#[test]
fn miss_verifies_and_writes_then_hit_skips_verify() {
let (_dir, cache) = temp_cache(true);
let mut executed = 0u64;
// First call: miss -> real verification runs and passes -> entry written.
cache
.verify(
CryptoSigVerifyPoint::EeCertFast,
b"tbs",
b"sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Ok(())
},
)
.expect("verify ok");
assert_eq!(executed, 1);
// Second call: hit -> verification skipped entirely.
cache
.verify(
CryptoSigVerifyPoint::EeCertFast,
b"tbs",
b"sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Ok(())
},
)
.expect("verify ok");
assert_eq!(executed, 1, "cache hit must skip the real verification");
let summary = cache.summary();
let stats = summary
.per_point
.get("ee_cert_fast")
.expect("stats present");
assert_eq!(stats.calls, 2);
assert_eq!(stats.new_keys, 1);
assert_eq!(stats.would_hit, 1);
assert_eq!(stats.verify_executed, 1);
assert_eq!(stats.verify_skipped, 1);
assert!(summary.reuse_enabled);
}
#[test]
fn preset_positive_key_skips_verify_that_would_fail() {
let (_dir, cache) = temp_cache(true);
// Tampered object bytes whose signature can never verify; pre-seed the positive
// conclusion directly (as if a previous run had verified the untampered object).
let key = compute_cache_key(
CryptoSigVerifyPoint::CmsSignedObject,
b"tampered-input",
b"tampered-sig",
b"spki",
);
cache.insert_positive(key);
let mut executed = 0u64;
let result = cache.verify(
CryptoSigVerifyPoint::CmsSignedObject,
b"tampered-input",
b"tampered-sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Err("signature does not verify".to_string())
},
);
assert_eq!(result, Ok(()), "cache hit must return Ok without verifying");
assert_eq!(executed, 0, "real verification must not run on a hit");
}
#[test]
fn failed_verify_is_not_cached_and_reruns_next_time() {
let (_dir, cache) = temp_cache(true);
let mut executed = 0u64;
for _ in 0..2 {
let result = cache.verify(
CryptoSigVerifyPoint::Crl,
b"tbs",
b"sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Err("bad signature".to_string())
},
);
assert!(result.is_err());
}
assert_eq!(executed, 2, "no negative caching: every call re-verifies");
let key = compute_cache_key(CryptoSigVerifyPoint::Crl, b"tbs", b"sig", b"spki");
assert!(!cache.contains(&key));
let summary = cache.summary();
let stats = summary.per_point.get("crl").expect("stats present");
assert_eq!(stats.new_keys, 2);
assert_eq!(stats.verify_executed, 2);
assert_eq!(stats.verify_skipped, 0);
}
#[test]
fn observe_only_mode_counts_would_hit_but_still_verifies() {
let (_dir, cache) = temp_cache(false);
let key = compute_cache_key(CryptoSigVerifyPoint::ChildCaCert, b"tbs", b"sig", b"spki");
cache.insert_positive(key);
let mut executed = 0u64;
cache
.verify(
CryptoSigVerifyPoint::ChildCaCert,
b"tbs",
b"sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Ok(())
},
)
.expect("verify ok");
assert_eq!(
executed, 1,
"observe-only mode must still run the verification"
);
let summary = cache.summary();
assert!(!summary.reuse_enabled);
let stats = summary
.per_point
.get("child_ca_cert")
.expect("stats present");
assert_eq!(stats.would_hit, 1);
assert_eq!(stats.verify_executed, 1);
assert_eq!(stats.verify_skipped, 0);
}
#[test]
fn cache_file_roundtrips_entries_across_reload() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("work-db.crypto-sig-cache");
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
let key = compute_cache_key(CryptoSigVerifyPoint::EeCertFast, b"tbs", b"sig", b"spki");
cache.insert_positive(key);
cache.persist().expect("persist");
let reloaded = CryptoSigCache::load_or_rebuild(path, true);
assert!(reloaded.contains(&key));
let summary = reloaded.summary();
assert_eq!(summary.entries_loaded, 1);
assert_eq!(summary.entries_total, 1);
assert_eq!(summary.schema_version, CACHE_SCHEMA_VERSION);
// Reloaded entry drives a skip.
let mut executed = 0u64;
reloaded
.verify(
CryptoSigVerifyPoint::EeCertFast,
b"tbs",
b"sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Ok(())
},
)
.expect("verify ok");
assert_eq!(executed, 0);
}
#[test]
fn cache_file_rebuilds_on_missing_corrupt_or_mismatched_schema() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("work-db.crypto-sig-cache");
// Missing file.
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
assert_eq!(cache.summary().entries_loaded, 0);
// Garbage content.
std::fs::write(&path, b"not a valid cache file").expect("write garbage");
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
assert_eq!(cache.summary().entries_loaded, 0);
// M1 seen-set format (RPKICSC1) must not load as a v2 cache: cold start.
let mut bytes = Vec::new();
bytes.extend_from_slice(b"RPKICSC1");
bytes.extend_from_slice(&1u64.to_le_bytes());
bytes.extend_from_slice(&[7u8; 32]);
std::fs::write(&path, &bytes).expect("write v1 file");
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
assert_eq!(cache.summary().entries_loaded, 0);
// Valid v2 header but truncated body.
let mut bytes = Vec::new();
bytes.extend_from_slice(CACHE_FILE_MAGIC);
bytes.extend_from_slice(&2u64.to_le_bytes());
bytes.extend_from_slice(&[9u8; RECORD_BYTES]);
std::fs::write(&path, &bytes).expect("write truncated");
let cache = CryptoSigCache::load_or_rebuild(path, true);
assert_eq!(cache.summary().entries_loaded, 0);
}
#[test]
fn size_cap_triggers_full_rebuild_eviction() {
let dir = tempfile::tempdir().expect("tempdir");
let cache = CryptoSigCache::load_or_rebuild_with_capacity(
dir.path().join("work-db.crypto-sig-cache"),
true,
2,
);
let k1 = compute_cache_key(CryptoSigVerifyPoint::Crl, b"a", b"s", b"k");
let k2 = compute_cache_key(CryptoSigVerifyPoint::Crl, b"b", b"s", b"k");
let k3 = compute_cache_key(CryptoSigVerifyPoint::Crl, b"c", b"s", b"k");
cache.insert_positive(k1);
cache.insert_positive(k2);
assert!(cache.contains(&k1) && cache.contains(&k2));
// Third insert exceeds the cap -> full rebuild, then insert.
cache.insert_positive(k3);
assert!(!cache.contains(&k1));
assert!(!cache.contains(&k2));
assert!(cache.contains(&k3));
let summary = cache.summary();
assert_eq!(summary.entries_total, 1);
assert_eq!(summary.evictions_total, 1);
}
#[test]
fn default_cache_file_path_mirrors_pp_cache_index_placement() {
let db = Path::new("/state/db/work-db");
assert_eq!(
default_cache_file_path(db),
PathBuf::from("/state/db/work-db.crypto-sig-cache")
);
}
#[test]
fn verify_with_cache_without_global_cache_runs_verify_and_collects_nothing() {
let _guard = GLOBAL_TEST_LOCK.lock().expect("test lock");
clear_global();
assert!(global_cache().is_none());
let mut executed = 0u64;
let result: Result<u64, String> = (|| {
verify_with_cache(
CryptoSigVerifyPoint::CmsSignedObject,
b"input",
b"sig",
b"spki",
|| -> Result<(), String> {
executed += 1;
Ok(())
},
)?;
Ok(42)
})();
assert_eq!(result, Ok(42));
assert_eq!(executed, 1);
assert!(global_cache().is_none());
}
#[test]
fn verify_with_cache_preserves_result_through_global_install() {
let _guard = GLOBAL_TEST_LOCK.lock().expect("test lock");
clear_global();
let dir = tempfile::tempdir().expect("tempdir");
let cache = Arc::new(CryptoSigCache::load_or_rebuild(
dir.path().join("work-db.crypto-sig-cache"),
true,
));
install_global(Arc::clone(&cache));
let ok: Result<(), String> = verify_with_cache(
CryptoSigVerifyPoint::ChildCaCert,
b"tbs",
b"sig",
b"spki",
|| Ok(()),
);
assert_eq!(ok, Ok(()));
let err: Result<(), String> = verify_with_cache(
CryptoSigVerifyPoint::ChildCaCert,
b"tbs2",
b"sig",
b"spki",
|| Err("bad signature".to_string()),
);
assert_eq!(err, Err("bad signature".to_string()));
let summary = cache.summary();
let stats = summary
.per_point
.get("child_ca_cert")
.expect("stats present");
assert_eq!(stats.calls, 2);
assert_eq!(stats.new_keys, 2);
clear_global();
assert!(global_cache().is_none());
}
}

View File

@ -0,0 +1,290 @@
use std::collections::{HashMap, HashSet};
use std::sync::{Arc, RwLock};
use crate::storage::{RepositoryViewEntry, RepositoryViewState};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CurrentRepoEntry {
pub current_hash: [u8; 32],
pub current_hash_hex: String,
pub repository_source: String,
pub object_type: Option<String>,
pub state: RepositoryViewState,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct CurrentRepoObject {
pub rsync_uri: String,
pub current_hash_hex: String,
pub repository_source: String,
pub object_type: Option<String>,
}
#[derive(Default, Debug)]
pub struct CurrentRepoIndex {
by_uri: HashMap<String, CurrentRepoEntry>,
}
/// Shared handle to the run-wide current repository index. Readers (phase 2
/// publication point staging, cache lookups, output snapshots) take
/// `.read()` and no longer exclude each other; writers (repo sync transports
/// applying repository view entries, run-state reset) take `.write()` and
/// stay exclusive.
pub type CurrentRepoIndexHandle = Arc<RwLock<CurrentRepoIndex>>;
impl CurrentRepoIndex {
pub fn new() -> Self {
Self::default()
}
pub fn shared() -> CurrentRepoIndexHandle {
Arc::new(RwLock::new(Self::new()))
}
pub fn get_by_uri(&self, rsync_uri: &str) -> Option<&CurrentRepoEntry> {
self.by_uri.get(rsync_uri)
}
pub fn list_scope_uris(&self, repository_source: &str) -> Vec<String> {
let mut out = self
.by_uri
.iter()
.filter_map(|(rsync_uri, entry)| {
(entry.repository_source == repository_source).then(|| rsync_uri.clone())
})
.collect::<Vec<_>>();
out.sort();
out
}
pub fn active_uri_count(&self) -> usize {
self.by_uri.len()
}
pub fn scope_count(&self) -> usize {
self.by_uri
.values()
.map(|entry| entry.repository_source.as_str())
.collect::<HashSet<_>>()
.len()
}
pub fn snapshot_objects(&self) -> Vec<CurrentRepoObject> {
let mut out = self
.by_uri
.iter()
.map(|(rsync_uri, entry)| CurrentRepoObject {
rsync_uri: rsync_uri.clone(),
current_hash_hex: entry.current_hash_hex.clone(),
repository_source: entry.repository_source.clone(),
object_type: entry.object_type.clone(),
})
.collect::<Vec<_>>();
out.sort();
out
}
pub fn clear(&mut self) {
self.by_uri.clear();
}
pub fn apply_repository_view_entries(
&mut self,
entries: &[RepositoryViewEntry],
) -> Result<(), String> {
for entry in entries {
self.apply_repository_view_entry(entry)?;
}
Ok(())
}
fn apply_repository_view_entry(&mut self, entry: &RepositoryViewEntry) -> Result<(), String> {
entry.validate_internal().map_err(|e| e.to_string())?;
match entry.state {
RepositoryViewState::Present | RepositoryViewState::Replaced => {
let repository_source = entry.repository_source.clone().ok_or_else(|| {
format!(
"repository_view entry missing repository_source for current object {}",
entry.rsync_uri
)
})?;
let current_hash_hex = entry.current_hash.clone().ok_or_else(|| {
format!(
"repository_view entry missing current_hash for current object {}",
entry.rsync_uri
)
})?;
let current_hash = decode_sha256_hex_32(&current_hash_hex)?;
self.by_uri.insert(
entry.rsync_uri.clone(),
CurrentRepoEntry {
current_hash,
current_hash_hex: current_hash_hex.to_ascii_lowercase(),
repository_source,
object_type: entry.object_type.clone(),
state: entry.state,
},
);
}
RepositoryViewState::Withdrawn => {
self.by_uri.remove(&entry.rsync_uri);
}
}
Ok(())
}
}
fn decode_sha256_hex_32(value: &str) -> Result<[u8; 32], String> {
if value.len() != 64 || !value.as_bytes().iter().all(u8::is_ascii_hexdigit) {
return Err(format!("invalid sha256 hex: {value}"));
}
let mut out = [0u8; 32];
hex::decode_to_slice(value, &mut out).map_err(|e| format!("hex decode failed: {e}"))?;
Ok(out)
}
#[cfg(test)]
mod tests {
use super::CurrentRepoIndex;
use crate::storage::{RepositoryViewEntry, RepositoryViewState};
fn present(source: &str, uri: &str, hash: &str) -> RepositoryViewEntry {
RepositoryViewEntry {
rsync_uri: uri.to_string(),
current_hash: Some(hash.to_string()),
repository_source: Some(source.to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
}
}
#[test]
fn current_repo_index_tracks_present_and_withdrawn_entries() {
let mut index = CurrentRepoIndex::new();
let uri = "rsync://example.test/repo/a.roa";
let source = "rsync://example.test/repo/";
let hash = &"11".repeat(32);
index
.apply_repository_view_entries(&[present(source, uri, hash)])
.expect("apply present");
let got = index.get_by_uri(uri).expect("current entry");
assert_eq!(got.current_hash_hex, hash.to_string());
assert_eq!(index.list_scope_uris(source), vec![uri.to_string()]);
index
.apply_repository_view_entries(&[RepositoryViewEntry {
rsync_uri: uri.to_string(),
current_hash: Some(hash.to_string()),
repository_source: Some(source.to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Withdrawn,
}])
.expect("apply withdrawn");
assert!(index.get_by_uri(uri).is_none());
assert!(index.list_scope_uris(source).is_empty());
}
#[test]
fn current_repo_index_moves_uri_between_scopes() {
let mut index = CurrentRepoIndex::new();
let uri = "rsync://example.test/repo/a.roa";
let old_scope = "rsync://example.test/repo/";
let new_scope = "https://rrdp.example.test/notification.xml";
index
.apply_repository_view_entries(&[present(old_scope, uri, &"22".repeat(32))])
.expect("apply old scope");
index
.apply_repository_view_entries(&[present(new_scope, uri, &"33".repeat(32))])
.expect("apply new scope");
assert!(index.list_scope_uris(old_scope).is_empty());
assert_eq!(index.list_scope_uris(new_scope), vec![uri.to_string()]);
assert_eq!(
index.get_by_uri(uri).expect("entry").current_hash_hex,
"33".repeat(32)
);
}
#[test]
fn current_repo_index_snapshot_objects_and_counts_are_sorted() {
let handle = CurrentRepoIndex::shared();
let mut index = handle.write().expect("write-lock index");
index
.apply_repository_view_entries(&[
present(
"rsync://example.test/repo-b/",
"rsync://example.test/repo-b/b.roa",
&"22".repeat(32),
),
present(
"rsync://example.test/repo-a/",
"rsync://example.test/repo-a/a.roa",
&"11".repeat(32),
),
])
.expect("apply present entries");
assert_eq!(index.active_uri_count(), 2);
assert_eq!(index.scope_count(), 2);
let snapshot = index.snapshot_objects();
assert_eq!(snapshot.len(), 2);
assert_eq!(snapshot[0].rsync_uri, "rsync://example.test/repo-a/a.roa");
assert_eq!(snapshot[1].rsync_uri, "rsync://example.test/repo-b/b.roa");
}
#[test]
fn current_repo_index_reports_missing_fields_and_invalid_hash() {
let mut index = CurrentRepoIndex::new();
let err = index
.apply_repository_view_entries(&[RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
current_hash: Some("11".repeat(32)),
repository_source: None,
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
}])
.expect_err("missing source should fail");
assert!(err.contains("missing repository_source"), "{err}");
let err = index
.apply_repository_view_entries(&[RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
current_hash: Some("not-a-valid-sha256".to_string()),
repository_source: Some("rsync://example.test/repo/".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
}])
.expect_err("invalid hash should fail");
assert!(err.contains("invalid"), "{err}");
index
.apply_repository_view_entries(&[RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/b.roa".to_string(),
current_hash: Some("22".repeat(32)),
repository_source: Some("rsync://example.test/repo/".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
}])
.expect("valid entry");
let got = index.get_by_uri("rsync://example.test/repo/b.roa").unwrap();
assert_eq!(got.current_hash_hex, "22".repeat(32));
}
#[test]
fn current_repo_index_withdraw_unknown_uri_is_noop() {
let mut index = CurrentRepoIndex::new();
index
.apply_repository_view_entries(&[RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/missing.roa".to_string(),
current_hash: None,
repository_source: Some("rsync://example.test/repo/".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Withdrawn,
}])
.expect("withdraw unknown should not fail");
assert_eq!(index.active_uri_count(), 0);
assert_eq!(index.scope_count(), 0);
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,414 @@
use crate::data_model::common::{DerReader, der_take_tlv};
use crate::data_model::oid::OID_CT_ASPA;
use crate::data_model::rc::ResourceCertificate;
use crate::data_model::signed_object::{
RpkiSignedObject, RpkiSignedObjectParsed, SignedObjectDecodeError, SignedObjectParseError,
SignedObjectValidateError,
};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AspaObject {
pub signed_object: RpkiSignedObject,
pub econtent_type: String,
pub aspa: AspaEContent,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AspaObjectParsed {
pub signed_object: RpkiSignedObjectParsed,
pub econtent_type: String,
pub aspa: Option<AspaEContentParsed>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AspaEContent {
pub version: u32,
pub customer_as_id: u32,
pub provider_as_ids: Vec<u32>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AspaEContentParsed {
der: Vec<u8>,
}
#[derive(Debug, thiserror::Error)]
pub enum AspaParseError {
#[error("signed object parse error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
SignedObject(#[from] SignedObjectParseError),
#[error("ASPA parse error: {0} (draft-ietf-sidrops-aspa-profile-21 §3; DER)")]
Parse(String),
#[error("ASPA trailing bytes: {0} bytes (draft-ietf-sidrops-aspa-profile-21 §3; DER)")]
TrailingBytes(usize),
}
#[derive(Debug, thiserror::Error)]
pub enum AspaProfileError {
#[error("signed object profile error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
SignedObject(#[from] SignedObjectValidateError),
#[error(
"ASPA eContentType must be {OID_CT_ASPA}, got {0} (draft-ietf-sidrops-aspa-profile-21 §2)"
)]
InvalidEContentType(String),
#[error("ASPA profile decode error: {0} (draft-ietf-sidrops-aspa-profile-21 §3; DER)")]
ProfileDecode(String),
#[error(
"ASProviderAttestation must be a SEQUENCE of 3 elements (draft-ietf-sidrops-aspa-profile-21 §3)"
)]
InvalidAttestationSequence,
#[error(
"ASPA version must be 1 and MUST be explicitly encoded (draft-ietf-sidrops-aspa-profile-21 §3.1)"
)]
VersionMustBeExplicitOne,
#[error(
"ASPA customerASID out of range (0..=4294967295), got {0} (draft-ietf-sidrops-aspa-profile-21 §3.2)"
)]
CustomerAsIdOutOfRange(u64),
#[error(
"ASPA providers must contain at least one ASID (draft-ietf-sidrops-aspa-profile-21 §3.3)"
)]
EmptyProviders,
#[error(
"ASPA provider ASID out of range (0..=4294967295), got {0} (draft-ietf-sidrops-aspa-profile-21 §3.3)"
)]
ProviderAsIdOutOfRange(u64),
#[error(
"ASPA providers must be in strictly increasing order (draft-ietf-sidrops-aspa-profile-21 §3.3)"
)]
ProvidersNotStrictlyIncreasing,
#[error(
"ASPA providers contains the customerASID ({0}) which is not allowed (draft-ietf-sidrops-aspa-profile-21 §3.3)"
)]
ProvidersContainCustomer(u32),
}
impl From<SignedObjectDecodeError> for AspaProfileError {
fn from(value: SignedObjectDecodeError) -> Self {
match value {
SignedObjectDecodeError::Parse(e) => AspaProfileError::ProfileDecode(e.to_string()),
SignedObjectDecodeError::Validate(e) => AspaProfileError::SignedObject(e),
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum AspaDecodeError {
#[error("{0}")]
Parse(#[from] AspaParseError),
#[error("{0}")]
Validate(#[from] AspaProfileError),
}
#[derive(Debug, thiserror::Error)]
pub enum AspaValidateError {
#[error(
"ASPA EE certificate must contain AS resources extension (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2)"
)]
EeAsResourcesMissing,
#[error(
"ASPA EE certificate AS resources must not use inherit (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2.3.3)"
)]
EeAsResourcesInherit,
#[error(
"ASPA EE certificate AS resources must not include ranges (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2.3.6-§3.2.3.7)"
)]
EeAsResourcesRangePresent,
#[error(
"ASPA EE certificate AS resources must not include RDI (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §3.2.3.5; RFC 6487 §4.8.11)"
)]
EeAsResourcesRdiPresent,
#[error(
"ASPA EE certificate AS resources must contain exactly one ASID (id element) (draft-ietf-sidrops-aspa-profile-21 §4)"
)]
EeAsResourcesNotSingleId,
#[error(
"ASPA customerASID ({customer_as_id}) does not match EE AS resources ({ee_as_id}) (draft-ietf-sidrops-aspa-profile-21 §4)"
)]
CustomerAsIdMismatch { customer_as_id: u32, ee_as_id: u32 },
#[error(
"ASPA EE certificate must not contain IP resources extension (draft-ietf-sidrops-aspa-profile-21 §4; RFC 3779 §2.2)"
)]
EeIpResourcesPresent,
}
impl AspaObject {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<AspaObjectParsed, AspaParseError> {
let signed_object = RpkiSignedObject::parse_der(der)?;
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
let aspa = signed_object
.signed_data
.encap_content_info
.econtent
.as_deref()
.map(AspaEContent::parse_der)
.transpose()?;
Ok(AspaObjectParsed {
signed_object,
econtent_type,
aspa,
})
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `AspaObject` is already profile-validated when constructed via `decode_der()` /
/// `AspaObjectParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), AspaProfileError> {
Ok(())
}
pub fn decode_der(der: &[u8]) -> Result<Self, AspaDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn decode_der_with_strict_options(
der: &[u8],
strict_cms_der: bool,
strict_name: bool,
) -> Result<Self, AspaDecodeError> {
let signed_object =
RpkiSignedObject::decode_der_with_strict_options(der, strict_cms_der, strict_name)
.map_err(AspaProfileError::from)?;
Self::from_signed_object(signed_object)
}
pub fn from_signed_object(signed_object: RpkiSignedObject) -> Result<Self, AspaDecodeError> {
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
if econtent_type != OID_CT_ASPA {
return Err(AspaProfileError::InvalidEContentType(econtent_type).into());
}
let aspa =
AspaEContent::decode_der(&signed_object.signed_data.encap_content_info.econtent)?;
Ok(Self {
aspa,
signed_object,
econtent_type: OID_CT_ASPA.to_string(),
})
}
/// Validate this ASPA's embedded EE certificate resources.
pub fn validate_embedded_ee_cert(&self) -> Result<(), AspaValidateError> {
let ee = &self.signed_object.signed_data.certificates[0].resource_cert;
self.aspa.validate_against_ee_cert(ee)
}
}
impl AspaEContent {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<AspaEContentParsed, AspaParseError> {
let (_tag, _value, rem) = der_take_tlv(der).map_err(AspaParseError::Parse)?;
if !rem.is_empty() {
return Err(AspaParseError::TrailingBytes(rem.len()));
}
Ok(AspaEContentParsed { der: der.to_vec() })
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `AspaEContent` is already profile-validated when constructed via `decode_der()` /
/// `AspaEContentParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), AspaProfileError> {
Ok(())
}
/// Decode the DER-encoded ASProviderAttestation defined in
/// draft-ietf-sidrops-aspa-profile-21 §3 (`parse + validate`).
pub fn decode_der(der: &[u8]) -> Result<Self, AspaDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
/// Validate ASPA payload against the embedded EE resource certificate.
///
/// This implements the EE/payload semantic checks described in
/// `draft-ietf-sidrops-aspa-profile-21` §4 (as summarized in `rpki/specs/08_aspa.md`).
pub fn validate_against_ee_cert(
&self,
ee: &ResourceCertificate,
) -> Result<(), AspaValidateError> {
if ee.tbs.extensions.ip_resources.is_some() {
return Err(AspaValidateError::EeIpResourcesPresent);
}
let asn = ee
.tbs
.extensions
.as_resources
.as_ref()
.ok_or(AspaValidateError::EeAsResourcesMissing)?;
if asn.rdi.is_some() {
return Err(AspaValidateError::EeAsResourcesRdiPresent);
}
if asn.is_asnum_inherit() {
return Err(AspaValidateError::EeAsResourcesInherit);
}
if asn.has_any_range() {
return Err(AspaValidateError::EeAsResourcesRangePresent);
}
let ee_as_id = asn
.asnum_single_id()
.ok_or(AspaValidateError::EeAsResourcesNotSingleId)?;
if ee_as_id != self.customer_as_id {
return Err(AspaValidateError::CustomerAsIdMismatch {
customer_as_id: self.customer_as_id,
ee_as_id,
});
}
Ok(())
}
}
impl AspaObjectParsed {
pub fn validate_profile(self) -> Result<AspaObject, AspaProfileError> {
let signed_object = self.signed_object.validate_profile()?;
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
if econtent_type != OID_CT_ASPA {
return Err(AspaProfileError::InvalidEContentType(econtent_type));
}
let aspa = self
.aspa
.ok_or_else(|| AspaProfileError::ProfileDecode("ASPA.eContent missing".into()))?
.validate_profile()?;
Ok(AspaObject {
signed_object,
econtent_type: OID_CT_ASPA.to_string(),
aspa,
})
}
}
impl AspaEContentParsed {
pub fn validate_profile(self) -> Result<AspaEContent, AspaProfileError> {
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
let mut n = 0usize;
while !r.is_empty() {
r.skip_any()?;
n += 1;
}
Ok(n)
}
let mut r = DerReader::new(&self.der);
let mut seq = r
.take_sequence()
.map_err(|e| AspaProfileError::ProfileDecode(e))?;
if !r.is_empty() {
return Err(AspaProfileError::ProfileDecode(
"trailing bytes after ASProviderAttestation".into(),
));
}
let elem_count =
count_elements(seq).map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
if elem_count != 3 {
return Err(AspaProfileError::InvalidAttestationSequence);
}
// version [0] EXPLICIT INTEGER MUST be present and MUST be 1.
if seq
.peek_tag()
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?
!= 0xA0
{
return Err(AspaProfileError::VersionMustBeExplicitOne);
}
let (inner_tag, inner_val) = seq
.take_explicit(0xA0)
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
if inner_tag != 0x02 {
return Err(AspaProfileError::VersionMustBeExplicitOne);
}
let v = crate::data_model::common::der_uint_from_bytes(inner_val)
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
if v != 1 {
return Err(AspaProfileError::VersionMustBeExplicitOne);
}
let customer_u64 = seq
.take_uint_u64()
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
if customer_u64 > u32::MAX as u64 {
return Err(AspaProfileError::CustomerAsIdOutOfRange(customer_u64));
}
let customer_as_id = customer_u64 as u32;
let providers_seq = seq
.take_sequence()
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
if !seq.is_empty() {
return Err(AspaProfileError::InvalidAttestationSequence);
}
let providers = parse_providers_cursor(providers_seq, customer_as_id)?;
Ok(AspaEContent {
version: 1,
customer_as_id,
provider_as_ids: providers,
})
}
}
fn parse_providers_cursor(
mut seq: DerReader<'_>,
customer_as_id: u32,
) -> Result<Vec<u32>, AspaProfileError> {
if seq.is_empty() {
return Err(AspaProfileError::EmptyProviders);
}
let mut out: Vec<u32> = Vec::new();
let mut prev: Option<u32> = None;
while !seq.is_empty() {
let v = seq
.take_uint_u64()
.map_err(|e| AspaProfileError::ProfileDecode(e.to_string()))?;
if v > u32::MAX as u64 {
return Err(AspaProfileError::ProviderAsIdOutOfRange(v));
}
let asn = v as u32;
if asn == customer_as_id {
return Err(AspaProfileError::ProvidersContainCustomer(customer_as_id));
}
if let Some(p) = prev {
if asn <= p {
return Err(AspaProfileError::ProvidersNotStrictlyIncreasing);
}
}
prev = Some(asn);
out.push(asn);
}
Ok(out)
}

View File

@ -0,0 +1,302 @@
use x509_parser::asn1_rs::Tag;
use x509_parser::prelude::FromDer;
use x509_parser::x509::AlgorithmIdentifier;
pub type UtcTime = time::OffsetDateTime;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Asn1TimeEncoding {
UtcTime,
GeneralizedTime,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Asn1TimeUtc {
pub utc: UtcTime,
pub encoding: Asn1TimeEncoding,
}
impl Asn1TimeUtc {
/// Validate Time encoding rules (RFC 5280): years 1950-2049 use UTCTime,
/// other years use GeneralizedTime.
pub fn validate_encoding_rfc5280(
&self,
field: &'static str,
) -> Result<(), InvalidTimeEncodingError> {
let year = self.utc.year();
let expected = if year <= 2049 {
Asn1TimeEncoding::UtcTime
} else {
Asn1TimeEncoding::GeneralizedTime
};
if self.encoding != expected {
return Err(InvalidTimeEncodingError {
field,
year,
encoding: self.encoding,
});
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BigUnsigned {
/// Minimal big-endian bytes. For zero, this is `[0]`.
pub bytes_be: Vec<u8>,
}
impl BigUnsigned {
pub fn from_biguint(n: &der_parser::num_bigint::BigUint) -> Self {
let mut bytes = n.to_bytes_be();
if bytes.is_empty() {
bytes.push(0);
}
Self { bytes_be: bytes }
}
pub fn to_hex_upper(&self) -> String {
hex::encode_upper(&self.bytes_be)
}
pub fn to_u64(&self) -> Option<u64> {
if self.bytes_be.len() > 8 {
return None;
}
let mut value: u64 = 0;
for &b in &self.bytes_be {
value = (value << 8) | (b as u64);
}
Some(value)
}
}
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
#[error(
"{field} time encoding invalid for year {year}: got {encoding:?} (RFC 5280 §4.1.2.5; RFC 5280 §5.1.2.4-§5.1.2.6)"
)]
pub struct InvalidTimeEncodingError {
pub field: &'static str,
pub year: i32,
pub encoding: Asn1TimeEncoding,
}
pub fn asn1_time_to_model(t: x509_parser::time::ASN1Time) -> Asn1TimeUtc {
let encoding = if t.is_utctime() {
Asn1TimeEncoding::UtcTime
} else {
Asn1TimeEncoding::GeneralizedTime
};
Asn1TimeUtc {
utc: t.to_datetime(),
encoding,
}
}
pub fn algorithm_params_absent_or_null(sig: &AlgorithmIdentifier<'_>) -> bool {
match sig.parameters.as_ref() {
None => true,
Some(p) if p.tag() == Tag::Null => true,
Some(_p) => false,
}
}
/// Take a single DER TLV (Tag-Length-Value) from the start of `input`.
///
/// This helper supports:
/// - short- and long-form lengths (up to 8 length bytes)
/// - only low-tag-number form tags (no high-tag-number form)
/// - definite length only (DER forbids indefinite length)
///
/// Returns: `(tag_byte, value_bytes, remaining_bytes)`.
pub(crate) fn der_take_tlv(input: &[u8]) -> Result<(u8, &[u8], &[u8]), String> {
if input.len() < 2 {
return Err("truncated DER (need tag+len)".into());
}
let tag = input[0];
if (tag & 0x1F) == 0x1F {
return Err("high-tag-number form not supported".into());
}
let len0 = input[1];
if len0 == 0x80 {
return Err("indefinite length not allowed in DER".into());
}
let (len, hdr_len) = if len0 & 0x80 == 0 {
(len0 as usize, 2usize)
} else {
let n = (len0 & 0x7F) as usize;
if n == 0 || n > 8 {
return Err("invalid DER length".into());
}
if input.len() < 2 + n {
return Err("truncated DER (length bytes)".into());
}
let mut l: usize = 0;
for &b in &input[2..2 + n] {
l = (l << 8) | (b as usize);
}
(l, 2 + n)
};
if input.len() < hdr_len + len {
return Err("truncated DER (value bytes)".into());
}
let value = &input[hdr_len..hdr_len + len];
let rem = &input[hdr_len + len..];
Ok((tag, value, rem))
}
/// Minimal streaming DER reader built on `der_take_tlv`.
///
/// This is intentionally small and only supports the subset of DER needed by
/// RPKI object eContent decoders (ROA/ASPA), to avoid constructing a generic AST
/// (which is expensive on large objects such as ROAs with thousands of prefixes).
#[derive(Clone, Copy)]
pub(crate) struct DerReader<'a> {
buf: &'a [u8],
}
impl<'a> DerReader<'a> {
pub(crate) fn new(buf: &'a [u8]) -> Self {
Self { buf }
}
pub(crate) fn is_empty(&self) -> bool {
self.buf.is_empty()
}
pub(crate) fn remaining_len(&self) -> usize {
self.buf.len()
}
pub(crate) fn peek_tag(&self) -> Result<u8, String> {
self.buf
.first()
.copied()
.ok_or_else(|| "truncated DER".into())
}
pub(crate) fn take_any(&mut self) -> Result<(u8, &'a [u8]), String> {
let (tag, value, rem) = der_take_tlv(self.buf)?;
self.buf = rem;
Ok((tag, value))
}
pub(crate) fn take_any_full(&mut self) -> Result<(u8, &'a [u8], &'a [u8]), String> {
let (tag, value, rem) = der_take_tlv(self.buf)?;
let consumed = self.buf.len() - rem.len();
let full = &self.buf[..consumed];
self.buf = rem;
Ok((tag, full, value))
}
pub(crate) fn skip_any(&mut self) -> Result<(), String> {
let _ = self.take_any()?;
Ok(())
}
pub(crate) fn take_tag(&mut self, expected_tag: u8) -> Result<&'a [u8], String> {
let (tag, value) = self.take_any()?;
if tag != expected_tag {
return Err(format!(
"unexpected tag: got 0x{tag:02X}, expected 0x{expected_tag:02X}"
));
}
Ok(value)
}
pub(crate) fn take_sequence(&mut self) -> Result<DerReader<'a>, String> {
let value = self.take_tag(0x30)?;
Ok(DerReader::new(value))
}
pub(crate) fn take_octet_string(&mut self) -> Result<&'a [u8], String> {
self.take_tag(0x04)
}
pub(crate) fn take_bit_string(&mut self) -> Result<(u8, &'a [u8]), String> {
let v = self.take_tag(0x03)?;
if v.is_empty() {
return Err("BIT STRING content is empty".into());
}
Ok((v[0], &v[1..]))
}
pub(crate) fn take_uint_u64(&mut self) -> Result<u64, String> {
let v = self.take_tag(0x02)?;
der_uint_from_bytes(v)
}
pub(crate) fn take_explicit(
&mut self,
expected_outer_tag: u8,
) -> Result<(u8, &'a [u8]), String> {
let inner_der = self.take_tag(expected_outer_tag)?;
let (tag, value, rem) = der_take_tlv(inner_der)?;
if !rem.is_empty() {
return Err("trailing bytes inside EXPLICIT value".into());
}
Ok((tag, value))
}
pub(crate) fn take_explicit_der(&mut self, expected_outer_tag: u8) -> Result<&'a [u8], String> {
let inner_der = self.take_tag(expected_outer_tag)?;
let (_tag, _value, rem) = der_take_tlv(inner_der)?;
if !rem.is_empty() {
return Err("trailing bytes inside EXPLICIT value".into());
}
Ok(inner_der)
}
}
pub(crate) fn der_uint_from_bytes(bytes: &[u8]) -> Result<u64, String> {
if bytes.is_empty() {
return Err("INTEGER has empty content".into());
}
// Disallow negative values.
if (bytes[0] & 0x80) != 0 {
return Err("INTEGER is negative".into());
}
// DER requires minimal encoding for INTEGER.
if bytes.len() > 1 && bytes[0] == 0x00 && (bytes[1] & 0x80) == 0 {
return Err("INTEGER not minimally encoded".into());
}
if bytes.len() > 8 {
return Err("INTEGER does not fit u64".into());
}
let mut v: u64 = 0;
for &b in bytes {
v = (v << 8) | (b as u64);
}
Ok(v)
}
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct X509NameDer(pub Vec<u8>);
impl X509NameDer {
pub fn as_raw(&self) -> &[u8] {
&self.0
}
}
impl std::fmt::Display for X509NameDer {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let Ok((rem, name)) = x509_parser::x509::X509Name::from_der(&self.0) else {
return write!(f, "<invalid X.509 Name DER>");
};
if !rem.is_empty() {
return write!(f, "<invalid X.509 Name DER (trailing bytes)>");
}
write!(f, "{name}")
}
}
/// Filename extensions registered in IANA "RPKI Repository Name Schemes".
///
/// Source: <https://www.iana.org/assignments/rpki/rpki.xhtml>
/// Snapshot date: 2026-01-28.
///
/// Notes:
/// - Includes entries marked TEMPORARY/DEPRECATED by IANA (e.g., `asa`, `gbr`).
pub const IANA_RPKI_REPOSITORY_FILENAME_EXTENSIONS: &[&str] =
&["asa", "cer", "crl", "gbr", "mft", "roa", "sig", "tak"];

View File

@ -0,0 +1,546 @@
pub use crate::data_model::common::{Asn1TimeEncoding, Asn1TimeUtc, BigUnsigned};
use crate::data_model::oid::{
OID_AUTHORITY_KEY_IDENTIFIER, OID_AUTHORITY_KEY_IDENTIFIER_RAW, OID_CRL_NUMBER,
OID_CRL_NUMBER_RAW, OID_SHA256_WITH_RSA_ENCRYPTION, OID_SHA256_WITH_RSA_ENCRYPTION_RAW,
OID_SUBJECT_KEY_IDENTIFIER_RAW,
};
use x509_parser::certificate::X509Certificate;
use x509_parser::extensions::{ParsedExtension, X509Extension};
use x509_parser::prelude::{FromDer, X509Version};
use x509_parser::revocation_list::CertificateRevocationList;
use x509_parser::x509::{AlgorithmIdentifier, SubjectPublicKeyInfo};
use x509_parser::{asn1_rs::Class as Asn1Class, asn1_rs::Tag as Asn1Tag};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RevokedCert {
pub serial_number: BigUnsigned,
pub revocation_date: Asn1TimeUtc,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CrlExtensions {
pub authority_key_identifier: Vec<u8>,
pub crl_number: BigUnsigned,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RpkixCrl {
pub raw_der: Vec<u8>,
pub version: u32,
pub issuer_dn: String,
pub signature_algorithm_oid: String,
pub this_update: Asn1TimeUtc,
pub next_update: Asn1TimeUtc,
pub revoked_certs: Vec<RevokedCert>,
pub extensions: CrlExtensions,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RpkixCrlParsed {
pub raw_der: Vec<u8>,
pub version: Option<X509Version>,
pub issuer_dn: String,
pub signature_algorithm: AlgorithmIdentifierValue,
pub tbs_signature_algorithm: AlgorithmIdentifierValue,
pub this_update: Asn1TimeUtc,
pub next_update: Option<Asn1TimeUtc>,
pub revoked_certs: Vec<RevokedCertParsed>,
pub extensions: Vec<CrlExtensionParsed>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RevokedCertParsed {
pub serial_number: BigUnsigned,
pub revocation_date: Asn1TimeUtc,
pub has_extensions: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum CrlExtensionParsed {
AuthorityKeyIdentifier {
key_identifier: Option<Vec<u8>>,
has_other_fields: bool,
critical: bool,
},
CrlNumber {
number: der_parser::num_bigint::BigUint,
critical: bool,
},
Other {
oid: String,
critical: bool,
},
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AlgorithmIdentifierValue {
pub oid: String,
pub parameters: Option<AlgorithmParametersValue>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AlgorithmParametersValue {
pub class: Asn1Class,
pub tag: Asn1Tag,
pub data: Vec<u8>,
}
impl AlgorithmIdentifierValue {
pub fn params_absent_or_null(&self) -> bool {
match &self.parameters {
None => true,
Some(p) if p.class == Asn1Class::Universal && p.tag == Asn1Tag::Null => true,
Some(_p) => false,
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum CrlParseError {
#[error("X.509 CRL parse error: {0} (RFC 5280 §5.1; RFC 6487 §5)")]
Parse(String),
#[error("trailing bytes after CRL DER: {0} bytes (DER; RFC 5280 §5.1)")]
TrailingBytes(usize),
}
#[derive(Debug, thiserror::Error)]
pub enum CrlProfileError {
#[error("CRL version must be v2, got {0:?} (RFC 5280 §5.1; RFC 6487 §5)")]
InvalidVersion(Option<u32>),
#[error("CRL signatureAlgorithm must match TBSCertList.signature (RFC 5280 §5.1)")]
SignatureAlgorithmMismatch,
#[error(
"CRL signatureAlgorithm must be sha256WithRSAEncryption ({OID_SHA256_WITH_RSA_ENCRYPTION}), got {0} (RFC 6487 §5; RFC 7935 §2)"
)]
InvalidSignatureAlgorithm(String),
#[error(
"CRL signature algorithm parameters must be absent or NULL (RFC 5280 §4.1.1.2; RFC 7935 §2)"
)]
InvalidSignatureAlgorithmParameters,
#[error("CRL extensions must be exactly two (AKI + CRLNumber), got {0} (RFC 9829 §3.1)")]
InvalidExtensionsCount(usize),
#[error("unsupported CRL extension OID {0} (RFC 9829 §3.1)")]
UnsupportedExtension(String),
#[error("duplicate CRL extension OID {0} (RFC 5280 §4.2; RFC 9829 §3.1)")]
DuplicateExtension(String),
#[error("AuthorityKeyIdentifier CRL extension missing (RFC 9829 §3.1; RFC 5280 §5.2.1)")]
AkiMissing,
#[error("AuthorityKeyIdentifier must contain keyIdentifier (RFC 5280 §5.2.1; RFC 9829 §3.1)")]
AkiMissingKeyIdentifier,
#[error(
"AuthorityKeyIdentifier must not contain authorityCertIssuer or authorityCertSerialNumber (RFC 5280 §5.2.1; RFC 9829 §3.1)"
)]
AkiHasOtherFields,
#[error("CRLNumber CRL extension missing (RFC 9829 §3.1; RFC 5280 §5.2.3)")]
CrlNumberMissing,
#[error("CRLNumber must be non-critical (RFC 9829 §3.1; RFC 5280 §5.2.3)")]
CrlNumberCritical,
#[error("CRLNumber out of range (must fit in 0..2^159-1) (RFC 9829 §3.1)")]
CrlNumberOutOfRange,
#[error("CRL entry extensions must not be present (RFC 6487 §5; RFC 5280 §5.1)")]
EntryExtensionsNotAllowed,
#[error("CRL nextUpdate must be present (RFC 5280 §5.1.2.5; RFC 6487 §5)")]
NextUpdateMissing,
#[error(
"{field} time encoding invalid for year {year}: got {encoding:?} (RFC 5280 §5.1.2.4-§5.1.2.6)"
)]
InvalidTimeEncoding {
field: &'static str,
year: i32,
encoding: Asn1TimeEncoding,
},
}
#[derive(Debug, thiserror::Error)]
pub enum CrlDecodeError {
#[error("{0}")]
Parse(#[from] CrlParseError),
#[error("{0}")]
Validate(#[from] CrlProfileError),
}
impl RpkixCrl {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<RpkixCrlParsed, CrlParseError> {
let (rem, crl) = CertificateRevocationList::from_der(der)
.map_err(|e| CrlParseError::Parse(e.to_string()))?;
if !rem.is_empty() {
return Err(CrlParseError::TrailingBytes(rem.len()));
}
let revoked_certs = crl
.iter_revoked_certificates()
.map(|rc| RevokedCertParsed {
serial_number: BigUnsigned::from_biguint(rc.serial()),
revocation_date: crate::data_model::common::asn1_time_to_model(rc.revocation_date),
has_extensions: !rc.extensions().is_empty(),
})
.collect::<Vec<_>>();
let this_update = crate::data_model::common::asn1_time_to_model(crl.last_update());
let next_update = crl
.next_update()
.map(crate::data_model::common::asn1_time_to_model);
let extensions =
parse_extensions_parse(crl.extensions()).map_err(|e| CrlParseError::Parse(e))?;
Ok(RpkixCrlParsed {
raw_der: der.to_vec(),
version: crl.version(),
issuer_dn: crl.issuer().to_string(),
signature_algorithm: algorithm_identifier_value(&crl.signature_algorithm),
tbs_signature_algorithm: algorithm_identifier_value(&crl.tbs_cert_list.signature),
this_update,
next_update,
revoked_certs,
extensions,
})
}
/// Decode a DER-encoded X.509 v2 CRL and enforce the RPKI profile constraints from
/// `specs/prepare/data_models/04_crl.md` (RFC 6487 §5; RFC 9829 §3.1; RFC 5280 §5.1).
pub fn decode_der(der: &[u8]) -> Result<Self, CrlDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `RpkixCrl` is already profile-validated when constructed via `decode_der()` /
/// `RpkixCrlParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), CrlProfileError> {
Ok(())
}
/// Verify the cryptographic signature on this CRL using the issuer certificate.
///
/// Signature verification needs the issuer public key (RFC 5280 §6.3.3 (f)-(g)).
/// In RPKI practice, this public key is obtained from the CRL issuer CA certificate
/// (and that certificate must already be validated up to the same trust anchor).
///
/// This helper also performs common binding checks:
/// - CRL `issuer_dn` must equal issuer certificate `subject`
/// - if issuer KeyUsage is present, require `cRLSign`
/// - if issuer SKI is present, require it matches CRL AKI.keyIdentifier
pub fn verify_signature_with_issuer_certificate_der(
&self,
issuer_cert_der: &[u8],
) -> Result<(), CrlVerifyError> {
let (rem, issuer_cert) = X509Certificate::from_der(issuer_cert_der)
.map_err(|e| CrlVerifyError::IssuerCertificateParse(e.to_string()))?;
if !rem.is_empty() {
return Err(CrlVerifyError::IssuerCertificateTrailingBytes(rem.len()));
}
let subject_dn = issuer_cert.subject().to_string();
if subject_dn != self.issuer_dn {
return Err(CrlVerifyError::IssuerSubjectMismatch {
crl_issuer_dn: self.issuer_dn.clone(),
issuer_subject_dn: subject_dn,
});
}
if let Some(ku) = issuer_cert
.key_usage()
.map_err(|e| CrlVerifyError::IssuerCertificateParse(e.to_string()))?
{
if !ku.value.crl_sign() {
return Err(CrlVerifyError::IssuerKeyUsageMissingCrlSign);
}
}
if let Some(issuer_ski) = get_subject_key_identifier(&issuer_cert) {
if issuer_ski != self.extensions.authority_key_identifier {
return Err(CrlVerifyError::AkiSkiMismatch);
}
}
self.verify_signature_with_issuer_spki(issuer_cert.public_key())
}
/// Verify the cryptographic signature on this CRL using the issuer SubjectPublicKeyInfo.
pub fn verify_signature_with_issuer_spki(
&self,
issuer_spki: &SubjectPublicKeyInfo<'_>,
) -> Result<(), CrlVerifyError> {
let (rem, crl) = CertificateRevocationList::from_der(&self.raw_der)
.map_err(|e| CrlVerifyError::CrlParse(e.to_string()))?;
if !rem.is_empty() {
return Err(CrlVerifyError::CrlTrailingBytes(rem.len()));
}
crate::crypto_sig_cache::verify_with_cache(
crate::crypto_sig_cache::CryptoSigVerifyPoint::Crl,
crl.tbs_cert_list.as_ref(),
crl.signature_value.data.as_ref(),
issuer_spki.raw,
|| {
crl.verify_signature(issuer_spki)
.map_err(|e| CrlVerifyError::InvalidSignature(e.to_string()))
},
)
}
/// Verify the cryptographic signature on this CRL using a DER-encoded SubjectPublicKeyInfo.
pub fn verify_signature_with_issuer_spki_der(
&self,
issuer_spki_der: &[u8],
) -> Result<(), CrlVerifyError> {
let (rem, spki) = SubjectPublicKeyInfo::from_der(issuer_spki_der)
.map_err(|e| CrlVerifyError::IssuerSpkiParse(e.to_string()))?;
if !rem.is_empty() {
return Err(CrlVerifyError::IssuerSpkiTrailingBytes(rem.len()));
}
self.verify_signature_with_issuer_spki(&spki)
}
}
impl RpkixCrlParsed {
/// Profile validate step of scheme A (`parse → validate → verify`).
pub fn validate_profile(self) -> Result<RpkixCrl, CrlProfileError> {
let version = match self.version {
Some(X509Version::V2) => 2,
Some(v) => return Err(CrlProfileError::InvalidVersion(Some(v.0))),
None => return Err(CrlProfileError::InvalidVersion(None)),
};
// signatureAlgorithm must match tbsCertList.signature
if self.signature_algorithm != self.tbs_signature_algorithm {
return Err(CrlProfileError::SignatureAlgorithmMismatch);
}
let sig_oid = self.signature_algorithm.oid.clone();
if sig_oid != OID_SHA256_WITH_RSA_ENCRYPTION {
return Err(CrlProfileError::InvalidSignatureAlgorithm(sig_oid));
}
if !self.signature_algorithm.params_absent_or_null() {
return Err(CrlProfileError::InvalidSignatureAlgorithmParameters);
}
let extensions = validate_extensions_profile(&self.extensions)?;
let mut revoked_out = Vec::with_capacity(self.revoked_certs.len());
for rc in self.revoked_certs {
if rc.has_extensions {
return Err(CrlProfileError::EntryExtensionsNotAllowed);
}
validate_time_encoding_rfc5280("revocationDate", &rc.revocation_date)?;
revoked_out.push(RevokedCert {
serial_number: rc.serial_number,
revocation_date: rc.revocation_date,
});
}
validate_time_encoding_rfc5280("thisUpdate", &self.this_update)?;
let next_update = self.next_update.ok_or(CrlProfileError::NextUpdateMissing)?;
validate_time_encoding_rfc5280("nextUpdate", &next_update)?;
Ok(RpkixCrl {
raw_der: self.raw_der,
version,
issuer_dn: self.issuer_dn,
signature_algorithm_oid: OID_SHA256_WITH_RSA_ENCRYPTION.to_string(),
this_update: self.this_update,
next_update,
revoked_certs: revoked_out,
extensions,
})
}
}
#[derive(Debug, thiserror::Error)]
pub enum CrlVerifyError {
#[error("issuer certificate parse error: {0} (RFC 5280 §4.1; RFC 6487 §4)")]
IssuerCertificateParse(String),
#[error("trailing bytes after issuer certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
IssuerCertificateTrailingBytes(usize),
#[error("issuer SubjectPublicKeyInfo parse error: {0} (RFC 5280 §4.1.2.7)")]
IssuerSpkiParse(String),
#[error(
"trailing bytes after issuer SubjectPublicKeyInfo DER: {0} bytes (DER; RFC 5280 §4.1.2.7)"
)]
IssuerSpkiTrailingBytes(usize),
#[error("CRL parse error: {0} (RFC 5280 §5.1; RFC 6487 §5)")]
CrlParse(String),
#[error("trailing bytes after CRL DER: {0} bytes (DER; RFC 5280 §5.1)")]
CrlTrailingBytes(usize),
#[error(
"CRL issuer DN does not match issuer certificate subject (RFC 5280 §5.1; RFC 5280 §6.3.3(b))"
)]
IssuerSubjectMismatch {
crl_issuer_dn: String,
issuer_subject_dn: String,
},
#[error(
"issuer certificate keyUsage present but missing cRLSign (RFC 5280 §4.2.1.3; RFC 5280 §6.3.3(f))"
)]
IssuerKeyUsageMissingCrlSign,
#[error(
"CRL AKI.keyIdentifier does not match issuer certificate SKI (RFC 5280 §4.2.1.1; RFC 5280 §4.2.1.2; RFC 5280 §6.3.3(c)/(f))"
)]
AkiSkiMismatch,
#[error("CRL signature verification failed: {0} (RFC 5280 §6.3.3(g); RFC 7935 §2)")]
InvalidSignature(String),
}
fn validate_time_encoding_rfc5280(
field: &'static str,
t: &Asn1TimeUtc,
) -> Result<(), CrlProfileError> {
let year = t.utc.year();
let expected = if year <= 2049 {
Asn1TimeEncoding::UtcTime
} else {
Asn1TimeEncoding::GeneralizedTime
};
if t.encoding != expected {
return Err(CrlProfileError::InvalidTimeEncoding {
field,
year,
encoding: t.encoding,
});
}
Ok(())
}
fn algorithm_identifier_value(ai: &AlgorithmIdentifier<'_>) -> AlgorithmIdentifierValue {
let parameters = ai.parameters.as_ref().map(|p| AlgorithmParametersValue {
class: p.class(),
tag: p.tag(),
data: p.as_bytes().to_vec(),
});
// NOTE(perf): Avoid `to_id_string()` allocations for the signature algorithms we expect
// in RPKI CRLs. Fall back to `to_id_string()` for unexpected algorithms (mostly error paths).
let oid = if ai.algorithm.as_bytes() == OID_SHA256_WITH_RSA_ENCRYPTION_RAW {
OID_SHA256_WITH_RSA_ENCRYPTION.to_string()
} else {
ai.algorithm.to_id_string()
};
AlgorithmIdentifierValue { oid, parameters }
}
fn parse_extensions_parse(exts: &[X509Extension<'_>]) -> Result<Vec<CrlExtensionParsed>, String> {
let mut out = Vec::with_capacity(exts.len());
for ext in exts {
let oid = ext.oid.as_bytes();
if oid == OID_AUTHORITY_KEY_IDENTIFIER_RAW {
let ParsedExtension::AuthorityKeyIdentifier(aki) = ext.parsed_extension() else {
return Err("AKI extension parse failed".to_string());
};
out.push(CrlExtensionParsed::AuthorityKeyIdentifier {
key_identifier: aki.key_identifier.as_ref().map(|k| k.0.to_vec()),
has_other_fields: aki.authority_cert_issuer.is_some()
|| aki.authority_cert_serial.is_some(),
critical: ext.critical,
});
} else if oid == OID_CRL_NUMBER_RAW {
match ext.parsed_extension() {
ParsedExtension::CRLNumber(n) => out.push(CrlExtensionParsed::CrlNumber {
number: n.clone(),
critical: ext.critical,
}),
_ => return Err("CRLNumber extension parse failed".to_string()),
}
} else {
out.push(CrlExtensionParsed::Other {
oid: ext.oid.to_id_string(),
critical: ext.critical,
})
}
}
Ok(out)
}
fn validate_extensions_profile(
exts: &[CrlExtensionParsed],
) -> Result<CrlExtensions, CrlProfileError> {
if exts.len() != 2 {
return Err(CrlProfileError::InvalidExtensionsCount(exts.len()));
}
let mut seen: Vec<String> = Vec::new();
let mut authority_key_identifier: Option<Vec<u8>> = None;
let mut crl_number: Option<BigUnsigned> = None;
for ext in exts {
match ext {
CrlExtensionParsed::AuthorityKeyIdentifier {
key_identifier,
has_other_fields,
critical: _,
} => {
let oid = OID_AUTHORITY_KEY_IDENTIFIER.to_string();
if seen.iter().any(|s| s == &oid) {
return Err(CrlProfileError::DuplicateExtension(oid));
}
seen.push(oid.clone());
if *has_other_fields {
return Err(CrlProfileError::AkiHasOtherFields);
}
let ki = key_identifier
.as_ref()
.ok_or(CrlProfileError::AkiMissingKeyIdentifier)?;
authority_key_identifier = Some(ki.clone());
}
CrlExtensionParsed::CrlNumber { number, critical } => {
let oid = OID_CRL_NUMBER.to_string();
if seen.iter().any(|s| s == &oid) {
return Err(CrlProfileError::DuplicateExtension(oid));
}
seen.push(oid.clone());
if *critical {
return Err(CrlProfileError::CrlNumberCritical);
}
if number.bits() > 159 {
return Err(CrlProfileError::CrlNumberOutOfRange);
}
crl_number = Some(BigUnsigned::from_biguint(number));
}
CrlExtensionParsed::Other { oid, .. } => {
return Err(CrlProfileError::UnsupportedExtension(oid.clone()));
}
}
}
Ok(CrlExtensions {
authority_key_identifier: authority_key_identifier.ok_or(CrlProfileError::AkiMissing)?,
crl_number: crl_number.ok_or(CrlProfileError::CrlNumberMissing)?,
})
}
fn get_subject_key_identifier(cert: &X509Certificate<'_>) -> Option<Vec<u8>> {
cert.extensions()
.iter()
.find(|ext| ext.oid.as_bytes() == OID_SUBJECT_KEY_IDENTIFIER_RAW)
.and_then(|ext| match ext.parsed_extension() {
ParsedExtension::SubjectKeyIdentifier(ki) => Some(ki.0.to_vec()),
_ => None,
})
}

View File

@ -0,0 +1,975 @@
use crate::data_model::common::der_take_tlv;
use crate::data_model::common::{BigUnsigned, UtcTime};
use crate::data_model::oid::{OID_CT_RPKI_MANIFEST, OID_SHA256};
use crate::data_model::rc::ResourceCertificate;
use crate::data_model::signed_object::{
RpkiSignedObject, RpkiSignedObjectParsed, SignedObjectDecodeError, SignedObjectParseError,
SignedObjectValidateError,
};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ManifestObject {
pub signed_object: RpkiSignedObject,
pub econtent_type: String,
pub manifest: ManifestEContent,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ManifestObjectParsed {
pub signed_object: RpkiSignedObjectParsed,
pub econtent_type: String,
pub manifest: Option<ManifestEContentParsed>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ManifestEContent {
pub version: u32,
pub manifest_number: BigUnsigned,
pub this_update: UtcTime,
pub next_update: UtcTime,
pub file_hash_alg: String,
/// DER-encoded content bytes of `Manifest.fileList` (SEQUENCE OF FileAndHash).
pub file_list_der: Vec<u8>,
/// Count of FileAndHash entries in `fileList`.
pub file_count: usize,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ManifestEContentParsed {
der: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FileAndHash {
pub file_name: String,
pub hash_bytes: [u8; 32],
}
#[derive(Debug, thiserror::Error)]
pub enum ManifestParseError {
#[error("signed object parse error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
SignedObject(#[from] SignedObjectParseError),
#[error("DER parse error: {0} (RFC 9286 §4.2; DER)")]
Parse(String),
#[error("trailing bytes after DER object: {0} bytes (RFC 9286 §4.2; DER)")]
TrailingBytes(usize),
}
#[derive(Debug, thiserror::Error)]
pub enum ManifestProfileError {
#[error("signed object profile error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
SignedObject(#[from] SignedObjectValidateError),
#[error(
"eContentType must be id-ct-rpkiManifest ({OID_CT_RPKI_MANIFEST}), got {0} (RFC 9286 §4.1; RFC 9286 §4.4(1))"
)]
InvalidEContentType(String),
#[error("manifest profile decode error: {0} (RFC 9286 §4.2; DER)")]
ProfileDecode(String),
#[error("Manifest must be a SEQUENCE of 5 or 6 elements, got {0} (RFC 9286 §4.2)")]
InvalidManifestSequenceLen(usize),
#[error("Manifest.version must be 0, got {0} (RFC 9286 §4.2.1)")]
InvalidManifestVersion(u64),
#[error(
"Manifest.manifestNumber must be non-negative INTEGER (RFC 9286 §4.2; RFC 9286 §4.2.1)"
)]
InvalidManifestNumber,
#[error("Manifest.manifestNumber longer than 20 octets (RFC 9286 §4.2.1)")]
ManifestNumberTooLong,
#[error("Manifest.thisUpdate must be GeneralizedTime (RFC 9286 §4.2)")]
InvalidThisUpdate,
#[error("Manifest.nextUpdate must be GeneralizedTime (RFC 9286 §4.2)")]
InvalidNextUpdate,
#[error("Manifest.nextUpdate must be later than thisUpdate (RFC 9286 §4.2.1)")]
NextUpdateNotLater,
#[error(
"Manifest.fileHashAlg must be id-sha256 ({OID_SHA256}), got {0} (RFC 9286 §4.2.1; RFC 7935 §2)"
)]
InvalidFileHashAlg(String),
#[error("Manifest.fileList must be a SEQUENCE (RFC 9286 §4.2)")]
InvalidFileList,
#[error("FileAndHash must be SEQUENCE of 2 (RFC 9286 §4.2)")]
InvalidFileAndHash,
#[error("fileList file name invalid: {0} (RFC 9286 §4.2.2)")]
InvalidFileName(String),
#[error("fileList hash must be BIT STRING (RFC 9286 §4.2)")]
InvalidHashType,
#[error(
"fileList hash BIT STRING must be octet-aligned (unused bits=0) (RFC 9286 §4.2.1; DER BIT STRING)"
)]
HashNotOctetAligned,
#[error(
"fileList hash length invalid for sha256: got {0} bytes (RFC 9286 §4.2.1; RFC 7935 §2)"
)]
InvalidHashLength(usize),
}
impl From<SignedObjectDecodeError> for ManifestProfileError {
fn from(value: SignedObjectDecodeError) -> Self {
match value {
SignedObjectDecodeError::Parse(e) => ManifestProfileError::ProfileDecode(e.to_string()),
SignedObjectDecodeError::Validate(e) => ManifestProfileError::SignedObject(e),
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum ManifestDecodeError {
#[error("{0}")]
Parse(#[from] ManifestParseError),
#[error("{0}")]
Validate(#[from] ManifestProfileError),
}
#[derive(Debug, thiserror::Error)]
pub enum ManifestValidateError {
#[error(
"Manifest EE certificate MUST include at least one RFC 3779 resource extension (IP or AS) (RFC 6487 §4.8.10-§4.8.11; RFC 3779; RFC 9286 §5.1)"
)]
EeResourcesMissing,
#[error(
"Manifest EE certificate IP resources MUST use inherit only (RFC 9286 §5.1; RFC 3779 §2.2.3.5)"
)]
EeIpResourcesNotInherit,
#[error(
"Manifest EE certificate AS resources MUST use inherit only (RFC 9286 §5.1; RFC 3779 §3.2.3.3)"
)]
EeAsResourcesNotInherit,
#[error(
"Manifest EE certificate AS resources rdi MUST be absent (RFC 6487 §4.8.11; RFC 3779 §3.2.3.5)"
)]
EeAsResourcesRdiPresent,
}
impl ManifestObject {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<ManifestObjectParsed, ManifestParseError> {
let signed_object = RpkiSignedObject::parse_der(der)?;
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
let manifest = signed_object
.signed_data
.encap_content_info
.econtent
.as_deref()
.map(ManifestEContent::parse_der)
.transpose()?;
Ok(ManifestObjectParsed {
signed_object,
econtent_type,
manifest,
})
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `ManifestObject` is already profile-validated when constructed via `decode_der()` /
/// `ManifestObjectParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), ManifestProfileError> {
Ok(())
}
pub fn decode_der(der: &[u8]) -> Result<Self, ManifestDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn decode_der_with_strict_options(
der: &[u8],
strict_cms_der: bool,
strict_name: bool,
) -> Result<Self, ManifestDecodeError> {
let signed_object =
RpkiSignedObject::decode_der_with_strict_options(der, strict_cms_der, strict_name)
.map_err(ManifestProfileError::from)?;
Self::from_signed_object(signed_object)
}
pub fn from_signed_object(
signed_object: RpkiSignedObject,
) -> Result<Self, ManifestDecodeError> {
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
if econtent_type != OID_CT_RPKI_MANIFEST {
return Err(ManifestProfileError::InvalidEContentType(econtent_type).into());
}
let manifest =
ManifestEContent::decode_der(&signed_object.signed_data.encap_content_info.econtent)?;
Ok(Self {
signed_object,
econtent_type: OID_CT_RPKI_MANIFEST.to_string(),
manifest,
})
}
/// Validate the embedded EE certificate resources against RFC 9286 §5.1.
///
/// This does **not** perform certificate path validation. It assumes `ee` is a parsed and
/// profile-validated RPKI EE resource certificate.
pub fn validate_against_ee_cert(
&self,
ee: &ResourceCertificate,
) -> Result<(), ManifestValidateError> {
let ip = ee.tbs.extensions.ip_resources.as_ref();
let asn = ee.tbs.extensions.as_resources.as_ref();
if ip.is_none() && asn.is_none() {
return Err(ManifestValidateError::EeResourcesMissing);
}
if let Some(ip) = ip {
if !ip.is_all_inherit() {
return Err(ManifestValidateError::EeIpResourcesNotInherit);
}
}
if let Some(asn) = asn {
if asn.rdi.is_some() {
return Err(ManifestValidateError::EeAsResourcesRdiPresent);
}
if !asn.is_asnum_inherit() {
return Err(ManifestValidateError::EeAsResourcesNotInherit);
}
}
Ok(())
}
/// Validate this manifest's embedded EE certificate resources.
pub fn validate_embedded_ee_cert(&self) -> Result<(), ManifestValidateError> {
let ee = &self.signed_object.signed_data.certificates[0].resource_cert;
self.validate_against_ee_cert(ee)
}
}
impl ManifestEContent {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<ManifestEContentParsed, ManifestParseError> {
let (_tag, _value, rem) = der_take_tlv(der).map_err(|e| ManifestParseError::Parse(e))?;
if !rem.is_empty() {
return Err(ManifestParseError::TrailingBytes(rem.len()));
}
Ok(ManifestEContentParsed { der: der.to_vec() })
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `ManifestEContent` is already profile-validated when constructed via `decode_der()` /
/// `ManifestEContentParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), ManifestProfileError> {
Ok(())
}
/// Decode the DER-encoded Manifest eContent defined in RFC 9286 §4.2 (`parse + validate`).
pub fn decode_der(der: &[u8]) -> Result<Self, ManifestDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
/// Parse and return the manifest fileList.
///
/// Note: `ManifestEContent` is profile-validated when produced via `decode_der()`, so this
/// should only fail due to internal inconsistencies (or if constructed manually).
pub fn parse_files(&self) -> Result<Vec<FileAndHash>, ManifestProfileError> {
parse_file_list_sha256_fast(&self.file_list_der)
}
pub fn file_count(&self) -> usize {
self.file_count
}
}
impl ManifestObjectParsed {
pub fn validate_profile(self) -> Result<ManifestObject, ManifestProfileError> {
let signed_object = self.signed_object.validate_profile()?;
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
if econtent_type != OID_CT_RPKI_MANIFEST {
return Err(ManifestProfileError::InvalidEContentType(econtent_type));
}
let manifest = self
.manifest
.ok_or_else(|| ManifestProfileError::ProfileDecode("Manifest.eContent missing".into()))?
.validate_profile()?;
Ok(ManifestObject {
signed_object,
econtent_type: OID_CT_RPKI_MANIFEST.to_string(),
manifest,
})
}
}
impl ManifestEContentParsed {
pub fn validate_profile(self) -> Result<ManifestEContent, ManifestProfileError> {
decode_manifest_econtent_fast(&self.der)
}
}
fn validate_file_name_bytes(bytes: &[u8]) -> Result<(), ManifestProfileError> {
// RFC 9286 §4.2.2:
// 1+ chars from a-zA-Z0-9-_ , then '.', then 3-letter extension.
if bytes.len() < 5 {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
};
// "followed by a single . (DOT), followed by a three letter extension"
// -> the dot must be exactly 4 bytes from the end.
let dot_pos = bytes.len() - 4;
if bytes[dot_pos] != b'.' {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
}
#[inline(always)]
fn valid_base_char(b: u8) -> bool {
// RFC 9286 allowed set: a-zA-Z0-9-_
(b'0'..=b'9').contains(&b)
|| (b'a'..=b'z').contains(&b)
|| (b'A'..=b'Z').contains(&b)
|| b == b'-'
|| b == b'_'
}
for &b in &bytes[..dot_pos] {
if (b & 0x80) != 0 || !valid_base_char(b) {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
}
}
let e0 = bytes[dot_pos + 1];
let e1 = bytes[dot_pos + 2];
let e2 = bytes[dot_pos + 3];
if (e0 & 0x80) != 0 || (e1 & 0x80) != 0 || (e2 & 0x80) != 0 {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
}
#[inline(always)]
fn lower_if_alpha(b: u8) -> Option<u8> {
match b {
b'a'..=b'z' => Some(b),
b'A'..=b'Z' => Some(b + 32),
_ => None,
}
}
let Some(l0) = lower_if_alpha(e0) else {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
};
let Some(l1) = lower_if_alpha(e1) else {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
};
let Some(l2) = lower_if_alpha(e2) else {
return Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
));
};
match [l0, l1, l2] {
// Full IANA list (see `common.rs`).
[b'a', b's', b'a']
| [b'c', b'e', b'r']
| [b'c', b'r', b'l']
| [b'g', b'b', b'r']
| [b'm', b'f', b't']
| [b'r', b'o', b'a']
| [b's', b'i', b'g']
| [b't', b'a', b'k'] => Ok(()),
_ => Err(ManifestProfileError::InvalidFileName(
String::from_utf8_lossy(bytes).into_owned(),
)),
}
}
fn decode_manifest_econtent_fast(der: &[u8]) -> Result<ManifestEContent, ManifestProfileError> {
let (tag, mut seq_content, rem) = der_take_tlv(der)
.map_err(|e| ManifestProfileError::ProfileDecode(format!("DER decode error: {e}")))?;
if !rem.is_empty() {
return Err(ManifestProfileError::ProfileDecode(format!(
"trailing bytes after DER object: {} bytes",
rem.len()
)));
}
if tag != 0x30 {
return Err(ManifestProfileError::ProfileDecode(
"Manifest eContent must be SEQUENCE".into(),
));
}
let seq_len =
der_count_elements(seq_content).map_err(|e| ManifestProfileError::ProfileDecode(e))?;
if seq_len != 5 && seq_len != 6 {
return Err(ManifestProfileError::InvalidManifestSequenceLen(seq_len));
}
let mut version: u32 = 0;
if seq_len == 6 {
let Some(&first_tag) = seq_content.first() else {
return Err(ManifestProfileError::InvalidManifestSequenceLen(0));
};
if first_tag != 0xA0 {
return Err(ManifestProfileError::ProfileDecode(
"Manifest.version must be [0] EXPLICIT INTEGER".into(),
));
}
let (_cs_tag, cs_value, after) = der_take_tlv(seq_content).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.version decode error: {e}"))
})?;
seq_content = after;
let (inner_tag, inner_value, inner_rem) = der_take_tlv(cs_value).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.version inner decode error: {e}"))
})?;
if !inner_rem.is_empty() {
return Err(ManifestProfileError::ProfileDecode(
"trailing bytes inside Manifest.version".into(),
));
}
if inner_tag != 0x02 {
return Err(ManifestProfileError::ProfileDecode(
"Manifest.version must be [0] EXPLICIT INTEGER".into(),
));
}
let v = der_integer_to_u64(inner_value).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.version decode error: {e}"))
})?;
if v != 0 {
return Err(ManifestProfileError::InvalidManifestVersion(v));
}
version = 0;
}
let (mn_tag, mn_value, after) = der_take_tlv(seq_content).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.manifestNumber decode error: {e}"))
})?;
seq_content = after;
if mn_tag != 0x02 {
return Err(ManifestProfileError::InvalidManifestNumber);
}
let manifest_number = der_integer_to_bigunsigned(mn_value)?;
let (tu_tag, tu_value, after) = der_take_tlv(seq_content).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.thisUpdate decode error: {e}"))
})?;
seq_content = after;
if tu_tag != 0x18 {
return Err(ManifestProfileError::InvalidThisUpdate);
}
let this_update = parse_generalized_time_bytes(tu_value)
.map_err(|e| ManifestProfileError::ProfileDecode(e))?;
let (nu_tag, nu_value, after) = der_take_tlv(seq_content).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.nextUpdate decode error: {e}"))
})?;
seq_content = after;
if nu_tag != 0x18 {
return Err(ManifestProfileError::InvalidNextUpdate);
}
let next_update = parse_generalized_time_bytes(nu_value)
.map_err(|e| ManifestProfileError::ProfileDecode(e))?;
if next_update <= this_update {
return Err(ManifestProfileError::NextUpdateNotLater);
}
let (oid_tag, oid_value, after) = der_take_tlv(seq_content).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.fileHashAlg decode error: {e}"))
})?;
seq_content = after;
if oid_tag != 0x06 {
return Err(ManifestProfileError::ProfileDecode(
"Manifest.fileHashAlg must be OBJECT IDENTIFIER".into(),
));
}
if !oid_content_is_sha256(oid_value) {
return Err(ManifestProfileError::InvalidFileHashAlg(
oid_content_to_string(oid_value),
));
}
let (fl_tag, fl_value, after) = der_take_tlv(seq_content).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("Manifest.fileList decode error: {e}"))
})?;
seq_content = after;
if fl_tag != 0x30 {
return Err(ManifestProfileError::InvalidFileList);
}
let file_count = validate_file_list_sha256_fast(fl_value)?;
let file_list_der = fl_value.to_vec();
if !seq_content.is_empty() {
return Err(ManifestProfileError::InvalidManifestSequenceLen(seq_len));
}
Ok(ManifestEContent {
version,
manifest_number,
this_update,
next_update,
file_hash_alg: OID_SHA256.to_string(),
file_list_der,
file_count,
})
}
fn validate_file_list_sha256_fast(content: &[u8]) -> Result<usize, ManifestProfileError> {
let mut cur = content;
let mut count: usize = 0;
while !cur.is_empty() {
let (tag, value, rem) = der_take_tlv(cur).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("fileList entry decode error: {e}"))
})?;
cur = rem;
if tag != 0x30 {
return Err(ManifestProfileError::InvalidFileAndHash);
}
let mut entry = value;
let (fn_tag, fn_value, entry_rem) = der_take_tlv(entry).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("fileList fileName decode error: {e}"))
})?;
entry = entry_rem;
if fn_tag != 0x16 {
return Err(ManifestProfileError::InvalidFileAndHash);
}
if entry.is_empty() {
return Err(ManifestProfileError::InvalidFileAndHash);
}
validate_file_name_bytes(fn_value)?;
let (hash_tag, hash_value, entry_rem) = der_take_tlv(entry).map_err(|_e| {
// Missing second element should map to "SEQUENCE of 2" shape error.
ManifestProfileError::InvalidFileAndHash
})?;
entry = entry_rem;
if !entry.is_empty() {
return Err(ManifestProfileError::InvalidFileAndHash);
}
if hash_tag != 0x03 {
return Err(ManifestProfileError::InvalidHashType);
}
if hash_value.is_empty() {
return Err(ManifestProfileError::InvalidHashLength(0));
}
let unused_bits = hash_value[0];
if unused_bits != 0 {
return Err(ManifestProfileError::HashNotOctetAligned);
}
let bits = &hash_value[1..];
if bits.len() != 32 {
return Err(ManifestProfileError::InvalidHashLength(bits.len()));
}
count += 1;
}
Ok(count)
}
fn parse_file_list_sha256_fast(content: &[u8]) -> Result<Vec<FileAndHash>, ManifestProfileError> {
// Heuristic initial capacity (avoid a full pre-scan, which is expensive for xlarge manifests).
// Each FileAndHash entry is typically tens of bytes; 80 is a conservative average.
let est = (content.len() / 80).clamp(16, 4096);
let mut cur = content;
let mut out: Vec<FileAndHash> = Vec::with_capacity(est);
while !cur.is_empty() {
let (tag, value, rem) = der_take_tlv(cur).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("fileList entry decode error: {e}"))
})?;
cur = rem;
if tag != 0x30 {
return Err(ManifestProfileError::InvalidFileAndHash);
}
let mut entry = value;
let (fn_tag, fn_value, entry_rem) = der_take_tlv(entry).map_err(|e| {
ManifestProfileError::ProfileDecode(format!("fileList fileName decode error: {e}"))
})?;
entry = entry_rem;
if fn_tag != 0x16 {
return Err(ManifestProfileError::InvalidFileAndHash);
}
if entry.is_empty() {
return Err(ManifestProfileError::InvalidFileAndHash);
}
let file_name = validate_and_copy_file_name(fn_value)?;
let (hash_tag, hash_value, entry_rem) = der_take_tlv(entry).map_err(|_e| {
// Missing second element should map to "SEQUENCE of 2" shape error.
ManifestProfileError::InvalidFileAndHash
})?;
entry = entry_rem;
if !entry.is_empty() {
return Err(ManifestProfileError::InvalidFileAndHash);
}
if hash_tag != 0x03 {
return Err(ManifestProfileError::InvalidHashType);
}
if hash_value.is_empty() {
return Err(ManifestProfileError::InvalidHashLength(0));
}
let unused_bits = hash_value[0];
if unused_bits != 0 {
return Err(ManifestProfileError::HashNotOctetAligned);
}
let bits = &hash_value[1..];
if bits.len() != 32 {
return Err(ManifestProfileError::InvalidHashLength(bits.len()));
}
let mut hash_bytes = [0u8; 32];
hash_bytes.copy_from_slice(bits);
out.push(FileAndHash {
file_name,
hash_bytes,
});
}
Ok(out)
}
fn validate_and_copy_file_name(bytes: &[u8]) -> Result<String, ManifestProfileError> {
validate_file_name_bytes(bytes)?;
Ok(unsafe { String::from_utf8_unchecked(bytes.to_vec()) })
}
fn der_count_elements(mut input: &[u8]) -> Result<usize, String> {
let mut count: usize = 0;
while !input.is_empty() {
let (_tag, _value, rem) = der_take_tlv(input)?;
input = rem;
count += 1;
}
Ok(count)
}
fn der_integer_to_u64(bytes: &[u8]) -> Result<u64, String> {
if bytes.is_empty() {
return Err("INTEGER empty".into());
}
// Reject negative (two's complement).
if bytes[0] & 0x80 != 0 {
return Err("INTEGER is negative".into());
}
if bytes.len() > 8 {
return Err("INTEGER too large".into());
}
let mut v: u64 = 0;
for &b in bytes {
v = (v << 8) | (b as u64);
}
Ok(v)
}
fn der_integer_to_bigunsigned(bytes: &[u8]) -> Result<BigUnsigned, ManifestProfileError> {
if bytes.is_empty() {
return Err(ManifestProfileError::InvalidManifestNumber);
}
// Two's complement: for non-negative values, a leading 0x00 may be present.
if bytes[0] & 0x80 != 0 {
return Err(ManifestProfileError::InvalidManifestNumber);
}
let mut start = 0usize;
while start + 1 < bytes.len() && bytes[start] == 0 {
start += 1;
}
let mut minimal = bytes[start..].to_vec();
if minimal.is_empty() {
minimal.push(0);
}
if minimal.len() > 20 {
return Err(ManifestProfileError::ManifestNumberTooLong);
}
Ok(BigUnsigned { bytes_be: minimal })
}
fn parse_generalized_time_bytes(bytes: &[u8]) -> Result<UtcTime, String> {
// Accept "YYYYMMDDHHMMSSZ" and also allow optional fractional seconds (".fff...Z").
if !bytes.is_ascii() {
return Err("GeneralizedTime not ASCII".into());
}
let s = std::str::from_utf8(bytes).map_err(|e| e.to_string())?;
if !s.ends_with('Z') {
return Err("GeneralizedTime must end with 'Z'".into());
}
let core = &s[..s.len() - 1];
let (main, frac) = core
.split_once('.')
.map_or((core, None), |(a, b)| (a, Some(b)));
if main.len() != 14 || !main.bytes().all(|b| b.is_ascii_digit()) {
return Err("GeneralizedTime must be YYYYMMDDHHMMSS[.fff]Z".into());
}
let year: i32 = main[0..4].parse().map_err(|_| "bad year")?;
let month: u8 = main[4..6].parse().map_err(|_| "bad month")?;
let day: u8 = main[6..8].parse().map_err(|_| "bad day")?;
let hour: u8 = main[8..10].parse().map_err(|_| "bad hour")?;
let minute: u8 = main[10..12].parse().map_err(|_| "bad minute")?;
let second: u8 = main[12..14].parse().map_err(|_| "bad second")?;
let nanosecond: u32 = if let Some(frac) = frac {
if frac.is_empty() || !frac.bytes().all(|b| b.is_ascii_digit()) {
return Err("bad fractional seconds".into());
}
let mut ns: u32 = 0;
let mut scale: u32 = 1_000_000_000;
for (i, ch) in frac.bytes().enumerate() {
if i >= 9 {
break;
}
scale /= 10;
ns += ((ch - b'0') as u32) * scale;
}
ns
} else {
0
};
let date = time::Date::from_calendar_date(
year,
time::Month::try_from(month).map_err(|_| "bad month")?,
day,
)
.map_err(|e| e.to_string())?;
let t =
time::Time::from_hms_nano(hour, minute, second, nanosecond).map_err(|e| e.to_string())?;
Ok(date.with_time(t).assume_utc())
}
fn oid_content_is_sha256(bytes: &[u8]) -> bool {
// 2.16.840.1.101.3.4.2.1
let mut arcs = oid_content_iter(bytes);
const EXPECTED: &[u64] = &[2, 16, 840, 1, 101, 3, 4, 2, 1];
for &e in EXPECTED {
match arcs.next() {
Some(v) if v == e => {}
_ => return false,
}
}
arcs.next().is_none()
}
fn oid_content_to_string(bytes: &[u8]) -> String {
let arcs: Vec<u64> = oid_content_iter(bytes).collect();
if arcs.is_empty() {
return "<invalid oid>".to_string();
}
let mut s = String::new();
for (i, a) in arcs.iter().enumerate() {
if i > 0 {
s.push('.');
}
s.push_str(&a.to_string());
}
s
}
fn oid_content_iter(bytes: &[u8]) -> impl Iterator<Item = u64> + '_ {
struct It<'a> {
bytes: &'a [u8],
pos: usize,
first_done: bool,
first_a0: u64,
first_a1: u64,
emit_first_idx: u8,
}
impl<'a> Iterator for It<'a> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
if !self.first_done {
if self.bytes.is_empty() {
self.first_done = true;
return None;
}
let first = self.bytes[0] as u64;
self.first_a0 = first / 40;
self.first_a1 = first % 40;
self.pos = 1;
self.first_done = true;
self.emit_first_idx = 0;
}
if self.emit_first_idx == 0 {
self.emit_first_idx = 1;
return Some(self.first_a0);
}
if self.emit_first_idx == 1 {
self.emit_first_idx = 2;
return Some(self.first_a1);
}
if self.pos >= self.bytes.len() {
return None;
}
let mut v: u64 = 0;
while self.pos < self.bytes.len() {
let b = self.bytes[self.pos];
self.pos += 1;
v = (v << 7) | ((b & 0x7F) as u64);
if b & 0x80 == 0 {
return Some(v);
}
}
None
}
}
It {
bytes,
pos: 0,
first_done: false,
first_a0: 0,
first_a1: 0,
emit_first_idx: 0,
}
}
#[cfg(test)]
mod tests {
use super::*;
fn tlv(tag: u8, value: &[u8]) -> Vec<u8> {
assert!(value.len() < 128);
let mut out = Vec::with_capacity(2 + value.len());
out.push(tag);
out.push(value.len() as u8);
out.extend_from_slice(value);
out
}
fn tlv_long_len(tag: u8, len_bytes: &[u8], value: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(2 + len_bytes.len() + value.len());
out.push(tag);
out.push(0x80 | (len_bytes.len() as u8));
out.extend_from_slice(len_bytes);
out.extend_from_slice(value);
out
}
#[test]
fn der_take_tlv_supports_short_and_long_form_lengths_and_errors() {
let v = b"abc";
let der = tlv(0x04, v);
let (tag, val, rem) = der_take_tlv(&der).expect("short len");
assert_eq!(tag, 0x04);
assert_eq!(val, v);
assert!(rem.is_empty());
// Long-form length with 1 length byte (130).
let v = vec![b'x'; 130];
let der = tlv_long_len(0x04, &[0x82], &v);
let (tag, val, rem) = der_take_tlv(&der).expect("long len 1");
assert_eq!(tag, 0x04);
assert_eq!(val.len(), 130);
assert!(rem.is_empty());
// Long-form length with 2 length bytes (256).
let v = vec![b'y'; 256];
let der = tlv_long_len(0x04, &[0x01, 0x00], &v);
let (tag, val, rem) = der_take_tlv(&der).expect("long len 2");
assert_eq!(tag, 0x04);
assert_eq!(val.len(), 256);
assert!(rem.is_empty());
assert!(der_take_tlv(&[]).is_err());
assert!(der_take_tlv(&[0x04]).is_err());
// High-tag-number form not supported.
assert!(der_take_tlv(&[0x1F, 0x01, 0x00]).is_err());
// Indefinite length is not allowed in DER.
assert!(der_take_tlv(&[0x04, 0x80]).is_err());
// Invalid long-form length encoding.
assert!(der_take_tlv(&[0x04, 0x81]).is_err());
assert!(der_take_tlv(&[0x04, 0x89]).is_err());
}
#[test]
fn parse_generalized_time_bytes_accepts_fraction_and_rejects_invalid() {
let t = parse_generalized_time_bytes(b"20260101000000Z").expect("basic time");
assert_eq!(t.year(), 2026);
let t = parse_generalized_time_bytes(b"20260101000000.1Z").expect("fractional");
assert_eq!(t.nanosecond(), 100_000_000);
assert!(parse_generalized_time_bytes(b"20260101000000").is_err());
assert!(parse_generalized_time_bytes(b"20260101000000+00").is_err());
assert!(parse_generalized_time_bytes(b"2026010100000Z").is_err());
assert!(parse_generalized_time_bytes(b"20261301000000Z").is_err());
assert!(parse_generalized_time_bytes(b"20260132000000Z").is_err());
assert!(parse_generalized_time_bytes(&[0xFF]).is_err());
}
#[test]
fn oid_helpers_accept_sha256_and_format_invalid() {
// 2.16.840.1.101.3.4.2.1
let sha256_oid_content = [0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01];
assert!(oid_content_is_sha256(&sha256_oid_content));
assert!(!oid_content_is_sha256(&[0x55, 0x04, 0x03])); // 2.5.4.3
assert_eq!(oid_content_to_string(&[]), "<invalid oid>".to_string());
}
#[test]
fn validate_file_list_sha256_fast_counts_and_rejects_bad_hash() {
fn file_and_hash(file: &str, digest: u8) -> Vec<u8> {
let mut hash = vec![0u8; 33];
hash[0] = 0; // unused bits
for b in &mut hash[1..] {
*b = digest;
}
let ia5 = tlv(0x16, file.as_bytes());
let bit = tlv(0x03, &hash);
let mut entry = Vec::new();
entry.extend_from_slice(&ia5);
entry.extend_from_slice(&bit);
tlv(0x30, &entry)
}
let mut list = Vec::new();
list.extend_from_slice(&file_and_hash("A.cer", 0xAA));
list.extend_from_slice(&file_and_hash("B.roa", 0xBB));
assert_eq!(validate_file_list_sha256_fast(&list).expect("count"), 2);
// Wrong hash length.
let mut bad = Vec::new();
let ia5 = tlv(0x16, b"A.cer");
let bit = tlv(0x03, &[0u8; 2]); // too short
let mut entry = Vec::new();
entry.extend_from_slice(&ia5);
entry.extend_from_slice(&bit);
bad.extend_from_slice(&tlv(0x30, &entry));
assert!(matches!(
validate_file_list_sha256_fast(&bad),
Err(ManifestProfileError::InvalidHashLength(_))
));
}
}

View File

@ -0,0 +1,13 @@
pub mod aspa;
pub mod common;
pub mod crl;
pub mod manifest;
pub mod oid;
pub mod rc;
pub mod roa;
pub mod signed_object;
pub mod ta;
pub mod tal;
#[cfg(feature = "full")]
pub mod router_cert;

View File

@ -0,0 +1,78 @@
pub const OID_SHA256: &str = "2.16.840.1.101.3.4.2.1";
pub const OID_SHA256_RAW: &[u8] = &asn1_rs::oid!(raw 2.16.840.1.101.3.4.2.1);
pub const OID_SIGNED_DATA: &str = "1.2.840.113549.1.7.2";
pub const OID_SIGNED_DATA_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.7.2);
pub const OID_CMS_ATTR_CONTENT_TYPE: &str = "1.2.840.113549.1.9.3";
pub const OID_CMS_ATTR_CONTENT_TYPE_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.3);
pub const OID_CMS_ATTR_MESSAGE_DIGEST: &str = "1.2.840.113549.1.9.4";
pub const OID_CMS_ATTR_MESSAGE_DIGEST_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.4);
pub const OID_CMS_ATTR_SIGNING_TIME: &str = "1.2.840.113549.1.9.5";
pub const OID_CMS_ATTR_SIGNING_TIME_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.5);
pub const OID_RSA_ENCRYPTION: &str = "1.2.840.113549.1.1.1";
pub const OID_RSA_ENCRYPTION_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.1.1);
pub const OID_SHA256_WITH_RSA_ENCRYPTION: &str = "1.2.840.113549.1.1.11";
pub const OID_SHA256_WITH_RSA_ENCRYPTION_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.1.11);
// X.509 extensions (RFC 5280 / RFC 6487)
pub const OID_BASIC_CONSTRAINTS: &str = "2.5.29.19";
pub const OID_BASIC_CONSTRAINTS_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.19);
pub const OID_KEY_USAGE: &str = "2.5.29.15";
pub const OID_KEY_USAGE_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.15);
pub const OID_EXTENDED_KEY_USAGE: &str = "2.5.29.37";
pub const OID_EXTENDED_KEY_USAGE_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.37);
pub const OID_CRL_DISTRIBUTION_POINTS: &str = "2.5.29.31";
pub const OID_CRL_DISTRIBUTION_POINTS_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.31);
pub const OID_AUTHORITY_INFO_ACCESS: &str = "1.3.6.1.5.5.7.1.1";
pub const OID_AUTHORITY_INFO_ACCESS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.1);
pub const OID_CERTIFICATE_POLICIES: &str = "2.5.29.32";
pub const OID_CERTIFICATE_POLICIES_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.32);
pub const OID_QT_CPS: &str = "1.3.6.1.5.5.7.2.1";
pub const OID_AUTHORITY_KEY_IDENTIFIER: &str = "2.5.29.35";
pub const OID_AUTHORITY_KEY_IDENTIFIER_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.35);
pub const OID_CRL_NUMBER: &str = "2.5.29.20";
pub const OID_CRL_NUMBER_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.20);
pub const OID_SUBJECT_KEY_IDENTIFIER: &str = "2.5.29.14";
pub const OID_SUBJECT_KEY_IDENTIFIER_RAW: &[u8] = &asn1_rs::oid!(raw 2.5.29.14);
pub const OID_CT_RPKI_MANIFEST: &str = "1.2.840.113549.1.9.16.1.26";
pub const OID_CT_RPKI_MANIFEST_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.26);
pub const OID_CT_ROUTE_ORIGIN_AUTHZ: &str = "1.2.840.113549.1.9.16.1.24";
pub const OID_CT_ROUTE_ORIGIN_AUTHZ_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.24);
pub const OID_CT_ASPA: &str = "1.2.840.113549.1.9.16.1.49";
pub const OID_CT_ASPA_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.49);
// X.509 extensions / access methods (RFC 5280 / RFC 6487)
pub const OID_SUBJECT_INFO_ACCESS: &str = "1.3.6.1.5.5.7.1.11";
pub const OID_SUBJECT_INFO_ACCESS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.11);
pub const OID_AD_SIGNED_OBJECT: &str = "1.3.6.1.5.5.7.48.11";
pub const OID_AD_SIGNED_OBJECT_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.11);
pub const OID_AD_CA_ISSUERS: &str = "1.3.6.1.5.5.7.48.2";
pub const OID_AD_CA_ISSUERS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.2);
pub const OID_AD_CA_REPOSITORY: &str = "1.3.6.1.5.5.7.48.5";
pub const OID_AD_CA_REPOSITORY_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.5);
pub const OID_AD_RPKI_MANIFEST: &str = "1.3.6.1.5.5.7.48.10";
pub const OID_AD_RPKI_MANIFEST_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.10);
pub const OID_AD_RPKI_NOTIFY: &str = "1.3.6.1.5.5.7.48.13";
pub const OID_AD_RPKI_NOTIFY_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.48.13);
// RFC 3779 resource extensions (RFC 6487 profile)
pub const OID_IP_ADDR_BLOCKS: &str = "1.3.6.1.5.5.7.1.7";
pub const OID_IP_ADDR_BLOCKS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.7);
pub const OID_AUTONOMOUS_SYS_IDS: &str = "1.3.6.1.5.5.7.1.8";
pub const OID_AUTONOMOUS_SYS_IDS_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.1.8);
// RPKI CP (RFC 6484 / RFC 6487)
pub const OID_CP_IPADDR_ASNUMBER: &str = "1.3.6.1.5.5.7.14.2";
pub const OID_CP_IPADDR_ASNUMBER_RAW: &[u8] = &asn1_rs::oid!(raw 1.3.6.1.5.5.7.14.2);
pub const OID_CT_RPKI_CCR: &str = "1.2.840.113549.1.9.16.1.54";
pub const OID_CT_RPKI_CCR_RAW: &[u8] = &asn1_rs::oid!(raw 1.2.840.113549.1.9.16.1.54);
pub const OID_KP_BGPSEC_ROUTER: &str = "1.3.6.1.5.5.7.3.30";
pub const OID_EC_PUBLIC_KEY: &str = "1.2.840.10045.2.1";
pub const OID_SECP256R1: &str = "1.2.840.10045.3.1.7";

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,633 @@
use crate::data_model::common::{DerReader, der_take_tlv};
use crate::data_model::oid::OID_CT_ROUTE_ORIGIN_AUTHZ;
use crate::data_model::rc::{Afi as RcAfi, IpPrefix as RcIpPrefix, ResourceCertificate};
use crate::data_model::signed_object::{
RpkiSignedObject, RpkiSignedObjectParsed, SignedObjectDecodeError, SignedObjectParseError,
SignedObjectValidateError,
};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaObject {
pub signed_object: RpkiSignedObject,
pub econtent_type: String,
pub roa: RoaEContent,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaObjectParsed {
pub signed_object: RpkiSignedObjectParsed,
pub econtent_type: String,
pub roa: Option<RoaEContentParsed>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaEContent {
pub version: u32,
pub as_id: u32,
pub ip_addr_blocks: Vec<RoaIpAddressFamily>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaEContentParsed {
der: Vec<u8>,
}
#[derive(Debug, thiserror::Error)]
pub enum RoaParseError {
#[error("signed object parse error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
SignedObject(#[from] SignedObjectParseError),
#[error("ROA parse error: {0} (RFC 9582 §4; DER)")]
Parse(String),
#[error("ROA trailing bytes: {0} bytes (RFC 9582 §4; DER)")]
TrailingBytes(usize),
}
#[derive(Debug, thiserror::Error)]
pub enum RoaProfileError {
#[error("signed object profile error: {0} (RFC 6488 §2-§3; RFC 9589 §4)")]
SignedObject(#[from] SignedObjectValidateError),
#[error("ROA eContentType must be {OID_CT_ROUTE_ORIGIN_AUTHZ}, got {0} (RFC 9582 §3)")]
InvalidEContentType(String),
#[error("ROA profile decode error: {0} (RFC 9582 §4; DER)")]
ProfileDecode(String),
#[error("RouteOriginAttestation must be a SEQUENCE of 2 or 3 elements, got {0} (RFC 9582 §4)")]
InvalidAttestationSequenceLen(usize),
#[error("ROA version must be 0, got {0} (RFC 9582 §4.1)")]
InvalidVersion(u64),
#[error("ROA asID out of range (0..=4294967295), got {0} (RFC 9582 §4.2)")]
AsIdOutOfRange(u64),
#[error("ROA ipAddrBlocks must have length 1..2, got {0} (RFC 9582 §4; RFC 9582 §4.3.1)")]
InvalidIpAddrBlocksLen(usize),
#[error("ROAIPAddressFamily must be a SEQUENCE of 2 elements (RFC 9582 §4.3.1)")]
InvalidIpAddressFamily,
#[error("ROA addressFamily must be an OCTET STRING of 2 bytes (RFC 9582 §4.3.1)")]
InvalidAddressFamily,
#[error("ROA addressFamily AFI not supported: {0:02X?} (RFC 9582 §4.3.1)")]
UnsupportedAfi(Vec<u8>),
#[error("ROA contains duplicate AFI {0:?} (RFC 9582 §4.3.1)")]
DuplicateAfi(RoaAfi),
#[error("ROAAddresses must have at least one entry (RFC 9582 §4.3.2)")]
EmptyAddressList,
#[error("ROAIPAddress must be a SEQUENCE of 1..2 elements (RFC 9582 §4.3.2)")]
InvalidRoaIpAddress,
#[error("ROAIPAddress.address must be a BIT STRING (RFC 9582 §4.3.2.1; RFC 3779 §2.2.3.8)")]
InvalidPrefixBitString,
#[error(
"ROAIPAddress.address has invalid unused bits encoding (RFC 9582 §4.3.2.1; RFC 3779 §2.2.3.8)"
)]
InvalidPrefixUnusedBits,
#[error(
"ROAIPAddress.address prefix length {prefix_len} out of range for {afi:?} (RFC 9582 §4.3.2.1)"
)]
PrefixLenOutOfRange { afi: RoaAfi, prefix_len: u16 },
#[error(
"ROAIPAddress.maxLength out of range for {afi:?}: prefix_len={prefix_len}, max_len={max_len} (RFC 9582 §4.3.2.2)"
)]
InvalidMaxLength {
afi: RoaAfi,
prefix_len: u16,
max_len: u16,
},
}
impl From<SignedObjectDecodeError> for RoaProfileError {
fn from(value: SignedObjectDecodeError) -> Self {
match value {
SignedObjectDecodeError::Parse(e) => RoaProfileError::ProfileDecode(e.to_string()),
SignedObjectDecodeError::Validate(e) => RoaProfileError::SignedObject(e),
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum RoaDecodeError {
#[error("{0}")]
Parse(#[from] RoaParseError),
#[error("{0}")]
Validate(#[from] RoaProfileError),
}
#[derive(Debug, thiserror::Error)]
pub enum RoaValidateError {
#[error("ROA EE certificate must not contain AS resources extension (RFC 9582 §5)")]
EeAsResourcesPresent,
#[error("ROA EE certificate must contain IP resources extension (RFC 9582 §5)")]
EeIpResourcesMissing,
#[error("ROA EE certificate IP resources must not use inherit (RFC 9582 §5)")]
EeIpResourcesInherit,
#[error(
"ROA prefix not covered by EE certificate IP resources: {afi:?} {addr:?}/{prefix_len} (RFC 9582 §5; RFC 3779 §2.3)"
)]
PrefixNotInEeResources {
afi: RoaAfi,
addr: Vec<u8>,
prefix_len: u16,
},
}
impl RoaObject {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<RoaObjectParsed, RoaParseError> {
let signed_object = RpkiSignedObject::parse_der(der)?;
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
let roa = signed_object
.signed_data
.encap_content_info
.econtent
.as_deref()
.map(RoaEContent::parse_der)
.transpose()?;
Ok(RoaObjectParsed {
signed_object,
econtent_type,
roa,
})
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `RoaObject` is already profile-validated when constructed via `decode_der()` /
/// `RoaObjectParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), RoaProfileError> {
Ok(())
}
pub fn decode_der(der: &[u8]) -> Result<Self, RoaDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn decode_der_with_strict_options(
der: &[u8],
strict_cms_der: bool,
strict_name: bool,
) -> Result<Self, RoaDecodeError> {
let signed_object =
RpkiSignedObject::decode_der_with_strict_options(der, strict_cms_der, strict_name)
.map_err(RoaProfileError::from)?;
Self::from_signed_object(signed_object)
}
pub fn from_signed_object(signed_object: RpkiSignedObject) -> Result<Self, RoaDecodeError> {
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
if econtent_type != OID_CT_ROUTE_ORIGIN_AUTHZ {
return Err(RoaProfileError::InvalidEContentType(econtent_type).into());
}
let roa = RoaEContent::decode_der(&signed_object.signed_data.encap_content_info.econtent)?;
Ok(Self {
roa,
signed_object,
econtent_type: OID_CT_ROUTE_ORIGIN_AUTHZ.to_string(),
})
}
/// Validate this ROA's embedded EE certificate resources.
pub fn validate_embedded_ee_cert(&self) -> Result<(), RoaValidateError> {
let ee = &self.signed_object.signed_data.certificates[0].resource_cert;
self.roa.validate_against_ee_cert(ee)
}
}
#[derive(
Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, serde::Serialize, serde::Deserialize,
)]
pub enum RoaAfi {
Ipv4,
Ipv6,
}
impl RoaAfi {
fn ub(self) -> u16 {
match self {
RoaAfi::Ipv4 => 32,
RoaAfi::Ipv6 => 128,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaIpAddressFamily {
pub afi: RoaAfi,
pub addresses: Vec<RoaIpAddress>,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct RoaIpAddress {
pub prefix: IpPrefix,
pub max_length: Option<u16>,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct IpPrefix {
pub afi: RoaAfi,
/// Prefix length in bits.
pub prefix_len: u16,
/// Network order address bytes (always 16 bytes), with host bits cleared.
///
/// For IPv4 prefixes, only the first 4 bytes are used and the remaining 12 bytes are zero.
pub addr: [u8; 16],
}
impl IpPrefix {
pub fn addr_bytes(&self) -> &[u8] {
match self.afi {
RoaAfi::Ipv4 => &self.addr[..4],
RoaAfi::Ipv6 => &self.addr[..16],
}
}
}
impl RoaEContent {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<RoaEContentParsed, RoaParseError> {
let (_tag, _value, rem) = der_take_tlv(der).map_err(RoaParseError::Parse)?;
if !rem.is_empty() {
return Err(RoaParseError::TrailingBytes(rem.len()));
}
Ok(RoaEContentParsed { der: der.to_vec() })
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `RoaEContent` is already profile-validated when constructed via `decode_der()` /
/// `RoaEContentParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), RoaProfileError> {
Ok(())
}
/// Decode the DER-encoded RouteOriginAttestation defined in RFC 9582 §4 (`parse + validate`).
pub fn decode_der(der: &[u8]) -> Result<Self, RoaDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn canonicalize(&mut self) {
self.ip_addr_blocks.sort_by_key(|f| f.afi);
for fam in &mut self.ip_addr_blocks {
fam.addresses.sort();
fam.addresses.dedup();
}
}
/// Validate ROA payload against the embedded EE resource certificate (RFC 9582 §5).
///
/// This performs the EE/payload semantic checks that do not require certificate path
/// validation.
pub fn validate_against_ee_cert(
&self,
ee: &ResourceCertificate,
) -> Result<(), RoaValidateError> {
if ee.tbs.extensions.as_resources.is_some() {
return Err(RoaValidateError::EeAsResourcesPresent);
}
let ip = ee
.tbs
.extensions
.ip_resources
.as_ref()
.ok_or(RoaValidateError::EeIpResourcesMissing)?;
if ip.has_any_inherit() {
return Err(RoaValidateError::EeIpResourcesInherit);
}
for fam in &self.ip_addr_blocks {
for entry in &fam.addresses {
let rc_prefix = roa_prefix_to_rc(&entry.prefix);
if !ip.contains_prefix(&rc_prefix) {
return Err(RoaValidateError::PrefixNotInEeResources {
afi: entry.prefix.afi,
addr: entry.prefix.addr_bytes().to_vec(),
prefix_len: entry.prefix.prefix_len,
});
}
}
}
Ok(())
}
}
impl RoaObjectParsed {
pub fn validate_profile(self) -> Result<RoaObject, RoaProfileError> {
let signed_object = self.signed_object.validate_profile()?;
let econtent_type = signed_object
.signed_data
.encap_content_info
.econtent_type
.clone();
if econtent_type != OID_CT_ROUTE_ORIGIN_AUTHZ {
return Err(RoaProfileError::InvalidEContentType(econtent_type));
}
let roa = self
.roa
.ok_or_else(|| RoaProfileError::ProfileDecode("ROA.eContent missing".into()))?
.validate_profile()?;
Ok(RoaObject {
signed_object,
econtent_type: OID_CT_ROUTE_ORIGIN_AUTHZ.to_string(),
roa,
})
}
}
impl RoaEContentParsed {
pub fn validate_profile(self) -> Result<RoaEContent, RoaProfileError> {
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
let mut n = 0usize;
while !r.is_empty() {
r.skip_any()?;
n += 1;
}
Ok(n)
}
let mut r = DerReader::new(&self.der);
let mut seq = r
.take_sequence()
.map_err(|e| RoaProfileError::ProfileDecode(e))?;
if !r.is_empty() {
return Err(RoaProfileError::ProfileDecode(
"trailing bytes after RouteOriginAttestation".into(),
));
}
let elem_count =
count_elements(seq).map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
if elem_count != 2 && elem_count != 3 {
return Err(RoaProfileError::InvalidAttestationSequenceLen(elem_count));
}
let mut version: u32 = 0;
if elem_count == 3 {
if seq
.peek_tag()
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?
!= 0xA0
{
return Err(RoaProfileError::ProfileDecode(
"RouteOriginAttestation.version must be [0] EXPLICIT INTEGER".into(),
));
}
let (inner_tag, inner_val) = seq
.take_explicit(0xA0)
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
if inner_tag != 0x02 {
return Err(RoaProfileError::ProfileDecode(
"RouteOriginAttestation.version must be [0] EXPLICIT INTEGER".into(),
));
}
let v = crate::data_model::common::der_uint_from_bytes(inner_val)
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
if v != 0 {
return Err(RoaProfileError::InvalidVersion(v));
}
version = 0;
}
let as_id_u64 = seq
.take_uint_u64()
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
if as_id_u64 > u32::MAX as u64 {
return Err(RoaProfileError::AsIdOutOfRange(as_id_u64));
}
let as_id = as_id_u64 as u32;
let ip_addr_blocks = parse_ip_addr_blocks_cursor(
seq.take_sequence()
.map_err(|e| RoaProfileError::ProfileDecode(format!("ipAddrBlocks: {e}")))?,
)?;
if !seq.is_empty() {
// Extra elements beyond the expected 2..3.
let extra =
count_elements(seq).map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
return Err(RoaProfileError::InvalidAttestationSequenceLen(
elem_count + extra,
));
}
let mut out = RoaEContent {
version,
as_id,
ip_addr_blocks,
};
out.canonicalize();
Ok(out)
}
}
fn roa_prefix_to_rc(p: &IpPrefix) -> RcIpPrefix {
let afi = match p.afi {
RoaAfi::Ipv4 => RcAfi::Ipv4,
RoaAfi::Ipv6 => RcAfi::Ipv6,
};
RcIpPrefix {
afi,
prefix_len: p.prefix_len,
addr: p.addr_bytes().to_vec(),
}
}
fn parse_ip_addr_blocks_cursor(
mut seq: DerReader<'_>,
) -> Result<Vec<RoaIpAddressFamily>, RoaProfileError> {
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
let mut n = 0usize;
while !r.is_empty() {
r.skip_any()?;
n += 1;
}
Ok(n)
}
let fam_count =
count_elements(seq).map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
if fam_count == 0 || fam_count > 2 {
return Err(RoaProfileError::InvalidIpAddrBlocksLen(fam_count));
}
let mut out: Vec<RoaIpAddressFamily> = Vec::with_capacity(fam_count);
while !seq.is_empty() {
let family = parse_ip_address_family_cursor(
seq.take_sequence()
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?,
)?;
if out.iter().any(|f| f.afi == family.afi) {
return Err(RoaProfileError::DuplicateAfi(family.afi));
}
out.push(family);
}
Ok(out)
}
fn parse_ip_address_family_cursor(
mut fam: DerReader<'_>,
) -> Result<RoaIpAddressFamily, RoaProfileError> {
let afi = {
let bytes = fam
.take_octet_string()
.map_err(|_e| RoaProfileError::InvalidAddressFamily)?;
if bytes.len() != 2 {
return Err(RoaProfileError::InvalidAddressFamily);
}
match bytes {
[0x00, 0x01] => RoaAfi::Ipv4,
[0x00, 0x02] => RoaAfi::Ipv6,
_ => return Err(RoaProfileError::UnsupportedAfi(bytes.to_vec())),
}
};
let mut addrs = fam
.take_sequence()
.map_err(|_e| RoaProfileError::InvalidIpAddressFamily)?;
if !fam.is_empty() {
return Err(RoaProfileError::InvalidIpAddressFamily);
}
if addrs.is_empty() {
return Err(RoaProfileError::EmptyAddressList);
}
let mut addresses: Vec<RoaIpAddress> = Vec::new();
while !addrs.is_empty() {
let entry = addrs
.take_sequence()
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
addresses.push(parse_roa_ip_address_cursor(afi, entry)?);
}
Ok(RoaIpAddressFamily { afi, addresses })
}
fn parse_roa_ip_address_cursor(
afi: RoaAfi,
mut seq: DerReader<'_>,
) -> Result<RoaIpAddress, RoaProfileError> {
if seq.is_empty() {
return Err(RoaProfileError::InvalidRoaIpAddress);
}
let (unused_bits, bytes) = seq
.take_bit_string()
.map_err(|_e| RoaProfileError::InvalidPrefixBitString)?;
let prefix = parse_prefix_bits_bytes(afi, unused_bits, bytes)?;
let max_length = if !seq.is_empty() {
let v = seq
.take_uint_u64()
.map_err(|e| RoaProfileError::ProfileDecode(e.to_string()))?;
let max_len: u16 = v
.try_into()
.map_err(|_e| RoaProfileError::InvalidMaxLength {
afi,
prefix_len: prefix.prefix_len,
max_len: u16::MAX,
})?;
Some(max_len)
} else {
None
};
if !seq.is_empty() {
return Err(RoaProfileError::InvalidRoaIpAddress);
}
if let Some(max_len) = max_length {
let ub = afi.ub();
if max_len > ub || max_len < prefix.prefix_len {
return Err(RoaProfileError::InvalidMaxLength {
afi,
prefix_len: prefix.prefix_len,
max_len,
});
}
}
Ok(RoaIpAddress { prefix, max_length })
}
fn parse_prefix_bits_bytes(
afi: RoaAfi,
unused_bits: u8,
bytes: &[u8],
) -> Result<IpPrefix, RoaProfileError> {
if unused_bits > 7 {
return Err(RoaProfileError::InvalidPrefixUnusedBits);
}
if bytes.is_empty() {
if unused_bits != 0 {
return Err(RoaProfileError::InvalidPrefixUnusedBits);
}
} else if unused_bits != 0 {
let mask = (1u8 << unused_bits) - 1;
if (bytes[bytes.len() - 1] & mask) != 0 {
return Err(RoaProfileError::InvalidPrefixUnusedBits);
}
}
let prefix_len = (bytes.len() * 8)
.checked_sub(unused_bits as usize)
.ok_or(RoaProfileError::InvalidPrefixUnusedBits)? as u16;
if prefix_len > afi.ub() {
return Err(RoaProfileError::PrefixLenOutOfRange { afi, prefix_len });
}
let addr = canonicalize_prefix_addr(afi, prefix_len, bytes);
Ok(IpPrefix {
afi,
prefix_len,
addr,
})
}
fn canonicalize_prefix_addr(afi: RoaAfi, prefix_len: u16, bytes: &[u8]) -> [u8; 16] {
let full_len = match afi {
RoaAfi::Ipv4 => 4,
RoaAfi::Ipv6 => 16,
};
let mut addr = [0u8; 16];
let copy_len = bytes.len().min(full_len);
addr[..copy_len].copy_from_slice(&bytes[..copy_len]);
if prefix_len == 0 {
return addr;
}
let last_prefix_bit = (prefix_len - 1) as usize;
let last_prefix_byte = last_prefix_bit / 8;
let rem = (prefix_len % 8) as u8;
if rem != 0 {
let mask: u8 = 0xFF << (8 - rem);
if last_prefix_byte < full_len {
addr[last_prefix_byte] &= mask;
}
}
addr
}

View File

@ -0,0 +1,363 @@
use crate::data_model::oid::{
OID_EC_PUBLIC_KEY, OID_EXTENDED_KEY_USAGE_RAW, OID_KP_BGPSEC_ROUTER, OID_SECP256R1,
};
use crate::data_model::rc::{
AsIdOrRange, AsIdentifierChoice, ResourceCertKind, ResourceCertificate,
ResourceCertificateParseError, ResourceCertificateParsed, ResourceCertificateProfileError,
ResourceCertificateRole,
};
use crate::validation::cert_path::{CertPathError, validate_ee_cert_path_with_predecoded_ee};
use x509_parser::extensions::ParsedExtension;
use x509_parser::prelude::{FromDer, X509Certificate};
use x509_parser::public_key::PublicKey;
use x509_parser::x509::SubjectPublicKeyInfo;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BgpsecRouterCertificateParsed {
pub rc_parsed: ResourceCertificateParsed,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BgpsecRouterCertificate {
pub raw_der: Vec<u8>,
pub resource_cert: ResourceCertificate,
pub subject_key_identifier: Vec<u8>,
pub spki_der: Vec<u8>,
pub asns: Vec<u32>,
}
#[derive(Debug, thiserror::Error)]
pub enum BgpsecRouterCertificateParseError {
#[error("resource certificate parse error: {0} (RFC 5280 §4.1; RFC 6487 §4; RFC 8209 §3.1)")]
ResourceCertificate(#[from] ResourceCertificateParseError),
#[error("X.509 parse error: {0} (RFC 5280 §4.1; RFC 8209 §3.1)")]
X509(String),
#[error("trailing bytes after router certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
TrailingBytes(usize),
#[error("router SubjectPublicKeyInfo parse error: {0} (RFC 5280 §4.1.2.7; RFC 8208 §3.1)")]
SpkiParse(String),
#[error("trailing bytes after router SubjectPublicKeyInfo DER: {0} bytes (DER; RFC 8208 §3.1)")]
SpkiTrailingBytes(usize),
}
#[derive(Debug, thiserror::Error)]
pub enum BgpsecRouterCertificateProfileError {
#[error("resource certificate profile error: {0} (RFC 6487 §4; RFC 8209 §3.1)")]
ResourceCertificate(#[from] ResourceCertificateProfileError),
#[error("BGPsec router certificate must be an EE certificate (RFC 8209 §3.1)")]
NotEe,
#[error(
"BGPsec router certificate must contain SubjectKeyIdentifier (RFC 6487 §4.8.2; RFC 8209 §3.3)"
)]
MissingSki,
#[error(
"BGPsec router certificate must include ExtendedKeyUsage (RFC 8209 §3.1.3.2; RFC 8209 §3.3)"
)]
MissingExtendedKeyUsage,
#[error(
"BGPsec router certificate ExtendedKeyUsage must be non-critical (RFC 6487 §4.8.4; RFC 8209 §3.1.3.2)"
)]
ExtendedKeyUsageCriticality,
#[error(
"BGPsec router certificate ExtendedKeyUsage must contain id-kp-bgpsec-router ({OID_KP_BGPSEC_ROUTER}) (RFC 8209 §3.1.3.2; RFC 8209 §3.3)"
)]
MissingBgpsecRouterEku,
#[error(
"BGPsec router certificate MUST NOT include Subject Information Access (RFC 8209 §3.1.3.3; RFC 8209 §3.3)"
)]
SubjectInfoAccessPresent,
#[error(
"BGPsec router certificate MUST NOT include IP resources extension (RFC 8209 §3.1.3.4; RFC 8209 §3.3)"
)]
IpResourcesPresent,
#[error(
"BGPsec router certificate MUST include AS resources extension (RFC 8209 §3.1.3.5; RFC 8209 §3.3)"
)]
AsResourcesMissing,
#[error(
"BGPsec router certificate AS resources MUST include one or more ASNs (RFC 8209 §3.1.3.5)"
)]
AsResourcesAsnumMissing,
#[error("BGPsec router certificate AS resources MUST NOT use inherit (RFC 8209 §3.1.3.5)")]
AsResourcesInherit,
#[error(
"BGPsec router certificate AS resources MUST contain explicit ASNs, not ranges (RFC 8209 §3.1.3.5)"
)]
AsResourcesRangeNotAllowed,
#[error(
"BGPsec router certificate subjectPublicKeyInfo.algorithm must be id-ecPublicKey ({OID_EC_PUBLIC_KEY}) (RFC 8208 §3.1)"
)]
SpkiAlgorithmNotEcPublicKey,
#[error(
"BGPsec router certificate subjectPublicKeyInfo.parameters must be secp256r1 ({OID_SECP256R1}) (RFC 8208 §3.1)"
)]
SpkiWrongCurve,
#[error(
"BGPsec router certificate subjectPublicKeyInfo.parameters missing or invalid (RFC 8208 §3.1)"
)]
SpkiParametersMissingOrInvalid,
#[error(
"BGPsec router certificate subjectPublicKey MUST be uncompressed P-256 ECPoint (RFC 8208 §3.1)"
)]
SpkiEcPointNotUncompressedP256,
}
#[derive(Debug, thiserror::Error)]
pub enum BgpsecRouterCertificateDecodeError {
#[error("{0}")]
Parse(#[from] BgpsecRouterCertificateParseError),
#[error("{0}")]
Validate(#[from] BgpsecRouterCertificateProfileError),
}
#[derive(Debug, thiserror::Error)]
pub enum BgpsecRouterCertificatePathError {
#[error("{0}")]
Decode(#[from] BgpsecRouterCertificateDecodeError),
#[error("{0}")]
CertPath(#[from] CertPathError),
}
impl BgpsecRouterCertificate {
pub fn parse_der(
der: &[u8],
) -> Result<BgpsecRouterCertificateParsed, BgpsecRouterCertificateParseError> {
let (rem, cert) = X509Certificate::from_der(der)
.map_err(|e| BgpsecRouterCertificateParseError::X509(e.to_string()))?;
if !rem.is_empty() {
return Err(BgpsecRouterCertificateParseError::TrailingBytes(rem.len()));
}
let (spki_rem, _spki) =
SubjectPublicKeyInfo::from_der(cert.tbs_certificate.subject_pki.raw)
.map_err(|e| BgpsecRouterCertificateParseError::SpkiParse(e.to_string()))?;
if !spki_rem.is_empty() {
return Err(BgpsecRouterCertificateParseError::SpkiTrailingBytes(
spki_rem.len(),
));
}
let rc_parsed = ResourceCertificate::parse_der(der)?;
Ok(BgpsecRouterCertificateParsed { rc_parsed })
}
pub fn validate_profile(&self) -> Result<(), BgpsecRouterCertificateProfileError> {
Ok(())
}
pub fn decode_der(der: &[u8]) -> Result<Self, BgpsecRouterCertificateDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn from_der(der: &[u8]) -> Result<Self, BgpsecRouterCertificateDecodeError> {
Self::decode_der(der)
}
pub fn validate_path_with_prevalidated_issuer(
der: &[u8],
issuer_ca: &ResourceCertificate,
issuer_spki: &SubjectPublicKeyInfo<'_>,
issuer_crl: &crate::data_model::crl::RpkixCrl,
issuer_crl_revoked_serials: &std::collections::HashSet<Vec<u8>>,
issuer_ca_rsync_uri: Option<&str>,
issuer_crl_rsync_uri: Option<&str>,
validation_time: time::OffsetDateTime,
) -> Result<Self, BgpsecRouterCertificatePathError> {
let cert = Self::decode_der(der)?;
validate_ee_cert_path_with_predecoded_ee(
&cert.resource_cert,
der,
issuer_ca,
issuer_spki,
issuer_crl,
issuer_crl_revoked_serials,
issuer_ca_rsync_uri,
issuer_crl_rsync_uri,
validation_time,
)?;
Ok(cert)
}
}
impl BgpsecRouterCertificateParsed {
pub fn validate_profile(
self,
) -> Result<BgpsecRouterCertificate, BgpsecRouterCertificateProfileError> {
let rc = self.rc_parsed.validate_profile()?;
rc.validate_rfc6487_profile(ResourceCertificateRole::RouterEe)?;
if rc.kind != ResourceCertKind::Ee {
return Err(BgpsecRouterCertificateProfileError::NotEe);
}
let ski = rc
.tbs
.extensions
.subject_key_identifier
.clone()
.ok_or(BgpsecRouterCertificateProfileError::MissingSki)?;
if rc.tbs.extensions.subject_info_access.is_some() {
return Err(BgpsecRouterCertificateProfileError::SubjectInfoAccessPresent);
}
if rc.tbs.extensions.ip_resources.is_some() {
return Err(BgpsecRouterCertificateProfileError::IpResourcesPresent);
}
let as_resources = rc
.tbs
.extensions
.as_resources
.as_ref()
.ok_or(BgpsecRouterCertificateProfileError::AsResourcesMissing)?;
let asns = extract_router_asns(as_resources)?;
let (rem, cert) = X509Certificate::from_der(&rc.raw_der).map_err(|e| {
BgpsecRouterCertificateProfileError::ResourceCertificate(
ResourceCertificateProfileError::InvalidCertificatePolicy(e.to_string()),
)
})?;
if !rem.is_empty() {
return Err(BgpsecRouterCertificateProfileError::ResourceCertificate(
ResourceCertificateProfileError::InvalidCertificatePolicy(format!(
"trailing bytes after router certificate DER: {}",
rem.len()
)),
));
}
validate_router_eku(&cert)?;
validate_router_spki(&rc.tbs.subject_public_key_info)?;
Ok(BgpsecRouterCertificate {
raw_der: rc.raw_der.clone(),
resource_cert: rc.clone(),
subject_key_identifier: ski,
spki_der: rc.tbs.subject_public_key_info.clone(),
asns,
})
}
}
fn extract_router_asns(
as_resources: &crate::data_model::rc::AsResourceSet,
) -> Result<Vec<u32>, BgpsecRouterCertificateProfileError> {
let asnum = as_resources
.asnum
.as_ref()
.ok_or(BgpsecRouterCertificateProfileError::AsResourcesAsnumMissing)?;
if matches!(asnum, AsIdentifierChoice::Inherit)
|| matches!(as_resources.rdi.as_ref(), Some(AsIdentifierChoice::Inherit))
{
return Err(BgpsecRouterCertificateProfileError::AsResourcesInherit);
}
let AsIdentifierChoice::AsIdsOrRanges(items) = asnum else {
return Err(BgpsecRouterCertificateProfileError::AsResourcesInherit);
};
if items.is_empty() {
return Err(BgpsecRouterCertificateProfileError::AsResourcesAsnumMissing);
}
let mut asns = Vec::with_capacity(items.len());
for item in items {
match item {
AsIdOrRange::Id(v) => asns.push(*v),
AsIdOrRange::Range { .. } => {
return Err(BgpsecRouterCertificateProfileError::AsResourcesRangeNotAllowed);
}
}
}
asns.sort_unstable();
asns.dedup();
Ok(asns)
}
fn validate_router_eku(
cert: &X509Certificate<'_>,
) -> Result<(), BgpsecRouterCertificateProfileError> {
let mut matches = cert
.tbs_certificate
.extensions()
.iter()
.filter(|ext| ext.oid.as_bytes() == OID_EXTENDED_KEY_USAGE_RAW);
let Some(ext) = matches.next() else {
return Err(BgpsecRouterCertificateProfileError::MissingExtendedKeyUsage);
};
if matches.next().is_some() {
return Err(BgpsecRouterCertificateProfileError::MissingExtendedKeyUsage);
}
if ext.critical {
return Err(BgpsecRouterCertificateProfileError::ExtendedKeyUsageCriticality);
}
let ParsedExtension::ExtendedKeyUsage(eku) = ext.parsed_extension() else {
return Err(BgpsecRouterCertificateProfileError::MissingExtendedKeyUsage);
};
let found = eku
.other
.iter()
.any(|oid| oid.to_id_string() == OID_KP_BGPSEC_ROUTER);
if !found {
return Err(BgpsecRouterCertificateProfileError::MissingBgpsecRouterEku);
}
Ok(())
}
fn validate_router_spki(spki_der: &[u8]) -> Result<(), BgpsecRouterCertificateProfileError> {
let (rem, spki) = SubjectPublicKeyInfo::from_der(spki_der)
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid)?;
if !rem.is_empty() {
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
}
if spki.algorithm.algorithm.to_id_string() != OID_EC_PUBLIC_KEY {
return Err(BgpsecRouterCertificateProfileError::SpkiAlgorithmNotEcPublicKey);
}
let Some(params) = spki.algorithm.parameters.as_ref() else {
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
};
if params.header.tag().0 != 0x06 {
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
}
let mut der = Vec::with_capacity(params.data.len() + 2);
der.push(0x06);
if params.data.len() >= 0x80 {
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
}
der.push(params.data.len() as u8);
der.extend_from_slice(params.data);
let (prem, oid) = der_parser::der::parse_der_oid(&der)
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid)?;
if !prem.is_empty() {
return Err(BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid);
}
let curve = oid
.as_oid_val()
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiParametersMissingOrInvalid)?
.to_string();
if curve != OID_SECP256R1 {
return Err(BgpsecRouterCertificateProfileError::SpkiWrongCurve);
}
let parsed = spki
.parsed()
.map_err(|_| BgpsecRouterCertificateProfileError::SpkiEcPointNotUncompressedP256)?;
let PublicKey::EC(ec) = parsed else {
return Err(BgpsecRouterCertificateProfileError::SpkiAlgorithmNotEcPublicKey);
};
if ec.data().len() != 65 || ec.data().first() != Some(&0x04) {
return Err(BgpsecRouterCertificateProfileError::SpkiEcPointNotUncompressedP256);
}
Ok(())
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,312 @@
use url::Url;
use x509_parser::prelude::{FromDer, X509Certificate};
use crate::data_model::oid::OID_CP_IPADDR_ASNUMBER;
use crate::data_model::rc::{
AsIdentifierChoice, IpAddressChoice, ResourceCertKind, ResourceCertificate,
ResourceCertificateParseError, ResourceCertificateParsed, ResourceCertificateProfileError,
ResourceCertificateRole,
};
use crate::data_model::tal::Tal;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaCertificate {
pub raw_der: Vec<u8>,
pub rc_ca: ResourceCertificate,
}
#[derive(Debug, thiserror::Error)]
pub enum TaCertificateParseError {
#[error("TA certificate parse error: {0} (RFC 5280 §4.1; RFC 6487 §4; RFC 8630 §2.3)")]
ResourceCertificate(#[from] ResourceCertificateParseError),
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaCertificateParsed {
pub rc_parsed: ResourceCertificateParsed,
}
#[derive(Debug, thiserror::Error)]
pub enum TaCertificateProfileError {
#[error("resource certificate profile error: {0} (RFC 5280 §4; RFC 6487 §4)")]
ResourceCertificate(#[from] ResourceCertificateProfileError),
#[error("TA certificate must be a CA certificate (RFC 8630 §2.3; RFC 6487 §4.8.1)")]
NotCa,
#[error(
"TA certificate must be self-signed (issuer DN must equal subject DN) (RFC 8630 §2.3; RFC 5280 §4.1.2.4)"
)]
NotSelfSignedIssuerSubject,
#[error(
"TA certificate must contain certificatePolicies ipAddr-asNumber ({OID_CP_IPADDR_ASNUMBER}) (RFC 6487 §4.8.9; RFC 8630 §2.3)"
)]
MissingOrInvalidCertificatePolicies,
#[error("TA certificate must contain SubjectKeyIdentifier (RFC 6487 §4.8.2; RFC 8630 §2.3)")]
MissingSubjectKeyIdentifier,
#[error(
"TA certificate must contain at least one RFC 3779 resource extension (IP or AS) (RFC 6487 §4.8.10-§4.8.11; RFC 8630 §2.3)"
)]
ResourcesMissing,
#[error("TA certificate resources must be non-empty (RFC 8630 §2.3)")]
ResourcesEmpty,
#[error(
"TA certificate MUST NOT use inherit in IP resources (RFC 8630 §2.3; RFC 3779 §2.2.3.5)"
)]
IpResourcesInherit,
#[error(
"TA certificate MUST NOT use inherit in AS resources (RFC 8630 §2.3; RFC 3779 §3.2.3.3)"
)]
AsResourcesInherit,
}
#[derive(Debug, thiserror::Error)]
pub enum TaCertificateDecodeError {
#[error("{0}")]
Parse(#[from] TaCertificateParseError),
#[error("{0}")]
Validate(#[from] TaCertificateProfileError),
}
/// Backwards-compatible name: TA certificate errors from parse+validate.
pub type TaCertificateError = TaCertificateDecodeError;
#[derive(Debug, thiserror::Error)]
pub enum TaCertificateVerifyError {
#[error("TA certificate parse error: {0} (RFC 5280 §4.1; RFC 8630 §2.3)")]
Parse(String),
#[error("trailing bytes after TA certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
TrailingBytes(usize),
#[error(
"TA certificate self-signature verification failed: {0} (RFC 8630 §2.3; RFC 5280 §6.1)"
)]
InvalidSelfSignature(String),
}
impl TaCertificate {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(der: &[u8]) -> Result<TaCertificateParsed, TaCertificateParseError> {
Ok(TaCertificateParsed {
rc_parsed: ResourceCertificate::parse_der(der)?,
})
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `TaCertificate` is already profile-validated when constructed via `decode_der()` /
/// `TaCertificateParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), TaCertificateProfileError> {
Ok(())
}
/// Decode a TA certificate (`parse + validate`).
pub fn decode_der(der: &[u8]) -> Result<Self, TaCertificateDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn decode_der_with_strict_name(der: &[u8]) -> Result<Self, TaCertificateDecodeError> {
let ta = Self::decode_der(der)?;
ta.rc_ca
.validate_strict_name_profile()
.map_err(TaCertificateProfileError::from)?;
Ok(ta)
}
/// Backwards-compatible helper (historical name).
pub fn from_der(der: &[u8]) -> Result<Self, TaCertificateError> {
Self::decode_der(der)
}
pub fn spki_der(&self) -> &[u8] {
&self.rc_ca.tbs.subject_public_key_info
}
/// Verify step of scheme A (`parse → validate → verify`).
pub fn verify_self_signature(&self) -> Result<(), TaCertificateVerifyError> {
let (rem, cert) = X509Certificate::from_der(&self.raw_der)
.map_err(|e| TaCertificateVerifyError::Parse(e.to_string()))?;
if !rem.is_empty() {
return Err(TaCertificateVerifyError::TrailingBytes(rem.len()));
}
cert.verify_signature(None)
.map_err(|e| TaCertificateVerifyError::InvalidSelfSignature(e.to_string()))?;
Ok(())
}
/// Validate TA-specific semantic constraints on a parsed Resource Certificate.
///
/// Note: this does not verify the X.509 signature; it is intended for higher-level logic and
/// for unit tests that exercise individual constraint branches.
pub fn validate_rc_constraints(
rc_ca: &ResourceCertificate,
) -> Result<(), TaCertificateProfileError> {
if rc_ca.kind != ResourceCertKind::Ca {
return Err(TaCertificateProfileError::NotCa);
}
if rc_ca.tbs.extensions.certificate_policies_oid.as_deref() != Some(OID_CP_IPADDR_ASNUMBER)
{
return Err(TaCertificateProfileError::MissingOrInvalidCertificatePolicies);
}
if rc_ca.tbs.extensions.subject_key_identifier.is_none() {
return Err(TaCertificateProfileError::MissingSubjectKeyIdentifier);
}
let ip = rc_ca.tbs.extensions.ip_resources.as_ref();
let asn = rc_ca.tbs.extensions.as_resources.as_ref();
if ip.is_none() && asn.is_none() {
return Err(TaCertificateProfileError::ResourcesMissing);
}
let mut has_any_resource = false;
if let Some(ip) = ip {
if ip.has_any_inherit() {
return Err(TaCertificateProfileError::IpResourcesInherit);
}
for fam in &ip.families {
match &fam.choice {
IpAddressChoice::Inherit => {
return Err(TaCertificateProfileError::IpResourcesInherit);
}
IpAddressChoice::AddressesOrRanges(items) => {
if !items.is_empty() {
has_any_resource = true;
}
}
}
}
}
if let Some(asn) = asn {
if matches!(asn.asnum, Some(AsIdentifierChoice::Inherit))
|| matches!(asn.rdi, Some(AsIdentifierChoice::Inherit))
{
return Err(TaCertificateProfileError::AsResourcesInherit);
}
if let Some(AsIdentifierChoice::AsIdsOrRanges(items)) = asn.asnum.as_ref() {
if !items.is_empty() {
has_any_resource = true;
}
}
if let Some(AsIdentifierChoice::AsIdsOrRanges(items)) = asn.rdi.as_ref() {
if !items.is_empty() {
has_any_resource = true;
}
}
}
if !has_any_resource {
return Err(TaCertificateProfileError::ResourcesEmpty);
}
Ok(())
}
}
impl TaCertificateParsed {
pub fn validate_profile(self) -> Result<TaCertificate, TaCertificateProfileError> {
let rc_ca = self.rc_parsed.validate_profile()?;
if rc_ca.kind != ResourceCertKind::Ca {
return Err(TaCertificateProfileError::NotCa);
}
rc_ca.validate_rfc6487_profile(ResourceCertificateRole::TrustAnchor)?;
if rc_ca.tbs.issuer_name != rc_ca.tbs.subject_name {
return Err(TaCertificateProfileError::NotSelfSignedIssuerSubject);
}
TaCertificate::validate_rc_constraints(&rc_ca)?;
Ok(TaCertificate {
raw_der: rc_ca.raw_der.clone(),
rc_ca,
})
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TrustAnchor {
pub tal: Tal,
pub ta_certificate: TaCertificate,
pub resolved_ta_uri: Option<Url>,
}
#[derive(Debug, thiserror::Error)]
pub enum TrustAnchorError {
#[error("TA certificate error: {0} (RFC 8630 §2.3)")]
TaCertificate(#[from] TaCertificateDecodeError),
#[error("TA certificate self-signature error: {0} (RFC 8630 §2.3)")]
TaSelfSignature(#[from] TaCertificateVerifyError),
#[error("{0}")]
Bind(#[from] TrustAnchorBindError),
}
#[derive(Debug, thiserror::Error)]
pub enum TrustAnchorBindError {
#[error("resolved TA URI not listed in TAL: {0} (RFC 8630 §2.2-§2.3)")]
ResolvedUriNotInTal(String),
#[error(
"TAL SPKI does not match TA certificate SubjectPublicKeyInfo (RFC 8630 §2.3; RFC 5280 §4.1.2.7)"
)]
TalSpkiMismatch,
}
impl TrustAnchor {
/// Bind a TAL and a downloaded TA certificate.
///
/// This does not download anything; it only validates the binding rules from RFC 8630 §2.3.
pub fn bind_der(
tal: Tal,
ta_der: &[u8],
resolved_uri: Option<&Url>,
) -> Result<Self, TrustAnchorError> {
let ta_certificate = TaCertificate::decode_der(ta_der)?;
ta_certificate.verify_self_signature()?;
Ok(Self::bind(tal, ta_certificate, resolved_uri)?)
}
pub fn bind_der_with_strict_name(
tal: Tal,
ta_der: &[u8],
resolved_uri: Option<&Url>,
) -> Result<Self, TrustAnchorError> {
let ta_certificate = TaCertificate::decode_der_with_strict_name(ta_der)?;
ta_certificate.verify_self_signature()?;
Ok(Self::bind(tal, ta_certificate, resolved_uri)?)
}
pub fn bind(
tal: Tal,
ta_certificate: TaCertificate,
resolved_uri: Option<&Url>,
) -> Result<Self, TrustAnchorBindError> {
if let Some(u) = resolved_uri {
if !tal.ta_uris.iter().any(|x| x == u) {
return Err(TrustAnchorBindError::ResolvedUriNotInTal(u.to_string()));
}
}
if tal.subject_public_key_info_der != ta_certificate.spki_der() {
return Err(TrustAnchorBindError::TalSpkiMismatch);
}
Ok(Self {
tal,
ta_certificate,
resolved_ta_uri: resolved_uri.cloned(),
})
}
}

View File

@ -0,0 +1,183 @@
use base64::Engine;
use url::Url;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TalParsed {
pub raw: Vec<u8>,
/// Lines split by '\n' and normalized by stripping a trailing '\r' per line.
pub lines: Vec<String>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Tal {
pub raw: Vec<u8>,
pub comments: Vec<String>,
pub ta_uris: Vec<Url>,
pub subject_public_key_info_der: Vec<u8>,
}
#[derive(Debug, thiserror::Error)]
pub enum TalParseError {
#[error("TAL must be valid UTF-8 (RFC 8630 §2.2)")]
InvalidUtf8,
}
#[derive(Debug, thiserror::Error)]
pub enum TalProfileError {
#[error("TAL comments must appear only at the beginning (RFC 8630 §2.2)")]
CommentAfterHeader,
#[error("TAL must contain at least one TA URI line (RFC 8630 §2.2)")]
MissingTaUris,
#[error(
"TAL must contain an empty line separator between URI list and SPKI base64 (RFC 8630 §2.2)"
)]
MissingSeparatorEmptyLine,
#[error("TAL TA URI invalid: {0} (RFC 8630 §2.2)")]
InvalidUri(String),
#[error("TAL TA URI scheme must be rsync or https, got {0} (RFC 8630 §2.2)")]
UnsupportedUriScheme(String),
#[error(
"TAL TA URI must reference a single object (must not end with '/'): {0} (RFC 8630 §2.3)"
)]
UriIsDirectory(String),
#[error(
"TAL must contain base64-encoded SubjectPublicKeyInfo after the separator (RFC 8630 §2.2)"
)]
MissingSpki,
#[error("TAL SPKI base64 decode failed (RFC 8630 §2.2)")]
SpkiBase64Decode,
#[error("TAL SPKI DER is empty (RFC 8630 §2.2)")]
SpkiDerEmpty,
}
#[derive(Debug, thiserror::Error)]
pub enum TalDecodeError {
#[error("{0}")]
Parse(#[from] TalParseError),
#[error("{0}")]
Validate(#[from] TalProfileError),
}
impl Tal {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_bytes(input: &[u8]) -> Result<TalParsed, TalParseError> {
let raw = input.to_vec();
let text = std::str::from_utf8(input).map_err(|_| TalParseError::InvalidUtf8)?;
let lines: Vec<String> = text
.split('\n')
.map(|l| l.strip_suffix('\r').unwrap_or(l).to_string())
.collect();
Ok(TalParsed { raw, lines })
}
/// Validate step of scheme A (`parse → validate → verify`).
///
/// `Tal` is already profile-validated when constructed via `decode_bytes()` /
/// `TalParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), TalProfileError> {
Ok(())
}
pub fn decode_bytes(input: &[u8]) -> Result<Self, TalDecodeError> {
Ok(Self::parse_bytes(input)?.validate_profile()?)
}
}
impl TalParsed {
pub fn validate_profile(self) -> Result<Tal, TalProfileError> {
let mut idx = 0usize;
// 1) Leading comments.
let mut comments: Vec<String> = Vec::new();
while idx < self.lines.len() && self.lines[idx].starts_with('#') {
comments.push(self.lines[idx][1..].to_string());
idx += 1;
}
// 2) URI list (one or more non-empty lines).
let mut ta_uris: Vec<Url> = Vec::new();
while idx < self.lines.len() {
let line = self.lines[idx].trim();
if line.is_empty() {
break;
}
if line.starts_with('#') {
return Err(TalProfileError::CommentAfterHeader);
}
let url = match Url::parse(line) {
Ok(u) => u,
Err(_) => {
if !ta_uris.is_empty() {
return Err(TalProfileError::MissingSeparatorEmptyLine);
}
return Err(TalProfileError::InvalidUri(line.to_string()));
}
};
match url.scheme() {
"rsync" | "https" => {}
s => return Err(TalProfileError::UnsupportedUriScheme(s.to_string())),
}
if url.path().ends_with('/') {
return Err(TalProfileError::UriIsDirectory(line.to_string()));
}
if url
.path_segments()
.and_then(|mut s| s.next_back())
.unwrap_or("")
.is_empty()
{
return Err(TalProfileError::UriIsDirectory(line.to_string()));
}
ta_uris.push(url);
idx += 1;
}
if ta_uris.is_empty() {
return Err(TalProfileError::MissingTaUris);
}
// 3) Empty line separator (must exist).
if idx >= self.lines.len() || !self.lines[idx].trim().is_empty() {
return Err(TalProfileError::MissingSeparatorEmptyLine);
}
idx += 1;
// 4) Base64(SPKI DER) remainder; allow line wrapping.
let mut b64 = String::new();
while idx < self.lines.len() {
let line = self.lines[idx].trim();
if !line.is_empty() {
b64.push_str(line);
}
idx += 1;
}
if b64.is_empty() {
return Err(TalProfileError::MissingSpki);
}
let spki_der = base64::engine::general_purpose::STANDARD
.decode(b64.as_bytes())
.map_err(|_| TalProfileError::SpkiBase64Decode)?;
if spki_der.is_empty() {
return Err(TalProfileError::SpkiDerEmpty);
}
Ok(Tal {
raw: self.raw,
comments,
ta_uris,
subject_public_key_info_der: spki_der,
})
}
}

View File

@ -0,0 +1,234 @@
use std::sync::Arc;
use crate::fetch::rsync::{RsyncFetchError, RsyncFetchResult, RsyncFetcher};
use crate::fetch::rsync_system::{
RsyncScopePolicy, scoped_rsync_failure_dedup_key, scoped_rsync_fetch_uri,
};
use crate::storage::{RepositoryViewState, RocksStore};
#[derive(Clone)]
pub struct CurrentRepositoryViewRsyncFetcher {
store: Arc<RocksStore>,
scope_policy: RsyncScopePolicy,
}
impl CurrentRepositoryViewRsyncFetcher {
pub fn new(store: Arc<RocksStore>, scope_policy: RsyncScopePolicy) -> Self {
Self {
store,
scope_policy,
}
}
}
impl RsyncFetcher for CurrentRepositoryViewRsyncFetcher {
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
// The source run's live fetcher may have fetched a module root rather
// than the publication-point URI that triggered it. Read that same
// frozen-view prefix here so transport prefetch request identities and
// their materialized object sets stay equivalent.
let base = scoped_rsync_fetch_uri(self.scope_policy, rsync_base_uri);
let entries = self
.store
.list_repository_view_entries_with_prefix(&base)
.map_err(|error| {
RsyncFetchError::Fetch(format!(
"list frozen repository view failed for {base}: {error}"
))
})?;
let mut objects = Vec::with_capacity(entries.len());
for entry in entries {
if !matches!(
entry.state,
RepositoryViewState::Present | RepositoryViewState::Replaced
) {
continue;
}
let bytes = self
.store
.load_current_object_bytes_by_uri(&entry.rsync_uri)
.map_err(|error| {
RsyncFetchError::Fetch(format!(
"load frozen repository object failed for {}: {error}",
entry.rsync_uri
))
})?
.ok_or_else(|| {
RsyncFetchError::Fetch(format!(
"frozen repository object missing for {}",
entry.rsync_uri
))
})?;
objects.push((entry.rsync_uri, bytes));
}
objects.sort_by(|left, right| left.0.cmp(&right.0));
if objects.is_empty() {
return Err(RsyncFetchError::Fetch(format!(
"frozen repository view contains no current objects under {base}"
)));
}
Ok(objects)
}
fn fetch_object(&self, rsync_uri: &str) -> RsyncFetchResult<Vec<u8>> {
self.store
.load_current_object_bytes_by_uri(rsync_uri)
.map_err(|error| {
RsyncFetchError::Fetch(format!(
"load frozen repository object failed for {rsync_uri}: {error}"
))
})?
.ok_or_else(|| {
RsyncFetchError::Fetch(format!("frozen repository object not found: {rsync_uri}"))
})
}
fn dedup_key(&self, rsync_base_uri: &str) -> String {
scoped_rsync_fetch_uri(self.scope_policy, rsync_base_uri)
}
fn failure_dedup_key(&self, rsync_base_uri: &str) -> Option<String> {
scoped_rsync_failure_dedup_key(self.scope_policy, rsync_base_uri)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::storage::{RepositoryViewEntry, RocksStore};
#[test]
fn current_repository_fetcher_returns_sorted_present_objects() {
let dir = tempfile::tempdir().expect("tempdir");
let store = Arc::new(RocksStore::open(dir.path()).expect("open store"));
let entries = [
("rsync://example.test/repo/b.roa", b"b".as_slice()),
("rsync://example.test/repo/a.mft", b"a".as_slice()),
];
for (uri, bytes) in entries {
let hash = hex::encode(crate::cir::sha256(bytes));
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.to_vec())])
.expect("put blob");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: uri.to_string(),
repository_source: Some("fixture".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
current_hash: Some(hash),
})
.expect("put view");
}
let fetcher = CurrentRepositoryViewRsyncFetcher::new(store, RsyncScopePolicy::default());
let objects = fetcher
.fetch_objects("rsync://example.test/repo/")
.expect("fetch objects");
assert_eq!(
objects
.iter()
.map(|(uri, _)| uri.as_str())
.collect::<Vec<_>>(),
vec![
"rsync://example.test/repo/a.mft",
"rsync://example.test/repo/b.roa"
]
);
assert_eq!(
fetcher
.fetch_object("rsync://example.test/repo/a.mft")
.expect("fetch one"),
b"a"
);
assert!(
fetcher
.fetch_object("rsync://example.test/repo/missing.roa")
.is_err()
);
assert_eq!(
fetcher.dedup_key("rsync://example.test/repo"),
"rsync://example.test/repo/"
);
assert!(
fetcher
.fetch_objects("rsync://empty.example/repo/")
.unwrap_err()
.to_string()
.contains("no current objects")
);
}
#[test]
fn current_repository_fetcher_ignores_withdrawn_and_reports_missing_blob() {
let dir = tempfile::tempdir().expect("tempdir");
let store = Arc::new(RocksStore::open(dir.path()).expect("open store"));
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/withdrawn.roa".to_string(),
repository_source: Some("fixture".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Withdrawn,
current_hash: None,
})
.expect("put withdrawn view");
let missing_hash = "ab".repeat(32);
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/missing.roa".to_string(),
repository_source: Some("fixture".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
current_hash: Some(missing_hash),
})
.expect("put missing view");
let fetcher = CurrentRepositoryViewRsyncFetcher::new(store, RsyncScopePolicy::default());
let error = fetcher
.fetch_objects("rsync://example.test/repo/")
.unwrap_err()
.to_string();
assert!(error.contains("blob bytes missing"));
}
#[test]
fn current_repository_fetcher_replays_module_scope_and_failure_dedup() {
let dir = tempfile::tempdir().expect("tempdir");
let store = Arc::new(RocksStore::open(dir.path()).expect("open store"));
for (uri, bytes) in [
("rsync://example.test/repo/ca/a.mft", b"a".as_slice()),
("rsync://example.test/repo/other/b.roa", b"b".as_slice()),
] {
let hash = hex::encode(crate::cir::sha256(bytes));
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.to_vec())])
.expect("put blob");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: uri.to_string(),
repository_source: Some("fixture".to_string()),
object_type: Some("fixture".to_string()),
state: RepositoryViewState::Present,
current_hash: Some(hash),
})
.expect("put view");
}
let module_fetcher = CurrentRepositoryViewRsyncFetcher::new(
Arc::clone(&store),
RsyncScopePolicy::ModuleRoot,
);
let objects = module_fetcher
.fetch_objects("rsync://example.test/repo/ca/")
.expect("fetch module scope");
assert_eq!(objects.len(), 2);
assert_eq!(
module_fetcher.dedup_key("rsync://example.test/repo/ca/"),
"rsync://example.test/repo/"
);
let host_fetcher = CurrentRepositoryViewRsyncFetcher::new(store, RsyncScopePolicy::Host);
assert_eq!(
host_fetcher.failure_dedup_key("rsync://example.test/repo/ca/"),
Some("rsync://example.test/".to_string())
);
}
}

View File

@ -0,0 +1,652 @@
use std::cell::RefCell;
use std::io::Write;
use std::time::Duration;
use reqwest::blocking::Client;
use reqwest::header::HeaderMap;
use crate::sync::rrdp::Fetcher;
thread_local! {
static HTTP_TIMEOUT_OVERRIDE: RefCell<Option<Duration>> = const { RefCell::new(None) };
}
pub fn with_scoped_http_timeout_override<R>(timeout: Duration, f: impl FnOnce() -> R) -> R {
HTTP_TIMEOUT_OVERRIDE.with(|cell| {
let previous = cell.replace(Some(timeout));
let result = f();
let _ = cell.replace(previous);
result
})
}
/// Default User-Agent sent with every outgoing HTTP request (RRDP
/// notification/snapshot/delta, TAL/TA downloads, daemon probes).
pub const DEFAULT_HTTP_USER_AGENT: &str = "panda-rpki/0.2";
/// Environment variable that overrides the HTTP User-Agent. Unset, empty, or
/// values containing characters that are illegal in a header value fall back
/// to [`DEFAULT_HTTP_USER_AGENT`].
pub const HTTP_USER_AGENT_ENV: &str = "RPKI_HTTP_USER_AGENT";
fn resolve_http_user_agent(env_value: Option<String>) -> String {
let Some(value) = env_value else {
return DEFAULT_HTTP_USER_AGENT.to_string();
};
let trimmed = value.trim();
let valid = !trimmed.is_empty() && trimmed.bytes().all(|b| (0x20..=0x7e).contains(&b));
if valid {
trimmed.to_string()
} else {
DEFAULT_HTTP_USER_AGENT.to_string()
}
}
#[derive(Clone, Debug)]
pub struct HttpFetcherConfig {
/// Connection-establishment timeout for HTTP requests.
pub connect_timeout: Duration,
/// Short timeout used for connection establishment and small metadata objects.
pub timeout: Duration,
/// Larger timeout used for RRDP snapshot / delta bodies.
pub large_body_timeout: Duration,
pub user_agent: String,
/// Extra PEM trust anchors for HTTPS transport tests or private RRDP endpoints.
pub extra_root_certificates_pem: Vec<Vec<u8>>,
}
impl Default for HttpFetcherConfig {
fn default() -> Self {
Self {
connect_timeout: Duration::from_secs(15),
timeout: Duration::from_secs(30),
large_body_timeout: Duration::from_secs(180),
user_agent: resolve_http_user_agent(std::env::var(HTTP_USER_AGENT_ENV).ok()),
extra_root_certificates_pem: Vec::new(),
}
}
}
/// Minimal blocking HTTP(S) fetcher for stage2.
///
/// This is used for:
/// - downloading TAL / TA certificates (RFC 8630 §2)
/// - fetching RRDP notification/snapshot files (RFC 8182 §3.4)
#[derive(Clone, Debug)]
pub struct BlockingHttpFetcher {
short_client: Client,
large_body_client: Client,
retry_short_client: Client,
short_timeout: Duration,
large_body_timeout: Duration,
}
impl BlockingHttpFetcher {
pub fn new(config: HttpFetcherConfig) -> Result<Self, String> {
let short_timeout = config.timeout;
let large_body_timeout = std::cmp::max(config.large_body_timeout, config.timeout);
let connect_timeout = std::cmp::min(config.connect_timeout, config.timeout);
let short_client = Self::client_builder(
&config,
connect_timeout,
config.timeout,
config.user_agent.clone(),
)?
.build()
.map_err(|e| e.to_string())?;
let large_body_client = Self::client_builder(
&config,
connect_timeout,
large_body_timeout,
config.user_agent.clone(),
)?
.build()
.map_err(|e| e.to_string())?;
let retry_short_client = Self::client_builder(
&config,
Duration::from_secs(1),
Duration::from_secs(1),
config.user_agent.clone(),
)?
.build()
.map_err(|e| e.to_string())?;
Ok(Self {
short_client,
large_body_client,
retry_short_client,
short_timeout,
large_body_timeout,
})
}
fn client_builder(
config: &HttpFetcherConfig,
connect_timeout: Duration,
timeout: Duration,
user_agent: String,
) -> Result<reqwest::blocking::ClientBuilder, String> {
let mut builder = Client::builder()
.connect_timeout(connect_timeout)
.timeout(timeout)
.user_agent(user_agent);
for (idx, pem) in config.extra_root_certificates_pem.iter().enumerate() {
let certificate = reqwest::Certificate::from_pem(pem)
.map_err(|e| format!("parse HTTP root certificate #{idx} failed: {e}"))?;
builder = builder.add_root_certificate(certificate);
}
Ok(builder)
}
pub fn fetch_bytes(&self, uri: &str) -> Result<Vec<u8>, String> {
let started = std::time::Instant::now();
let (client, timeout_profile, timeout_value) = self.client_for_uri(uri);
let resp = client.get(uri).send().map_err(|e| {
let msg = format!("http request failed: {e:?}");
crate::progress_log::emit(
"http_fetch_failed",
serde_json::json!({
"uri": uri,
"stage": "request",
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": started.elapsed().as_millis() as u64,
"error": msg,
}),
);
msg
})?;
let status = resp.status();
let headers = resp.headers().clone();
if !status.is_success() {
let body_preview = resp
.text()
.ok()
.map(|text| text.chars().take(160).collect::<String>());
let body_prefix = body_preview
.clone()
.unwrap_or_else(|| "<unavailable>".to_string());
let msg = format!(
"http status {status}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}; body_prefix={}",
header_value(&headers, "content-type"),
header_value(&headers, "content-encoding"),
header_value(&headers, "content-length"),
header_value(&headers, "transfer-encoding"),
body_prefix,
);
crate::progress_log::emit(
"http_fetch_failed",
serde_json::json!({
"uri": uri,
"stage": "status",
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": started.elapsed().as_millis() as u64,
"status": status.as_u16(),
"content_type": header_value_opt(&headers, "content-type"),
"content_encoding": header_value_opt(&headers, "content-encoding"),
"content_length": header_value_opt(&headers, "content-length"),
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
"body_prefix": body_preview,
"error": msg,
}),
);
return Err(msg);
}
match resp.bytes() {
Ok(bytes) => {
let duration_ms = started.elapsed().as_millis() as u64;
if (duration_ms as f64) / 1000.0 >= crate::progress_log::slow_threshold_secs() {
crate::progress_log::emit(
"http_fetch_slow",
serde_json::json!({
"uri": uri,
"status": status.as_u16(),
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": duration_ms,
"bytes": bytes.len(),
"content_type": header_value_opt(&headers, "content-type"),
"content_encoding": header_value_opt(&headers, "content-encoding"),
"content_length": header_value_opt(&headers, "content-length"),
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
}),
);
}
Ok(bytes.to_vec())
}
Err(e) => {
let msg = format!(
"http read body failed: {e}; status={}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}",
status,
header_value(&headers, "content-type"),
header_value(&headers, "content-encoding"),
header_value(&headers, "content-length"),
header_value(&headers, "transfer-encoding"),
);
crate::progress_log::emit(
"http_fetch_failed",
serde_json::json!({
"uri": uri,
"stage": "read_body",
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": started.elapsed().as_millis() as u64,
"status": status.as_u16(),
"content_type": header_value_opt(&headers, "content-type"),
"content_encoding": header_value_opt(&headers, "content-encoding"),
"content_length": header_value_opt(&headers, "content-length"),
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
"error": msg,
}),
);
Err(msg)
}
}
}
fn client_for_uri(&self, uri: &str) -> (&Client, &'static str, Duration) {
if let Some(timeout) = HTTP_TIMEOUT_OVERRIDE.with(|cell| *cell.borrow()) {
return (&self.retry_short_client, "retry_short", timeout);
}
if uses_large_body_timeout(uri) {
(
&self.large_body_client,
"large_body",
self.large_body_timeout,
)
} else {
(&self.short_client, "short", self.short_timeout)
}
}
}
impl Fetcher for BlockingHttpFetcher {
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
self.fetch_bytes(uri)
}
fn fetch_to_writer(&self, uri: &str, out: &mut dyn Write) -> Result<u64, String> {
let started = std::time::Instant::now();
let (client, timeout_profile, timeout_value) = self.client_for_uri(uri);
let resp = client.get(uri).send().map_err(|e| {
let msg = format!("http request failed: {e:?}");
crate::progress_log::emit(
"http_fetch_failed",
serde_json::json!({
"uri": uri,
"stage": "request",
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": started.elapsed().as_millis() as u64,
"error": msg,
}),
);
msg
})?;
let status = resp.status();
let headers = resp.headers().clone();
if !status.is_success() {
let body_preview = resp
.text()
.ok()
.map(|text| text.chars().take(160).collect::<String>());
let body_prefix = body_preview
.clone()
.unwrap_or_else(|| "<unavailable>".to_string());
let msg = format!(
"http status {status}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}; body_prefix={}",
header_value(&headers, "content-type"),
header_value(&headers, "content-encoding"),
header_value(&headers, "content-length"),
header_value(&headers, "transfer-encoding"),
body_prefix,
);
crate::progress_log::emit(
"http_fetch_failed",
serde_json::json!({
"uri": uri,
"stage": "status",
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": started.elapsed().as_millis() as u64,
"status": status.as_u16(),
"content_type": header_value_opt(&headers, "content-type"),
"content_encoding": header_value_opt(&headers, "content-encoding"),
"content_length": header_value_opt(&headers, "content-length"),
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
"body_prefix": body_preview,
"error": msg,
}),
);
return Err(msg);
}
let mut resp = resp;
match resp.copy_to(out) {
Ok(bytes) => {
let duration_ms = started.elapsed().as_millis() as u64;
if (duration_ms as f64) / 1000.0 >= crate::progress_log::slow_threshold_secs() {
crate::progress_log::emit(
"http_fetch_slow",
serde_json::json!({
"uri": uri,
"status": status.as_u16(),
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": duration_ms,
"bytes": bytes,
"content_type": header_value_opt(&headers, "content-type"),
"content_encoding": header_value_opt(&headers, "content-encoding"),
"content_length": header_value_opt(&headers, "content-length"),
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
}),
);
}
Ok(bytes)
}
Err(e) => {
let msg = format!(
"http stream body failed: {e}; status={}; content_type={}; content_encoding={}; content_length={}; transfer_encoding={}",
status,
header_value(&headers, "content-type"),
header_value(&headers, "content-encoding"),
header_value(&headers, "content-length"),
header_value(&headers, "transfer-encoding"),
);
crate::progress_log::emit(
"http_fetch_failed",
serde_json::json!({
"uri": uri,
"stage": "stream_body",
"timeout_profile": timeout_profile,
"request_timeout_ms": timeout_value.as_millis() as u64,
"duration_ms": started.elapsed().as_millis() as u64,
"status": status.as_u16(),
"content_type": header_value_opt(&headers, "content-type"),
"content_encoding": header_value_opt(&headers, "content-encoding"),
"content_length": header_value_opt(&headers, "content-length"),
"transfer_encoding": header_value_opt(&headers, "transfer-encoding"),
"error": msg,
}),
);
Err(msg)
}
}
}
}
fn header_value(headers: &HeaderMap, name: &str) -> String {
header_value_opt(headers, name).unwrap_or_else(|| "<none>".to_string())
}
fn header_value_opt(headers: &HeaderMap, name: &str) -> Option<String> {
headers
.get(name)
.and_then(|v| v.to_str().ok())
.map(|v| v.to_string())
}
fn uses_large_body_timeout(uri: &str) -> bool {
uri.starts_with("https://") && uri.ends_with(".xml") && !uri.ends_with("notification.xml")
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::{Read, Write};
use std::net::TcpListener;
use std::thread;
use std::time::Duration as StdDuration;
fn spawn_one_shot_http_server(status_line: &'static str, body: &'static [u8]) -> String {
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
let addr = listener.local_addr().expect("addr");
thread::spawn(move || {
let (mut stream, _) = listener.accept().expect("accept");
let mut buf = [0u8; 1024];
let _ = stream.read(&mut buf);
let hdr = format!(
"{status_line}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
);
stream.write_all(hdr.as_bytes()).expect("write hdr");
stream.write_all(body).expect("write body");
});
format!("http://{}/", addr)
}
#[test]
fn fetch_bytes_returns_body_on_success() {
let url = spawn_one_shot_http_server("HTTP/1.1 200 OK", b"hello");
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(2),
..HttpFetcherConfig::default()
})
.expect("http");
let got = http.fetch_bytes(&url).expect("fetch");
assert_eq!(got, b"hello");
}
#[test]
fn fetch_bytes_rejects_non_success_status() {
let url = spawn_one_shot_http_server("HTTP/1.1 404 Not Found", b"");
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(2),
..HttpFetcherConfig::default()
})
.expect("http");
let err = http.fetch_bytes(&url).unwrap_err();
assert!(err.contains("http status"), "{err}");
}
#[test]
fn fetch_bytes_times_out_on_idle_body_read() {
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
let addr = listener.local_addr().expect("addr");
thread::spawn(move || {
let (mut stream, _) = listener.accept().expect("accept");
let mut buf = [0u8; 1024];
let _ = stream.read(&mut buf);
stream
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\nConnection: close\r\n\r\nh")
.expect("write partial body");
std::thread::sleep(StdDuration::from_secs(2));
let _ = stream.write_all(b"ello");
});
let url = format!("http://{}/", addr);
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(1),
..HttpFetcherConfig::default()
})
.expect("http");
let err = http.fetch_bytes(&url).unwrap_err();
assert!(err.contains("http read body failed"), "{err}");
}
#[test]
fn fetch_to_writer_streams_body_on_success() {
let url = spawn_one_shot_http_server("HTTP/1.1 200 OK", b"writer-body");
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(2),
..HttpFetcherConfig::default()
})
.expect("http");
let mut out = Vec::new();
let bytes = http.fetch_to_writer(&url, &mut out).expect("stream");
assert_eq!(bytes, 11);
assert_eq!(out, b"writer-body");
}
#[test]
fn fetch_to_writer_rejects_non_success_status() {
let url = spawn_one_shot_http_server("HTTP/1.1 500 Internal Server Error", b"boom");
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(2),
..HttpFetcherConfig::default()
})
.expect("http");
let mut out = Vec::new();
let err = http.fetch_to_writer(&url, &mut out).unwrap_err();
assert!(err.contains("http status"), "{err}");
assert!(out.is_empty());
}
#[test]
fn fetch_to_writer_times_out_on_idle_stream_read() {
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
let addr = listener.local_addr().expect("addr");
thread::spawn(move || {
let (mut stream, _) = listener.accept().expect("accept");
let mut buf = [0u8; 1024];
let _ = stream.read(&mut buf);
stream
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\nConnection: close\r\n\r\nh")
.expect("write partial body");
std::thread::sleep(StdDuration::from_secs(2));
let _ = stream.write_all(b"ello");
});
let url = format!("http://{}/", addr);
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(1),
..HttpFetcherConfig::default()
})
.expect("http");
let mut out = Vec::new();
let err = http.fetch_to_writer(&url, &mut out).unwrap_err();
assert!(err.contains("http stream body failed"), "{err}");
}
#[test]
fn uses_large_body_timeout_selects_rrdp_snapshot_and_delta_not_notification() {
assert!(!uses_large_body_timeout(
"https://rrdp.example.test/notification.xml"
));
assert!(uses_large_body_timeout(
"https://rrdp.example.test/session/123/snapshot.xml"
));
assert!(uses_large_body_timeout(
"https://rrdp.example.test/session/123/delta-42.xml"
));
assert!(!uses_large_body_timeout(
"https://tal.example.test/example.tal"
));
}
#[test]
fn client_for_uri_selects_expected_timeout_profile() {
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(3),
large_body_timeout: Duration::from_secs(9),
..HttpFetcherConfig::default()
})
.expect("http");
let (_, profile_short, timeout_short) =
http.client_for_uri("https://example.test/root.tal");
assert_eq!(profile_short, "short");
assert_eq!(timeout_short, Duration::from_secs(3));
let (_, profile_large, timeout_large) =
http.client_for_uri("https://rrdp.example.test/session/1/snapshot.xml");
assert_eq!(profile_large, "large_body");
assert_eq!(timeout_large, Duration::from_secs(9));
}
#[test]
fn resolve_http_user_agent_falls_back_to_default() {
assert_eq!(resolve_http_user_agent(None), DEFAULT_HTTP_USER_AGENT);
assert_eq!(
resolve_http_user_agent(Some(String::new())),
DEFAULT_HTTP_USER_AGENT
);
assert_eq!(
resolve_http_user_agent(Some(" ".to_string())),
DEFAULT_HTTP_USER_AGENT
);
// Control characters are illegal in header values; fall back instead
// of letting reqwest fail to build the client.
assert_eq!(
resolve_http_user_agent(Some("bad\u{1}ua".to_string())),
DEFAULT_HTTP_USER_AGENT
);
assert_eq!(
resolve_http_user_agent(Some("non-ascii-ua-中".to_string())),
DEFAULT_HTTP_USER_AGENT
);
}
#[test]
fn resolve_http_user_agent_accepts_custom_value() {
assert_eq!(
resolve_http_user_agent(Some("panda-rpki/9.9 (test)".to_string())),
"panda-rpki/9.9 (test)"
);
// Surrounding whitespace is trimmed.
assert_eq!(
resolve_http_user_agent(Some(" custom/1.0 ".to_string())),
"custom/1.0"
);
}
fn spawn_user_agent_capture_server() -> (String, std::sync::mpsc::Receiver<String>) {
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind");
let addr = listener.local_addr().expect("addr");
let (tx, rx) = std::sync::mpsc::channel();
thread::spawn(move || {
let (mut stream, _) = listener.accept().expect("accept");
let mut buf = [0u8; 4096];
let read = stream.read(&mut buf).expect("read request");
let request = String::from_utf8_lossy(&buf[..read]).to_string();
let body = b"ok";
let hdr = format!(
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
);
stream.write_all(hdr.as_bytes()).expect("write hdr");
stream.write_all(body).expect("write body");
let _ = tx.send(request);
});
(format!("http://{}/", addr), rx)
}
#[test]
fn fetch_bytes_sends_configured_user_agent_header() {
let (url, rx) = spawn_user_agent_capture_server();
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(2),
user_agent: "panda-rpki/0.2".to_string(),
..HttpFetcherConfig::default()
})
.expect("http");
http.fetch_bytes(&url).expect("fetch");
let request = rx.recv().expect("request captured");
assert!(
request
.to_ascii_lowercase()
.contains("user-agent: panda-rpki/0.2"),
"{request}"
);
}
#[test]
fn fetch_bytes_sends_custom_user_agent_header() {
let (url, rx) = spawn_user_agent_capture_server();
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: Duration::from_secs(2),
user_agent: "custom-marker/7.7".to_string(),
..HttpFetcherConfig::default()
})
.expect("http");
http.fetch_bytes(&url).expect("fetch");
let request = rx.recv().expect("request captured");
assert!(
request
.to_ascii_lowercase()
.contains("user-agent: custom-marker/7.7"),
"{request}"
);
}
}

View File

@ -0,0 +1,4 @@
pub mod current_repository;
pub mod http;
pub mod rsync;
pub mod rsync_system;

View File

@ -0,0 +1,206 @@
use std::path::{Path, PathBuf};
#[derive(Debug, thiserror::Error)]
pub enum RsyncFetchError {
#[error("rsync fetch error: {0}")]
Fetch(String),
}
pub type RsyncFetchResult<T> = Result<T, RsyncFetchError>;
pub fn normalize_rsync_base_uri(s: &str) -> String {
if s.ends_with('/') {
s.to_string()
} else {
format!("{s}/")
}
}
/// Fetch repository objects from a publication point.
///
/// v1: this is intentionally abstract so unit tests can use a mock, and later we can
/// back it by calling the system `rsync` binary (RFC 6481 §5; RFC 8182 §3.4.5).
pub trait RsyncFetcher: Send + Sync {
/// Return a list of objects as `(rsync_uri, bytes)` pairs.
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>>;
/// Fetch one object by exact rsync URI.
///
/// The default implementation fetches the parent directory and filters the exact
/// object. Live fetchers should override this to avoid widening one-object TAL
/// bootstrap fetches into whole publication point or module synchronizations.
fn fetch_object(&self, rsync_uri: &str) -> RsyncFetchResult<Vec<u8>> {
let base = parent_rsync_uri(rsync_uri).map_err(RsyncFetchError::Fetch)?;
self.fetch_objects(&base)?
.into_iter()
.find(|(uri, _)| uri == rsync_uri)
.map(|(_, bytes)| bytes)
.ok_or_else(|| RsyncFetchError::Fetch(format!("rsync object not found: {rsync_uri}")))
}
/// Stream fetched objects to a visitor without requiring callers to materialize the
/// full result vector in memory.
fn visit_objects(
&self,
rsync_base_uri: &str,
visitor: &mut dyn FnMut(String, Vec<u8>) -> Result<(), String>,
) -> RsyncFetchResult<(usize, u64)> {
let objects = self.fetch_objects(rsync_base_uri)?;
let mut count = 0usize;
let mut bytes_total = 0u64;
for (uri, bytes) in objects {
bytes_total += bytes.len() as u64;
count += 1;
visitor(uri, bytes).map_err(RsyncFetchError::Fetch)?;
}
Ok((count, bytes_total))
}
/// Return the deduplication key used by orchestration layers.
///
/// By default this is the normalized publication point base URI. Fetchers that
/// intentionally widen their fetch scope (for example to a full rsync module)
/// should override this so callers can safely deduplicate at the same scope.
fn dedup_key(&self, rsync_base_uri: &str) -> String {
normalize_rsync_base_uri(rsync_base_uri)
}
/// Return an optional failure-only deduplication key.
///
/// This key is only used to short-circuit repeated failed rsync fallbacks. It must
/// not be used to reuse successful fetch results unless it is identical to
/// `dedup_key`, because a successful fetch for one publication point does not imply
/// that another publication point under the same host has been fetched.
fn failure_dedup_key(&self, _rsync_base_uri: &str) -> Option<String> {
None
}
}
fn parent_rsync_uri(rsync_uri: &str) -> Result<String, String> {
let parsed = url::Url::parse(rsync_uri).map_err(|e| e.to_string())?;
if parsed.scheme() != "rsync" {
return Err(format!("not an rsync URI: {rsync_uri}"));
}
let host = parsed
.host_str()
.ok_or_else(|| format!("missing host in rsync URI: {rsync_uri}"))?;
let segments = parsed
.path_segments()
.ok_or_else(|| format!("missing path in rsync URI: {rsync_uri}"))?
.collect::<Vec<_>>();
if segments.is_empty() || segments.last().copied().unwrap_or_default().is_empty() {
return Err(format!(
"rsync URI must reference a file object: {rsync_uri}"
));
}
let parent_segments = &segments[..segments.len() - 1];
let mut parent = format!("rsync://{host}/");
if !parent_segments.is_empty() {
parent.push_str(&parent_segments.join("/"));
parent.push('/');
}
Ok(parent)
}
/// A simple "rsync" implementation backed by a local directory.
///
/// This is primarily meant for offline tests and fixtures. The key generation mimics rsync URIs:
/// `rsync_base_uri` + relative path (with `/` separators).
#[derive(Clone, Debug)]
pub struct LocalDirRsyncFetcher {
pub root_dir: PathBuf,
}
impl LocalDirRsyncFetcher {
pub fn new(root_dir: impl Into<PathBuf>) -> Self {
Self {
root_dir: root_dir.into(),
}
}
}
impl RsyncFetcher for LocalDirRsyncFetcher {
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
let base = normalize_rsync_base_uri(rsync_base_uri);
let mut out = Vec::new();
walk_dir_collect(&self.root_dir, &self.root_dir, &base, &mut out)
.map_err(|e| RsyncFetchError::Fetch(e))?;
Ok(out)
}
}
fn walk_dir_collect(
root: &Path,
current: &Path,
rsync_base_uri: &str,
out: &mut Vec<(String, Vec<u8>)>,
) -> Result<(), String> {
let rd = std::fs::read_dir(current).map_err(|e| e.to_string())?;
for entry in rd {
let entry = entry.map_err(|e| e.to_string())?;
let path = entry.path();
let meta = entry.metadata().map_err(|e| e.to_string())?;
if meta.is_dir() {
walk_dir_collect(root, &path, rsync_base_uri, out)?;
continue;
}
if !meta.is_file() {
continue;
}
let rel = path
.strip_prefix(root)
.map_err(|e| e.to_string())?
.to_string_lossy()
.replace('\\', "/");
let uri = format!("{rsync_base_uri}{rel}");
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
out.push((uri, bytes));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn local_dir_rsync_fetcher_collects_files_and_normalizes_base_uri() {
let tmp = tempfile::tempdir().expect("tempdir");
std::fs::create_dir_all(tmp.path().join("nested")).expect("mkdir");
std::fs::write(tmp.path().join("a.mft"), b"a").expect("write");
std::fs::write(tmp.path().join("nested").join("b.roa"), b"b").expect("write");
let f = LocalDirRsyncFetcher::new(tmp.path());
let mut objects = f
.fetch_objects("rsync://example.net/repo")
.expect("fetch_objects");
objects.sort_by(|(a, _), (b, _)| a.cmp(b));
assert_eq!(objects.len(), 2);
assert_eq!(objects[0].0, "rsync://example.net/repo/a.mft");
assert_eq!(objects[0].1, b"a");
assert_eq!(objects[1].0, "rsync://example.net/repo/nested/b.roa");
assert_eq!(objects[1].1, b"b");
}
#[test]
fn local_dir_rsync_fetcher_reports_read_dir_errors() {
let tmp = tempfile::tempdir().expect("tempdir");
let missing = tmp.path().join("missing");
let f = LocalDirRsyncFetcher::new(missing);
let err = f.fetch_objects("rsync://example.net/repo").unwrap_err();
match err {
RsyncFetchError::Fetch(msg) => assert!(!msg.is_empty()),
}
}
#[test]
fn default_dedup_key_is_normalized_base_uri() {
let tmp = tempfile::tempdir().expect("tempdir");
let fetcher = LocalDirRsyncFetcher::new(tmp.path());
assert_eq!(
fetcher.dedup_key("rsync://example.net/repo"),
"rsync://example.net/repo/"
);
}
}

View File

@ -0,0 +1,944 @@
use std::cell::RefCell;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::process::Stdio;
use std::thread;
use std::time::Duration;
use std::time::Instant;
use sha2::Digest;
use uuid::Uuid;
use crate::fetch::rsync::{
RsyncFetchError, RsyncFetchResult, RsyncFetcher, normalize_rsync_base_uri,
};
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum RsyncScopePolicy {
Host,
PublicationPoint,
ModuleRoot,
}
impl Default for RsyncScopePolicy {
fn default() -> Self {
Self::ModuleRoot
}
}
impl RsyncScopePolicy {
pub fn parse_cli_value(value: &str) -> Result<Self, String> {
match value {
"host" => Ok(Self::Host),
"publication-point" => Ok(Self::PublicationPoint),
"module-root" => Ok(Self::ModuleRoot),
_ => Err(format!(
"invalid --rsync-scope: {value}; expected host, publication-point, or module-root"
)),
}
}
}
#[derive(Clone, Debug)]
pub struct SystemRsyncConfig {
pub rsync_bin: PathBuf,
pub connect_timeout: Duration,
pub timeout: Duration,
pub extra_args: Vec<String>,
/// Optional root directory for persistent rsync mirrors.
///
/// When set, callers may choose to sync into stable subdirectories under this
/// root (instead of a temporary directory) to benefit from rsync's incremental
/// behavior across runs.
///
/// Note: actual mirror behavior is implemented separately from config wiring.
pub mirror_root: Option<PathBuf>,
pub scope_policy: RsyncScopePolicy,
}
impl Default for SystemRsyncConfig {
fn default() -> Self {
Self {
rsync_bin: PathBuf::from("rsync"),
connect_timeout: Duration::from_secs(15),
timeout: Duration::from_secs(30),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
}
}
}
/// A `RsyncFetcher` implementation backed by the system `rsync` binary.
///
/// This is intended for live stage2 runs. For unit tests and offline fixtures,
/// prefer `LocalDirRsyncFetcher`.
#[derive(Clone, Debug)]
pub struct SystemRsyncFetcher {
config: SystemRsyncConfig,
}
thread_local! {
static RSYNC_TIMEOUT_OVERRIDE: RefCell<Option<Duration>> = const { RefCell::new(None) };
static RSYNC_FAIL_FAST_PROFILE: RefCell<Option<RsyncFailFastProfile>> = const { RefCell::new(None) };
}
pub fn with_scoped_rsync_timeout_override<R>(timeout: Duration, f: impl FnOnce() -> R) -> R {
RSYNC_TIMEOUT_OVERRIDE.with(|cell| {
let previous = cell.replace(Some(timeout));
let result = f();
let _ = cell.replace(previous);
result
})
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct RsyncFailFastProfile {
pub initial_wall_clock_timeout: Duration,
pub max_wall_clock_timeout: Duration,
pub max_attempts: usize,
}
pub fn with_scoped_rsync_fail_fast_profile<R>(
profile: RsyncFailFastProfile,
f: impl FnOnce() -> R,
) -> R {
RSYNC_FAIL_FAST_PROFILE.with(|cell| {
let previous = cell.replace(Some(profile));
let result = f();
let _ = cell.replace(previous);
result
})
}
impl SystemRsyncFetcher {
pub fn new(config: SystemRsyncConfig) -> Self {
Self { config }
}
fn mirror_dst_dir(&self, normalized_rsync_base_uri: &str) -> Result<Option<PathBuf>, String> {
let Some(root) = self.config.mirror_root.as_ref() else {
return Ok(None);
};
std::fs::create_dir_all(root)
.map_err(|e| format!("create rsync mirror root failed: {}: {e}", root.display()))?;
let hash = hex::encode(sha2::Sha256::digest(normalized_rsync_base_uri.as_bytes()));
let dir = root.join(hash);
std::fs::create_dir_all(&dir).map_err(|e| {
format!(
"create rsync mirror directory failed: {}: {e}",
dir.display()
)
})?;
Ok(Some(dir))
}
fn run_rsync(&self, src: &str, dst: &Path) -> Result<(), String> {
let fail_fast = RSYNC_FAIL_FAST_PROFILE.with(|cell| *cell.borrow());
if let Some(profile) = fail_fast {
return self.run_rsync_fail_fast(src, dst, profile);
}
self.run_rsync_once(src, dst, None, false)
}
fn run_rsync_once(
&self,
src: &str,
dst: &Path,
wall_clock_timeout: Option<Duration>,
keep_partial: bool,
) -> Result<(), String> {
// `--timeout` is I/O timeout in seconds (applies to network reads/writes).
let timeout =
RSYNC_TIMEOUT_OVERRIDE.with(|cell| cell.borrow().unwrap_or(self.config.timeout));
let connect_timeout_secs = self.config.connect_timeout.as_secs().max(1).to_string();
let timeout_secs = timeout.as_secs().max(1).to_string();
let is_remote_rsync = src.starts_with("rsync://");
let mut cmd = Command::new(&self.config.rsync_bin);
cmd.arg("-rt")
.arg("--delete")
.arg("--timeout")
.arg(timeout_secs)
.args(&self.config.extra_args);
if is_remote_rsync {
cmd.arg("--contimeout").arg(connect_timeout_secs);
}
if keep_partial {
cmd.arg("--partial");
}
cmd.arg(src)
.arg(dst)
.stdout(Stdio::piped())
.stderr(Stdio::piped());
let mut child = cmd
.spawn()
.map_err(|e| format!("rsync spawn failed: {e}"))?;
if let Some(limit) = wall_clock_timeout {
let started = Instant::now();
loop {
match child
.try_wait()
.map_err(|e| format!("rsync wait failed: {e}"))?
{
Some(_status) => {
let out = child
.wait_with_output()
.map_err(|e| format!("rsync wait_with_output failed: {e}"))?;
if out.status.success() {
return Ok(());
}
let stderr = String::from_utf8_lossy(&out.stderr);
let stdout = String::from_utf8_lossy(&out.stdout);
return Err(format!(
"rsync failed: status={} stdout={} stderr={}",
out.status,
stdout.trim(),
stderr.trim()
));
}
None => {
if started.elapsed() >= limit {
let _ = child.kill();
let out = child
.wait_with_output()
.map_err(|e| format!("rsync wait_with_output failed: {e}"))?;
let stderr = String::from_utf8_lossy(&out.stderr);
let stdout = String::from_utf8_lossy(&out.stdout);
return Err(format!(
"rsync wall-clock timeout after {}s: stdout={} stderr={}",
limit.as_secs(),
stdout.trim(),
stderr.trim()
));
}
thread::sleep(Duration::from_millis(100));
}
}
}
}
let out = child
.wait_with_output()
.map_err(|e| format!("rsync wait_with_output failed: {e}"))?;
if out.status.success() {
return Ok(());
}
let stderr = String::from_utf8_lossy(&out.stderr);
let stdout = String::from_utf8_lossy(&out.stdout);
Err(format!(
"rsync failed: status={} stdout={} stderr={}",
out.status,
stdout.trim(),
stderr.trim()
))
}
fn run_rsync_fail_fast(
&self,
src: &str,
dst: &Path,
profile: RsyncFailFastProfile,
) -> Result<(), String> {
let mut attempt = 0usize;
let mut timeout = profile.initial_wall_clock_timeout;
let mut previous_progress = (0usize, 0u64);
let mut zero_progress_attempts = 0usize;
let max_timeout = std::cmp::max(
profile.max_wall_clock_timeout,
profile.initial_wall_clock_timeout,
);
loop {
attempt += 1;
match self.run_rsync_once(src, dst, Some(timeout), true) {
Ok(()) => return Ok(()),
Err(err) => {
if is_hard_fail_rsync_error(&err) {
return Err(format!(
"rsync fail-fast hard-fail on attempt {}: {}",
attempt, err
));
}
if !err.contains("wall-clock timeout") {
return Err(err);
}
let progress = dir_progress(dst)
.map_err(|e| format!("rsync fail-fast progress stat failed: {e}"))?;
if progress == (0, 0) {
zero_progress_attempts += 1;
if zero_progress_attempts >= 2 || attempt >= profile.max_attempts {
return Err(format!(
"rsync fail-fast gave up after {} attempts with no progress: {}",
attempt, err
));
}
} else if progress == previous_progress {
return Err(format!(
"rsync fail-fast gave up after {} attempts with no additional progress: {}",
attempt, err
));
} else {
previous_progress = progress;
}
if attempt >= profile.max_attempts {
return Err(format!(
"rsync fail-fast exhausted {} attempts: {}",
profile.max_attempts, err
));
}
timeout = std::cmp::min(timeout.saturating_mul(2), max_timeout);
}
}
}
}
fn scope_fetch_uri(&self, rsync_base_uri: &str) -> String {
scoped_rsync_fetch_uri(self.config.scope_policy, rsync_base_uri)
}
}
impl RsyncFetcher for SystemRsyncFetcher {
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
let mut out = Vec::new();
self.visit_objects(rsync_base_uri, &mut |uri, bytes| {
out.push((uri, bytes));
Ok(())
})?;
Ok(out)
}
fn fetch_object(&self, rsync_uri: &str) -> RsyncFetchResult<Vec<u8>> {
let parsed =
url::Url::parse(rsync_uri).map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
if parsed.scheme() != "rsync" {
return Err(RsyncFetchError::Fetch(format!(
"not an rsync URI: {rsync_uri}"
)));
}
let file_name = parsed
.path_segments()
.and_then(|segments| segments.filter(|segment| !segment.is_empty()).next_back())
.ok_or_else(|| {
RsyncFetchError::Fetch(format!(
"rsync URI must reference a file object: {rsync_uri}"
))
})?;
let tmp = TempDir::new().map_err(RsyncFetchError::Fetch)?;
self.run_rsync(rsync_uri, tmp.path())
.map_err(RsyncFetchError::Fetch)?;
let object_path = tmp.path().join(file_name);
std::fs::read(&object_path).map_err(|e| {
RsyncFetchError::Fetch(format!(
"read fetched rsync object failed: {}: {e}",
object_path.display()
))
})
}
fn visit_objects(
&self,
rsync_base_uri: &str,
visitor: &mut dyn FnMut(String, Vec<u8>) -> Result<(), String>,
) -> RsyncFetchResult<(usize, u64)> {
let base = self.scope_fetch_uri(rsync_base_uri);
let mut count = 0usize;
let mut bytes_total = 0u64;
let mut wrapped = |uri: String, bytes: Vec<u8>| -> Result<(), String> {
bytes_total += bytes.len() as u64;
count += 1;
visitor(uri, bytes)
};
if let Some(dst) = self
.mirror_dst_dir(&base)
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?
{
self.run_rsync(&base, &dst)
.map_err(RsyncFetchError::Fetch)?;
walk_dir_visit(&dst, &dst, &base, &mut wrapped).map_err(RsyncFetchError::Fetch)?;
return Ok((count, bytes_total));
}
let tmp = TempDir::new().map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
self.run_rsync(&base, tmp.path())
.map_err(RsyncFetchError::Fetch)?;
walk_dir_visit(tmp.path(), tmp.path(), &base, &mut wrapped)
.map_err(RsyncFetchError::Fetch)?;
Ok((count, bytes_total))
}
fn dedup_key(&self, rsync_base_uri: &str) -> String {
self.scope_fetch_uri(rsync_base_uri)
}
fn failure_dedup_key(&self, rsync_base_uri: &str) -> Option<String> {
scoped_rsync_failure_dedup_key(self.config.scope_policy, rsync_base_uri)
}
}
/// Return the exact successful-fetch scope used by the live rsync fetcher.
///
/// The request scope is shared by live and replay transport paths so their
/// request keys and object projection scope remain equivalent.
pub fn scoped_rsync_fetch_uri(scope_policy: RsyncScopePolicy, rsync_base_uri: &str) -> String {
match scope_policy {
RsyncScopePolicy::Host | RsyncScopePolicy::PublicationPoint => {
normalize_rsync_base_uri(rsync_base_uri)
}
RsyncScopePolicy::ModuleRoot => rsync_module_root_uri(rsync_base_uri)
.unwrap_or_else(|| normalize_rsync_base_uri(rsync_base_uri)),
}
}
/// Return the live fetcher's failure-deduplication key for the configured
/// scope. This is part of the transport request identity, even though host
/// scope deliberately does not widen the successful fetch URI.
pub fn scoped_rsync_failure_dedup_key(
scope_policy: RsyncScopePolicy,
rsync_base_uri: &str,
) -> Option<String> {
match scope_policy {
RsyncScopePolicy::Host => rsync_host_scope_uri(rsync_base_uri),
RsyncScopePolicy::PublicationPoint | RsyncScopePolicy::ModuleRoot => None,
}
}
fn rsync_host_scope_uri(rsync_base_uri: &str) -> Option<String> {
let parsed = url::Url::parse(rsync_base_uri).ok()?;
if parsed.scheme() != "rsync" {
return None;
}
Some(format!("rsync://{}/", parsed.host_str()?))
}
struct TempDir {
path: PathBuf,
}
impl TempDir {
fn new() -> Result<Self, String> {
let mut p = std::env::temp_dir();
p.push(format!("rpki-system-rsync-{}", Uuid::new_v4()));
std::fs::create_dir_all(&p).map_err(|e| e.to_string())?;
Ok(Self { path: p })
}
fn path(&self) -> &Path {
&self.path
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.path);
}
}
fn rsync_module_root_uri(s: &str) -> Option<String> {
let normalized = normalize_rsync_base_uri(s);
let rest = normalized.strip_prefix("rsync://")?;
let mut host_and_path = rest.splitn(2, '/');
let authority = host_and_path.next()?;
let path = host_and_path.next()?;
let mut segments: Vec<&str> = path
.split('/')
.filter(|segment| !segment.is_empty())
.collect();
if segments.is_empty() {
return None;
}
let module = segments.remove(0);
Some(format!("rsync://{authority}/{module}/"))
}
fn walk_dir_collect(
root: &Path,
current: &Path,
rsync_base_uri: &str,
out: &mut Vec<(String, Vec<u8>)>,
) -> Result<(), String> {
let rd = std::fs::read_dir(current).map_err(|e| e.to_string())?;
for entry in rd {
let entry = entry.map_err(|e| e.to_string())?;
let path = entry.path();
let meta = entry.metadata().map_err(|e| e.to_string())?;
if meta.is_dir() {
walk_dir_collect(root, &path, rsync_base_uri, out)?;
continue;
}
if !meta.is_file() {
continue;
}
let rel = path
.strip_prefix(root)
.map_err(|e| e.to_string())?
.to_string_lossy()
.replace('\\', "/");
let uri = format!("{rsync_base_uri}{rel}");
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
out.push((uri, bytes));
}
Ok(())
}
fn dir_progress(root: &Path) -> Result<(usize, u64), String> {
if !root.exists() {
return Ok((0, 0));
}
let mut files = 0usize;
let mut bytes = 0u64;
let mut stack = vec![root.to_path_buf()];
while let Some(path) = stack.pop() {
let rd = std::fs::read_dir(&path).map_err(|e| e.to_string())?;
for entry in rd {
let entry = entry.map_err(|e| e.to_string())?;
let path = entry.path();
let meta = entry.metadata().map_err(|e| e.to_string())?;
if meta.is_dir() {
stack.push(path);
} else if meta.is_file() {
files += 1;
bytes += meta.len();
}
}
}
Ok((files, bytes))
}
fn is_hard_fail_rsync_error(msg: &str) -> bool {
let lower = msg.to_ascii_lowercase();
lower.contains("no route to host")
|| lower.contains("network is unreachable")
|| lower.contains("connection refused")
|| lower.contains("name or service not known")
}
fn walk_dir_visit(
root: &Path,
current: &Path,
rsync_base_uri: &str,
visitor: &mut dyn FnMut(String, Vec<u8>) -> Result<(), String>,
) -> Result<(), String> {
let rd = std::fs::read_dir(current).map_err(|e| e.to_string())?;
for entry in rd {
let entry = entry.map_err(|e| e.to_string())?;
let path = entry.path();
let meta = entry.metadata().map_err(|e| e.to_string())?;
if meta.is_dir() {
walk_dir_visit(root, &path, rsync_base_uri, visitor)?;
continue;
}
if !meta.is_file() {
continue;
}
let rel = path
.strip_prefix(root)
.map_err(|e| e.to_string())?
.to_string_lossy()
.replace('\\', "/");
let uri = format!("{rsync_base_uri}{rel}");
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
visitor(uri, bytes)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn normalize_rsync_base_uri_appends_slash_when_missing() {
assert_eq!(
normalize_rsync_base_uri("rsync://example.net/repo"),
"rsync://example.net/repo/".to_string()
);
assert_eq!(
normalize_rsync_base_uri("rsync://example.net/repo/"),
"rsync://example.net/repo/".to_string()
);
}
#[test]
fn walk_dir_collect_collects_files_and_normalizes_backslashes_in_uri() {
let temp = tempfile::tempdir().expect("tempdir");
let root = temp.path();
std::fs::create_dir_all(root.join("sub")).expect("mkdir");
std::fs::write(root.join("sub").join("a.cer"), b"x").expect("write");
std::fs::write(root.join("b\\c.mft"), b"y").expect("write backslash file");
let mut out: Vec<(String, Vec<u8>)> = Vec::new();
walk_dir_collect(root, root, "rsync://example.net/repo/", &mut out).expect("walk");
out.sort_by(|a, b| a.0.cmp(&b.0));
assert_eq!(out.len(), 2);
assert_eq!(out[0].0, "rsync://example.net/repo/b/c.mft");
assert_eq!(out[0].1, b"y");
assert_eq!(out[1].0, "rsync://example.net/repo/sub/a.cer");
assert_eq!(out[1].1, b"x");
}
#[test]
fn rsync_module_root_uri_returns_host_and_module_only() {
assert_eq!(
rsync_module_root_uri("rsync://example.net/repo/ta/ca/publication-point/"),
Some("rsync://example.net/repo/".to_string())
);
assert_eq!(
rsync_module_root_uri("rsync://example.net/repo/ta/"),
Some("rsync://example.net/repo/".to_string())
);
assert_eq!(
rsync_module_root_uri("rsync://example.net/repo/"),
Some("rsync://example.net/repo/".to_string())
);
assert_eq!(rsync_module_root_uri("https://example.net/repo"), None);
}
#[test]
fn rsync_host_scope_uri_returns_host_only() {
assert_eq!(
rsync_host_scope_uri("rsync://example.net/repo/ta/ca/publication-point/"),
Some("rsync://example.net/".to_string())
);
assert_eq!(rsync_host_scope_uri("https://example.net/repo"), None);
}
#[test]
fn system_rsync_dedup_key_uses_module_root_by_default() {
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig::default());
assert_eq!(
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
"rsync://example.net/repo/"
);
assert_eq!(
fetcher.failure_dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
None
);
}
#[test]
fn system_rsync_host_scope_does_not_widen_success_fetch_scope() {
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
scope_policy: RsyncScopePolicy::Host,
..SystemRsyncConfig::default()
});
assert_eq!(
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
"rsync://example.net/repo/ta/ca/publication-point/"
);
assert_eq!(
fetcher.scope_fetch_uri("rsync://example.net/repo/ta/ca/publication-point/"),
"rsync://example.net/repo/ta/ca/publication-point/"
);
assert_eq!(
fetcher.failure_dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
Some("rsync://example.net/".to_string())
);
}
#[test]
fn system_rsync_dedup_key_uses_publication_point_scope_when_configured() {
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
scope_policy: RsyncScopePolicy::PublicationPoint,
..SystemRsyncConfig::default()
});
assert_eq!(
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
"rsync://example.net/repo/ta/ca/publication-point/"
);
}
#[test]
fn system_rsync_dedup_key_uses_module_root_when_configured() {
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
scope_policy: RsyncScopePolicy::ModuleRoot,
..SystemRsyncConfig::default()
});
assert_eq!(
fetcher.dedup_key("rsync://example.net/repo/ta/ca/publication-point/"),
"rsync://example.net/repo/"
);
}
#[test]
fn system_rsync_fetcher_reports_spawn_and_exit_errors() {
let dst = tempfile::tempdir().expect("tempdir");
// 1) Spawn error.
let f = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: PathBuf::from("/this/does/not/exist/rsync"),
connect_timeout: Duration::from_secs(1),
timeout: Duration::from_secs(1),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
});
let e = f
.run_rsync("rsync://example.net/repo/", dst.path())
.expect_err("spawn must fail");
assert!(e.contains("rsync spawn failed:"), "{e}");
// 2) Non-zero exit status.
let f = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: PathBuf::from("false"),
connect_timeout: Duration::from_secs(1),
timeout: Duration::from_secs(1),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
});
let e = f
.run_rsync("rsync://example.net/repo/", dst.path())
.expect_err("false must fail");
assert!(e.contains("rsync failed:"), "{e}");
assert!(e.contains("status="), "{e}");
}
#[test]
fn mirror_dst_dir_reports_root_creation_error() {
let temp = tempfile::tempdir().expect("tempdir");
let root_file = temp.path().join("mirror-root-file");
std::fs::write(&root_file, b"not a directory").expect("write root file");
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: PathBuf::from("rsync"),
connect_timeout: Duration::from_secs(1),
timeout: Duration::from_secs(1),
extra_args: Vec::new(),
mirror_root: Some(root_file.clone()),
scope_policy: RsyncScopePolicy::default(),
});
let err = fetcher
.mirror_dst_dir("rsync://example.net/repo/")
.expect_err("file mirror root must fail");
assert!(err.contains("create rsync mirror root failed"), "{err}");
assert!(err.contains(&root_file.display().to_string()), "{err}");
}
#[cfg(unix)]
#[test]
fn mirror_dst_dir_reports_directory_creation_error_inside_root() {
use std::os::unix::fs::PermissionsExt;
let temp = tempfile::tempdir().expect("tempdir");
let root = temp.path().join("mirror");
std::fs::create_dir_all(&root).expect("mkdir root");
let mut perms = std::fs::metadata(&root).expect("metadata").permissions();
perms.set_mode(0o555);
std::fs::set_permissions(&root, perms).expect("chmod root readonly");
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: PathBuf::from("rsync"),
connect_timeout: Duration::from_secs(1),
timeout: Duration::from_secs(1),
extra_args: Vec::new(),
mirror_root: Some(root.clone()),
scope_policy: RsyncScopePolicy::default(),
});
let err = fetcher
.mirror_dst_dir("rsync://example.net/repo/")
.expect_err("readonly mirror root must fail");
assert!(
err.contains("create rsync mirror directory failed"),
"{err}"
);
let mut perms = std::fs::metadata(&root).expect("metadata").permissions();
perms.set_mode(0o755);
std::fs::set_permissions(&root, perms).expect("restore perms");
}
#[cfg(unix)]
#[test]
fn walk_dir_collect_ignores_non_file_entries() {
use std::os::unix::net::UnixListener;
let temp = tempfile::tempdir().expect("tempdir");
let root = temp.path();
std::fs::write(root.join("a.cer"), b"x").expect("write file");
let socket_path = root.join("skip.sock");
let _listener = UnixListener::bind(&socket_path).expect("bind socket");
let mut out: Vec<(String, Vec<u8>)> = Vec::new();
walk_dir_collect(root, root, "rsync://example.net/repo/", &mut out).expect("walk");
assert_eq!(out.len(), 1);
assert_eq!(out[0].0, "rsync://example.net/repo/a.cer");
}
#[cfg(unix)]
#[test]
fn rsync_fail_fast_retries_when_progress_is_made() {
use std::os::unix::fs::PermissionsExt;
let temp = tempfile::tempdir().expect("tempdir");
let script = temp.path().join("fake-rsync.sh");
let state = temp.path().join("state.txt");
std::fs::write(
&script,
format!(
"#!/usr/bin/env bash\nset -euo pipefail\nSTATE=\"{}\"\nDST=\"${{@: -1}}\"\nCOUNT=0\nif [[ -f \"$STATE\" ]]; then COUNT=$(cat \"$STATE\"); fi\nCOUNT=$((COUNT+1))\necho \"$COUNT\" > \"$STATE\"\nmkdir -p \"$DST\"\nif [[ \"$COUNT\" -eq 1 ]]; then\n echo first > \"$DST/part1\"\n sleep 2\nelse\n echo second > \"$DST/part2\"\nfi\n",
state.display()
),
)
.expect("write script");
let mut perms = std::fs::metadata(&script).unwrap().permissions();
perms.set_mode(0o755);
std::fs::set_permissions(&script, perms).unwrap();
let dst = temp.path().join("dst");
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: script,
connect_timeout: Duration::from_secs(15),
timeout: Duration::from_secs(60),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
});
fetcher
.run_rsync_fail_fast(
"rsync://example.net/repo/",
&dst,
RsyncFailFastProfile {
initial_wall_clock_timeout: Duration::from_secs(1),
max_wall_clock_timeout: Duration::from_secs(4),
max_attempts: 3,
},
)
.expect("eventual success");
assert!(dst.join("part1").exists());
assert!(dst.join("part2").exists());
}
#[cfg(unix)]
#[test]
fn rsync_fail_fast_gives_up_after_two_zero_progress_timeouts() {
use std::os::unix::fs::PermissionsExt;
let temp = tempfile::tempdir().expect("tempdir");
let script = temp.path().join("fake-rsync.sh");
std::fs::write(&script, "#!/usr/bin/env bash\nset -euo pipefail\nsleep 5\n")
.expect("write script");
let mut perms = std::fs::metadata(&script).unwrap().permissions();
perms.set_mode(0o755);
std::fs::set_permissions(&script, perms).unwrap();
let dst = temp.path().join("dst");
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: script,
connect_timeout: Duration::from_secs(15),
timeout: Duration::from_secs(60),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
});
let err = fetcher
.run_rsync_fail_fast(
"rsync://example.net/repo/",
&dst,
RsyncFailFastProfile {
initial_wall_clock_timeout: Duration::from_secs(1),
max_wall_clock_timeout: Duration::from_secs(2),
max_attempts: 4,
},
)
.expect_err("must fail");
assert!(err.contains("no progress"), "{err}");
}
#[cfg(unix)]
#[test]
fn rsync_fail_fast_hard_fail_stops_after_first_attempt() {
use std::os::unix::fs::PermissionsExt;
let temp = tempfile::tempdir().expect("tempdir");
let script = temp.path().join("fake-rsync.sh");
let state = temp.path().join("state.txt");
std::fs::write(
&script,
format!(
"#!/usr/bin/env bash\nset -euo pipefail\nSTATE=\"{}\"\nCOUNT=0\nif [[ -f \"$STATE\" ]]; then COUNT=$(cat \"$STATE\"); fi\nCOUNT=$((COUNT+1))\necho \"$COUNT\" > \"$STATE\"\necho 'rsync: [Receiver] failed to connect to host (1.2.3.4): Connection refused (111)' >&2\nexit 10\n",
state.display()
),
)
.expect("write script");
let mut perms = std::fs::metadata(&script).unwrap().permissions();
perms.set_mode(0o755);
std::fs::set_permissions(&script, perms).unwrap();
let dst = temp.path().join("dst");
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: script,
connect_timeout: Duration::from_secs(15),
timeout: Duration::from_secs(60),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
});
let err = fetcher
.run_rsync_fail_fast(
"rsync://example.net/repo/",
&dst,
RsyncFailFastProfile {
initial_wall_clock_timeout: Duration::from_secs(10),
max_wall_clock_timeout: Duration::from_secs(80),
max_attempts: 4,
},
)
.expect_err("must hard fail");
assert!(err.contains("hard-fail"), "{err}");
let count = std::fs::read_to_string(&state).unwrap();
assert_eq!(count.trim(), "1");
}
#[cfg(unix)]
#[test]
fn run_rsync_once_passes_contimeout_and_timeout_args() {
use std::os::unix::fs::PermissionsExt;
let temp = tempfile::tempdir().expect("tempdir");
let script = temp.path().join("capture-rsync.sh");
let args_file = temp.path().join("args.txt");
std::fs::write(
&script,
format!(
"#!/usr/bin/env bash\nset -euo pipefail\nprintf '%s\\n' \"$@\" > \"{}\"\nDST=\"${{@: -1}}\"\nmkdir -p \"$DST\"\n",
args_file.display()
),
)
.expect("write script");
let mut perms = std::fs::metadata(&script).unwrap().permissions();
perms.set_mode(0o755);
std::fs::set_permissions(&script, perms).unwrap();
let dst = temp.path().join("dst");
let fetcher = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: script,
connect_timeout: Duration::from_secs(15),
timeout: Duration::from_secs(30),
extra_args: Vec::new(),
mirror_root: None,
scope_policy: RsyncScopePolicy::default(),
});
fetcher
.run_rsync_once("rsync://example.net/repo/", &dst, None, false)
.expect("rsync");
let args = std::fs::read_to_string(&args_file).expect("read args");
assert!(args.contains("--contimeout\n15\n"), "{args}");
assert!(args.contains("--timeout\n30\n"), "{args}");
}
}

View File

@ -0,0 +1,54 @@
//! Public package boundary for the Panda RPKI synchronization validator.
//!
//! The package intentionally keeps the audited implementation in one crate
//! while exposing only the normal synchronization/validation CLI. A separate
//! offline revalidation workflow remains in the original private repository
//! until a follow-up backlog item defines its public contract.
//!
//! The implementation is extracted from the private `rpki` tree. During the
//! parity window we intentionally keep the upstream implementation's lint
//! profile unchanged; functional and output-equivalence gates take precedence
//! over behavior-changing lint cleanup.
#![allow(clippy::all)]
#![allow(clippy::pedantic, clippy::nursery, clippy::restriction)]
#![allow(dead_code, deprecated, unused_mut)]
pub mod analysis;
pub mod audit;
pub mod audit_downloads;
pub mod blob_store;
pub mod ccr;
pub mod cir;
pub mod cli;
pub mod contract;
pub mod crypto_sig_cache;
pub mod current_repo_index;
pub mod daemon;
pub mod data_model;
pub mod fetch;
pub mod memory_telemetry;
pub mod object_projection;
pub mod parallel;
pub mod policy;
pub mod progress_log;
pub mod replay;
pub mod report;
pub mod storage;
pub mod sync;
pub mod ta_constraints;
pub mod validation;
pub const COMPONENT_NAME: &str = "panda-rpki-validator";
pub const COMPONENT_VERSION: &str = env!("CARGO_PKG_VERSION");
/// Runs the normal synchronization/validation CLI.
pub fn run<I, S>(args: I) -> Result<(), String>
where
I: IntoIterator<Item = S>,
S: Into<String>,
{
let mut argv = Vec::with_capacity(1);
argv.push(COMPONENT_NAME.to_string());
argv.extend(args.into_iter().map(Into::into));
cli::run(&argv)
}

View File

@ -0,0 +1,6 @@
fn main() {
if let Err(error) = panda_rpki_validator::run(std::env::args().skip(1)) {
eprintln!("panda-rpki-validator: {error}");
std::process::exit(2);
}
}

View File

@ -0,0 +1,348 @@
use serde::Serialize;
use crate::storage::RocksDbMemorySnapshot;
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct MallocTrimProbe {
pub supported: bool,
pub return_value: Option<i32>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct ProcessMemorySnapshot {
pub label: String,
pub vm_rss_kb: Option<u64>,
pub vm_size_kb: Option<u64>,
pub vm_data_kb: Option<u64>,
pub vm_swap_kb: Option<u64>,
pub rss_anon_kb: Option<u64>,
pub rss_file_kb: Option<u64>,
pub rss_shmem_kb: Option<u64>,
pub threads: Option<u64>,
pub fd_count: Option<u64>,
pub smaps_rollup: Option<SmapsRollupSnapshot>,
pub smaps_mapping_summary: Option<SmapsMappingSummary>,
pub errors: Vec<String>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct SmapsRollupSnapshot {
pub rss_kb: Option<u64>,
pub pss_kb: Option<u64>,
pub shared_clean_kb: Option<u64>,
pub shared_dirty_kb: Option<u64>,
pub private_clean_kb: Option<u64>,
pub private_dirty_kb: Option<u64>,
pub anonymous_kb: Option<u64>,
pub swap_kb: Option<u64>,
pub swap_pss_kb: Option<u64>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct SmapsMappingSummary {
pub heap: SmapsMappingCategory,
pub anonymous_mmap: SmapsMappingCategory,
pub file_backed: SmapsMappingCategory,
pub stack: SmapsMappingCategory,
pub special: SmapsMappingCategory,
pub total: SmapsMappingCategory,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct SmapsMappingCategory {
pub mappings: u64,
pub size_kb: u64,
pub rss_kb: u64,
pub pss_kb: u64,
pub private_clean_kb: u64,
pub private_dirty_kb: u64,
pub anonymous_kb: u64,
pub largest_mapping_rss_kb: u64,
pub large_mapping_count_64m: u64,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct MemoryTelemetryCheckpoint {
pub label: String,
pub elapsed_ms: u64,
pub process: ProcessMemorySnapshot,
pub rocksdb: RocksDbMemorySnapshot,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct MemoryTelemetrySummary {
pub checkpoints: Vec<MemoryTelemetryCheckpoint>,
#[serde(skip_serializing_if = "Option::is_none")]
pub object_graph: Option<ObjectGraphMemorySummary>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub malloc_trim_probes: Vec<MallocTrimProbe>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct ObjectGraphMemorySummary {
pub captured_at_label: String,
pub total_estimated_bytes: u64,
pub sections: Vec<ObjectGraphMemorySection>,
pub notes: Vec<String>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct ObjectGraphMemorySection {
pub name: String,
pub item_count: u64,
pub shallow_bytes: u64,
pub heap_bytes: u64,
pub estimated_bytes: u64,
pub string_count: u64,
pub string_bytes: u64,
pub string_capacity_bytes: u64,
pub vec_count: u64,
pub vec_heap_bytes: u64,
pub vec_capacity_bytes: u64,
pub details: Vec<ObjectGraphMemoryMetric>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct ObjectGraphMemoryMetric {
pub name: String,
pub value: u64,
}
pub fn process_memory_snapshot(label: impl Into<String>) -> ProcessMemorySnapshot {
let label = label.into();
let mut snapshot = ProcessMemorySnapshot {
label,
vm_rss_kb: None,
vm_size_kb: None,
vm_data_kb: None,
vm_swap_kb: None,
rss_anon_kb: None,
rss_file_kb: None,
rss_shmem_kb: None,
threads: None,
fd_count: current_fd_count(),
smaps_rollup: None,
smaps_mapping_summary: None,
errors: Vec::new(),
};
match std::fs::read_to_string("/proc/self/status") {
Ok(status) => parse_status(&status, &mut snapshot),
Err(err) => snapshot
.errors
.push(format!("read /proc/self/status failed: {err}")),
}
match std::fs::read_to_string("/proc/self/smaps_rollup") {
Ok(smaps) => snapshot.smaps_rollup = Some(parse_smaps_rollup(&smaps)),
Err(err) => snapshot
.errors
.push(format!("read /proc/self/smaps_rollup failed: {err}")),
}
match std::fs::read_to_string("/proc/self/smaps") {
Ok(smaps) => snapshot.smaps_mapping_summary = Some(parse_smaps_mapping_summary(&smaps)),
Err(err) => snapshot
.errors
.push(format!("read /proc/self/smaps failed: {err}")),
}
snapshot
}
pub fn malloc_trim_probe() -> MallocTrimProbe {
#[cfg(all(target_os = "linux", target_env = "gnu"))]
{
MallocTrimProbe {
supported: true,
return_value: Some(unsafe { malloc_trim(0) }),
}
}
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
{
MallocTrimProbe {
supported: false,
return_value: None,
}
}
}
#[cfg(all(target_os = "linux", target_env = "gnu"))]
unsafe extern "C" {
fn malloc_trim(pad: usize) -> i32;
}
fn current_fd_count() -> Option<u64> {
std::fs::read_dir("/proc/self/fd")
.ok()
.map(|entries| entries.filter_map(Result::ok).count() as u64)
}
fn parse_status(status: &str, snapshot: &mut ProcessMemorySnapshot) {
for line in status.lines() {
let Some((key, value)) = line.split_once(':') else {
continue;
};
let parsed = parse_kb_or_plain_u64(value);
match key {
"VmRSS" => snapshot.vm_rss_kb = parsed,
"VmSize" => snapshot.vm_size_kb = parsed,
"VmData" => snapshot.vm_data_kb = parsed,
"VmSwap" => snapshot.vm_swap_kb = parsed,
"RssAnon" => snapshot.rss_anon_kb = parsed,
"RssFile" => snapshot.rss_file_kb = parsed,
"RssShmem" => snapshot.rss_shmem_kb = parsed,
"Threads" => snapshot.threads = parsed,
_ => {}
}
}
}
fn parse_smaps_rollup(smaps: &str) -> SmapsRollupSnapshot {
let mut snapshot = SmapsRollupSnapshot::default();
for line in smaps.lines() {
let Some((key, value)) = line.split_once(':') else {
continue;
};
let parsed = parse_kb_or_plain_u64(value);
match key {
"Rss" => snapshot.rss_kb = parsed,
"Pss" => snapshot.pss_kb = parsed,
"Shared_Clean" => snapshot.shared_clean_kb = parsed,
"Shared_Dirty" => snapshot.shared_dirty_kb = parsed,
"Private_Clean" => snapshot.private_clean_kb = parsed,
"Private_Dirty" => snapshot.private_dirty_kb = parsed,
"Anonymous" => snapshot.anonymous_kb = parsed,
"Swap" => snapshot.swap_kb = parsed,
"SwapPss" => snapshot.swap_pss_kb = parsed,
_ => {}
}
}
snapshot
}
fn parse_smaps_mapping_summary(smaps: &str) -> SmapsMappingSummary {
let mut summary = SmapsMappingSummary::default();
let mut current_path = String::new();
let mut current = SmapsMappingCategory::default();
let mut have_mapping = false;
for line in smaps.lines() {
if is_smaps_mapping_header(line) {
if have_mapping {
add_mapping(&mut summary, &current_path, &current);
}
current_path = smaps_header_path(line);
current = SmapsMappingCategory {
mappings: 1,
..SmapsMappingCategory::default()
};
have_mapping = true;
continue;
}
if !have_mapping {
continue;
}
let Some((key, value)) = line.split_once(':') else {
continue;
};
let parsed = parse_kb_or_plain_u64(value).unwrap_or(0);
match key {
"Size" => current.size_kb = parsed,
"Rss" => current.rss_kb = parsed,
"Pss" => current.pss_kb = parsed,
"Private_Clean" => current.private_clean_kb = parsed,
"Private_Dirty" => current.private_dirty_kb = parsed,
"Anonymous" => current.anonymous_kb = parsed,
_ => {}
}
}
if have_mapping {
add_mapping(&mut summary, &current_path, &current);
}
summary
}
fn is_smaps_mapping_header(line: &str) -> bool {
let mut parts = line.split_whitespace();
let Some(range) = parts.next() else {
return false;
};
let Some(perms) = parts.next() else {
return false;
};
let Some((start, end)) = range.split_once('-') else {
return false;
};
!start.is_empty()
&& !end.is_empty()
&& start.as_bytes().iter().all(u8::is_ascii_hexdigit)
&& end.as_bytes().iter().all(u8::is_ascii_hexdigit)
&& perms.len() == 4
&& perms
.as_bytes()
.iter()
.all(|b| matches!(b, b'r' | b'w' | b'x' | b's' | b'p' | b'-'))
}
fn smaps_header_path(line: &str) -> String {
line.split_whitespace()
.skip(5)
.collect::<Vec<_>>()
.join(" ")
}
fn add_mapping(summary: &mut SmapsMappingSummary, path: &str, mapping: &SmapsMappingCategory) {
add_category(&mut summary.total, mapping);
match mapping_category(path) {
MappingCategory::Heap => add_category(&mut summary.heap, mapping),
MappingCategory::AnonymousMmap => add_category(&mut summary.anonymous_mmap, mapping),
MappingCategory::FileBacked => add_category(&mut summary.file_backed, mapping),
MappingCategory::Stack => add_category(&mut summary.stack, mapping),
MappingCategory::Special => add_category(&mut summary.special, mapping),
}
}
fn add_category(target: &mut SmapsMappingCategory, source: &SmapsMappingCategory) {
target.mappings += source.mappings;
target.size_kb += source.size_kb;
target.rss_kb += source.rss_kb;
target.pss_kb += source.pss_kb;
target.private_clean_kb += source.private_clean_kb;
target.private_dirty_kb += source.private_dirty_kb;
target.anonymous_kb += source.anonymous_kb;
target.largest_mapping_rss_kb = target.largest_mapping_rss_kb.max(source.rss_kb);
if source.rss_kb >= 64 * 1024 {
target.large_mapping_count_64m += source.mappings;
}
}
enum MappingCategory {
Heap,
AnonymousMmap,
FileBacked,
Stack,
Special,
}
fn mapping_category(path: &str) -> MappingCategory {
if path == "[heap]" {
MappingCategory::Heap
} else if path.starts_with("[stack") {
MappingCategory::Stack
} else if path.is_empty() {
MappingCategory::AnonymousMmap
} else if path.starts_with('/') {
MappingCategory::FileBacked
} else {
MappingCategory::Special
}
}
fn parse_kb_or_plain_u64(value: &str) -> Option<u64> {
value.split_whitespace().next()?.parse::<u64>().ok()
}

View File

@ -0,0 +1,668 @@
use std::net::{Ipv4Addr, Ipv6Addr};
use std::path::Path;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use sha2::{Digest, Sha256};
use crate::data_model::aspa::AspaObject;
use crate::data_model::crl::RpkixCrl;
use crate::data_model::manifest::ManifestObject;
use crate::data_model::rc::{
AccessDescription, RcExtensions, ResourceCertificate, SubjectInfoAccess,
};
use crate::data_model::roa::{IpPrefix as RoaIpPrefix, RoaAfi, RoaObject};
use crate::data_model::signed_object::{
ResourceEeCertificate, RpkiSignedObject, SignedAttrsProfiled, SignerInfoProfiled,
};
use crate::data_model::ta::TaCertificate;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ObjectType {
Auto,
Cer,
Mft,
Crl,
Roa,
Aspa,
}
impl ObjectType {
pub fn parse(value: &str) -> Result<Self, String> {
match value.to_ascii_lowercase().as_str() {
"auto" => Ok(Self::Auto),
"cer" | ".cer" | "cert" | "certificate" => Ok(Self::Cer),
"mft" | ".mft" | "manifest" => Ok(Self::Mft),
"crl" | ".crl" => Ok(Self::Crl),
"roa" | ".roa" => Ok(Self::Roa),
"asa" | ".asa" | "aspa" => Ok(Self::Aspa),
_ => Err(format!("unsupported object type: {value}")),
}
}
pub fn label(self) -> &'static str {
object_type_label(self)
}
}
pub fn resolve_object_type(object_type: ObjectType, path: &Path) -> Result<ObjectType, String> {
if object_type != ObjectType::Auto {
return Ok(object_type);
}
match path
.extension()
.and_then(|v| v.to_str())
.map(|v| v.to_ascii_lowercase())
.as_deref()
{
Some("cer") => Ok(ObjectType::Cer),
Some("mft") => Ok(ObjectType::Mft),
Some("crl") => Ok(ObjectType::Crl),
Some("roa") => Ok(ObjectType::Roa),
Some("asa") | Some("aspa") => Ok(ObjectType::Aspa),
_ => Err(format!(
"cannot infer object type from path: {}",
path.display()
)),
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ObjectProjectionRecord {
pub schema_version: u32,
pub sha256: String,
pub object_type: String,
pub parse_status: String,
pub error_summary: Option<String>,
pub projection: Value,
}
pub fn build_object_projection(
object_type: ObjectType,
input_path: &Path,
bytes: &[u8],
entry_limit: usize,
) -> ObjectProjectionRecord {
let resolved = match resolve_object_type(object_type, input_path) {
Ok(value) => value,
Err(err) => {
return ObjectProjectionRecord {
schema_version: 1,
sha256: sha256_hex(bytes),
object_type: "unknown".to_string(),
parse_status: "error".to_string(),
error_summary: Some(err),
projection: json!({"decode": {"profileValid": false}}),
};
}
};
let projection = parse_object_json(resolved, input_path, bytes, entry_limit);
let parse_status = if projection
.get("object")
.and_then(|v| v.get("decode"))
.and_then(|v| v.get("profileValid"))
.and_then(Value::as_bool)
.unwrap_or(false)
{
"ok"
} else {
"error"
};
let error_summary = projection
.get("object")
.and_then(|v| v.get("decode"))
.and_then(|v| v.get("error"))
.and_then(Value::as_str)
.map(str::to_string);
ObjectProjectionRecord {
schema_version: 1,
sha256: sha256_hex(bytes),
object_type: resolved.label().to_string(),
parse_status: parse_status.to_string(),
error_summary,
projection,
}
}
pub fn parse_object_json(
object_type: ObjectType,
input_path: &Path,
bytes: &[u8],
entry_limit: usize,
) -> Value {
let object = match object_type {
ObjectType::Auto => unreachable!("auto must be resolved"),
ObjectType::Cer => parse_cer_json(bytes),
ObjectType::Mft => parse_mft_json(bytes, entry_limit),
ObjectType::Crl => parse_crl_json(bytes, entry_limit),
ObjectType::Roa => parse_roa_json(bytes, entry_limit),
ObjectType::Aspa => parse_aspa_json(bytes, entry_limit),
};
json!({
"tool": "rpki_object_parse",
"schemaVersion": 1,
"input": {
"path": input_path.display().to_string(),
"type": object_type_label(object_type),
"bytes": bytes_summary(bytes),
},
"object": object,
})
}
pub fn parse_cer_json(bytes: &[u8]) -> Value {
match ResourceCertificate::decode_der(bytes) {
Ok(cert) => {
let ta_profile = match TaCertificate::decode_der(bytes) {
Ok(ta) => json!({
"valid": true,
"selfSignature": result_json(ta.verify_self_signature().map_err(|e| e.to_string())),
}),
Err(err) => json!({
"valid": false,
"error": err.to_string(),
}),
};
json!({
"type": "cer",
"decode": {"profileValid": true},
"resourceCertificate": resource_certificate_json(&cert),
"trustAnchorProfile": ta_profile,
})
}
Err(err) => json!({
"type": "cer",
"decode": {"profileValid": false, "error": err.to_string()},
}),
}
}
pub fn parse_mft_json(bytes: &[u8], entry_limit: usize) -> Value {
match ManifestObject::decode_der(bytes) {
Ok(mft) => {
let files = mft.manifest.parse_files();
let (file_sample, file_list_error) = match files {
Ok(entries) => (
json!({
"count": entries.len(),
"truncated": entries.len() > entry_limit,
"entries": entries.iter().take(entry_limit).map(|item| {
json!({"fileName": item.file_name, "hashHex": hex::encode(item.hash_bytes)})
}).collect::<Vec<_>>(),
}),
Value::Null,
),
Err(err) => (Value::Null, json!(err.to_string())),
};
json!({
"type": "mft",
"decode": {"profileValid": true},
"eContentType": mft.econtent_type,
"signedObject": signed_object_json(&mft.signed_object),
"manifest": {
"version": mft.manifest.version,
"manifestNumberHex": mft.manifest.manifest_number.to_hex_upper(),
"thisUpdate": format_time(mft.manifest.this_update),
"nextUpdate": format_time(mft.manifest.next_update),
"fileHashAlg": mft.manifest.file_hash_alg,
"fileCount": mft.manifest.file_count(),
"fileList": file_sample,
"fileListError": file_list_error,
},
"embeddedEeProfile": result_json(mft.validate_embedded_ee_cert().map_err(|e| e.to_string())),
"cmsSignature": result_json(mft.signed_object.verify_signature().map_err(|e| e.to_string())),
})
}
Err(err) => json!({
"type": "mft",
"decode": {"profileValid": false, "error": err.to_string()},
}),
}
}
pub fn parse_crl_json(bytes: &[u8], entry_limit: usize) -> Value {
match RpkixCrl::decode_der(bytes) {
Ok(crl) => json!({
"type": "crl",
"decode": {"profileValid": true},
"rawDer": bytes_summary(&crl.raw_der),
"version": crl.version,
"issuer": crl.issuer_dn,
"signatureAlgorithm": crl.signature_algorithm_oid,
"thisUpdate": format_time(crl.this_update.utc),
"nextUpdate": format_time(crl.next_update.utc),
"extensions": {
"authorityKeyIdentifier": hex::encode(&crl.extensions.authority_key_identifier),
"crlNumberHex": crl.extensions.crl_number.to_hex_upper(),
"crlNumber": crl.extensions.crl_number.to_u64(),
},
"revokedCertificates": {
"count": crl.revoked_certs.len(),
"truncated": crl.revoked_certs.len() > entry_limit,
"entries": crl.revoked_certs.iter().take(entry_limit).map(|item| {
json!({
"serialNumberHex": item.serial_number.to_hex_upper(),
"serialNumber": item.serial_number.to_u64(),
"revocationDate": format_time(item.revocation_date.utc),
})
}).collect::<Vec<_>>(),
},
}),
Err(err) => json!({
"type": "crl",
"decode": {"profileValid": false, "error": err.to_string()},
}),
}
}
pub fn manifest_file_entries_page(
bytes: &[u8],
offset: usize,
limit: usize,
) -> Result<(usize, Vec<Value>), String> {
let mft = ManifestObject::decode_der(bytes).map_err(|err| err.to_string())?;
let entries = mft.manifest.parse_files().map_err(|err| err.to_string())?;
let total = entries.len();
let end = (offset + limit).min(total);
let page = entries[offset.min(total)..end]
.iter()
.map(|item| json!({"fileName": item.file_name, "hashHex": hex::encode(&item.hash_bytes)}))
.collect::<Vec<_>>();
Ok((total, page))
}
pub fn crl_revoked_entries_page(
bytes: &[u8],
offset: usize,
limit: usize,
) -> Result<(usize, Vec<Value>), String> {
let crl = RpkixCrl::decode_der(bytes).map_err(|err| err.to_string())?;
let total = crl.revoked_certs.len();
let end = (offset + limit).min(total);
let page = crl.revoked_certs[offset.min(total)..end]
.iter()
.map(|item| {
json!({
"serialNumberHex": item.serial_number.to_hex_upper(),
"serialNumber": item.serial_number.to_u64(),
"revocationDate": format_time(item.revocation_date.utc),
})
})
.collect::<Vec<_>>();
Ok((total, page))
}
pub fn parse_roa_json(bytes: &[u8], entry_limit: usize) -> Value {
match RoaObject::decode_der(bytes) {
Ok(roa) => json!({
"type": "roa",
"decode": {"profileValid": true},
"eContentType": roa.econtent_type,
"signedObject": signed_object_json(&roa.signed_object),
"roa": {
"version": roa.roa.version,
"asId": roa.roa.as_id,
"ipAddressFamilies": roa.roa.ip_addr_blocks.iter().map(|family| {
json!({
"afi": format!("{:?}", family.afi),
"addressCount": family.addresses.len(),
"truncated": family.addresses.len() > entry_limit,
"addresses": family.addresses.iter().take(entry_limit).map(|entry| {
json!({
"prefix": roa_prefix_string(&entry.prefix),
"maxLength": entry.max_length,
})
}).collect::<Vec<_>>(),
})
}).collect::<Vec<_>>(),
},
"embeddedEeProfile": result_json(roa.validate_embedded_ee_cert().map_err(|e| e.to_string())),
"cmsSignature": result_json(roa.signed_object.verify_signature().map_err(|e| e.to_string())),
}),
Err(err) => json!({
"type": "roa",
"decode": {"profileValid": false, "error": err.to_string()},
}),
}
}
pub fn parse_aspa_json(bytes: &[u8], entry_limit: usize) -> Value {
match AspaObject::decode_der(bytes) {
Ok(aspa) => json!({
"type": "aspa",
"decode": {"profileValid": true},
"eContentType": aspa.econtent_type,
"signedObject": signed_object_json(&aspa.signed_object),
"aspa": {
"version": aspa.aspa.version,
"customerAsId": aspa.aspa.customer_as_id,
"providerCount": aspa.aspa.provider_as_ids.len(),
"providersTruncated": aspa.aspa.provider_as_ids.len() > entry_limit,
"providerAsIds": aspa.aspa.provider_as_ids.iter().take(entry_limit).copied().collect::<Vec<_>>(),
},
"embeddedEeProfile": result_json(aspa.validate_embedded_ee_cert().map_err(|e| e.to_string())),
"cmsSignature": result_json(aspa.signed_object.verify_signature().map_err(|e| e.to_string())),
}),
Err(err) => json!({
"type": "aspa",
"decode": {"profileValid": false, "error": err.to_string()},
}),
}
}
fn resource_certificate_json(cert: &ResourceCertificate) -> Value {
let tbs = &cert.tbs;
json!({
"rawDer": bytes_summary(&cert.raw_der),
"kind": format!("{:?}", cert.kind),
"version": tbs.version,
"serialNumberHex": hex::encode(tbs.serial_number.to_bytes_be()),
"signatureAlgorithm": tbs.signature_algorithm,
"issuer": tbs.issuer_name.to_string(),
"subject": tbs.subject_name.to_string(),
"validity": {
"notBefore": format_time(tbs.validity_not_before),
"notAfter": format_time(tbs.validity_not_after),
},
"subjectPublicKeyInfo": bytes_summary(&tbs.subject_public_key_info),
"extensions": rc_extensions_json(&tbs.extensions),
})
}
fn rc_extensions_json(ext: &RcExtensions) -> Value {
json!({
"basicConstraintsCa": ext.basic_constraints_ca,
"subjectKeyIdentifier": ext.subject_key_identifier.as_ref().map(|v| hex::encode(v)),
"authorityKeyIdentifier": ext.authority_key_identifier.as_ref().map(|v| hex::encode(v)),
"crlDistributionPointsUris": ext.crl_distribution_points_uris,
"caIssuersUris": ext.ca_issuers_uris,
"subjectInfoAccess": subject_info_access_json(ext.subject_info_access.as_ref()),
"certificatePoliciesOid": ext.certificate_policies_oid,
"ipResources": serde_json::to_value(&ext.ip_resources).unwrap_or(Value::Null),
"asResources": serde_json::to_value(&ext.as_resources).unwrap_or(Value::Null),
})
}
fn subject_info_access_json(value: Option<&SubjectInfoAccess>) -> Value {
match value {
None => Value::Null,
Some(SubjectInfoAccess::Ca(ca)) => json!({
"kind": "ca",
"accessDescriptions": ca.access_descriptions.iter().map(access_description_json).collect::<Vec<_>>(),
}),
Some(SubjectInfoAccess::Ee(ee)) => json!({
"kind": "ee",
"signedObjectUris": ee.signed_object_uris,
"accessDescriptions": ee.access_descriptions.iter().map(access_description_json).collect::<Vec<_>>(),
}),
}
}
fn access_description_json(value: &AccessDescription) -> Value {
json!({
"accessMethodOid": value.access_method_oid,
"accessLocation": value.access_location,
})
}
fn signed_object_json(signed_object: &RpkiSignedObject) -> Value {
let signed_data = &signed_object.signed_data;
json!({
"rawDer": bytes_summary(&signed_object.raw_der),
"contentInfoContentType": signed_object.content_info_content_type,
"signedData": {
"version": signed_data.version,
"digestAlgorithms": signed_data.digest_algorithms,
"encapContentInfo": {
"eContentType": signed_data.encap_content_info.econtent_type,
"eContent": bytes_summary(&signed_data.encap_content_info.econtent),
},
"certificates": signed_data.certificates.iter().map(ee_certificate_json).collect::<Vec<_>>(),
"crlsPresent": signed_data.crls_present,
"signerInfos": signed_data.signer_infos.iter().map(signer_info_json).collect::<Vec<_>>(),
},
})
}
fn ee_certificate_json(cert: &ResourceEeCertificate) -> Value {
json!({
"rawDer": bytes_summary(&cert.raw_der),
"subjectKeyIdentifier": hex::encode(&cert.subject_key_identifier),
"spkiDer": bytes_summary(&cert.spki_der),
"rsaPublicKey": {
"modulus": bytes_summary(&cert.rsa_public_modulus),
"exponent": bytes_summary(&cert.rsa_public_exponent),
},
"tbsCertificate": bytes_summary(&cert.tbs_certificate_der),
"certificateSignature": bytes_summary(&cert.signature_bytes),
"keyUsageSummary": format!("{:?}", cert.key_usage_summary),
"siaSignedObjectUris": cert.sia_signed_object_uris,
"resourceCertificate": resource_certificate_json(&cert.resource_cert),
})
}
fn signer_info_json(info: &SignerInfoProfiled) -> Value {
json!({
"version": info.version,
"sidSki": hex::encode(&info.sid_ski),
"digestAlgorithm": info.digest_algorithm,
"signatureAlgorithm": info.signature_algorithm,
"signedAttrs": signed_attrs_json(&info.signed_attrs),
"unsignedAttrsPresent": info.unsigned_attrs_present,
"signature": bytes_summary(&info.signature),
"signedAttrsDerForSignature": bytes_summary(&info.signed_attrs_der_for_signature),
})
}
fn signed_attrs_json(attrs: &SignedAttrsProfiled) -> Value {
json!({
"contentType": attrs.content_type,
"messageDigest": hex::encode(&attrs.message_digest),
"signingTime": {
"utc": format_time(attrs.signing_time.utc),
"encoding": format!("{:?}", attrs.signing_time.encoding),
},
"otherAttrsPresent": attrs.other_attrs_present,
})
}
fn result_json(result: Result<(), String>) -> Value {
match result {
Ok(()) => json!({"valid": true}),
Err(err) => json!({"valid": false, "error": err}),
}
}
fn object_type_label(object_type: ObjectType) -> &'static str {
match object_type {
ObjectType::Auto => "auto",
ObjectType::Cer => "cer",
ObjectType::Mft => "mft",
ObjectType::Crl => "crl",
ObjectType::Roa => "roa",
ObjectType::Aspa => "aspa",
}
}
fn bytes_summary(bytes: &[u8]) -> Value {
let head_len = bytes.len().min(16);
let tail_len = bytes.len().min(16);
json!({
"len": bytes.len(),
"sha256": sha256_hex(bytes),
"headHex": hex::encode(&bytes[..head_len]),
"tailHex": hex::encode(&bytes[bytes.len().saturating_sub(tail_len)..]),
})
}
fn sha256_hex(bytes: &[u8]) -> String {
hex::encode(Sha256::digest(bytes))
}
fn format_time(value: time::OffsetDateTime) -> String {
value
.to_offset(time::UtcOffset::UTC)
.format(&time::format_description::well_known::Rfc3339)
.unwrap_or_else(|_| value.unix_timestamp().to_string())
}
fn roa_prefix_string(prefix: &RoaIpPrefix) -> String {
let bytes = prefix.addr_bytes();
match prefix.afi {
RoaAfi::Ipv4 => {
let octets = [bytes[0], bytes[1], bytes[2], bytes[3]];
format!("{}/{}", Ipv4Addr::from(octets), prefix.prefix_len)
}
RoaAfi::Ipv6 => {
let mut octets = [0u8; 16];
octets.copy_from_slice(bytes);
format!("{}/{}", Ipv6Addr::from(octets), prefix.prefix_len)
}
}
}
#[cfg(test)]
mod tests {
use std::path::Path;
use super::*;
#[test]
fn object_type_parser_and_resolver_cover_aliases() {
assert_eq!(ObjectType::parse("auto").unwrap(), ObjectType::Auto);
assert_eq!(ObjectType::parse(".cer").unwrap(), ObjectType::Cer);
assert_eq!(ObjectType::parse("certificate").unwrap(), ObjectType::Cer);
assert_eq!(ObjectType::parse("manifest").unwrap(), ObjectType::Mft);
assert_eq!(ObjectType::parse(".crl").unwrap(), ObjectType::Crl);
assert_eq!(ObjectType::parse("roa").unwrap(), ObjectType::Roa);
assert_eq!(ObjectType::parse("aspa").unwrap(), ObjectType::Aspa);
assert_eq!(ObjectType::parse(".asa").unwrap(), ObjectType::Aspa);
assert!(ObjectType::parse("unknown").is_err());
assert_eq!(ObjectType::Aspa.label(), "aspa");
assert_eq!(
resolve_object_type(ObjectType::Auto, Path::new("repo/a.cer")).unwrap(),
ObjectType::Cer
);
assert_eq!(
resolve_object_type(ObjectType::Auto, Path::new("repo/a.mft")).unwrap(),
ObjectType::Mft
);
assert_eq!(
resolve_object_type(ObjectType::Auto, Path::new("repo/a.crl")).unwrap(),
ObjectType::Crl
);
assert_eq!(
resolve_object_type(ObjectType::Auto, Path::new("repo/a.roa")).unwrap(),
ObjectType::Roa
);
assert_eq!(
resolve_object_type(ObjectType::Auto, Path::new("repo/a.asa")).unwrap(),
ObjectType::Aspa
);
assert_eq!(
resolve_object_type(ObjectType::Roa, Path::new("repo/a.bin")).unwrap(),
ObjectType::Roa
);
assert!(resolve_object_type(ObjectType::Auto, Path::new("repo/a.bin")).is_err());
}
#[test]
fn invalid_der_returns_error_projection_for_all_object_types() {
let bytes = b"not der";
for object_type in [
ObjectType::Cer,
ObjectType::Mft,
ObjectType::Crl,
ObjectType::Roa,
ObjectType::Aspa,
] {
let value = parse_object_json(object_type, Path::new("bad.der"), bytes, 1);
assert_eq!(
value["object"]["decode"]["profileValid"].as_bool(),
Some(false)
);
assert!(value["object"]["decode"]["error"].as_str().is_some());
}
let record = build_object_projection(ObjectType::Auto, Path::new("bad.bin"), bytes, 1);
assert_eq!(record.object_type, "unknown");
assert_eq!(record.parse_status, "error");
assert!(record.error_summary.is_some());
}
#[test]
fn parses_fixture_objects_into_human_readable_projection() {
let cases = [
(
ObjectType::Cer,
"tests/fixtures/ta/apnic-ta.cer",
"cer",
"resourceCertificate",
),
(
ObjectType::Mft,
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft",
"mft",
"manifest",
),
(
ObjectType::Crl,
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.crl",
"crl",
"revokedCertificates",
),
(
ObjectType::Roa,
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/AS4538.roa",
"roa",
"roa",
),
(
ObjectType::Aspa,
"tests/fixtures/repository/chloe.sobornost.net/rpki/RIPE-nljobsnijders/5m80fwYws_3FiFD7JiQjAqZ1RYQ.asa",
"aspa",
"aspa",
),
];
for (object_type, path, expected_type, expected_section) in cases {
let bytes = std::fs::read(path).expect("fixture");
let record = build_object_projection(object_type, Path::new(path), &bytes, 1);
assert_eq!(record.object_type, expected_type);
assert_eq!(record.parse_status, "ok");
assert_eq!(
record.projection["object"]["decode"]["profileValid"].as_bool(),
Some(true)
);
assert!(record.projection["object"][expected_section].is_object());
}
}
#[test]
fn large_projection_lists_are_paged_from_raw_bytes() {
let mft_bytes = std::fs::read(
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.mft",
)
.expect("mft");
let (total, page) = manifest_file_entries_page(&mft_bytes, 1, 3).expect("mft page");
assert!(total >= 3);
assert_eq!(page.len(), 3);
assert!(page[0]["fileName"].as_str().is_some());
let (_, empty_page) =
manifest_file_entries_page(&mft_bytes, total + 10, 3).expect("empty page");
assert!(empty_page.is_empty());
let crl_bytes = std::fs::read(
"tests/fixtures/repository/rpki.cernet.net/repo/cernet/0/05FC9C5B88506F7C0D3F862C8895BED67E9F8EBA.crl",
)
.expect("crl");
let (total, page) = crl_revoked_entries_page(&crl_bytes, 0, 5).expect("crl page");
assert!(page.len() <= total);
let (_, empty_page) =
crl_revoked_entries_page(&crl_bytes, total + 10, 5).expect("empty crl page");
assert!(empty_page.is_empty());
}
}

View File

@ -0,0 +1,79 @@
use crate::parallel::dead_repo_blacklist::DeadRepoBlacklistConfig;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ParallelPhase1Config {
pub max_repo_sync_workers_global: usize,
pub max_inflight_snapshot_bytes_global: usize,
pub max_pending_repo_results: usize,
/// Dead-repo transport blacklist (#141). `None` disables the feature
/// entirely (default, behavior unchanged).
pub dead_repo_blacklist: Option<DeadRepoBlacklistConfig>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ParallelPhase2Config {
pub object_workers: usize,
pub worker_queue_capacity: usize,
pub ready_batch_size: usize,
pub ready_batch_wall_time_budget_ms: u64,
pub object_result_drain_batch_size: usize,
pub publication_point_finalize_batch_size: usize,
pub publication_point_finalize_wall_time_budget_ms: u64,
pub publication_point_finalize_queue_capacity: usize,
/// Experimental: workers of the ready publication point stage pool. `0`
/// disables the pool and keeps the inline compute+apply staging path.
pub stage_workers: usize,
}
impl Default for ParallelPhase2Config {
fn default() -> Self {
Self {
object_workers: 8,
worker_queue_capacity: 256,
ready_batch_size: 256,
ready_batch_wall_time_budget_ms: 100,
object_result_drain_batch_size: 2048,
publication_point_finalize_batch_size: 256,
publication_point_finalize_wall_time_budget_ms: 100,
publication_point_finalize_queue_capacity: 32768,
stage_workers: 0,
}
}
}
impl Default for ParallelPhase1Config {
fn default() -> Self {
Self {
max_repo_sync_workers_global: 4,
max_inflight_snapshot_bytes_global: 512 * 1024 * 1024,
max_pending_repo_results: 1024,
dead_repo_blacklist: None,
}
}
}
#[cfg(test)]
mod tests {
use super::{ParallelPhase1Config, ParallelPhase2Config};
#[test]
fn default_parallel_phase1_config_is_bounded() {
let cfg = ParallelPhase1Config::default();
assert!(cfg.max_repo_sync_workers_global > 0);
assert!(cfg.max_inflight_snapshot_bytes_global > 0);
assert!(cfg.max_pending_repo_results > 0);
}
#[test]
fn default_parallel_phase2_config_is_bounded() {
let cfg = ParallelPhase2Config::default();
assert!(cfg.object_workers > 0);
assert!(cfg.worker_queue_capacity > 0);
assert!(cfg.ready_batch_size > 0);
assert!(cfg.ready_batch_wall_time_budget_ms > 0);
assert!(cfg.object_result_drain_batch_size > 0);
assert!(cfg.publication_point_finalize_batch_size > 0);
assert!(cfg.publication_point_finalize_wall_time_budget_ms > 0);
assert!(cfg.publication_point_finalize_queue_capacity > 0);
}
}

View File

@ -0,0 +1,391 @@
//! Persistent dead-repo transport blacklist (#141).
//!
//! Tracks per-(repo, transport) consecutive transport-layer fetch failures
//! across runs. Once an entry reaches the configured failure threshold it is
//! admitted to the blacklist; the live transport scheduler then skips the
//! dead transport (rrdp-blacklisted -> straight to rsync, dual-blacklisted ->
//! immediate terminal failure). Entries are removed either by a successful
//! fetch (self-heal) or by the daemon-side health probe.
use std::collections::HashMap;
use std::fs;
use std::io;
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
use crate::parallel::types::{RepoTransportErrorClass, RepoTransportMode};
pub const DEAD_REPO_BLACKLIST_SCHEMA_VERSION: u32 = 1;
pub const DEAD_REPO_BLACKLIST_DEFAULT_CAPACITY: usize = 256;
/// Probe backoff is capped at interval * 2^PROBE_BACKOFF_CAP_SHIFT.
pub const PROBE_BACKOFF_CAP_SHIFT: u32 = 4;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct DeadRepoBlacklistConfig {
pub path: PathBuf,
pub fail_threshold: u32,
pub capacity: usize,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct DeadRepoBlacklistEntry {
pub transport: RepoTransportMode,
pub uri: String,
/// Consecutive runs with transport-class fetch failure. Reset on success.
pub consecutive_failures: u32,
/// True once `consecutive_failures` reached the admission threshold.
#[serde(default)]
pub blacklisted: bool,
#[serde(default)]
pub first_failure_at_unix: u64,
#[serde(default)]
pub last_failure_at_unix: u64,
/// When the entry was admitted to the blacklist.
#[serde(default)]
pub added_at_unix: u64,
#[serde(default)]
pub last_probe_at_unix: Option<u64>,
#[serde(default)]
pub probe_failures: u32,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum DeadRepoFailureOutcome {
/// Failure counted, entry not (or not yet) blacklisted.
Counted { consecutive_failures: u32 },
/// Entry just reached the threshold and was admitted.
Admitted,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum DeadRepoSuccessOutcome {
/// A counting (not yet admitted) entry was reset.
CounterReset,
/// A blacklisted entry was removed (repo revived).
BlacklistRemoved,
}
#[derive(Serialize, Deserialize)]
struct DeadRepoBlacklistFile {
schema_version: u32,
updated_at_unix: u64,
#[serde(default)]
entries: Vec<DeadRepoBlacklistEntry>,
}
/// In-memory blacklist state for one writer at a time (child run, or daemon
/// supervisor between runs). Lookups go through a HashMap keyed by
/// (transport, uri).
#[derive(Clone, Debug, Default)]
pub struct DeadRepoBlacklist {
entries: HashMap<(RepoTransportMode, String), DeadRepoBlacklistEntry>,
}
impl DeadRepoBlacklist {
pub fn new() -> Self {
Self::default()
}
/// Load from disk. A missing file or an unreadable/incompatible file
/// degrades to an empty blacklist (logged by the caller via `warning`).
pub fn load(path: &Path) -> (Self, Option<String>) {
let bytes = match fs::read(path) {
Ok(bytes) => bytes,
Err(err) if err.kind() == io::ErrorKind::NotFound => return (Self::new(), None),
Err(err) => {
return (
Self::new(),
Some(format!(
"dead repo blacklist {} unreadable ({err}); starting empty",
path.display()
)),
);
}
};
match serde_json::from_slice::<DeadRepoBlacklistFile>(&bytes) {
Ok(file) if file.schema_version == DEAD_REPO_BLACKLIST_SCHEMA_VERSION => {
let entries = file
.entries
.into_iter()
.map(|entry| ((entry.transport, entry.uri.clone()), entry))
.collect();
(Self { entries }, None)
}
Ok(file) => (
Self::new(),
Some(format!(
"dead repo blacklist {} schema_version {} != {}; starting empty",
path.display(),
file.schema_version,
DEAD_REPO_BLACKLIST_SCHEMA_VERSION
)),
),
Err(err) => (
Self::new(),
Some(format!(
"dead repo blacklist {} unparsable ({err}); starting empty",
path.display()
)),
),
}
}
/// Persist atomically (tmp file + rename).
pub fn store_atomic(&self, path: &Path, now_unix: u64) -> io::Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let mut entries: Vec<&DeadRepoBlacklistEntry> = self.entries.values().collect();
entries.sort_by(|a, b| a.uri.cmp(&b.uri).then(a.transport.cmp(&b.transport)));
let file = DeadRepoBlacklistFile {
schema_version: DEAD_REPO_BLACKLIST_SCHEMA_VERSION,
updated_at_unix: now_unix,
entries: entries.into_iter().cloned().collect(),
};
let bytes = serde_json::to_vec_pretty(&file)
.map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err))?;
let mut tmp_name = path
.file_name()
.map(|name| name.to_os_string())
.unwrap_or_default();
tmp_name.push(".tmp");
let tmp_path = path.with_file_name(tmp_name);
fs::write(&tmp_path, &bytes)?;
// Best-effort durability before the rename.
if let Ok(file) = fs::File::open(&tmp_path) {
let _ = file.sync_all();
}
fs::rename(&tmp_path, path)?;
Ok(())
}
pub fn is_blacklisted(&self, transport: RepoTransportMode, uri: &str) -> bool {
self.entries
.get(&(transport, uri.to_string()))
.map(|entry| entry.blacklisted)
.unwrap_or(false)
}
/// Record a transport-class fetch failure. Returns whether the failure
/// newly admitted the entry to the blacklist.
pub fn record_transport_failure(
&mut self,
transport: RepoTransportMode,
uri: &str,
now_unix: u64,
fail_threshold: u32,
capacity: usize,
) -> DeadRepoFailureOutcome {
let key = (transport, uri.to_string());
if !self.entries.contains_key(&key) {
self.evict_if_full(capacity);
}
let threshold = fail_threshold.max(1);
let entry = self
.entries
.entry(key)
.or_insert_with(|| DeadRepoBlacklistEntry {
transport,
uri: uri.to_string(),
consecutive_failures: 0,
blacklisted: false,
first_failure_at_unix: now_unix,
last_failure_at_unix: now_unix,
added_at_unix: 0,
last_probe_at_unix: None,
probe_failures: 0,
});
entry.consecutive_failures = entry.consecutive_failures.saturating_add(1);
entry.last_failure_at_unix = now_unix;
if entry.first_failure_at_unix == 0 {
entry.first_failure_at_unix = now_unix;
}
if !entry.blacklisted && entry.consecutive_failures >= threshold {
entry.blacklisted = true;
entry.added_at_unix = now_unix;
// A fresh admission restarts probe bookkeeping.
entry.last_probe_at_unix = None;
entry.probe_failures = 0;
return DeadRepoFailureOutcome::Admitted;
}
DeadRepoFailureOutcome::Counted {
consecutive_failures: entry.consecutive_failures,
}
}
/// Record a successful fetch: reset a counting entry, or remove a
/// blacklisted entry entirely (self-heal).
pub fn record_transport_success(
&mut self,
transport: RepoTransportMode,
uri: &str,
) -> Option<DeadRepoSuccessOutcome> {
match self.entries.remove(&(transport, uri.to_string())) {
Some(entry) if entry.blacklisted => Some(DeadRepoSuccessOutcome::BlacklistRemoved),
Some(_) => Some(DeadRepoSuccessOutcome::CounterReset),
None => None,
}
}
/// Blacklisted entries whose next probe is due at `now_unix`.
pub fn probe_due_entries(
&self,
now_unix: u64,
probe_interval_secs: u64,
) -> Vec<DeadRepoBlacklistEntry> {
let mut due: Vec<DeadRepoBlacklistEntry> = self
.entries
.values()
.filter(|entry| entry.blacklisted)
.filter(|entry| {
let interval = probe_backoff_secs(probe_interval_secs, entry.probe_failures);
match entry.last_probe_at_unix {
None => true,
Some(last) => now_unix.saturating_sub(last) >= interval,
}
})
.cloned()
.collect();
due.sort_by(|a, b| a.uri.cmp(&b.uri));
due
}
/// Probe succeeded: remove the entry. Returns true if it was present.
pub fn record_probe_success(&mut self, transport: RepoTransportMode, uri: &str) -> bool {
self.entries.remove(&(transport, uri.to_string())).is_some()
}
pub fn record_probe_failure(&mut self, transport: RepoTransportMode, uri: &str, now_unix: u64) {
if let Some(entry) = self.entries.get_mut(&(transport, uri.to_string())) {
entry.last_probe_at_unix = Some(now_unix);
entry.probe_failures = entry.probe_failures.saturating_add(1);
}
}
pub fn len(&self) -> usize {
self.entries.len()
}
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
pub fn blacklisted_len(&self) -> usize {
self.entries
.values()
.filter(|entry| entry.blacklisted)
.count()
}
/// Deterministic snapshot of all entries (for status/observability).
pub fn entries_sorted(&self) -> Vec<DeadRepoBlacklistEntry> {
let mut entries: Vec<DeadRepoBlacklistEntry> = self.entries.values().cloned().collect();
entries.sort_by(|a, b| a.uri.cmp(&b.uri).then(a.transport.cmp(&b.transport)));
entries
}
fn evict_if_full(&mut self, capacity: usize) {
let capacity = capacity.max(1);
if self.entries.len() < capacity {
return;
}
// Prefer evicting the stalest counting entry; fall back to the oldest
// blacklisted entry.
let victim = self
.entries
.values()
.filter(|entry| !entry.blacklisted)
.min_by_key(|entry| entry.last_failure_at_unix)
.map(|entry| (entry.transport, entry.uri.clone()))
.or_else(|| {
self.entries
.values()
.min_by_key(|entry| entry.added_at_unix)
.map(|entry| (entry.transport, entry.uri.clone()))
});
if let Some(key) = victim {
self.entries.remove(&key);
}
}
}
pub fn probe_backoff_secs(base_interval_secs: u64, probe_failures: u32) -> u64 {
// First retry after a probe failure still uses the base interval;
// exponential backoff kicks in from the second consecutive failure.
let shift = probe_failures
.saturating_sub(1)
.min(PROBE_BACKOFF_CAP_SHIFT);
base_interval_secs.saturating_mul(1u64 << shift)
}
/// Classify an HTTP fetch error string (formats from
/// `fetch/http.rs::BlockingHttpFetcher::fetch_bytes`).
pub fn classify_http_fetch_error(detail: &str) -> RepoTransportErrorClass {
if detail.starts_with("http status") {
// The peer answered: transport is fine, the failure is protocol-level.
RepoTransportErrorClass::Protocol
} else if detail.starts_with("http request failed")
|| detail.starts_with("http read body failed")
{
RepoTransportErrorClass::TransportFetch
} else {
RepoTransportErrorClass::Unknown
}
}
/// Classify an rsync fetch error string using the same host-level substring
/// set as the run-local rsync failure scope short-circuit.
pub fn classify_rsync_fetch_error(detail: &str) -> RepoTransportErrorClass {
if crate::parallel::repo_scheduler::is_host_level_rsync_failure(detail)
// Stall failures are transport death too: the fail-fast mechanism
// gives up when wall-clock windows pass with no (additional) progress
// (#141 M9). Content-level errors fail immediately and never carry
// this marker, so they still do not count.
|| detail.contains("rsync fail-fast gave up")
{
RepoTransportErrorClass::TransportFetch
} else {
// Content/protocol-level rsync failures (digest mismatch, vanished
// files, partial transfers) never count toward the blacklist.
RepoTransportErrorClass::Unknown
}
}
/// Map a structured RRDP sync error to a transport error class.
pub fn classify_rrdp_sync_error(err: &crate::sync::rrdp::RrdpSyncError) -> RepoTransportErrorClass {
match err {
crate::sync::rrdp::RrdpSyncError::Fetch(detail) => classify_http_fetch_error(detail),
crate::sync::rrdp::RrdpSyncError::Rrdp(_) => RepoTransportErrorClass::Protocol,
crate::sync::rrdp::RrdpSyncError::Storage(_) => RepoTransportErrorClass::Storage,
}
}
/// Map a structured rsync-side repo sync error to a transport error class.
pub fn classify_rsync_repo_sync_error(
err: &crate::sync::repo::RepoSyncError,
) -> RepoTransportErrorClass {
match err {
crate::sync::repo::RepoSyncError::Rsync(crate::fetch::rsync::RsyncFetchError::Fetch(
detail,
)) => classify_rsync_fetch_error(detail),
crate::sync::repo::RepoSyncError::Storage(_) => RepoTransportErrorClass::Storage,
_ => RepoTransportErrorClass::Protocol,
}
}
impl RepoTransportMode {
pub fn as_str(self) -> &'static str {
match self {
RepoTransportMode::Rrdp => "rrdp",
RepoTransportMode::Rsync => "rsync",
}
}
}
// Keep the public surface free of internal key types; the (transport, uri)
// tuple key is an implementation detail of the lookup map.
#[cfg(test)]
#[path = "dead_repo_blacklist_tests.rs"]
mod tests;

View File

@ -0,0 +1,430 @@
use super::*;
use crate::parallel::types::RepoTransportMode;
const RRDP_URI: &str = "https://dead.example.com/notification.xml";
const RSYNC_URI: &str = "rsync://dead.example.com/repo/";
fn now() -> u64 {
1_700_000_000
}
#[test]
fn admission_requires_threshold_consecutive_failures() {
let mut bl = DeadRepoBlacklist::new();
assert!(!bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
let out1 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
assert_eq!(
out1,
DeadRepoFailureOutcome::Counted {
consecutive_failures: 1
}
);
assert!(!bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
let out2 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 1, 3, 256);
assert_eq!(
out2,
DeadRepoFailureOutcome::Counted {
consecutive_failures: 2
}
);
assert!(!bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
let out3 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 2, 3, 256);
assert_eq!(out3, DeadRepoFailureOutcome::Admitted);
assert!(bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
// Further failures keep counting but do not re-admit.
let out4 = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 3, 3, 256);
assert!(matches!(
out4,
DeadRepoFailureOutcome::Counted {
consecutive_failures: 4
}
));
}
#[test]
fn threshold_of_one_admits_immediately() {
let mut bl = DeadRepoBlacklist::new();
let out = bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now(), 1, 256);
assert_eq!(out, DeadRepoFailureOutcome::Admitted);
}
#[test]
fn success_resets_counting_entry() {
let mut bl = DeadRepoBlacklist::new();
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
assert_eq!(bl.len(), 1);
let out = bl.record_transport_success(RepoTransportMode::Rrdp, RRDP_URI);
assert_eq!(out, Some(DeadRepoSuccessOutcome::CounterReset));
assert!(bl.is_empty());
// Next failure starts from zero again.
let out = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
assert_eq!(
out,
DeadRepoFailureOutcome::Counted {
consecutive_failures: 1
}
);
}
#[test]
fn success_on_blacklisted_entry_self_heals() {
let mut bl = DeadRepoBlacklist::new();
for i in 0..3 {
bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now() + i, 3, 256);
}
assert!(bl.is_blacklisted(RepoTransportMode::Rsync, RSYNC_URI));
let out = bl.record_transport_success(RepoTransportMode::Rsync, RSYNC_URI);
assert_eq!(out, Some(DeadRepoSuccessOutcome::BlacklistRemoved));
assert!(!bl.is_blacklisted(RepoTransportMode::Rsync, RSYNC_URI));
assert!(bl.is_empty());
}
#[test]
fn success_on_unknown_entry_is_none() {
let mut bl = DeadRepoBlacklist::new();
assert_eq!(
bl.record_transport_success(RepoTransportMode::Rrdp, RRDP_URI),
None
);
}
#[test]
fn entries_are_per_transport() {
let mut bl = DeadRepoBlacklist::new();
for i in 0..3 {
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
}
assert!(bl.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
assert!(!bl.is_blacklisted(RepoTransportMode::Rsync, RRDP_URI));
}
#[test]
fn capacity_evicts_stale_counting_entries_first() {
let mut bl = DeadRepoBlacklist::new();
// One blacklisted entry (old) and two counting entries.
for i in 0..3 {
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://a.example/n.xml",
100 + i,
1,
3,
);
}
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://b.example/n.xml",
200,
3,
3,
);
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://c.example/n.xml",
300,
3,
3,
);
assert_eq!(bl.len(), 3);
assert_eq!(bl.blacklisted_len(), 1);
// Inserting a fourth entry evicts the stalest counting entry (b).
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://d.example/n.xml",
400,
3,
3,
);
assert_eq!(bl.len(), 3);
assert!(
bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://a.example/n.xml")
);
assert!(
!bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://b.example/n.xml")
);
assert!(
bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://c.example/n.xml")
);
assert!(
bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://d.example/n.xml")
);
}
#[test]
fn capacity_evicts_oldest_blacklisted_when_no_counting_entries() {
let mut bl = DeadRepoBlacklist::new();
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://a.example/n.xml",
100,
1,
2,
);
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://b.example/n.xml",
200,
1,
2,
);
assert_eq!(bl.blacklisted_len(), 2);
bl.record_transport_failure(
RepoTransportMode::Rrdp,
"https://c.example/n.xml",
300,
1,
2,
);
assert_eq!(bl.len(), 2);
assert!(
!bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://a.example/n.xml")
);
assert!(
bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://b.example/n.xml")
);
assert!(
bl.entries_sorted()
.iter()
.any(|e| e.uri == "https://c.example/n.xml")
);
}
#[test]
fn store_load_roundtrip() {
let dir = std::env::temp_dir().join(format!(
"dead_repo_blacklist_test_roundtrip_{}_{}",
std::process::id(),
now()
));
let path = dir.join("state").join("dead-repo-blacklist.json");
let mut bl = DeadRepoBlacklist::new();
for i in 0..3 {
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
}
bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now(), 3, 256);
bl.store_atomic(&path, now() + 10).unwrap();
let (loaded, warning) = DeadRepoBlacklist::load(&path);
assert!(warning.is_none());
assert!(loaded.is_blacklisted(RepoTransportMode::Rrdp, RRDP_URI));
assert!(!loaded.is_blacklisted(RepoTransportMode::Rsync, RSYNC_URI));
assert_eq!(loaded.len(), 2);
let entry = loaded
.entries_sorted()
.into_iter()
.find(|e| e.uri == RRDP_URI)
.unwrap();
assert_eq!(entry.consecutive_failures, 3);
assert!(entry.blacklisted);
assert_eq!(entry.first_failure_at_unix, now());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn load_missing_file_is_empty_without_warning() {
let path = std::env::temp_dir().join(format!(
"dead_repo_blacklist_test_missing_{}_{}.json",
std::process::id(),
now()
));
let (bl, warning) = DeadRepoBlacklist::load(&path);
assert!(bl.is_empty());
assert!(warning.is_none());
}
#[test]
fn load_corrupted_file_degrades_to_empty_with_warning() {
let path = std::env::temp_dir().join(format!(
"dead_repo_blacklist_test_corrupt_{}_{}.json",
std::process::id(),
now()
));
std::fs::write(&path, b"{not json").unwrap();
let (bl, warning) = DeadRepoBlacklist::load(&path);
assert!(bl.is_empty());
assert!(warning.is_some());
let _ = std::fs::remove_file(&path);
}
#[test]
fn load_wrong_schema_version_degrades_to_empty() {
let path = std::env::temp_dir().join(format!(
"dead_repo_blacklist_test_schema_{}_{}.json",
std::process::id(),
now()
));
std::fs::write(
&path,
r#"{"schema_version": 99, "updated_at_unix": 0, "entries": []}"#,
)
.unwrap();
let (bl, warning) = DeadRepoBlacklist::load(&path);
assert!(bl.is_empty());
assert!(warning.unwrap().contains("schema_version"));
let _ = std::fs::remove_file(&path);
}
#[test]
fn probe_due_uses_exponential_backoff() {
let mut bl = DeadRepoBlacklist::new();
for i in 0..3 {
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
}
// Never probed: due immediately.
let due = bl.probe_due_entries(now() + 100, 600);
assert_eq!(due.len(), 1);
// Probe failure: not due until base interval has passed.
bl.record_probe_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 100);
assert!(bl.probe_due_entries(now() + 100 + 599, 600).is_empty());
assert_eq!(bl.probe_due_entries(now() + 100 + 600, 600).len(), 1);
// Second probe failure: backoff doubles to 1200s.
bl.record_probe_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 700);
assert!(bl.probe_due_entries(now() + 700 + 1199, 600).is_empty());
assert_eq!(bl.probe_due_entries(now() + 700 + 1200, 600).len(), 1);
}
#[test]
fn probe_success_removes_entry() {
let mut bl = DeadRepoBlacklist::new();
for i in 0..3 {
bl.record_transport_failure(RepoTransportMode::Rsync, RSYNC_URI, now() + i, 3, 256);
}
assert!(bl.record_probe_success(RepoTransportMode::Rsync, RSYNC_URI));
assert!(bl.is_empty());
assert!(!bl.record_probe_success(RepoTransportMode::Rsync, RSYNC_URI));
}
#[test]
fn counting_entries_are_not_probed() {
let mut bl = DeadRepoBlacklist::new();
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now(), 3, 256);
assert!(bl.probe_due_entries(now() + 10_000, 600).is_empty());
}
#[test]
fn probe_backoff_caps_at_max_shift() {
assert_eq!(probe_backoff_secs(600, 0), 600);
assert_eq!(probe_backoff_secs(600, 1), 600);
assert_eq!(probe_backoff_secs(600, 2), 1200);
assert_eq!(probe_backoff_secs(600, 5), 9600);
assert_eq!(probe_backoff_secs(600, 100), 9600);
}
#[test]
fn http_fetch_error_classification() {
use crate::parallel::types::RepoTransportErrorClass;
assert_eq!(
classify_http_fetch_error("http request failed: reqwest::Error { kind: Connect, .. }"),
RepoTransportErrorClass::TransportFetch
);
assert_eq!(
classify_http_fetch_error("http read body failed: connection closed; status=200"),
RepoTransportErrorClass::TransportFetch
);
assert_eq!(
classify_http_fetch_error("http status 404 Not Found; content_type=; .."),
RepoTransportErrorClass::Protocol
);
assert_eq!(
classify_http_fetch_error("something unexpected"),
RepoTransportErrorClass::Unknown
);
}
#[test]
fn rsync_fetch_error_classification() {
use crate::parallel::types::RepoTransportErrorClass;
assert_eq!(
classify_rsync_fetch_error("rsync error: timeout waiting for daemon connection"),
RepoTransportErrorClass::TransportFetch
);
assert_eq!(
classify_rsync_fetch_error("rsync: failed to connect to host: Connection refused"),
RepoTransportErrorClass::TransportFetch
);
assert_eq!(
classify_rsync_fetch_error("temporary failure in name resolution"),
RepoTransportErrorClass::TransportFetch
);
// Content-level failures must not count.
assert_eq!(
classify_rsync_fetch_error("rsync file digest mismatch after download"),
RepoTransportErrorClass::Unknown
);
// Stall failures (#141 M9): fail-fast give-up means the host delivered no
// bytes inside the wall-clock window — transport death, must count.
assert_eq!(
classify_rsync_fetch_error(
"rsync fail-fast gave up after 1 attempts with no progress: rsync wall-clock window expired"
),
RepoTransportErrorClass::TransportFetch
);
assert_eq!(
classify_rsync_fetch_error(
"rsync fail-fast gave up after 3 attempts with no additional progress: rsync wall-clock window expired"
),
RepoTransportErrorClass::TransportFetch
);
// Content-level give-up text without the fail-fast marker stays unknown.
assert_eq!(
classify_rsync_fetch_error("rsync error: some files vanished before transfer"),
RepoTransportErrorClass::Unknown
);
}
#[test]
fn rrdp_sync_error_classification() {
use crate::parallel::types::RepoTransportErrorClass;
use crate::sync::rrdp::RrdpSyncError;
assert_eq!(
classify_rrdp_sync_error(&RrdpSyncError::Fetch(
"http request failed: connect timeout".to_string()
)),
RepoTransportErrorClass::TransportFetch
);
assert_eq!(
classify_rrdp_sync_error(&RrdpSyncError::Fetch("http status 500".to_string())),
RepoTransportErrorClass::Protocol
);
assert_eq!(
classify_rrdp_sync_error(&RrdpSyncError::Storage("db full".to_string())),
RepoTransportErrorClass::Storage
);
}
#[test]
fn readmission_after_probe_removal_starts_fresh() {
let mut bl = DeadRepoBlacklist::new();
for i in 0..3 {
bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + i, 3, 256);
}
assert!(bl.record_probe_success(RepoTransportMode::Rrdp, RRDP_URI));
// No half-open: needs the full threshold again.
let out = bl.record_transport_failure(RepoTransportMode::Rrdp, RRDP_URI, now() + 10, 3, 256);
assert_eq!(
out,
DeadRepoFailureOutcome::Counted {
consecutive_failures: 1
}
);
}

View File

@ -0,0 +1,11 @@
pub mod config;
pub mod dead_repo_blacklist;
pub mod object_worker;
pub mod phase2_scheduler;
pub mod repo_runtime;
pub mod repo_scheduler;
pub mod repo_worker;
pub mod run_coordinator;
pub mod stats;
pub mod transport_prefetch;
pub mod types;

View File

@ -0,0 +1,485 @@
use std::marker::PhantomData;
use std::sync::Arc;
use std::sync::mpsc::{self, Receiver, RecvTimeoutError, SyncSender, TrySendError};
use std::thread::{self, JoinHandle, Scope};
use std::time::Duration;
/// Executor shared by every worker of a pool. The `'static` bound required to
/// move the executor into detached threads is expressed on the pool types
/// instead of this trait so scoped pools can borrow their environment.
pub trait ObjectTaskExecutor<T, R>: Send + Sync {
fn execute(&self, worker_index: usize, task: T) -> R;
}
enum ObjectWorkerMessage<T> {
Task(T),
Shutdown,
}
#[derive(Debug)]
pub enum ObjectWorkerSubmitError<T> {
QueueFull { worker_index: usize, task: T },
Disconnected { worker_index: usize, task: T },
}
pub struct ObjectWorkerPool<T, R, E>
where
T: Send + 'static,
R: Send + 'static,
E: ObjectTaskExecutor<T, R> + 'static,
{
task_txs: Vec<SyncSender<ObjectWorkerMessage<T>>>,
result_rx: Receiver<R>,
workers: Vec<JoinHandle<()>>,
next_worker_idx: usize,
_executor: Arc<E>,
}
impl<T, R, E> ObjectWorkerPool<T, R, E>
where
T: Send + 'static,
R: Send + 'static,
E: ObjectTaskExecutor<T, R> + 'static,
{
pub fn new(worker_count: usize, queue_capacity: usize, executor: E) -> Result<Self, String> {
if worker_count == 0 {
return Err("ObjectWorkerPool requires at least one worker".to_string());
}
if queue_capacity == 0 {
return Err("ObjectWorkerPool requires queue_capacity > 0".to_string());
}
let executor = Arc::new(executor);
let (result_tx, result_rx) = mpsc::channel::<R>();
let mut task_txs = Vec::with_capacity(worker_count);
let mut workers = Vec::with_capacity(worker_count);
for worker_index in 0..worker_count {
let (task_tx, task_rx) = mpsc::sync_channel::<ObjectWorkerMessage<T>>(queue_capacity);
let result_tx = result_tx.clone();
let executor = Arc::clone(&executor);
let handle = thread::Builder::new()
.name(format!("object-validation-worker-{worker_index}"))
.spawn(move || object_worker_loop(worker_index, task_rx, result_tx, executor))
.map_err(|e| format!("spawn object worker failed: {e}"))?;
task_txs.push(task_tx);
workers.push(handle);
}
Ok(Self {
task_txs,
result_rx,
workers,
next_worker_idx: 0,
_executor: executor,
})
}
pub fn worker_count(&self) -> usize {
self.task_txs.len()
}
pub fn next_worker_index(&self) -> usize {
self.next_worker_idx
}
pub fn try_submit_round_robin(&mut self, task: T) -> Result<usize, ObjectWorkerSubmitError<T>> {
let worker_index = self.next_worker_idx % self.task_txs.len();
match self.task_txs[worker_index].try_send(ObjectWorkerMessage::Task(task)) {
Ok(()) => {
self.next_worker_idx = (worker_index + 1) % self.task_txs.len();
Ok(worker_index)
}
Err(TrySendError::Full(ObjectWorkerMessage::Task(task))) => {
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task })
}
Err(TrySendError::Disconnected(ObjectWorkerMessage::Task(task))) => {
Err(ObjectWorkerSubmitError::Disconnected { worker_index, task })
}
Err(TrySendError::Full(ObjectWorkerMessage::Shutdown))
| Err(TrySendError::Disconnected(ObjectWorkerMessage::Shutdown)) => {
unreachable!("shutdown is never submitted via try_submit_round_robin")
}
}
}
pub fn recv_result_timeout(&self, timeout: Duration) -> Result<Option<R>, String> {
match self.result_rx.recv_timeout(timeout) {
Ok(result) => Ok(Some(result)),
Err(RecvTimeoutError::Timeout) => Ok(None),
Err(RecvTimeoutError::Disconnected) => {
Err("object worker result channel disconnected".to_string())
}
}
}
pub fn shutdown(mut self) -> Result<(), String> {
self.shutdown_inner()
}
fn shutdown_inner(&mut self) -> Result<(), String> {
if self.workers.is_empty() {
return Ok(());
}
for tx in &self.task_txs {
tx.send(ObjectWorkerMessage::Shutdown)
.map_err(|e| format!("send shutdown to object worker failed: {e}"))?;
}
let mut first_err = None;
for handle in self.workers.drain(..) {
if let Err(e) = handle.join() {
if first_err.is_none() {
first_err = Some(format!("join object worker failed: {e:?}"));
}
}
}
if let Some(err) = first_err {
return Err(err);
}
Ok(())
}
}
impl<T, R, E> Drop for ObjectWorkerPool<T, R, E>
where
T: Send + 'static,
R: Send + 'static,
E: ObjectTaskExecutor<T, R> + 'static,
{
fn drop(&mut self) {
let _ = self.shutdown_inner();
}
}
fn object_worker_loop<T, R, E>(
worker_index: usize,
task_rx: Receiver<ObjectWorkerMessage<T>>,
result_tx: mpsc::Sender<R>,
executor: Arc<E>,
) where
T: Send,
R: Send,
E: ObjectTaskExecutor<T, R>,
{
loop {
match task_rx.recv() {
Ok(ObjectWorkerMessage::Task(task)) => {
let result = executor.execute(worker_index, task);
if result_tx.send(result).is_err() {
break;
}
}
Ok(ObjectWorkerMessage::Shutdown) | Err(_) => break,
}
}
}
/// Scoped variant of `ObjectWorkerPool`: workers are spawned on a
/// `std::thread::Scope`, so tasks, results and the executor may borrow their
/// environment (`'env`) instead of being `'static`. The pool never sends
/// `Shutdown`; workers exit when every task sender is dropped, which happens
/// when the pool itself is dropped ahead of the scope join.
pub struct ScopedObjectWorkerPool<'scope, 'env, T, R, E>
where
T: Send + 'env,
R: Send + 'env,
E: ObjectTaskExecutor<T, R> + 'env,
{
task_txs: Vec<SyncSender<ObjectWorkerMessage<T>>>,
result_rx: Receiver<R>,
next_worker_idx: usize,
_executor: Arc<E>,
// Join handles are intentionally not stored: dropping a `ScopedJoinHandle`
// detaches the worker and the enclosing scope joins it on exit, after the
// dropped task senders have made every worker return.
_marker: PhantomData<(&'scope (), &'env ())>,
}
impl<'scope, 'env, T, R, E> ScopedObjectWorkerPool<'scope, 'env, T, R, E>
where
T: Send + 'env,
R: Send + 'env,
E: ObjectTaskExecutor<T, R> + 'env,
{
pub fn new(
scope: &'scope Scope<'scope, 'env>,
worker_count: usize,
queue_capacity: usize,
executor: E,
) -> Result<Self, String> {
if worker_count == 0 {
return Err("ScopedObjectWorkerPool requires at least one worker".to_string());
}
if queue_capacity == 0 {
return Err("ScopedObjectWorkerPool requires queue_capacity > 0".to_string());
}
let executor = Arc::new(executor);
let (result_tx, result_rx) = mpsc::channel::<R>();
let mut task_txs = Vec::with_capacity(worker_count);
for worker_index in 0..worker_count {
let (task_tx, task_rx) = mpsc::sync_channel::<ObjectWorkerMessage<T>>(queue_capacity);
let result_tx = result_tx.clone();
let executor = Arc::clone(&executor);
thread::Builder::new()
.name(format!("object-validation-worker-{worker_index}"))
.spawn_scoped(scope, move || {
object_worker_loop(worker_index, task_rx, result_tx, executor)
})
.map_err(|e| format!("spawn scoped object worker failed: {e}"))?;
task_txs.push(task_tx);
}
Ok(Self {
task_txs,
result_rx,
next_worker_idx: 0,
_executor: executor,
_marker: PhantomData,
})
}
pub fn worker_count(&self) -> usize {
self.task_txs.len()
}
pub fn try_submit_round_robin(&mut self, task: T) -> Result<usize, ObjectWorkerSubmitError<T>> {
let worker_index = self.next_worker_idx % self.task_txs.len();
match self.task_txs[worker_index].try_send(ObjectWorkerMessage::Task(task)) {
Ok(()) => {
self.next_worker_idx = (worker_index + 1) % self.task_txs.len();
Ok(worker_index)
}
Err(TrySendError::Full(ObjectWorkerMessage::Task(task))) => {
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task })
}
Err(TrySendError::Disconnected(ObjectWorkerMessage::Task(task))) => {
Err(ObjectWorkerSubmitError::Disconnected { worker_index, task })
}
Err(TrySendError::Full(ObjectWorkerMessage::Shutdown))
| Err(TrySendError::Disconnected(ObjectWorkerMessage::Shutdown)) => {
unreachable!("shutdown is never submitted via try_submit_round_robin")
}
}
}
pub fn recv_result_timeout(&self, timeout: Duration) -> Result<Option<R>, String> {
match self.result_rx.recv_timeout(timeout) {
Ok(result) => Ok(Some(result)),
Err(RecvTimeoutError::Timeout) => Ok(None),
Err(RecvTimeoutError::Disconnected) => {
Err("scoped object worker result channel disconnected".to_string())
}
}
}
}
impl<'scope, 'env, T, R, E> Drop for ScopedObjectWorkerPool<'scope, 'env, T, R, E>
where
T: Send + 'env,
R: Send + 'env,
E: ObjectTaskExecutor<T, R> + 'env,
{
fn drop(&mut self) {
// Close every worker input queue so blocked `recv` calls return and
// the scoped workers exit before the enclosing scope joins them.
self.task_txs.clear();
}
}
#[cfg(test)]
mod tests {
use super::{ObjectTaskExecutor, ObjectWorkerPool, ObjectWorkerSubmitError};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Barrier};
use std::time::Duration;
#[derive(Clone)]
struct EchoExecutor;
impl ObjectTaskExecutor<u32, (usize, u32)> for EchoExecutor {
fn execute(&self, worker_index: usize, task: u32) -> (usize, u32) {
(worker_index, task)
}
}
#[test]
fn object_worker_pool_rejects_invalid_config_and_shutdowns_explicitly() {
let err = match ObjectWorkerPool::new(0, 1, EchoExecutor) {
Ok(_) => panic!("zero workers should be rejected"),
Err(err) => err,
};
assert!(err.contains("at least one worker"));
let err = match ObjectWorkerPool::new(1, 0, EchoExecutor) {
Ok(_) => panic!("zero queue should be rejected"),
Err(err) => err,
};
assert!(err.contains("queue_capacity > 0"));
let pool = ObjectWorkerPool::new(2, 1, EchoExecutor).expect("pool");
assert_eq!(pool.worker_count(), 2);
assert_eq!(pool.next_worker_index(), 0);
pool.shutdown().expect("shutdown");
}
#[test]
fn object_worker_pool_round_robin_submits_to_worker_queues() {
let mut pool = ObjectWorkerPool::new(3, 4, EchoExecutor).expect("pool");
assert_eq!(pool.try_submit_round_robin(10).expect("submit 10"), 0);
assert_eq!(pool.try_submit_round_robin(11).expect("submit 11"), 1);
assert_eq!(pool.try_submit_round_robin(12).expect("submit 12"), 2);
assert_eq!(pool.try_submit_round_robin(13).expect("submit 13"), 0);
let mut results = Vec::new();
for _ in 0..4 {
results.push(
pool.recv_result_timeout(Duration::from_secs(1))
.expect("result channel")
.expect("result"),
);
}
results.sort_by_key(|(_, task)| *task);
assert_eq!(results, vec![(0, 10), (1, 11), (2, 12), (0, 13)]);
}
struct BlockingExecutor {
barrier: Arc<Barrier>,
started: Arc<AtomicBool>,
}
impl ObjectTaskExecutor<u32, u32> for BlockingExecutor {
fn execute(&self, _worker_index: usize, task: u32) -> u32 {
self.started.store(true, Ordering::SeqCst);
self.barrier.wait();
task
}
}
#[test]
fn object_worker_pool_reports_full_worker_queue_without_advancing_round_robin() {
let barrier = Arc::new(Barrier::new(2));
let started = Arc::new(AtomicBool::new(false));
let mut pool = ObjectWorkerPool::new(
1,
1,
BlockingExecutor {
barrier: Arc::clone(&barrier),
started: Arc::clone(&started),
},
)
.expect("pool");
assert_eq!(pool.try_submit_round_robin(1).expect("first task"), 0);
let deadline = std::time::Instant::now() + Duration::from_secs(1);
while !started.load(Ordering::SeqCst) {
assert!(
std::time::Instant::now() < deadline,
"worker did not start first task"
);
std::thread::sleep(Duration::from_millis(1));
}
assert_eq!(pool.try_submit_round_robin(2).expect("queued task"), 0);
match pool.try_submit_round_robin(3) {
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task }) => {
assert_eq!(worker_index, 0);
assert_eq!(task, 3);
}
other => panic!("expected queue full, got {other:?}"),
}
assert_eq!(pool.next_worker_index(), 0);
barrier.wait();
assert_eq!(
pool.recv_result_timeout(Duration::from_secs(1))
.expect("result channel"),
Some(1)
);
barrier.wait();
assert_eq!(
pool.recv_result_timeout(Duration::from_secs(1))
.expect("result channel"),
Some(2)
);
}
struct BorrowingEchoExecutor<'a> {
base: &'a u32,
}
impl<'a> ObjectTaskExecutor<u32, u32> for BorrowingEchoExecutor<'a> {
fn execute(&self, _worker_index: usize, task: u32) -> u32 {
task + *self.base
}
}
#[test]
fn scoped_object_worker_pool_borrows_environment_and_processes_tasks() {
let base = 100u32;
std::thread::scope(|scope| {
let mut pool = super::ScopedObjectWorkerPool::new(
scope,
2,
2,
BorrowingEchoExecutor { base: &base },
)
.expect("scoped pool");
assert_eq!(pool.worker_count(), 2);
pool.try_submit_round_robin(1).expect("submit 1");
pool.try_submit_round_robin(2).expect("submit 2");
let mut results = Vec::new();
for _ in 0..2 {
results.push(
pool.recv_result_timeout(Duration::from_secs(1))
.expect("result channel")
.expect("result"),
);
}
results.sort();
assert_eq!(results, vec![101, 102]);
// Dropping the pool inside the scope closes the task queues; the
// workers exit on their own and the scope join below must not hang.
drop(pool);
});
}
#[test]
fn scoped_object_worker_pool_reports_full_queue() {
std::thread::scope(|scope| {
let barrier = Arc::new(Barrier::new(2));
let started = Arc::new(AtomicBool::new(false));
let mut pool = super::ScopedObjectWorkerPool::new(
scope,
1,
1,
BlockingExecutor {
barrier: Arc::clone(&barrier),
started: Arc::clone(&started),
},
)
.expect("scoped pool");
pool.try_submit_round_robin(1).expect("first task");
let deadline = std::time::Instant::now() + Duration::from_secs(1);
while !started.load(Ordering::SeqCst) {
assert!(
std::time::Instant::now() < deadline,
"scoped worker did not start first task"
);
std::thread::sleep(Duration::from_millis(1));
}
pool.try_submit_round_robin(2).expect("queued task");
match pool.try_submit_round_robin(3) {
Err(ObjectWorkerSubmitError::QueueFull { worker_index, task }) => {
assert_eq!(worker_index, 0);
assert_eq!(task, 3);
}
other => panic!("expected queue full, got {other:?}"),
}
// Dropping the pool closes the task queues; releasing the barrier
// afterwards lets the blocked worker finish task 1, observe the
// closed result channel, and exit before the scope join.
drop(pool);
barrier.wait();
});
}
}

View File

@ -0,0 +1,332 @@
use std::collections::{HashMap, VecDeque};
use crate::parallel::types::RepoIdentity;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct CaInstanceId(pub u64);
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct PublicationPointId(pub u64);
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PublicationPointState {
pub ca_instance_id: CaInstanceId,
pub pending_roa_tasks: usize,
pub child_discovery_released: bool,
pub finalized: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Phase2CompletionSnapshot {
pub ca_ready_queue_empty: bool,
pub ca_waiting_repo_empty: bool,
pub repo_tasks_idle: bool,
pub pending_roa_dispatch_empty: bool,
pub worker_queues_empty: bool,
pub object_result_queue_empty: bool,
pub object_workers_idle: bool,
pub inflight_publication_points_empty: bool,
}
impl Phase2CompletionSnapshot {
pub fn is_complete(&self) -> bool {
self.ca_ready_queue_empty
&& self.ca_waiting_repo_empty
&& self.repo_tasks_idle
&& self.pending_roa_dispatch_empty
&& self.worker_queues_empty
&& self.object_result_queue_empty
&& self.object_workers_idle
&& self.inflight_publication_points_empty
}
}
#[derive(Default)]
pub struct Phase2SchedulerState<T> {
ca_waiting_repo_by_identity: HashMap<RepoIdentity, Vec<CaInstanceId>>,
ca_ready_queue: VecDeque<CaInstanceId>,
inflight_publication_points: HashMap<PublicationPointId, PublicationPointState>,
pending_roa_dispatch: VecDeque<T>,
}
impl<T> Phase2SchedulerState<T> {
pub fn new() -> Self {
Self {
ca_waiting_repo_by_identity: HashMap::new(),
ca_ready_queue: VecDeque::new(),
inflight_publication_points: HashMap::new(),
pending_roa_dispatch: VecDeque::new(),
}
}
pub fn wait_for_repo(&mut self, identity: RepoIdentity, ca_id: CaInstanceId) {
self.ca_waiting_repo_by_identity
.entry(identity)
.or_default()
.push(ca_id);
}
pub fn release_repo_waiters(&mut self, identity: &RepoIdentity) -> Vec<CaInstanceId> {
let released = self
.ca_waiting_repo_by_identity
.remove(identity)
.unwrap_or_default();
for ca_id in &released {
self.ca_ready_queue.push_back(*ca_id);
}
released
}
pub fn push_ready_ca(&mut self, ca_id: CaInstanceId) {
self.ca_ready_queue.push_back(ca_id);
}
pub fn pop_ready_ca(&mut self) -> Option<CaInstanceId> {
self.ca_ready_queue.pop_front()
}
pub fn start_publication_point(
&mut self,
pp_id: PublicationPointId,
ca_id: CaInstanceId,
pending_roa_tasks: usize,
) {
self.inflight_publication_points.insert(
pp_id,
PublicationPointState {
ca_instance_id: ca_id,
pending_roa_tasks,
child_discovery_released: false,
finalized: false,
},
);
}
pub fn mark_child_discovery_released(&mut self, pp_id: PublicationPointId) {
if let Some(state) = self.inflight_publication_points.get_mut(&pp_id) {
state.child_discovery_released = true;
}
}
pub fn enqueue_roa_task(&mut self, task: T) {
self.pending_roa_dispatch.push_back(task);
}
pub fn pop_pending_roa_dispatch(&mut self) -> Option<T> {
self.pending_roa_dispatch.pop_front()
}
pub fn push_front_pending_roa_dispatch(&mut self, task: T) {
self.pending_roa_dispatch.push_front(task);
}
pub fn record_roa_result(&mut self, pp_id: PublicationPointId) -> Option<PublicationPointId> {
let state = self.inflight_publication_points.get_mut(&pp_id)?;
state.pending_roa_tasks = state.pending_roa_tasks.saturating_sub(1);
if state.pending_roa_tasks == 0 {
state.finalized = true;
self.inflight_publication_points.remove(&pp_id);
Some(pp_id)
} else {
None
}
}
pub fn waiting_repo_len(&self) -> usize {
self.ca_waiting_repo_by_identity
.values()
.map(Vec::len)
.sum()
}
pub fn ready_queue_len(&self) -> usize {
self.ca_ready_queue.len()
}
pub fn inflight_len(&self) -> usize {
self.inflight_publication_points.len()
}
pub fn pending_roa_dispatch_len(&self) -> usize {
self.pending_roa_dispatch.len()
}
pub fn publication_point_state(
&self,
pp_id: PublicationPointId,
) -> Option<&PublicationPointState> {
self.inflight_publication_points.get(&pp_id)
}
pub fn completion_snapshot(
&self,
repo_tasks_idle: bool,
worker_queues_empty: bool,
object_result_queue_empty: bool,
object_workers_idle: bool,
) -> Phase2CompletionSnapshot {
Phase2CompletionSnapshot {
ca_ready_queue_empty: self.ca_ready_queue.is_empty(),
ca_waiting_repo_empty: self.ca_waiting_repo_by_identity.is_empty(),
repo_tasks_idle,
pending_roa_dispatch_empty: self.pending_roa_dispatch.is_empty(),
worker_queues_empty,
object_result_queue_empty,
object_workers_idle,
inflight_publication_points_empty: self.inflight_publication_points.is_empty(),
}
}
}
#[cfg(test)]
mod tests {
use super::{CaInstanceId, Phase2SchedulerState, PublicationPointId};
use crate::parallel::object_worker::{ObjectTaskExecutor, ObjectWorkerPool};
use crate::parallel::types::RepoIdentity;
use std::time::Duration;
fn identity(name: &str) -> RepoIdentity {
RepoIdentity::new(
Some(format!("https://example.test/{name}/notification.xml")),
format!("rsync://example.test/{name}/"),
)
}
#[test]
fn scheduler_repo_ready_moves_waiting_ca_to_ready_queue() {
let mut state = Phase2SchedulerState::<u64>::new();
let repo = identity("arin");
state.wait_for_repo(repo.clone(), CaInstanceId(1));
state.wait_for_repo(repo.clone(), CaInstanceId(2));
assert_eq!(state.waiting_repo_len(), 2);
let released = state.release_repo_waiters(&repo);
assert_eq!(released, vec![CaInstanceId(1), CaInstanceId(2)]);
assert_eq!(state.waiting_repo_len(), 0);
assert_eq!(state.ready_queue_len(), 2);
assert_eq!(state.pop_ready_ca(), Some(CaInstanceId(1)));
assert_eq!(state.pop_ready_ca(), Some(CaInstanceId(2)));
}
#[test]
fn scheduler_releases_child_before_roa_results_finalize_parent() {
let mut state = Phase2SchedulerState::<u64>::new();
let pp = PublicationPointId(10);
state.start_publication_point(pp, CaInstanceId(1), 2);
state.mark_child_discovery_released(pp);
state.push_ready_ca(CaInstanceId(2));
let pp_state = state.publication_point_state(pp).expect("inflight pp");
assert!(pp_state.child_discovery_released);
assert_eq!(pp_state.pending_roa_tasks, 2);
assert_eq!(state.pop_ready_ca(), Some(CaInstanceId(2)));
assert_eq!(state.record_roa_result(pp), None);
assert_eq!(state.inflight_len(), 1);
assert_eq!(state.record_roa_result(pp), Some(pp));
assert_eq!(state.inflight_len(), 0);
}
#[test]
fn scheduler_completion_requires_all_queues_and_inflight_to_be_empty() {
let mut state = Phase2SchedulerState::new();
assert!(
state
.completion_snapshot(true, true, true, true)
.is_complete()
);
state.enqueue_roa_task(1u64);
assert!(
!state
.completion_snapshot(true, true, true, true)
.is_complete()
);
assert_eq!(state.pop_pending_roa_dispatch(), Some(1));
assert!(
state
.completion_snapshot(true, true, true, true)
.is_complete()
);
}
#[test]
fn scheduler_can_retry_pending_roa_task_at_front() {
let mut state = Phase2SchedulerState::new();
state.enqueue_roa_task(1u64);
state.enqueue_roa_task(2u64);
assert_eq!(state.pop_pending_roa_dispatch(), Some(1));
state.push_front_pending_roa_dispatch(3);
assert_eq!(state.pop_pending_roa_dispatch(), Some(3));
assert_eq!(state.pop_pending_roa_dispatch(), Some(2));
assert_eq!(state.pop_pending_roa_dispatch(), None);
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct TestRoaTask {
pp_id: PublicationPointId,
value: u64,
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct TestRoaResult {
worker_index: usize,
pp_id: PublicationPointId,
value: u64,
}
#[derive(Clone)]
struct TestRoaExecutor;
impl ObjectTaskExecutor<TestRoaTask, TestRoaResult> for TestRoaExecutor {
fn execute(&self, worker_index: usize, task: TestRoaTask) -> TestRoaResult {
TestRoaResult {
worker_index,
pp_id: task.pp_id,
value: task.value,
}
}
}
#[test]
fn scheduler_dispatches_pending_roa_tasks_to_workers_and_finalizes_on_results() {
let pp = PublicationPointId(42);
let mut state = Phase2SchedulerState::new();
state.start_publication_point(pp, CaInstanceId(7), 3);
state.mark_child_discovery_released(pp);
for value in 0..3 {
state.enqueue_roa_task(TestRoaTask { pp_id: pp, value });
}
let mut pool = ObjectWorkerPool::new(2, 4, TestRoaExecutor).expect("object pool");
while let Some(task) = state.pop_pending_roa_dispatch() {
pool.try_submit_round_robin(task).expect("submit task");
}
assert_eq!(state.pending_roa_dispatch_len(), 0);
let mut results = Vec::new();
for _ in 0..3 {
let result = pool
.recv_result_timeout(Duration::from_secs(1))
.expect("result channel")
.expect("result");
let finalized = state.record_roa_result(result.pp_id);
results.push(result);
if results.len() < 3 {
assert_eq!(finalized, None);
} else {
assert_eq!(finalized, Some(pp));
}
}
results.sort_by_key(|result| result.value);
assert_eq!(results[0].worker_index, 0);
assert_eq!(results[1].worker_index, 1);
assert_eq!(results[2].worker_index, 0);
assert_eq!(state.inflight_len(), 0);
assert!(
state
.completion_snapshot(true, true, true, true)
.is_complete()
);
}
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,879 @@
use std::collections::VecDeque;
use crate::current_repo_index::{CurrentRepoIndex, CurrentRepoIndexHandle};
use crate::parallel::config::ParallelPhase1Config;
use crate::parallel::dead_repo_blacklist::{
DeadRepoBlacklist, DeadRepoFailureOutcome, DeadRepoSuccessOutcome,
};
use crate::parallel::repo_scheduler::{
InFlightRepoTable, RepoCompletion, RepoRequestAction, TransportCompletion,
TransportRequestAction, TransportStateTables,
};
use crate::parallel::stats::ParallelRunStats;
use crate::parallel::types::{
RepoDedupKey, RepoIdentity, RepoKey, RepoRequester, RepoSyncResultEnvelope, RepoSyncTask,
RepoTransportErrorClass, RepoTransportMode, RepoTransportResultEnvelope,
RepoTransportResultKind, RepoTransportTask, TalInputSpec,
};
use crate::policy::SyncPreference;
pub struct GlobalRunCoordinator {
pub config: ParallelPhase1Config,
pub tal_inputs: Vec<TalInputSpec>,
pub current_repo_index: CurrentRepoIndexHandle,
pub in_flight_repos: InFlightRepoTable,
pub transport_tables: TransportStateTables,
pub pending_repo_tasks: VecDeque<RepoSyncTask>,
pub pending_transport_tasks: VecDeque<RepoTransportTask>,
pub stats: ParallelRunStats,
/// Mutable working copy of the dead-repo blacklist (#141): failure/success
/// counters update during the run and persist at run end. Skip decisions
/// inside `transport_tables` use the frozen run-start snapshot.
pub dead_repo_blacklist: Option<DeadRepoBlacklist>,
}
impl GlobalRunCoordinator {
pub fn new(config: ParallelPhase1Config, tal_inputs: Vec<TalInputSpec>) -> Self {
let dead_repo_blacklist = config.dead_repo_blacklist.as_ref().map(|blacklist_config| {
let (blacklist, warning) = DeadRepoBlacklist::load(&blacklist_config.path);
if let Some(warning) = warning {
crate::progress_log::emit(
"dead_repo_blacklist_load_warning",
serde_json::json!({ "warning": warning }),
);
}
blacklist
});
let mut transport_tables = TransportStateTables::new();
if let Some(blacklist) = dead_repo_blacklist.as_ref() {
// Frozen run-start snapshot for skip decisions.
transport_tables.set_dead_repo_blacklist(blacklist.clone());
}
Self {
config,
tal_inputs,
current_repo_index: CurrentRepoIndex::shared(),
in_flight_repos: InFlightRepoTable::new(),
transport_tables,
pending_repo_tasks: VecDeque::new(),
pending_transport_tasks: VecDeque::new(),
stats: ParallelRunStats::default(),
dead_repo_blacklist,
}
}
pub fn current_repo_index_handle(&self) -> CurrentRepoIndexHandle {
self.current_repo_index.clone()
}
pub fn register_repo_request(
&mut self,
repo_key: RepoKey,
requester: RepoRequester,
validation_time: time::OffsetDateTime,
sync_preference: SyncPreference,
priority: u8,
) -> RepoRequestAction {
let action = self.in_flight_repos.register_request(
repo_key,
requester,
validation_time,
sync_preference,
priority,
);
match &action {
RepoRequestAction::Enqueued(task) => {
self.stats.repo_tasks_total += 1;
self.pending_repo_tasks.push_back(task.clone());
self.stats.repo_queue_depth = self.pending_repo_tasks.len();
}
RepoRequestAction::Reused(_) | RepoRequestAction::FailedReuse { .. } => {
self.stats.repo_tasks_reused += 1;
}
RepoRequestAction::Waiting => {}
}
action
}
pub fn pop_next_repo_task(&mut self) -> Option<RepoSyncTask> {
let next = self.pending_repo_tasks.pop_front();
self.stats.repo_queue_depth = self.pending_repo_tasks.len();
next
}
pub fn mark_repo_running(
&mut self,
repo_key: &RepoKey,
started_at: time::OffsetDateTime,
) -> Result<(), String> {
self.in_flight_repos.mark_running(repo_key, started_at)?;
self.stats.repo_tasks_running += 1;
Ok(())
}
pub fn complete_repo_success(
&mut self,
result: RepoSyncResultEnvelope,
finished_at: time::OffsetDateTime,
) -> Result<RepoCompletion, String> {
let repo_key = result.repo_key.clone();
let completion = self
.in_flight_repos
.complete_success(&repo_key, result, finished_at)?;
self.stats.repo_tasks_running = self.stats.repo_tasks_running.saturating_sub(1);
Ok(completion)
}
pub fn complete_repo_failure(
&mut self,
result: RepoSyncResultEnvelope,
finished_at: time::OffsetDateTime,
) -> Result<RepoCompletion, String> {
let repo_key = result.repo_key.clone();
let completion = self
.in_flight_repos
.complete_failure(&repo_key, result, finished_at)?;
self.stats.repo_tasks_running = self.stats.repo_tasks_running.saturating_sub(1);
self.stats.repo_tasks_failed += 1;
Ok(completion)
}
pub fn register_transport_request(
&mut self,
identity: RepoIdentity,
requester: RepoRequester,
validation_time: time::OffsetDateTime,
priority: u8,
rsync_scope_uri: String,
rsync_failure_scope_uri: Option<String>,
sync_preference: SyncPreference,
retry_short_timeout: bool,
) -> TransportRequestAction {
let mut action = self.transport_tables.register_transport_request(
identity,
requester,
validation_time,
priority,
rsync_scope_uri,
rsync_failure_scope_uri,
sync_preference,
);
match &mut action {
TransportRequestAction::Enqueue(task) => {
if retry_short_timeout {
task.retry_short_timeout = true;
}
self.stats.repo_tasks_total += 1;
self.pending_transport_tasks.push_back(task.clone());
self.stats.repo_queue_depth = self.pending_transport_tasks.len();
}
TransportRequestAction::ReusedSuccess(_)
| TransportRequestAction::ReusedTerminalFailure(_) => {
self.stats.repo_tasks_reused += 1;
}
TransportRequestAction::Waiting { .. } => {}
}
action
}
pub fn push_transport_task(&mut self, task: RepoTransportTask) {
self.stats.repo_tasks_total += 1;
self.pending_transport_tasks.push_back(task);
self.stats.repo_queue_depth = self.pending_transport_tasks.len();
}
pub fn pop_next_transport_task(&mut self) -> Option<RepoTransportTask> {
let next = self.pending_transport_tasks.pop_front();
self.stats.repo_queue_depth = self.pending_transport_tasks.len();
next
}
pub fn mark_transport_running(
&mut self,
dedup_key: &crate::parallel::types::RepoDedupKey,
started_at: time::OffsetDateTime,
) -> Result<(), String> {
self.transport_tables
.mark_transport_running(dedup_key, started_at)?;
self.stats.repo_tasks_running += 1;
Ok(())
}
pub fn complete_transport_result(
&mut self,
result: RepoTransportResultEnvelope,
finished_at: time::OffsetDateTime,
) -> Result<TransportCompletion, String> {
let completion = self
.transport_tables
.complete_transport_result(result.clone(), finished_at)?;
self.stats.repo_tasks_running = self.stats.repo_tasks_running.saturating_sub(1);
if matches!(
result.result,
crate::parallel::types::RepoTransportResultKind::Failed { .. }
) && result.mode == crate::parallel::types::RepoTransportMode::Rsync
{
self.stats.repo_tasks_failed += 1;
}
self.update_dead_repo_blacklist(&result, finished_at);
Ok(completion)
}
/// Update dead-repo blacklist counters (#141) from a real worker result.
/// Runs on the single pump thread; synthesized terminal envelopes never
/// pass through here, so no double counting is possible.
fn update_dead_repo_blacklist(
&mut self,
result: &RepoTransportResultEnvelope,
finished_at: time::OffsetDateTime,
) {
let Some(blacklist_config) = self.config.dead_repo_blacklist.clone() else {
return;
};
let Some(blacklist) = self.dead_repo_blacklist.as_mut() else {
return;
};
let now_unix = finished_at.unix_timestamp().max(0) as u64;
let key: Option<(RepoTransportMode, String)> = match (&result.dedup_key, &result.result) {
(
RepoDedupKey::RrdpNotify { notification_uri },
RepoTransportResultKind::Success { .. },
) => Some((RepoTransportMode::Rrdp, notification_uri.clone())),
(RepoDedupKey::RsyncScope { .. }, RepoTransportResultKind::Success { .. }) => Some((
RepoTransportMode::Rsync,
result.repo_identity.rsync_base_uri.clone(),
)),
(
RepoDedupKey::RrdpNotify { notification_uri },
RepoTransportResultKind::Failed {
error_class: RepoTransportErrorClass::TransportFetch,
..
},
) => Some((RepoTransportMode::Rrdp, notification_uri.clone())),
(
RepoDedupKey::RsyncScope { .. },
RepoTransportResultKind::Failed {
error_class: RepoTransportErrorClass::TransportFetch,
..
},
) => Some((
RepoTransportMode::Rsync,
result.repo_identity.rsync_base_uri.clone(),
)),
_ => None,
};
let Some((transport, uri)) = key else {
return;
};
match &result.result {
RepoTransportResultKind::Success { .. } => {
if let Some(outcome) = blacklist.record_transport_success(transport, &uri) {
crate::progress_log::emit(
"dead_repo_blacklist_remove",
serde_json::json!({
"transport": transport.as_str(),
"uri": uri,
"reason": match outcome {
DeadRepoSuccessOutcome::CounterReset => "counter_reset",
DeadRepoSuccessOutcome::BlacklistRemoved => "fetch_success",
},
}),
);
}
}
RepoTransportResultKind::Failed { .. } => {
if let DeadRepoFailureOutcome::Admitted = blacklist.record_transport_failure(
transport,
&uri,
now_unix,
blacklist_config.fail_threshold,
blacklist_config.capacity,
) {
crate::progress_log::emit(
"dead_repo_blacklist_admit",
serde_json::json!({
"transport": transport.as_str(),
"uri": uri,
"fail_threshold": blacklist_config.fail_threshold,
}),
);
}
}
}
}
pub fn dead_repo_blacklist(&self) -> Option<&DeadRepoBlacklist> {
self.dead_repo_blacklist.as_ref()
}
pub fn runtime_record(
&self,
identity: &RepoIdentity,
) -> Option<&crate::parallel::repo_scheduler::RepoRuntimeRecord> {
self.transport_tables.runtime_record(identity)
}
pub fn finalized_runtime_records_for_transport(
&self,
dedup_key: &RepoDedupKey,
) -> Vec<crate::parallel::repo_scheduler::RepoRuntimeRecord> {
self.transport_tables
.finalized_runtime_records_for_transport(dedup_key)
}
pub fn finalized_runtime_records_for_transport_result(
&self,
result: &RepoTransportResultEnvelope,
) -> Vec<crate::parallel::repo_scheduler::RepoRuntimeRecord> {
self.transport_tables
.finalized_runtime_records_for_transport_result(result)
}
pub fn reset_run_state(&mut self) {
self.in_flight_repos.reset_run_state();
self.transport_tables.reset_run_state();
self.pending_repo_tasks.clear();
self.pending_transport_tasks.clear();
self.stats = ParallelRunStats::default();
if let Ok(mut index) = self.current_repo_index.write() {
index.clear();
}
}
}
#[cfg(test)]
mod tests {
use crate::parallel::config::ParallelPhase1Config;
use crate::parallel::repo_scheduler::RepoRequestAction;
use crate::parallel::run_coordinator::GlobalRunCoordinator;
use crate::parallel::types::{
RepoIdentity, RepoKey, RepoRequester, RepoSyncResultEnvelope, RepoSyncResultKind,
RepoSyncResultRef, TalInputSpec,
};
use crate::policy::SyncPreference;
use crate::storage::{RepositoryViewEntry, RepositoryViewState};
fn requester(tal_id: &str, rir_id: &str, manifest: &str) -> RepoRequester {
RepoRequester {
tal_id: tal_id.to_string(),
rir_id: rir_id.to_string(),
parent_node_id: None,
ca_instance_handle_id: format!("{tal_id}:{manifest}"),
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
manifest_rsync_uri: manifest.to_string(),
}
}
#[test]
fn coordinator_holds_tal_inputs_and_default_stats() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
assert_eq!(coordinator.tal_inputs.len(), 1);
assert_eq!(coordinator.tal_inputs[0].tal_id, "arin");
assert_eq!(coordinator.stats.repo_tasks_total, 0);
assert!(coordinator.in_flight_repos.is_empty());
}
#[test]
fn coordinator_enqueues_new_repo_task_once() {
let mut coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let key = RepoKey::new("rsync://example.test/repo/", None);
let action = coordinator.register_repo_request(
key.clone(),
requester("arin", "arin", "rsync://example.test/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
7,
);
let task = match action {
RepoRequestAction::Enqueued(task) => task,
other => panic!("expected enqueue, got {other:?}"),
};
assert_eq!(task.priority, 7);
assert_eq!(coordinator.stats.repo_tasks_total, 1);
assert_eq!(coordinator.stats.repo_queue_depth, 1);
let popped = coordinator.pop_next_repo_task().expect("task queued");
assert_eq!(popped.repo_key, key);
assert_eq!(coordinator.stats.repo_queue_depth, 0);
}
#[test]
fn coordinator_reset_run_state_clears_runtime_only_state() {
let mut coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/",
);
let requester = requester("arin", "arin", "rsync://example.test/repo/root.mft");
let action = coordinator.register_transport_request(
identity.clone(),
requester,
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/".to_string(),
None,
SyncPreference::RrdpThenRsync,
false,
);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
assert!(coordinator.runtime_record(&identity).is_some());
assert_eq!(coordinator.pending_transport_tasks.len(), 1);
{
let mut index = coordinator
.current_repo_index
.write()
.expect("index write lock");
index
.apply_repository_view_entries(&[RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
current_hash: Some("aa".repeat(32)),
repository_source: Some("rsync://example.test/repo/".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
}])
.expect("apply current object");
assert_eq!(index.active_uri_count(), 1);
}
coordinator.reset_run_state();
assert!(coordinator.runtime_record(&identity).is_none());
assert!(coordinator.pending_transport_tasks.is_empty());
assert!(coordinator.pending_repo_tasks.is_empty());
assert_eq!(
coordinator
.current_repo_index
.read()
.expect("index read lock")
.active_uri_count(),
0
);
assert_eq!(coordinator.stats.repo_tasks_total, 0);
}
#[test]
fn coordinator_merges_waiting_requesters_and_reuses_success() {
let mut coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![
TalInputSpec::from_url("https://example.test/arin.tal"),
TalInputSpec::from_url("https://example.test/apnic.tal"),
],
);
let key = RepoKey::new("rsync://shared.example/repo/", None);
let _ = coordinator.register_repo_request(
key.clone(),
requester("arin", "arin", "rsync://shared.example/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
let _ = coordinator.pop_next_repo_task();
coordinator
.mark_repo_running(&key, time::OffsetDateTime::UNIX_EPOCH)
.expect("running");
let wait_action = coordinator.register_repo_request(
key.clone(),
requester("apnic", "apnic", "rsync://shared.example/repo/child.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert_eq!(wait_action, RepoRequestAction::Waiting);
let completion = coordinator
.complete_repo_success(
RepoSyncResultEnvelope {
repo_key: key.clone(),
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
result: RepoSyncResultKind::Success(RepoSyncResultRef {
repo_key: key.clone(),
source: "rrdp".to_string(),
}),
phase: Some("rrdp_ok".to_string()),
timing_ms: 12,
warnings: Vec::new(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("success");
assert_eq!(completion.released_requesters.len(), 1);
assert_eq!(coordinator.stats.repo_tasks_running, 0);
let reuse_action = coordinator.register_repo_request(
key,
requester("ripe", "ripe", "rsync://shared.example/repo/again.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert!(matches!(reuse_action, RepoRequestAction::Reused(_)));
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
}
#[test]
fn coordinator_reuses_failures_without_enqueuing_again() {
let mut coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let key = RepoKey::new("rsync://shared.example/repo/", None);
let _ = coordinator.register_repo_request(
key.clone(),
requester("arin", "arin", "rsync://shared.example/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
let _ = coordinator.pop_next_repo_task();
coordinator
.mark_repo_running(&key, time::OffsetDateTime::UNIX_EPOCH)
.expect("running");
coordinator
.complete_repo_failure(
RepoSyncResultEnvelope {
repo_key: key.clone(),
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
result: RepoSyncResultKind::Failed {
detail: "timeout".to_string(),
},
phase: Some("rrdp_failed_rsync_failed".to_string()),
timing_ms: 12,
warnings: Vec::new(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("failure");
assert_eq!(coordinator.stats.repo_tasks_failed, 1);
let action = coordinator.register_repo_request(
key,
requester("ripe", "ripe", "rsync://shared.example/repo/child.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert_eq!(
action,
RepoRequestAction::FailedReuse {
detail: "timeout".to_string()
}
);
assert_eq!(coordinator.stats.repo_tasks_total, 1);
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
}
// ---- Dead-repo blacklist (#141) integration tests ----
use crate::parallel::dead_repo_blacklist::{DeadRepoBlacklist, DeadRepoBlacklistConfig};
use crate::parallel::types::{
RepoDedupKey, RepoTransportErrorClass, RepoTransportMode, RepoTransportResultEnvelope,
RepoTransportResultKind,
};
const DEAD_NOTIFY: &str = "https://dead.example/notification.xml";
const DEAD_BASE: &str = "rsync://dead.example/repo/";
fn blacklist_config(fail_threshold: u32) -> DeadRepoBlacklistConfig {
DeadRepoBlacklistConfig {
path: std::env::temp_dir().join(format!(
"dead_repo_blacklist_coordinator_test_{}.json",
std::process::id()
)),
fail_threshold,
capacity: 256,
}
}
fn coordinator_with_blacklist(
fail_threshold: u32,
seed: DeadRepoBlacklist,
) -> GlobalRunCoordinator {
let mut config = ParallelPhase1Config::default();
config.dead_repo_blacklist = Some(blacklist_config(fail_threshold));
let mut coordinator = GlobalRunCoordinator::new(
config,
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
coordinator.dead_repo_blacklist = Some(seed.clone());
coordinator.transport_tables.set_dead_repo_blacklist(seed);
coordinator
}
fn dead_identity() -> RepoIdentity {
RepoIdentity::new(Some(DEAD_NOTIFY.to_string()), DEAD_BASE)
}
fn dead_requester() -> RepoRequester {
requester("arin", "arin", "rsync://dead.example/repo/root.mft")
}
fn transport_envelope(
mode: RepoTransportMode,
error_class: RepoTransportErrorClass,
) -> RepoTransportResultEnvelope {
let (dedup_key, detail) = match mode {
RepoTransportMode::Rrdp => (
RepoDedupKey::RrdpNotify {
notification_uri: DEAD_NOTIFY.to_string(),
},
"http request failed: connect timeout".to_string(),
),
RepoTransportMode::Rsync => (
RepoDedupKey::RsyncScope {
rsync_scope_uri: DEAD_BASE.to_string(),
},
"rsync error: timeout waiting for daemon connection".to_string(),
),
};
RepoTransportResultEnvelope {
dedup_key,
rsync_failure_scope_uri: None,
repo_identity: dead_identity(),
mode,
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
timing_ms: 1,
result: RepoTransportResultKind::Failed {
detail,
warnings: Vec::new(),
error_class,
},
}
}
fn register_dead(
coordinator: &mut GlobalRunCoordinator,
) -> crate::parallel::repo_scheduler::TransportRequestAction {
coordinator.register_transport_request(
dead_identity(),
dead_requester(),
time::OffsetDateTime::UNIX_EPOCH,
0,
DEAD_BASE.to_string(),
None,
SyncPreference::RrdpThenRsync,
false,
)
}
#[test]
fn blacklist_admission_after_threshold_transport_failures() {
let mut coordinator = coordinator_with_blacklist(2, DeadRepoBlacklist::new());
for _ in 0..2 {
let action = register_dead(&mut coordinator);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
coordinator
.complete_transport_result(
transport_envelope(
RepoTransportMode::Rrdp,
RepoTransportErrorClass::TransportFetch,
),
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete");
// Simulate the next run: run-local scheduler state resets, the
// persistent blacklist working copy carries over.
coordinator.reset_run_state();
}
let blacklist = coordinator.dead_repo_blacklist().expect("blacklist");
assert!(blacklist.is_blacklisted(RepoTransportMode::Rrdp, DEAD_NOTIFY));
// Admission mid-run does not affect this run's frozen snapshot: the
// scheduler still enqueues rrdp for a fresh identity registration.
coordinator.reset_run_state();
let action = register_dead(&mut coordinator);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
}
#[test]
fn blacklist_ignores_protocol_and_unknown_failures() {
let mut coordinator = coordinator_with_blacklist(1, DeadRepoBlacklist::new());
for error_class in [
RepoTransportErrorClass::Protocol,
RepoTransportErrorClass::Unknown,
RepoTransportErrorClass::Storage,
] {
let action = register_dead(&mut coordinator);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
coordinator
.complete_transport_result(
transport_envelope(RepoTransportMode::Rrdp, error_class),
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete");
coordinator.reset_run_state();
}
assert!(
coordinator
.dead_repo_blacklist()
.expect("blacklist")
.is_empty()
);
}
#[test]
fn blacklist_skip_rrdp_routes_straight_to_rsync() {
let mut seed = DeadRepoBlacklist::new();
seed.record_transport_failure(RepoTransportMode::Rrdp, DEAD_NOTIFY, 100, 1, 256);
let mut coordinator = coordinator_with_blacklist(3, seed);
let action = register_dead(&mut coordinator);
let task = match action {
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(task) => task,
other => panic!("expected rsync enqueue, got {other:?}"),
};
assert_eq!(task.mode, RepoTransportMode::Rsync);
// register_dead passes retry_short_timeout=false (mirroring a prefetch
// snapshot that recorded no real rrdp failure for skipped runs); the
// skip path must still keep the short retry profile (#141).
assert!(task.retry_short_timeout);
}
#[test]
fn blacklist_dual_dead_terminates_immediately() {
let mut seed = DeadRepoBlacklist::new();
seed.record_transport_failure(RepoTransportMode::Rrdp, DEAD_NOTIFY, 100, 1, 256);
seed.record_transport_failure(RepoTransportMode::Rsync, DEAD_BASE, 100, 1, 256);
let mut coordinator = coordinator_with_blacklist(3, seed);
let action = register_dead(&mut coordinator);
let envelope = match action {
crate::parallel::repo_scheduler::TransportRequestAction::ReusedTerminalFailure(
envelope,
) => envelope,
other => panic!("expected terminal failure, got {other:?}"),
};
match &envelope.result {
RepoTransportResultKind::Failed {
detail, warnings, ..
} => {
assert!(detail.contains("dead repo blacklist"));
assert!(
warnings
.iter()
.any(|warning| warning.message.contains("dead_repo_blacklist_skip_all"))
);
}
other => panic!("expected failed result, got {other:?}"),
}
let record = coordinator
.runtime_record(&dead_identity())
.expect("record");
assert_eq!(
record.state,
crate::parallel::types::RepoRuntimeState::FailedTerminal
);
assert_eq!(coordinator.stats.repo_tasks_total, 0);
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
}
#[test]
fn blacklist_rsync_dead_terminates_after_rrdp_failure() {
// Only rsync blacklisted: rrdp is attempted; when it fails, the rsync
// fallback is short-circuited into an instant terminal failure.
let mut seed = DeadRepoBlacklist::new();
seed.record_transport_failure(RepoTransportMode::Rsync, DEAD_BASE, 100, 1, 256);
let mut coordinator = coordinator_with_blacklist(3, seed);
let action = register_dead(&mut coordinator);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
let completion = coordinator
.complete_transport_result(
transport_envelope(
RepoTransportMode::Rrdp,
RepoTransportErrorClass::TransportFetch,
),
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete");
assert!(completion.follow_up_tasks.is_empty());
let record = coordinator
.runtime_record(&dead_identity())
.expect("record");
assert_eq!(
record.state,
crate::parallel::types::RepoRuntimeState::FailedTerminal
);
let terminal = record.terminal_failure.as_ref().expect("terminal failure");
match &terminal.result {
RepoTransportResultKind::Failed { detail, .. } => {
assert!(detail.contains("dead repo blacklist"));
}
other => panic!("expected failed result, got {other:?}"),
}
}
#[test]
fn blacklist_success_resets_failure_counter() {
let mut coordinator = coordinator_with_blacklist(3, DeadRepoBlacklist::new());
// One transport failure starts counting.
let action = register_dead(&mut coordinator);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
coordinator
.complete_transport_result(
transport_envelope(
RepoTransportMode::Rrdp,
RepoTransportErrorClass::TransportFetch,
),
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete");
assert_eq!(
coordinator.dead_repo_blacklist().expect("blacklist").len(),
1
);
// A later success clears the entry.
coordinator.reset_run_state();
let action = register_dead(&mut coordinator);
assert!(matches!(
action,
crate::parallel::repo_scheduler::TransportRequestAction::Enqueue(_)
));
let mut success =
transport_envelope(RepoTransportMode::Rrdp, RepoTransportErrorClass::Unknown);
success.result = RepoTransportResultKind::Success {
source: "rrdp".to_string(),
warnings: Vec::new(),
};
coordinator
.complete_transport_result(success, time::OffsetDateTime::UNIX_EPOCH)
.expect("complete");
assert!(
coordinator
.dead_repo_blacklist()
.expect("blacklist")
.is_empty()
);
}
}

View File

@ -0,0 +1,25 @@
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct ParallelRunStats {
pub repo_tasks_total: usize,
pub repo_tasks_reused: usize,
pub repo_tasks_running: usize,
pub repo_tasks_failed: usize,
pub inflight_snapshot_bytes: usize,
pub repo_queue_depth: usize,
}
#[cfg(test)]
mod tests {
use super::ParallelRunStats;
#[test]
fn parallel_run_stats_default_to_zero() {
let stats = ParallelRunStats::default();
assert_eq!(stats.repo_tasks_total, 0);
assert_eq!(stats.repo_tasks_reused, 0);
assert_eq!(stats.repo_tasks_running, 0);
assert_eq!(stats.repo_tasks_failed, 0);
assert_eq!(stats.inflight_snapshot_bytes, 0);
assert_eq!(stats.repo_queue_depth, 0);
}
}

View File

@ -0,0 +1,512 @@
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use crate::parallel::types::{
RepoDedupKey, RepoIdentity, RepoRequester, RepoTransportMode, RepoTransportTask,
};
use crate::policy::SyncPreference;
pub const TRANSPORT_PREFETCH_SCHEMA_VERSION: u32 = 1;
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct TransportPrefetchSnapshot {
pub schema_version: u32,
pub generated_at_unix: i64,
pub sync_preference: SyncPreference,
pub requests: Vec<TransportPrefetchRequest>,
}
impl TransportPrefetchSnapshot {
pub fn new(sync_preference: SyncPreference, requests: Vec<TransportPrefetchRequest>) -> Self {
Self {
schema_version: TRANSPORT_PREFETCH_SCHEMA_VERSION,
generated_at_unix: time::OffsetDateTime::now_utc().unix_timestamp(),
sync_preference,
requests,
}
}
pub fn is_compatible_with(&self, sync_preference: SyncPreference) -> bool {
self.schema_version == TRANSPORT_PREFETCH_SCHEMA_VERSION
&& self.sync_preference == sync_preference
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct TransportPrefetchRequest {
pub dedup_key: TransportPrefetchDedupKey,
pub rsync_scope_uri: String,
pub rsync_failure_scope_uri: Option<String>,
pub repo_identity: TransportPrefetchRepoIdentity,
pub mode: TransportPrefetchMode,
#[serde(default)]
pub last_result: Option<TransportPrefetchLastResult>,
#[serde(default)]
pub last_rsync_result: Option<TransportPrefetchLastResult>,
pub tal_id: String,
pub rir_id: String,
pub priority: u8,
pub requesters: Vec<TransportPrefetchRequester>,
}
impl TransportPrefetchRequest {
pub fn from_registered_request(
identity: &RepoIdentity,
requester: &RepoRequester,
priority: u8,
rsync_scope_uri: String,
rsync_failure_scope_uri: Option<String>,
sync_preference: SyncPreference,
) -> Self {
let (dedup_key, mode) = if sync_preference == SyncPreference::RrdpThenRsync {
if let Some(notification_uri) = identity.notification_uri.clone() {
(
TransportPrefetchDedupKey::RrdpNotify { notification_uri },
TransportPrefetchMode::Rrdp,
)
} else {
(
TransportPrefetchDedupKey::RsyncScope {
rsync_scope_uri: rsync_scope_uri.clone(),
},
TransportPrefetchMode::Rsync,
)
}
} else {
(
TransportPrefetchDedupKey::RsyncScope {
rsync_scope_uri: rsync_scope_uri.clone(),
},
TransportPrefetchMode::Rsync,
)
};
Self {
dedup_key,
rsync_scope_uri,
rsync_failure_scope_uri,
repo_identity: TransportPrefetchRepoIdentity::from_identity(identity),
mode,
last_result: None,
last_rsync_result: None,
tal_id: requester.tal_id.clone(),
rir_id: requester.rir_id.clone(),
priority,
requesters: vec![TransportPrefetchRequester::from_requester(requester)],
}
}
pub fn from_task(task: &RepoTransportTask, rsync_scope_uri: String) -> Self {
Self {
dedup_key: TransportPrefetchDedupKey::from_repo_key(&task.dedup_key),
rsync_scope_uri,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: TransportPrefetchRepoIdentity::from_identity(&task.repo_identity),
mode: TransportPrefetchMode::from_mode(task.mode),
last_result: None,
last_rsync_result: None,
tal_id: task.tal_id.clone(),
rir_id: task.rir_id.clone(),
priority: task.priority,
requesters: task
.requesters
.iter()
.map(TransportPrefetchRequester::from_requester)
.collect(),
}
}
pub fn to_identity(&self) -> RepoIdentity {
self.repo_identity.to_identity()
}
pub fn to_requester(&self) -> RepoRequester {
self.requesters
.first()
.map(TransportPrefetchRequester::to_requester)
.unwrap_or_else(|| RepoRequester {
tal_id: self.tal_id.clone(),
rir_id: self.rir_id.clone(),
parent_node_id: None,
ca_instance_handle_id: format!(
"{}:{}",
self.tal_id, self.repo_identity.rsync_base_uri
),
publication_point_rsync_uri: self.repo_identity.rsync_base_uri.clone(),
manifest_rsync_uri: format!("{}prefetch.mft", self.repo_identity.rsync_base_uri),
})
}
fn recorder_key(&self) -> String {
self.dedup_key.stable_key()
}
pub fn retry_short_timeout(&self) -> bool {
matches!(
self.last_result,
Some(TransportPrefetchLastResult { ok: false })
)
}
pub fn retry_short_rsync_timeout(&self) -> bool {
matches!(
self.last_rsync_result,
Some(TransportPrefetchLastResult { ok: false })
) || (self.mode == TransportPrefetchMode::Rsync && self.retry_short_timeout())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum TransportPrefetchDedupKey {
RrdpNotify { notification_uri: String },
RsyncScope { rsync_scope_uri: String },
}
impl TransportPrefetchDedupKey {
fn from_repo_key(key: &RepoDedupKey) -> Self {
match key {
RepoDedupKey::RrdpNotify { notification_uri } => Self::RrdpNotify {
notification_uri: notification_uri.clone(),
},
RepoDedupKey::RsyncScope { rsync_scope_uri } => Self::RsyncScope {
rsync_scope_uri: rsync_scope_uri.clone(),
},
}
}
fn stable_key(&self) -> String {
match self {
Self::RrdpNotify { notification_uri } => format!("rrdp:{notification_uri}"),
Self::RsyncScope { rsync_scope_uri } => format!("rsync:{rsync_scope_uri}"),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct TransportPrefetchRepoIdentity {
pub notification_uri: Option<String>,
pub rsync_base_uri: String,
}
impl TransportPrefetchRepoIdentity {
fn from_identity(identity: &RepoIdentity) -> Self {
Self {
notification_uri: identity.notification_uri.clone(),
rsync_base_uri: identity.rsync_base_uri.clone(),
}
}
fn to_identity(&self) -> RepoIdentity {
RepoIdentity::new(self.notification_uri.clone(), self.rsync_base_uri.clone())
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum TransportPrefetchMode {
Rrdp,
Rsync,
}
impl TransportPrefetchMode {
fn from_mode(mode: RepoTransportMode) -> Self {
match mode {
RepoTransportMode::Rrdp => Self::Rrdp,
RepoTransportMode::Rsync => Self::Rsync,
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct TransportPrefetchLastResult {
pub ok: bool,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct TransportPrefetchRequester {
pub tal_id: String,
pub rir_id: String,
pub parent_node_id: Option<u64>,
pub ca_instance_handle_id: String,
pub publication_point_rsync_uri: String,
pub manifest_rsync_uri: String,
}
impl TransportPrefetchRequester {
fn from_requester(requester: &RepoRequester) -> Self {
Self {
tal_id: requester.tal_id.clone(),
rir_id: requester.rir_id.clone(),
parent_node_id: requester.parent_node_id,
ca_instance_handle_id: requester.ca_instance_handle_id.clone(),
publication_point_rsync_uri: requester.publication_point_rsync_uri.clone(),
manifest_rsync_uri: requester.manifest_rsync_uri.clone(),
}
}
fn to_requester(&self) -> RepoRequester {
RepoRequester {
tal_id: self.tal_id.clone(),
rir_id: self.rir_id.clone(),
parent_node_id: self.parent_node_id,
ca_instance_handle_id: self.ca_instance_handle_id.clone(),
publication_point_rsync_uri: self.publication_point_rsync_uri.clone(),
manifest_rsync_uri: self.manifest_rsync_uri.clone(),
}
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct TransportPrefetchRecorder {
requests_by_key: BTreeMap<String, TransportPrefetchRequest>,
results_by_key: BTreeMap<String, TransportPrefetchLastResult>,
rsync_results_by_scope: BTreeMap<String, TransportPrefetchLastResult>,
request_order: Vec<String>,
}
impl TransportPrefetchRecorder {
pub fn record_registered_request(
&mut self,
identity: &RepoIdentity,
requester: &RepoRequester,
priority: u8,
rsync_scope_uri: String,
rsync_failure_scope_uri: Option<String>,
sync_preference: SyncPreference,
) {
let request = TransportPrefetchRequest::from_registered_request(
identity,
requester,
priority,
rsync_scope_uri,
rsync_failure_scope_uri,
sync_preference,
);
self.record_request(request);
}
pub fn record_task(&mut self, task: &RepoTransportTask, rsync_scope_uri: String) {
let request = TransportPrefetchRequest::from_task(task, rsync_scope_uri);
self.record_request(request);
}
pub fn record_result(&mut self, result: &crate::parallel::types::RepoTransportResultEnvelope) {
let key = TransportPrefetchDedupKey::from_repo_key(&result.dedup_key).stable_key();
let last_result = TransportPrefetchLastResult {
ok: matches!(
result.result,
crate::parallel::types::RepoTransportResultKind::Success { .. }
),
};
self.results_by_key.insert(key.clone(), last_result);
if let Some(request) = self.requests_by_key.get_mut(&key) {
request.last_result = Some(last_result);
}
if let crate::parallel::types::RepoDedupKey::RsyncScope { rsync_scope_uri } =
&result.dedup_key
{
self.rsync_results_by_scope
.insert(rsync_scope_uri.clone(), last_result);
for request in self.requests_by_key.values_mut() {
if request.rsync_scope_uri == *rsync_scope_uri {
request.last_rsync_result = Some(last_result);
}
}
}
}
pub fn snapshot(&self, sync_preference: SyncPreference) -> TransportPrefetchSnapshot {
TransportPrefetchSnapshot::new(
sync_preference,
self.request_order
.iter()
.filter_map(|key| self.requests_by_key.get(key))
.cloned()
.collect(),
)
}
pub fn len(&self) -> usize {
self.requests_by_key.len()
}
fn record_request(&mut self, request: TransportPrefetchRequest) {
let key = request.recorder_key();
if !self.requests_by_key.contains_key(&key) {
self.request_order.push(key.clone());
let mut request = request;
if request.last_result.is_none() {
request.last_result = self.results_by_key.get(&key).copied();
}
if request.last_rsync_result.is_none() {
request.last_rsync_result = self
.rsync_results_by_scope
.get(&request.rsync_scope_uri)
.copied();
}
self.requests_by_key.insert(key, request);
}
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct TransportPrefetchDispatchStats {
pub loaded_requests: u64,
pub enqueued_tasks: u64,
pub waiting_requests: u64,
pub reused_results: u64,
pub skipped_incompatible: u64,
}
#[cfg(test)]
mod tests {
use super::*;
fn requester(uri: &str) -> RepoRequester {
RepoRequester {
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
parent_node_id: None,
ca_instance_handle_id: format!("apnic:{uri}"),
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
manifest_rsync_uri: uri.to_string(),
}
}
fn task(notification_uri: &str, manifest_uri: &str) -> RepoTransportTask {
RepoTransportTask {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: notification_uri.to_string(),
},
rsync_failure_scope_uri: Some("rsync://example.test/".to_string()),
repo_identity: RepoIdentity::new(
Some(notification_uri.to_string()),
"rsync://example.test/repo/",
),
mode: RepoTransportMode::Rrdp,
retry_short_timeout: false,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
validation_time: time::OffsetDateTime::UNIX_EPOCH,
priority: 0,
requesters: vec![requester(manifest_uri)],
}
}
#[test]
fn recorder_deduplicates_by_transport_key() {
let mut recorder = TransportPrefetchRecorder::default();
recorder.record_task(
&task(
"https://example.test/notification.xml",
"rsync://example.test/repo/a.mft",
),
"rsync://example.test/repo/".to_string(),
);
recorder.record_task(
&task(
"https://example.test/notification.xml",
"rsync://example.test/repo/b.mft",
),
"rsync://example.test/repo/".to_string(),
);
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
assert_eq!(snapshot.requests.len(), 1);
assert_eq!(
snapshot.requests[0].requesters[0].manifest_rsync_uri,
"rsync://example.test/repo/a.mft"
);
}
#[test]
fn recorder_preserves_first_discovery_order() {
let mut recorder = TransportPrefetchRecorder::default();
recorder.record_task(
&task(
"https://z.example.test/notification.xml",
"rsync://z.example.test/repo/root.mft",
),
"rsync://z.example.test/repo/".to_string(),
);
recorder.record_task(
&task(
"https://a.example.test/notification.xml",
"rsync://a.example.test/repo/root.mft",
),
"rsync://a.example.test/repo/".to_string(),
);
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
assert_eq!(
snapshot.requests[0]
.repo_identity
.notification_uri
.as_deref(),
Some("https://z.example.test/notification.xml")
);
assert_eq!(
snapshot.requests[1]
.repo_identity
.notification_uri
.as_deref(),
Some("https://a.example.test/notification.xml")
);
}
#[test]
fn snapshot_checks_schema_and_sync_preference() {
let snapshot = TransportPrefetchSnapshot::new(SyncPreference::RrdpThenRsync, Vec::new());
assert!(snapshot.is_compatible_with(SyncPreference::RrdpThenRsync));
assert!(!snapshot.is_compatible_with(SyncPreference::RsyncOnly));
}
#[test]
fn recorder_marks_failed_rrdp_and_rsync_for_next_run_short_timeouts() {
let mut recorder = TransportPrefetchRecorder::default();
let rrdp_task = task(
"https://example.test/notification.xml",
"rsync://example.test/repo/a.mft",
);
recorder.record_task(&rrdp_task, "rsync://example.test/repo/".to_string());
recorder.record_result(&crate::parallel::types::RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notification.xml".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: rrdp_task.repo_identity.clone(),
mode: RepoTransportMode::Rrdp,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 1,
result: crate::parallel::types::RepoTransportResultKind::Failed {
detail: "timeout".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
});
recorder.record_result(&crate::parallel::types::RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/repo/".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: rrdp_task.repo_identity.clone(),
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 1,
result: crate::parallel::types::RepoTransportResultKind::Failed {
detail: "timeout".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
});
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
assert_eq!(snapshot.requests.len(), 1);
assert!(snapshot.requests[0].retry_short_timeout());
assert!(snapshot.requests[0].retry_short_rsync_timeout());
}
}

View File

@ -0,0 +1,537 @@
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
use crate::policy::SyncPreference;
use crate::report::Warning;
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum TalSource {
Url(String),
DerBytes {
tal_url: String,
tal_bytes: Vec<u8>,
ta_der: Vec<u8>,
},
FilePath(PathBuf),
FilePathWithTa {
tal_path: PathBuf,
ta_path: PathBuf,
},
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TalInputSpec {
pub tal_id: String,
pub rir_id: String,
pub source: TalSource,
}
impl TalInputSpec {
pub fn from_url(url: impl Into<String>) -> Self {
let url = url.into();
let tal_id = derive_tal_id_from_url_like(&url);
Self {
rir_id: tal_id.clone(),
tal_id,
source: TalSource::Url(url),
}
}
pub fn from_file_path(path: impl Into<PathBuf>) -> Self {
let path = path.into();
let tal_id = derive_tal_id_from_path(&path);
Self {
rir_id: tal_id.clone(),
tal_id,
source: TalSource::FilePath(path),
}
}
pub fn from_file_path_with_ta(
tal_path: impl Into<PathBuf>,
ta_path: impl Into<PathBuf>,
) -> Self {
let tal_path = tal_path.into();
let ta_path = ta_path.into();
let tal_id = derive_tal_id_from_path(&tal_path);
Self {
rir_id: tal_id.clone(),
tal_id,
source: TalSource::FilePathWithTa { tal_path, ta_path },
}
}
pub fn from_ta_der(tal_url: impl Into<String>, tal_bytes: Vec<u8>, ta_der: Vec<u8>) -> Self {
let tal_url = tal_url.into();
let tal_id = derive_tal_id_from_url_like(&tal_url);
Self {
rir_id: tal_id.clone(),
tal_id,
source: TalSource::DerBytes {
tal_url,
tal_bytes,
ta_der,
},
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct RepoIdentity {
pub notification_uri: Option<String>,
pub rsync_base_uri: String,
}
impl RepoIdentity {
pub fn new(notification_uri: Option<String>, rsync_base_uri: impl Into<String>) -> Self {
Self {
notification_uri,
rsync_base_uri: rsync_base_uri.into(),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub enum RepoDedupKey {
RrdpNotify { notification_uri: String },
RsyncScope { rsync_scope_uri: String },
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RepoTransportMode {
Rrdp,
Rsync,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoTransportTask {
pub dedup_key: RepoDedupKey,
pub rsync_failure_scope_uri: Option<String>,
pub repo_identity: RepoIdentity,
pub mode: RepoTransportMode,
pub retry_short_timeout: bool,
pub tal_id: String,
pub rir_id: String,
pub validation_time: time::OffsetDateTime,
pub priority: u8,
pub requesters: Vec<RepoRequester>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RepoTransportErrorClass {
/// Transport-layer fetch failure (connect timeout/refused/TLS/DNS/broken
/// connection). Only this class counts toward the dead-repo blacklist.
TransportFetch,
/// Protocol-level failure (HTTP status, XML/parse, content mismatch).
Protocol,
/// Local storage failure.
Storage,
/// Unclassified (synthesized or legacy results); never counted.
Unknown,
}
impl Default for RepoTransportErrorClass {
fn default() -> Self {
RepoTransportErrorClass::Unknown
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum RepoTransportResultKind {
Success {
source: String,
warnings: Vec<Warning>,
},
Failed {
detail: String,
warnings: Vec<Warning>,
error_class: RepoTransportErrorClass,
},
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoTransportResultEnvelope {
pub dedup_key: RepoDedupKey,
pub rsync_failure_scope_uri: Option<String>,
pub repo_identity: RepoIdentity,
pub mode: RepoTransportMode,
pub tal_id: String,
pub rir_id: String,
pub timing_ms: u64,
pub result: RepoTransportResultKind,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RepoRuntimeState {
Init,
WaitingRrdp,
RrdpOk,
RrdpFailedPendingRsync,
WaitingRsync,
RsyncOk,
FailedTerminal,
}
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct RepoKey {
pub rsync_base_uri: String,
pub notification_uri: Option<String>,
}
impl RepoKey {
pub fn new(rsync_base_uri: impl Into<String>, notification_uri: Option<String>) -> Self {
Self {
rsync_base_uri: rsync_base_uri.into(),
notification_uri,
}
}
pub fn as_identity(&self) -> RepoIdentity {
RepoIdentity {
notification_uri: self.notification_uri.clone(),
rsync_base_uri: self.rsync_base_uri.clone(),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoRequester {
pub tal_id: String,
pub rir_id: String,
pub parent_node_id: Option<u64>,
pub ca_instance_handle_id: String,
pub publication_point_rsync_uri: String,
pub manifest_rsync_uri: String,
}
impl RepoRequester {
pub fn with_tal_rir(
tal_id: impl Into<String>,
rir_id: impl Into<String>,
manifest_rsync_uri: impl Into<String>,
publication_point_rsync_uri: impl Into<String>,
ca_instance_handle_id: impl Into<String>,
) -> Self {
Self {
tal_id: tal_id.into(),
rir_id: rir_id.into(),
parent_node_id: None,
ca_instance_handle_id: ca_instance_handle_id.into(),
publication_point_rsync_uri: publication_point_rsync_uri.into(),
manifest_rsync_uri: manifest_rsync_uri.into(),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncTask {
pub repo_key: RepoKey,
pub validation_time: time::OffsetDateTime,
pub sync_preference: SyncPreference,
pub tal_id: String,
pub rir_id: String,
pub priority: u8,
pub requesters: Vec<RepoRequester>,
}
impl RepoSyncTask {
pub fn as_transport_task(
&self,
dedup_key: RepoDedupKey,
mode: RepoTransportMode,
) -> RepoTransportTask {
RepoTransportTask {
dedup_key,
rsync_failure_scope_uri: None,
repo_identity: self.repo_key.as_identity(),
mode,
retry_short_timeout: false,
tal_id: self.tal_id.clone(),
rir_id: self.rir_id.clone(),
validation_time: self.validation_time,
priority: self.priority,
requesters: self.requesters.clone(),
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RepoTaskState {
Pending,
Running,
Succeeded,
Failed,
Reused,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncResultRef {
pub repo_key: RepoKey,
pub source: String,
}
impl RepoSyncResultRef {
pub fn as_identity(&self) -> RepoIdentity {
self.repo_key.as_identity()
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct InFlightRepoEntry {
pub state: RepoTaskState,
pub task_ref: Option<RepoSyncTask>,
pub waiting_requesters: Vec<RepoRequester>,
pub result_ref: Option<RepoSyncResultRef>,
pub last_result: Option<RepoSyncResultEnvelope>,
pub last_error: Option<String>,
pub started_at: Option<time::OffsetDateTime>,
pub finished_at: Option<time::OffsetDateTime>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncResultEnvelope {
pub repo_key: RepoKey,
pub tal_id: String,
pub rir_id: String,
pub result: RepoSyncResultKind,
pub phase: Option<String>,
pub timing_ms: u64,
pub warnings: Vec<Warning>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum RepoSyncResultKind {
Success(RepoSyncResultRef),
Failed { detail: String },
Reused(RepoSyncResultRef),
}
fn derive_tal_id_from_url_like(s: &str) -> String {
if let Ok(url) = url::Url::parse(s) {
if let Some(last) = url
.path_segments()
.and_then(|segments| segments.filter(|seg| !seg.is_empty()).next_back())
{
let stem = last.rsplit_once('.').map(|(stem, _)| stem).unwrap_or(last);
let trimmed = stem.trim();
if !trimmed.is_empty() {
return trimmed.to_string();
}
}
if let Some(host) = url.host_str() {
return host.to_string();
}
}
"unknown-tal".to_string()
}
fn derive_tal_id_from_path(path: &Path) -> String {
path.file_stem()
.and_then(|stem| stem.to_str())
.map(|s| s.trim())
.filter(|s| !s.is_empty())
.unwrap_or("unknown-tal")
.to_string()
}
#[cfg(test)]
mod tests {
use std::path::Path;
use crate::policy::SyncPreference;
use crate::report::Warning;
use super::{
RepoDedupKey, RepoIdentity, RepoKey, RepoRequester, RepoRuntimeState, RepoSyncTask,
RepoTaskState, RepoTransportMode, RepoTransportResultEnvelope, RepoTransportResultKind,
TalInputSpec, TalSource, derive_tal_id_from_path, derive_tal_id_from_url_like,
};
#[test]
fn tal_input_spec_from_url_derives_tal_and_rir_ids() {
let spec = TalInputSpec::from_url("https://example.test/tals/apnic.tal");
assert_eq!(spec.tal_id, "apnic");
assert_eq!(spec.rir_id, "apnic");
assert_eq!(
spec.source,
TalSource::Url("https://example.test/tals/apnic.tal".to_string())
);
}
#[test]
fn tal_input_spec_from_file_path_derives_file_stem() {
let spec = TalInputSpec::from_file_path("local/arin.tal");
assert_eq!(spec.tal_id, "arin");
assert_eq!(spec.rir_id, "arin");
}
#[test]
fn tal_input_spec_from_ta_der_preserves_payload() {
let spec = TalInputSpec::from_ta_der(
"https://example.test/ripe.tal",
vec![4, 5, 6],
vec![1, 2, 3],
);
assert_eq!(spec.tal_id, "ripe");
assert_eq!(spec.rir_id, "ripe");
assert_eq!(
spec.source,
TalSource::DerBytes {
tal_url: "https://example.test/ripe.tal".to_string(),
tal_bytes: vec![4, 5, 6],
ta_der: vec![1, 2, 3],
}
);
}
#[test]
fn repo_key_equality_uses_rsync_base_and_notification() {
let a = RepoKey::new(
"rsync://example.test/repo/",
Some("https://example.test/notify.xml".to_string()),
);
let b = RepoKey::new(
"rsync://example.test/repo/",
Some("https://example.test/notify.xml".to_string()),
);
let c = RepoKey::new("rsync://example.test/repo/", None);
assert_eq!(a, b);
assert_ne!(a, c);
}
#[test]
fn repo_task_state_variants_are_distinct() {
assert_ne!(RepoTaskState::Pending, RepoTaskState::Running);
assert_ne!(RepoTaskState::Succeeded, RepoTaskState::Failed);
assert_ne!(RepoTaskState::Failed, RepoTaskState::Reused);
}
#[test]
fn repo_identity_preserves_raw_inputs() {
let ident = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/",
);
assert_eq!(
ident.notification_uri.as_deref(),
Some("https://example.test/notify.xml")
);
assert_eq!(ident.rsync_base_uri, "rsync://example.test/repo/");
}
#[test]
fn repo_key_can_be_viewed_as_repo_identity() {
let key = RepoKey::new(
"rsync://example.test/repo/",
Some("https://example.test/notify.xml".to_string()),
);
let ident = key.as_identity();
assert_eq!(ident.rsync_base_uri, "rsync://example.test/repo/");
assert_eq!(
ident.notification_uri.as_deref(),
Some("https://example.test/notify.xml")
);
}
#[test]
fn repo_sync_task_maps_to_rrdp_transport_task() {
let task = RepoSyncTask {
repo_key: RepoKey::new(
"rsync://example.test/repo/",
Some("https://example.test/notify.xml".to_string()),
),
validation_time: time::OffsetDateTime::UNIX_EPOCH,
sync_preference: SyncPreference::RrdpThenRsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
priority: 1,
requesters: vec![RepoRequester::with_tal_rir(
"apnic",
"apnic",
"rsync://example.test/repo/root.mft",
"rsync://example.test/repo/",
"node:1",
)],
};
let transport = task.as_transport_task(
RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
RepoTransportMode::Rrdp,
);
assert_eq!(transport.mode, RepoTransportMode::Rrdp);
assert_eq!(transport.tal_id, "apnic");
assert_eq!(transport.rir_id, "apnic");
assert_eq!(transport.requesters.len(), 1);
assert_eq!(
transport.repo_identity.notification_uri.as_deref(),
Some("https://example.test/notify.xml")
);
}
#[test]
fn repo_transport_result_envelope_supports_success_and_failure_shapes() {
let identity = RepoIdentity::new(None, "rsync://example.test/repo/");
let ok = RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rsync,
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
timing_ms: 12,
result: RepoTransportResultKind::Success {
source: "rsync".to_string(),
warnings: vec![Warning::new("ok")],
},
};
let fail = RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity,
mode: RepoTransportMode::Rsync,
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
timing_ms: 30,
result: RepoTransportResultKind::Failed {
detail: "timeout".to_string(),
warnings: vec![Warning::new("timeout")],
error_class: super::RepoTransportErrorClass::Unknown,
},
};
assert!(matches!(ok.result, RepoTransportResultKind::Success { .. }));
assert!(matches!(
fail.result,
RepoTransportResultKind::Failed { .. }
));
}
#[test]
fn repo_runtime_state_variants_are_distinct() {
assert_ne!(RepoRuntimeState::Init, RepoRuntimeState::WaitingRrdp);
assert_ne!(RepoRuntimeState::RrdpOk, RepoRuntimeState::RsyncOk);
assert_ne!(
RepoRuntimeState::RrdpFailedPendingRsync,
RepoRuntimeState::FailedTerminal
);
}
#[test]
fn derive_tal_id_helpers_fall_back_safely() {
assert_eq!(
derive_tal_id_from_url_like("https://example.test/path/afrinic.tal"),
"afrinic"
);
assert_eq!(
derive_tal_id_from_path(Path::new("foo/lacnic.tal")),
"lacnic"
);
}
}

View File

@ -0,0 +1,235 @@
use serde::{Deserialize, Serialize};
use crate::ta_constraints::TaConstraintsByTal;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum SyncPreference {
RrdpThenRsync,
RsyncOnly,
}
impl Default for SyncPreference {
fn default() -> Self {
Self::RrdpThenRsync
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum CaFailedFetchPolicy {
ReuseCurrentInstanceVcir,
StopAllOutput,
}
impl Default for CaFailedFetchPolicy {
fn default() -> Self {
Self::ReuseCurrentInstanceVcir
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum SignedObjectFailurePolicy {
DropObject,
DropPublicationPoint,
}
impl Default for SignedObjectFailurePolicy {
fn default() -> Self {
Self::DropObject
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum ResourceValidationMode {
#[serde(rename = "validation-update-03")]
ValidationUpdate03,
#[serde(rename = "rfc6487")]
Rfc6487,
}
impl Default for ResourceValidationMode {
fn default() -> Self {
Self::ValidationUpdate03
}
}
impl ResourceValidationMode {
pub fn parse_cli_value(raw: &str) -> Result<Self, String> {
match raw.trim() {
"validation-update-03" => Ok(Self::ValidationUpdate03),
"rfc6487" => Ok(Self::Rfc6487),
value => Err(format!(
"unknown resource validation mode: {value}; supported: validation-update-03,rfc6487"
)),
}
}
}
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default)]
pub struct StrictPolicy {
pub name: bool,
pub cms_der: bool,
pub signed_attrs: bool,
}
impl StrictPolicy {
pub fn none() -> Self {
Self::default()
}
pub fn all() -> Self {
Self {
name: true,
cms_der: true,
signed_attrs: true,
}
}
pub fn parse_cli_spec(spec: Option<&str>) -> Result<Self, String> {
let Some(spec) = spec else {
return Ok(Self::all());
};
let spec = spec.trim();
if spec.is_empty() || spec == "all" {
return Ok(Self::all());
}
if spec == "none" {
return Ok(Self::none());
}
let mut out = Self::none();
for raw in spec.split(',') {
let item = raw.trim();
match item {
"name" => out.name = true,
"cms-der" | "cms_der" => out.cms_der = true,
"signed-attrs" | "signed_attrs" => out.signed_attrs = true,
"all" => out = Self::all(),
"none" => out = Self::none(),
"" => return Err("empty strict policy name".to_string()),
_ => {
return Err(format!(
"unknown strict policy: {item}; supported: name,cms-der,signed-attrs,all,none"
));
}
}
}
Ok(out)
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default)]
pub struct Policy {
pub sync_preference: SyncPreference,
pub ca_failed_fetch_policy: CaFailedFetchPolicy,
pub signed_object_failure_policy: SignedObjectFailurePolicy,
pub resource_validation_mode: ResourceValidationMode,
pub strict: StrictPolicy,
/// Locally configured, per-TAL EE certificate resource constraints. They
/// are intentionally CLI/runtime-only rather than policy-file input.
#[serde(skip)]
pub ta_constraints: TaConstraintsByTal,
}
impl Default for Policy {
fn default() -> Self {
Self {
sync_preference: SyncPreference::default(),
ca_failed_fetch_policy: CaFailedFetchPolicy::default(),
signed_object_failure_policy: SignedObjectFailurePolicy::default(),
resource_validation_mode: ResourceValidationMode::default(),
strict: StrictPolicy::default(),
ta_constraints: TaConstraintsByTal::default(),
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum PolicyParseError {
#[error("policy TOML parse error: {0}")]
Toml(String),
}
impl Policy {
pub fn from_toml_str(s: &str) -> Result<Self, PolicyParseError> {
toml::from_str(s).map_err(|e| PolicyParseError::Toml(e.to_string()))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn strict_policy_parses_cli_specs() {
assert_eq!(
StrictPolicy::parse_cli_spec(None).unwrap(),
StrictPolicy::all()
);
assert_eq!(
StrictPolicy::parse_cli_spec(Some("name,signed-attrs")).unwrap(),
StrictPolicy {
name: true,
cms_der: false,
signed_attrs: true,
}
);
assert_eq!(
StrictPolicy::parse_cli_spec(Some("none")).unwrap(),
StrictPolicy::none()
);
assert!(StrictPolicy::parse_cli_spec(Some("bogus")).is_err());
}
#[test]
fn resource_validation_mode_parses_cli_values() {
assert_eq!(
ResourceValidationMode::parse_cli_value("validation-update-03").unwrap(),
ResourceValidationMode::ValidationUpdate03
);
assert_eq!(
ResourceValidationMode::parse_cli_value("rfc6487").unwrap(),
ResourceValidationMode::Rfc6487
);
assert!(ResourceValidationMode::parse_cli_value("bogus").is_err());
}
#[test]
fn policy_toml_accepts_strict_table() {
let policy = Policy::from_toml_str(
r#"
[strict]
name = true
cms_der = true
signed_attrs = false
"#,
)
.expect("parse policy");
assert_eq!(
policy.strict,
StrictPolicy {
name: true,
cms_der: true,
signed_attrs: false,
}
);
}
#[test]
fn policy_toml_accepts_resource_validation_mode() {
let policy = Policy::from_toml_str(
r#"
resource_validation_mode = "rfc6487"
"#,
)
.expect("parse policy");
assert_eq!(
policy.resource_validation_mode,
ResourceValidationMode::Rfc6487
);
}
}

View File

@ -0,0 +1,68 @@
use serde_json::Value;
use time::format_description::well_known::Rfc3339;
fn progress_enabled() -> bool {
std::env::var("RPKI_PROGRESS_LOG")
.ok()
.map(|v| matches!(v.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"))
.unwrap_or(false)
}
pub fn slow_threshold_secs() -> f64 {
std::env::var("RPKI_PROGRESS_SLOW_SECS")
.ok()
.and_then(|v| v.parse::<f64>().ok())
.filter(|v| *v >= 0.0)
.unwrap_or(30.0)
}
pub fn stage_fresh_slow_threshold_ms() -> u64 {
std::env::var("RPKI_PROGRESS_STAGE_FRESH_SLOW_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(1_000)
}
pub fn pp_control_slow_threshold_ms() -> u64 {
std::env::var("RPKI_PROGRESS_PP_CONTROL_SLOW_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(100)
}
pub fn pp_cache_slow_threshold_ms() -> u64 {
std::env::var("RPKI_PROGRESS_PP_CACHE_SLOW_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(50)
}
pub fn control_loop_slow_threshold_ms() -> u64 {
std::env::var("RPKI_PROGRESS_CONTROL_LOOP_SLOW_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(1_000)
}
pub fn emit(kind: &str, payload: Value) {
if !progress_enabled() {
return;
}
let ts = time::OffsetDateTime::now_utc()
.format(&Rfc3339)
.unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_string());
let mut map = serde_json::Map::new();
map.insert("ts".to_string(), Value::String(ts));
map.insert("kind".to_string(), Value::String(kind.to_string()));
match payload {
Value::Object(obj) => {
for (k, v) in obj {
map.insert(k, v);
}
}
other => {
map.insert("value".to_string(), other);
}
}
eprintln!("[progress] {}", Value::Object(map));
}

View File

@ -0,0 +1,986 @@
use std::collections::BTreeMap;
use std::fs;
use std::path::{Path, PathBuf};
use serde::Deserialize;
use sha2::Digest;
#[derive(Debug, thiserror::Error)]
pub enum ReplayArchiveError {
#[error("read {entity} failed: {path}: {detail}")]
ReadFile {
entity: &'static str,
path: String,
detail: String,
},
#[error("parse {entity} JSON failed: {path}: {detail}")]
ParseJson {
entity: &'static str,
path: String,
detail: String,
},
#[error("unsupported {entity} version: expected 1, got {version}")]
UnsupportedVersion { entity: &'static str, version: u32 },
#[error("capture directory not found: {0}")]
MissingCaptureDirectory(String),
#[error("capture.json captureId mismatch: locks={locks_capture}, capture={capture_json}")]
CaptureIdMismatch {
locks_capture: String,
capture_json: String,
},
#[error("RRDP lock entry invalid for {notify_uri}: {detail}")]
InvalidRrdpLock { notify_uri: String, detail: String },
#[error("RRDP repo bucket not found for {notify_uri}: {path}")]
MissingRrdpRepoBucket { notify_uri: String, path: String },
#[error("RRDP repo meta rpkiNotify mismatch: expected {expected}, actual {actual}")]
RrdpMetaMismatch { expected: String, actual: String },
#[error("RRDP session directory not found for {notify_uri}: {path}")]
MissingRrdpSessionDir { notify_uri: String, path: String },
#[error("locked notification file not found for {notify_uri}: {path}")]
MissingLockedNotification { notify_uri: String, path: String },
#[error("locked snapshot file not found for {notify_uri} at serial {serial} in {session_dir}")]
MissingLockedSnapshot {
notify_uri: String,
serial: u64,
session_dir: String,
},
#[error(
"multiple locked snapshot files found for {notify_uri} at serial {serial} in {session_dir}"
)]
AmbiguousLockedSnapshot {
notify_uri: String,
serial: u64,
session_dir: String,
},
#[error("rsync URI is not a valid module/base URI: {uri}: {detail}")]
InvalidRsyncUri { uri: String, detail: String },
#[error("rsync module bucket not found for {module_uri}: {path}")]
MissingRsyncModuleBucket { module_uri: String, path: String },
#[error("rsync module meta mismatch: expected {expected}, actual {actual}")]
RsyncMetaMismatch { expected: String, actual: String },
#[error("rsync module tree not found for {module_uri}: {path}")]
MissingRsyncTree { module_uri: String, path: String },
#[error("no replay lock found for RRDP notification URI: {0}")]
MissingRrdpLock(String),
#[error("no replay lock found for rsync module: {0}")]
MissingRsyncLock(String),
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ReplayTransport {
Rrdp,
Rsync,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayRrdpLock {
pub transport: ReplayTransport,
pub session: Option<String>,
pub serial: Option<u64>,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayRsyncLock {
pub transport: ReplayTransport,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayLocks {
pub version: u32,
pub capture: String,
pub rrdp: BTreeMap<String, ReplayRrdpLock>,
pub rsync: BTreeMap<String, ReplayRsyncLock>,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayCaptureMeta {
pub version: u32,
#[serde(rename = "captureId")]
pub capture_id: String,
#[serde(rename = "createdAt")]
pub created_at: String,
pub notes: String,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayRrdpRepoMeta {
pub version: u32,
#[serde(rename = "rpkiNotify")]
pub rpki_notify: String,
#[serde(rename = "createdAt")]
pub created_at: String,
#[serde(rename = "lastSeenAt")]
pub last_seen_at: String,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayRsyncModuleMeta {
pub version: u32,
pub module: String,
#[serde(rename = "createdAt")]
pub created_at: String,
#[serde(rename = "lastSeenAt")]
pub last_seen_at: String,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ReplayRrdpRepo {
pub notify_uri: String,
pub bucket_hash: String,
pub bucket_dir: PathBuf,
pub meta: ReplayRrdpRepoMeta,
pub locked_session: String,
pub locked_serial: u64,
pub session_dir: PathBuf,
pub locked_notification_path: PathBuf,
pub locked_snapshot_path: PathBuf,
pub available_delta_paths: Vec<PathBuf>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ReplayRsyncModule {
pub module_uri: String,
pub bucket_hash: String,
pub bucket_dir: PathBuf,
pub meta: ReplayRsyncModuleMeta,
pub tree_dir: PathBuf,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ReplayArchiveIndex {
pub archive_root: PathBuf,
pub capture_root: PathBuf,
pub locks_path: PathBuf,
pub locks: ReplayLocks,
pub capture_meta: ReplayCaptureMeta,
pub rrdp_repos: BTreeMap<String, ReplayRrdpRepo>,
pub rsync_modules: BTreeMap<String, ReplayRsyncModule>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum ReplayArchiveLoadMode {
Strict,
AllowMissingRsyncModules,
}
impl ReplayArchiveIndex {
pub fn load(
archive_root: impl AsRef<Path>,
locks_path: impl AsRef<Path>,
) -> Result<Self, ReplayArchiveError> {
Self::load_with_mode(archive_root, locks_path, ReplayArchiveLoadMode::Strict)
}
pub fn load_allow_missing_rsync_modules(
archive_root: impl AsRef<Path>,
locks_path: impl AsRef<Path>,
) -> Result<Self, ReplayArchiveError> {
Self::load_with_mode(
archive_root,
locks_path,
ReplayArchiveLoadMode::AllowMissingRsyncModules,
)
}
fn load_with_mode(
archive_root: impl AsRef<Path>,
locks_path: impl AsRef<Path>,
load_mode: ReplayArchiveLoadMode,
) -> Result<Self, ReplayArchiveError> {
let archive_root = archive_root.as_ref().to_path_buf();
let locks_path = locks_path.as_ref().to_path_buf();
let locks: ReplayLocks = read_json_file(&locks_path, "payload replay locks")?;
ensure_version("payload replay locks", locks.version)?;
let capture_root = archive_root
.join("v1")
.join("captures")
.join(&locks.capture);
if !capture_root.is_dir() {
return Err(ReplayArchiveError::MissingCaptureDirectory(
capture_root.display().to_string(),
));
}
let capture_meta_path = capture_root.join("capture.json");
let capture_meta: ReplayCaptureMeta = read_json_file(&capture_meta_path, "capture meta")?;
ensure_version("capture meta", capture_meta.version)?;
if capture_meta.capture_id != locks.capture {
return Err(ReplayArchiveError::CaptureIdMismatch {
locks_capture: locks.capture.clone(),
capture_json: capture_meta.capture_id.clone(),
});
}
let mut rrdp_repos = BTreeMap::new();
for (notify_uri, lock) in &locks.rrdp {
match lock.transport {
ReplayTransport::Rrdp => {
let repo = load_rrdp_repo(&capture_root, notify_uri, lock)?;
rrdp_repos.insert(notify_uri.clone(), repo);
}
ReplayTransport::Rsync => {
validate_rsync_transport_lock(notify_uri, lock)?;
}
}
}
let mut rsync_modules = BTreeMap::new();
for (module_uri, lock) in &locks.rsync {
if lock.transport != ReplayTransport::Rsync {
return Err(ReplayArchiveError::InvalidRrdpLock {
notify_uri: module_uri.clone(),
detail: "rsync lock transport must be rsync".to_string(),
});
}
match load_rsync_module(&capture_root, module_uri) {
Ok(module) => {
rsync_modules.insert(module.module_uri.clone(), module);
}
Err(ReplayArchiveError::MissingRsyncModuleBucket { .. })
if load_mode == ReplayArchiveLoadMode::AllowMissingRsyncModules => {}
Err(err) => return Err(err),
}
}
Ok(Self {
archive_root,
capture_root,
locks_path,
locks,
capture_meta,
rrdp_repos,
rsync_modules,
})
}
pub fn rrdp_lock(&self, notify_uri: &str) -> Option<&ReplayRrdpLock> {
self.locks.rrdp.get(notify_uri)
}
pub fn rrdp_repo(&self, notify_uri: &str) -> Option<&ReplayRrdpRepo> {
self.rrdp_repos.get(notify_uri)
}
pub fn require_rrdp_repo(
&self,
notify_uri: &str,
) -> Result<&ReplayRrdpRepo, ReplayArchiveError> {
self.rrdp_repos
.get(notify_uri)
.ok_or_else(|| ReplayArchiveError::MissingRrdpLock(notify_uri.to_string()))
}
pub fn rsync_module(&self, module_uri: &str) -> Option<&ReplayRsyncModule> {
self.rsync_modules.get(module_uri)
}
pub fn resolve_rsync_module_for_base_uri(
&self,
rsync_base_uri: &str,
) -> Result<&ReplayRsyncModule, ReplayArchiveError> {
let module_uri = canonical_rsync_module(rsync_base_uri)?;
self.rsync_modules
.get(&module_uri)
.ok_or(ReplayArchiveError::MissingRsyncLock(module_uri))
}
}
fn load_rrdp_repo(
capture_root: &Path,
notify_uri: &str,
lock: &ReplayRrdpLock,
) -> Result<ReplayRrdpRepo, ReplayArchiveError> {
let session = lock
.session
.as_deref()
.ok_or_else(|| ReplayArchiveError::InvalidRrdpLock {
notify_uri: notify_uri.to_string(),
detail: "transport=rrdp requires non-null session".to_string(),
})?;
let serial = lock
.serial
.ok_or_else(|| ReplayArchiveError::InvalidRrdpLock {
notify_uri: notify_uri.to_string(),
detail: "transport=rrdp requires non-null serial".to_string(),
})?;
let bucket_hash = sha256_hex(notify_uri.as_bytes());
let bucket_dir = capture_root.join("rrdp").join("repos").join(&bucket_hash);
if !bucket_dir.is_dir() {
return Err(ReplayArchiveError::MissingRrdpRepoBucket {
notify_uri: notify_uri.to_string(),
path: bucket_dir.display().to_string(),
});
}
let meta_path = bucket_dir.join("meta.json");
let meta: ReplayRrdpRepoMeta = read_json_file(&meta_path, "RRDP repo meta")?;
ensure_version("RRDP repo meta", meta.version)?;
if meta.rpki_notify != notify_uri {
return Err(ReplayArchiveError::RrdpMetaMismatch {
expected: notify_uri.to_string(),
actual: meta.rpki_notify.clone(),
});
}
let session_dir = bucket_dir.join(session);
if !session_dir.is_dir() {
return Err(ReplayArchiveError::MissingRrdpSessionDir {
notify_uri: notify_uri.to_string(),
path: session_dir.display().to_string(),
});
}
let locked_notification_path = session_dir.join(format!("notification-{serial}.xml"));
if !locked_notification_path.is_file() {
return Err(ReplayArchiveError::MissingLockedNotification {
notify_uri: notify_uri.to_string(),
path: locked_notification_path.display().to_string(),
});
}
let mut snapshot_candidates: Vec<PathBuf> = fs::read_dir(&session_dir)
.map_err(|e| ReplayArchiveError::ReadFile {
entity: "RRDP session directory",
path: session_dir.display().to_string(),
detail: e.to_string(),
})?
.filter_map(|entry| entry.ok().map(|e| e.path()))
.filter(|path| path.is_file())
.filter(|path| {
path.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| {
name.starts_with(&format!("snapshot-{serial}-")) && name.ends_with(".xml")
})
})
.collect();
snapshot_candidates.sort();
let locked_snapshot_path = match snapshot_candidates.len() {
0 => {
return Err(ReplayArchiveError::MissingLockedSnapshot {
notify_uri: notify_uri.to_string(),
serial,
session_dir: session_dir.display().to_string(),
});
}
1 => snapshot_candidates.remove(0),
_ => {
return Err(ReplayArchiveError::AmbiguousLockedSnapshot {
notify_uri: notify_uri.to_string(),
serial,
session_dir: session_dir.display().to_string(),
});
}
};
let mut available_delta_paths: Vec<PathBuf> = fs::read_dir(&session_dir)
.map_err(|e| ReplayArchiveError::ReadFile {
entity: "RRDP session directory",
path: session_dir.display().to_string(),
detail: e.to_string(),
})?
.filter_map(|entry| entry.ok().map(|e| e.path()))
.filter(|path| path.is_file())
.filter(|path| {
path.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| name.starts_with("delta-") && name.ends_with(".xml"))
})
.collect();
available_delta_paths.sort();
Ok(ReplayRrdpRepo {
notify_uri: notify_uri.to_string(),
bucket_hash,
bucket_dir,
meta,
locked_session: session.to_string(),
locked_serial: serial,
session_dir,
locked_notification_path,
locked_snapshot_path,
available_delta_paths,
})
}
fn validate_rsync_transport_lock(
notify_uri: &str,
lock: &ReplayRrdpLock,
) -> Result<(), ReplayArchiveError> {
if lock.session.is_some() || lock.serial.is_some() {
return Err(ReplayArchiveError::InvalidRrdpLock {
notify_uri: notify_uri.to_string(),
detail: "transport=rsync requires null session and serial".to_string(),
});
}
Ok(())
}
fn load_rsync_module(
capture_root: &Path,
module_uri: &str,
) -> Result<ReplayRsyncModule, ReplayArchiveError> {
let canonical_module = canonical_rsync_module(module_uri)?;
let bucket_hash = sha256_hex(canonical_module.as_bytes());
let bucket_dir = capture_root
.join("rsync")
.join("modules")
.join(&bucket_hash);
if !bucket_dir.is_dir() {
return Err(ReplayArchiveError::MissingRsyncModuleBucket {
module_uri: canonical_module.clone(),
path: bucket_dir.display().to_string(),
});
}
let meta_path = bucket_dir.join("meta.json");
let meta: ReplayRsyncModuleMeta = if meta_path.is_file() {
let meta: ReplayRsyncModuleMeta = read_json_file(&meta_path, "rsync module meta")?;
ensure_version("rsync module meta", meta.version)?;
if meta.module != canonical_module {
return Err(ReplayArchiveError::RsyncMetaMismatch {
expected: canonical_module.clone(),
actual: meta.module.clone(),
});
}
meta
} else {
ReplayRsyncModuleMeta {
version: 1,
module: canonical_module.clone(),
created_at: String::new(),
last_seen_at: String::new(),
}
};
let tree_dir = bucket_dir.join("tree");
if !tree_dir.is_dir() {
return Err(ReplayArchiveError::MissingRsyncTree {
module_uri: canonical_module.clone(),
path: tree_dir.display().to_string(),
});
}
Ok(ReplayRsyncModule {
module_uri: canonical_module,
bucket_hash,
bucket_dir,
meta,
tree_dir,
})
}
pub fn canonical_rsync_module(rsync_uri: &str) -> Result<String, ReplayArchiveError> {
let raw = rsync_uri.trim();
let prefix = "rsync://";
let rest = raw
.strip_prefix(prefix)
.ok_or_else(|| ReplayArchiveError::InvalidRsyncUri {
uri: rsync_uri.to_string(),
detail: "URI must start with rsync://".to_string(),
})?;
let mut parts = rest.split('/');
let authority = parts.next().unwrap_or_default();
let module = parts.next().unwrap_or_default();
if authority.is_empty() || module.is_empty() {
return Err(ReplayArchiveError::InvalidRsyncUri {
uri: rsync_uri.to_string(),
detail: "URI must contain authority and module".to_string(),
});
}
Ok(format!("rsync://{authority}/{module}/"))
}
pub fn sha256_hex(bytes: &[u8]) -> String {
let digest = sha2::Sha256::digest(bytes);
hex::encode(digest)
}
fn ensure_version(entity: &'static str, version: u32) -> Result<(), ReplayArchiveError> {
if version == 1 {
Ok(())
} else {
Err(ReplayArchiveError::UnsupportedVersion { entity, version })
}
}
fn read_json_file<T: for<'de> Deserialize<'de>>(
path: &Path,
entity: &'static str,
) -> Result<T, ReplayArchiveError> {
let bytes = fs::read(path).map_err(|e| ReplayArchiveError::ReadFile {
entity,
path: path.display().to_string(),
detail: e.to_string(),
})?;
serde_json::from_slice(&bytes).map_err(|e| ReplayArchiveError::ParseJson {
entity,
path: path.display().to_string(),
detail: e.to_string(),
})
}
#[cfg(test)]
mod tests {
use super::*;
fn build_minimal_archive() -> (tempfile::TempDir, PathBuf, PathBuf, String, String) {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-archive");
let capture = "capture-001";
let capture_root = archive_root.join("v1").join("captures").join(capture);
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
std::fs::write(
capture_root.join("capture.json"),
format!(
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
),
)
.expect("write capture json");
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
let session = "11111111-1111-1111-1111-111111111111".to_string();
let serial = 42u64;
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = capture_root.join("rrdp").join("repos").join(&repo_hash);
let session_dir = repo_dir.join(&session);
std::fs::create_dir_all(&session_dir).expect("mkdir session dir");
std::fs::write(
repo_dir.join("meta.json"),
format!(
r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write repo meta");
std::fs::write(session_dir.join("notification-42.xml"), b"<notification/>")
.expect("write notification");
std::fs::write(session_dir.join("snapshot-42-deadbeef.xml"), b"<snapshot/>")
.expect("write snapshot");
let module_uri = "rsync://rsync.example.test/repo/".to_string();
let mod_hash = sha256_hex(module_uri.as_bytes());
let mod_dir = capture_root.join("rsync").join("modules").join(&mod_hash);
let tree_dir = mod_dir.join("tree").join("rsync.example.test").join("repo");
std::fs::create_dir_all(&tree_dir).expect("mkdir tree dir");
std::fs::write(
mod_dir.join("meta.json"),
format!(
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write rsync meta");
std::fs::write(tree_dir.join("a.roa"), b"roa").expect("write rsync object");
let locks_path = temp.path().join("locks.json");
std::fs::write(
&locks_path,
format!(
r#"{{
"version":1,
"capture":"{capture}",
"rrdp":{{
"{notify_uri}":{{"transport":"rrdp","session":"{session}","serial":{serial}}},
"https://rrdp-fallback.example.test/notification.xml":{{"transport":"rsync","session":null,"serial":null}}
}},
"rsync":{{
"{module_uri}":{{"transport":"rsync"}}
}}
}}"#
),
)
.expect("write locks");
(temp, archive_root, locks_path, notify_uri, module_uri)
}
#[test]
fn canonical_rsync_module_normalizes_subpaths_and_trailing_slash() {
let normalized = canonical_rsync_module("rsync://example.test/repo/sub/dir/file.roa")
.expect("normalize module");
assert_eq!(normalized, "rsync://example.test/repo/");
let normalized = canonical_rsync_module("rsync://example.test/repo")
.expect("normalize module without slash");
assert_eq!(normalized, "rsync://example.test/repo/");
}
#[test]
fn replay_archive_index_loads_rrdp_and_rsync_entries() {
let (_temp, archive_root, locks_path, notify_uri, module_uri) = build_minimal_archive();
let index =
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
assert_eq!(index.capture_meta.capture_id, "capture-001");
assert_eq!(index.rrdp_repos.len(), 1);
assert_eq!(index.locks.rrdp.len(), 2);
assert_eq!(index.rsync_modules.len(), 1);
let repo = index.require_rrdp_repo(&notify_uri).expect("rrdp repo");
assert_eq!(repo.locked_serial, 42);
assert!(
repo.locked_notification_path
.ends_with("notification-42.xml")
);
assert!(
repo.locked_snapshot_path
.ends_with("snapshot-42-deadbeef.xml")
);
assert!(repo.available_delta_paths.is_empty());
let module = index
.resolve_rsync_module_for_base_uri("rsync://rsync.example.test/repo/sub/path")
.expect("resolve module from base");
assert_eq!(module.module_uri, module_uri);
assert!(
module
.tree_dir
.join("rsync.example.test")
.join("repo")
.join("a.roa")
.is_file()
);
}
#[test]
fn replay_archive_index_rejects_rsync_transport_with_session_or_serial() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
let bad = locks_path.with_file_name("bad-locks.json");
std::fs::write(
&bad,
r#"{
"version":1,
"capture":"capture-001",
"rrdp":{
"https://rrdp-fallback.example.test/notification.xml":{"transport":"rsync","session":"oops","serial":1}
},
"rsync":{}
}"#,
)
.expect("write bad locks");
let err = ReplayArchiveIndex::load(&archive_root, &bad).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::InvalidRrdpLock { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_mismatched_rrdp_meta() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let meta_path = archive_root
.join("v1/captures/capture-001/rrdp/repos")
.join(repo_hash)
.join("meta.json");
std::fs::write(
&meta_path,
r#"{"version":1,"rpkiNotify":"https://other.example/notification.xml","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}"#,
)
.expect("rewrite meta");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::RrdpMetaMismatch { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_missing_snapshot() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = archive_root
.join("v1/captures/capture-001/rrdp/repos")
.join(repo_hash)
.join("11111111-1111-1111-1111-111111111111");
std::fs::remove_file(session_dir.join("snapshot-42-deadbeef.xml"))
.expect("remove snapshot");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingLockedSnapshot { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_ambiguous_snapshot_candidates() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = archive_root
.join("v1/captures/capture-001/rrdp/repos")
.join(repo_hash)
.join("11111111-1111-1111-1111-111111111111");
std::fs::write(session_dir.join("snapshot-42-another.xml"), b"<snapshot/>")
.expect("write second snapshot");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::AmbiguousLockedSnapshot { .. }),
"{err}"
);
}
#[test]
fn canonical_rsync_module_rejects_invalid_uris() {
let err = canonical_rsync_module("https://example.test/repo/").unwrap_err();
assert!(
matches!(err, ReplayArchiveError::InvalidRsyncUri { .. }),
"{err}"
);
let err = canonical_rsync_module("rsync://example.test").unwrap_err();
assert!(
matches!(err, ReplayArchiveError::InvalidRsyncUri { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_missing_capture_directory() {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-archive");
std::fs::create_dir_all(&archive_root).expect("mkdir archive root");
let locks_path = temp.path().join("locks.json");
std::fs::write(
&locks_path,
r#"{"version":1,"capture":"missing","rrdp":{},"rsync":{}}"#,
)
.expect("write locks");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingCaptureDirectory(_)),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_missing_rrdp_bucket_and_session_and_notification() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/capture-001/rrdp/repos")
.join(&repo_hash);
std::fs::remove_dir_all(&repo_dir).expect("remove repo dir");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingRrdpRepoBucket { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = archive_root
.join("v1/captures/capture-001/rrdp/repos")
.join(&repo_hash)
.join("11111111-1111-1111-1111-111111111111");
std::fs::remove_dir_all(&session_dir).expect("remove session dir");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingRrdpSessionDir { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_minimal_archive();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = archive_root
.join("v1/captures/capture-001/rrdp/repos")
.join(&repo_hash)
.join("11111111-1111-1111-1111-111111111111");
std::fs::remove_file(session_dir.join("notification-42.xml")).expect("remove notification");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingLockedNotification { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_invalid_json_and_missing_rsync_bucket() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
std::fs::write(&locks_path, b"not json").expect("corrupt locks");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(
err,
ReplayArchiveError::ParseJson {
entity: "payload replay locks",
..
}
),
"{err}"
);
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
let mod_hash = sha256_hex(module_uri.as_bytes());
let mod_dir = archive_root
.join("v1/captures/capture-001/rsync/modules")
.join(&mod_hash);
std::fs::remove_dir_all(&mod_dir).expect("remove module dir");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingRsyncModuleBucket { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_can_skip_missing_rsync_modules_in_lenient_mode() {
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
let mod_hash = sha256_hex(module_uri.as_bytes());
let mod_dir = archive_root
.join("v1/captures/capture-001/rsync/modules")
.join(mod_hash);
std::fs::remove_dir_all(&mod_dir).expect("remove module dir");
let index =
ReplayArchiveIndex::load_allow_missing_rsync_modules(&archive_root, &locks_path)
.expect("load lenient replay index");
assert!(index.rsync_modules.is_empty());
}
#[test]
fn replay_archive_index_accepts_missing_rsync_module_meta_when_tree_exists() {
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
let mod_hash = sha256_hex(module_uri.as_bytes());
let meta_path = archive_root
.join("v1/captures/capture-001/rsync/modules")
.join(mod_hash)
.join("meta.json");
std::fs::remove_file(&meta_path).expect("remove rsync module meta");
let index =
ReplayArchiveIndex::load_allow_missing_rsync_modules(&archive_root, &locks_path)
.expect("load replay index without rsync meta");
let module = index
.rsync_modules
.get(&module_uri)
.expect("module present");
assert_eq!(module.meta.module, module_uri);
assert_eq!(module.meta.version, 1);
}
#[test]
fn replay_archive_index_rejects_capture_id_mismatch() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
let capture_json = archive_root.join("v1/captures/capture-001/capture.json");
std::fs::write(
&capture_json,
r#"{"version":1,"captureId":"other-capture","createdAt":"2026-03-13T00:00:00Z","notes":""}"#,
)
.expect("rewrite capture json");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::CaptureIdMismatch { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_unsupported_locks_version() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
std::fs::write(
&locks_path,
r#"{"version":2,"capture":"capture-001","rrdp":{},"rsync":{}}"#,
)
.expect("rewrite locks version");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(
err,
ReplayArchiveError::UnsupportedVersion {
entity: "payload replay locks",
version: 2
}
),
"{err}"
);
}
#[test]
fn replay_archive_index_reports_missing_rrdp_lock_and_missing_rsync_lock() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_minimal_archive();
let index =
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
let err = index
.require_rrdp_repo("https://missing.example/notification.xml")
.unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingRrdpLock(_)),
"{err}"
);
let err = index
.resolve_rsync_module_for_base_uri("rsync://missing.example/repo/path")
.unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingRsyncLock(_)),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_rsync_meta_mismatch() {
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
let mod_hash = sha256_hex(module_uri.as_bytes());
let meta_path = archive_root
.join("v1/captures/capture-001/rsync/modules")
.join(mod_hash)
.join("meta.json");
std::fs::write(
&meta_path,
r#"{"version":1,"module":"rsync://other.example/repo/","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}"#,
)
.expect("rewrite rsync meta");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::RsyncMetaMismatch { .. }),
"{err}"
);
}
#[test]
fn replay_archive_index_rejects_missing_rsync_module_tree() {
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_minimal_archive();
let mod_hash = sha256_hex(module_uri.as_bytes());
let tree_dir = archive_root
.join("v1/captures/capture-001/rsync/modules")
.join(mod_hash)
.join("tree");
std::fs::remove_dir_all(&tree_dir).expect("remove tree dir");
let err = ReplayArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayArchiveError::MissingRsyncTree { .. }),
"{err}"
);
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,368 @@
use std::collections::BTreeMap;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::replay::delta_archive::{
ReplayDeltaArchiveError, ReplayDeltaArchiveIndex, ReplayDeltaRrdpKind,
};
use crate::sync::rrdp::{Fetcher, parse_notification};
#[derive(Debug, thiserror::Error)]
pub enum PayloadDeltaReplayHttpFetcherError {
#[error(transparent)]
DeltaIndex(#[from] ReplayDeltaArchiveError),
#[error("read delta replay RRDP file failed: {path}: {detail}")]
ReadFile { path: String, detail: String },
#[error("parse target notification failed for {notify_uri}: {detail}")]
ParseNotification { notify_uri: String, detail: String },
#[error(
"target notification session/serial mismatch for {notify_uri}: expected session={expected_session} serial={expected_serial}, got session={actual_session} serial={actual_serial}"
)]
NotificationTargetMismatch {
notify_uri: String,
expected_session: String,
expected_serial: u64,
actual_session: String,
actual_serial: u64,
},
#[error(
"delta serial list mismatch between target notification and transition for {notify_uri}"
)]
DeltaSerialMismatch { notify_uri: String },
#[error("duplicate delta replay HTTP URI mapping for {uri}: {first_path} vs {second_path}")]
DuplicateUriMapping {
uri: String,
first_path: String,
second_path: String,
},
#[error(
"delta replay notification URI is {kind} and should not be fetched as RRDP: {notify_uri}"
)]
NotificationKindNotFetchable { notify_uri: String, kind: String },
#[error("delta replay HTTP URI not found in archive: {0}")]
MissingUri(String),
}
#[derive(Clone, Debug)]
pub struct PayloadDeltaReplayHttpFetcher {
index: Arc<ReplayDeltaArchiveIndex>,
routes: BTreeMap<String, PathBuf>,
repo_kinds: BTreeMap<String, ReplayDeltaRrdpKind>,
}
impl PayloadDeltaReplayHttpFetcher {
pub fn new(
index: Arc<ReplayDeltaArchiveIndex>,
) -> Result<Self, PayloadDeltaReplayHttpFetcherError> {
let mut routes = BTreeMap::new();
let mut repo_kinds = BTreeMap::new();
for (notify_uri, repo) in &index.rrdp_repos {
repo_kinds.insert(notify_uri.clone(), repo.transition.kind);
if repo.transition.kind != ReplayDeltaRrdpKind::Delta {
continue;
}
let notification_path = repo
.target_notification_path
.as_ref()
.expect("delta repo target notification indexed");
insert_unique_route(&mut routes, notify_uri, notification_path)?;
let notification_xml = fs::read(notification_path).map_err(|e| {
PayloadDeltaReplayHttpFetcherError::ReadFile {
path: notification_path.display().to_string(),
detail: e.to_string(),
}
})?;
let notification = parse_notification(&notification_xml).map_err(|e| {
PayloadDeltaReplayHttpFetcherError::ParseNotification {
notify_uri: notify_uri.clone(),
detail: e.to_string(),
}
})?;
let expected_session = repo.transition.target.session.as_deref().unwrap_or("");
let expected_serial = repo.transition.target.serial.unwrap_or_default();
let actual_session = notification.session_id.to_string();
if actual_session != expected_session || notification.serial != expected_serial {
return Err(
PayloadDeltaReplayHttpFetcherError::NotificationTargetMismatch {
notify_uri: notify_uri.clone(),
expected_session: expected_session.to_string(),
expected_serial,
actual_session,
actual_serial: notification.serial,
},
);
}
let transition_serials = repo
.delta_paths
.iter()
.map(|(serial, _)| *serial)
.collect::<Vec<_>>();
let mut notification_delta_map = BTreeMap::new();
for dref in notification.deltas {
notification_delta_map.insert(dref.serial, dref.uri);
}
for serial in &transition_serials {
if !notification_delta_map.contains_key(serial) {
return Err(PayloadDeltaReplayHttpFetcherError::DeltaSerialMismatch {
notify_uri: notify_uri.clone(),
});
}
}
for (serial, path) in &repo.delta_paths {
let uri = notification_delta_map
.get(serial)
.expect("delta uri present for transition serial");
insert_unique_route(&mut routes, uri, path)?;
}
}
Ok(Self {
index,
routes,
repo_kinds,
})
}
pub fn from_index(
index: Arc<ReplayDeltaArchiveIndex>,
) -> Result<Self, PayloadDeltaReplayHttpFetcherError> {
Self::new(index)
}
pub fn archive_index(&self) -> &ReplayDeltaArchiveIndex {
self.index.as_ref()
}
}
impl Fetcher for PayloadDeltaReplayHttpFetcher {
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
if let Some(path) = self.routes.get(uri) {
return fs::read(path).map_err(|e| {
PayloadDeltaReplayHttpFetcherError::ReadFile {
path: path.display().to_string(),
detail: e.to_string(),
}
.to_string()
});
}
if let Some(kind) = self.repo_kinds.get(uri) {
return Err(
PayloadDeltaReplayHttpFetcherError::NotificationKindNotFetchable {
notify_uri: uri.to_string(),
kind: kind.as_str().to_string(),
}
.to_string(),
);
}
Err(PayloadDeltaReplayHttpFetcherError::MissingUri(uri.to_string()).to_string())
}
}
fn insert_unique_route(
routes: &mut BTreeMap<String, PathBuf>,
uri: &str,
path: &Path,
) -> Result<(), PayloadDeltaReplayHttpFetcherError> {
if let Some(existing) = routes.get(uri) {
if existing != path {
return Err(PayloadDeltaReplayHttpFetcherError::DuplicateUriMapping {
uri: uri.to_string(),
first_path: existing.display().to_string(),
second_path: path.display().to_string(),
});
}
return Ok(());
}
routes.insert(uri.to_string(), path.to_path_buf());
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::replay::archive::sha256_hex;
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
fn build_delta_http_fixture(
kind: ReplayDeltaRrdpKind,
) -> (tempfile::TempDir, PathBuf, PathBuf, String, String, String) {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-delta-archive");
let capture = "delta-http";
let capture_root = archive_root.join("v1").join("captures").join(capture);
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
std::fs::write(
capture_root.join("capture.json"),
format!(r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-16T00:00:00Z","notes":""}}"#),
)
.expect("write capture meta");
std::fs::write(
capture_root.join("base.json"),
r#"{"version":1,"baseCapture":"base-cap","baseLocksSha256":"deadbeef","createdAt":"2026-03-16T00:00:00Z"}"#,
)
.expect("write base meta");
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
let snapshot_uri = "https://rrdp.example.test/snapshot.xml".to_string();
let delta1_uri = "https://rrdp.example.test/d1.xml".to_string();
let delta2_uri = "https://rrdp.example.test/d2.xml".to_string();
let session = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string();
let target_serial = 12u64;
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = capture_root
.join("rrdp/repos")
.join(&repo_hash)
.join(&session);
let deltas_dir = session_dir.join("deltas");
std::fs::create_dir_all(&deltas_dir).expect("mkdir deltas dir");
std::fs::write(
session_dir.parent().unwrap().join("meta.json"),
format!(r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
)
.expect("write repo meta");
let notification_xml = format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="{target_serial}">
<snapshot uri="{snapshot_uri}" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
<delta serial="11" uri="{delta1_uri}" hash="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" />
<delta serial="12" uri="{delta2_uri}" hash="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" />
</notification>"#
);
std::fs::write(
session_dir.join("notification-target-12.xml"),
notification_xml,
)
.expect("write target notification");
std::fs::write(
deltas_dir.join("delta-11-aaaa.xml"),
b"<delta serial='11'/>",
)
.expect("write delta1");
std::fs::write(
deltas_dir.join("delta-12-bbbb.xml"),
b"<delta serial='12'/>",
)
.expect("write delta2");
std::fs::write(
session_dir.parent().unwrap().join("transition.json"),
format!(
r#"{{"kind":"{}","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}"#,
kind.as_str()
),
)
.expect("write transition");
let locks_path = temp.path().join("locks-delta.json");
std::fs::write(
&locks_path,
format!(
r#"{{"version":1,"capture":"{capture}","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{{"{notify_uri}":{{"kind":"{}","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}}},"rsync":{{}}}}"#,
kind.as_str()
),
)
.expect("write locks");
(
temp,
archive_root,
locks_path,
notify_uri,
delta1_uri,
delta2_uri,
)
}
#[test]
fn delta_http_fetcher_rejects_session_reset_and_gap_notification_kind() {
for kind in [ReplayDeltaRrdpKind::SessionReset, ReplayDeltaRrdpKind::Gap] {
let (_temp, archive_root, locks_path, notify_uri, _delta1_uri, _delta2_uri) =
build_delta_http_fixture(kind);
let index = Arc::new(
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path)
.expect("load delta index"),
);
let fetcher = PayloadDeltaReplayHttpFetcher::from_index(index).expect("build fetcher");
let err = fetcher.fetch(&notify_uri).unwrap_err();
assert!(err.contains(kind.as_str()), "{err}");
}
}
#[test]
fn delta_http_fetcher_reads_target_notification_and_delta_files() {
let (_temp, archive_root, locks_path, notify_uri, delta1_uri, delta2_uri) =
build_delta_http_fixture(ReplayDeltaRrdpKind::Delta);
let index = Arc::new(
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayHttpFetcher::from_index(index).expect("build fetcher");
let notification = fetcher.fetch(&notify_uri).expect("fetch notification");
assert!(
std::str::from_utf8(&notification)
.unwrap()
.contains("notification")
);
assert_eq!(
fetcher.fetch(&delta1_uri).expect("fetch delta1"),
b"<delta serial='11'/>".to_vec()
);
assert_eq!(
fetcher.fetch(&delta2_uri).expect("fetch delta2"),
b"<delta serial='12'/>".to_vec()
);
}
#[test]
fn delta_http_fetcher_rejects_non_delta_notification_kinds_and_missing_uri() {
let (_temp, archive_root, locks_path, notify_uri, _delta1_uri, _delta2_uri) =
build_delta_http_fixture(ReplayDeltaRrdpKind::Unchanged);
let index = Arc::new(
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayHttpFetcher::from_index(index).expect("build fetcher");
let err = fetcher.fetch(&notify_uri).unwrap_err();
assert!(err.contains("unchanged"), "{err}");
let err = fetcher
.fetch("https://missing.example/test.xml")
.unwrap_err();
assert!(err.contains("not found in archive"), "{err}");
}
#[test]
fn delta_http_fetcher_rejects_target_notification_mismatch() {
let (_temp, archive_root, locks_path, notify_uri, _delta1_uri, _delta2_uri) =
build_delta_http_fixture(ReplayDeltaRrdpKind::Delta);
let repo_hash = sha256_hex(notify_uri.as_bytes());
let notification = archive_root
.join("v1/captures/delta-http/rrdp/repos")
.join(repo_hash)
.join("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
.join("notification-target-12.xml");
std::fs::write(
&notification,
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" serial="12">
<snapshot uri="https://rrdp.example.test/snapshot.xml" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
<delta serial="11" uri="https://rrdp.example.test/d1.xml" hash="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" />
<delta serial="12" uri="https://rrdp.example.test/d2.xml" hash="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" />
</notification>"#,
)
.expect("rewrite notification");
let index = Arc::new(
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index"),
);
let err = PayloadDeltaReplayHttpFetcher::from_index(index).unwrap_err();
assert!(
matches!(
err,
PayloadDeltaReplayHttpFetcherError::NotificationTargetMismatch { .. }
),
"{err}"
);
}
}

View File

@ -0,0 +1,471 @@
use std::collections::BTreeMap;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::fetch::rsync::{RsyncFetchError, RsyncFetchResult, RsyncFetcher};
use crate::replay::archive::{ReplayArchiveIndex, canonical_rsync_module};
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
use crate::storage::{RepositoryViewState, RocksStore};
#[derive(Clone, Debug)]
pub struct PayloadDeltaReplayRsyncFetcher {
base_index: Arc<ReplayArchiveIndex>,
delta_index: Arc<ReplayDeltaArchiveIndex>,
}
impl PayloadDeltaReplayRsyncFetcher {
pub fn new(
base_index: Arc<ReplayArchiveIndex>,
delta_index: Arc<ReplayDeltaArchiveIndex>,
) -> Self {
Self {
base_index,
delta_index,
}
}
pub fn base_index(&self) -> &ReplayArchiveIndex {
self.base_index.as_ref()
}
pub fn delta_index(&self) -> &ReplayDeltaArchiveIndex {
self.delta_index.as_ref()
}
}
pub struct PayloadDeltaReplayCurrentStoreRsyncFetcher<'a> {
store: &'a RocksStore,
delta_index: Arc<ReplayDeltaArchiveIndex>,
}
impl<'a> PayloadDeltaReplayCurrentStoreRsyncFetcher<'a> {
pub fn new(store: &'a RocksStore, delta_index: Arc<ReplayDeltaArchiveIndex>) -> Self {
Self { store, delta_index }
}
pub fn delta_index(&self) -> &ReplayDeltaArchiveIndex {
self.delta_index.as_ref()
}
}
impl RsyncFetcher for PayloadDeltaReplayRsyncFetcher {
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
let module_uri = canonical_rsync_module(rsync_base_uri)
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
let normalized_base = if rsync_base_uri.ends_with('/') {
rsync_base_uri.to_string()
} else {
format!("{rsync_base_uri}/")
};
let mut merged: BTreeMap<String, Vec<u8>> = BTreeMap::new();
let mut saw_base = false;
let overlay_only = self
.delta_index
.rsync_module(&module_uri)
.map(|module| module.overlay_only)
.unwrap_or(false);
if !overlay_only {
if let Ok(base_module) = self
.base_index
.resolve_rsync_module_for_base_uri(rsync_base_uri)
{
let base_tree_root = module_tree_root(&module_uri, &base_module.tree_dir)
.map_err(RsyncFetchError::Fetch)?;
if base_tree_root.is_dir() {
let mut base_objects = Vec::new();
walk_dir_collect(
&base_tree_root,
&base_tree_root,
&module_uri,
&mut base_objects,
)
.map_err(RsyncFetchError::Fetch)?;
for (uri, bytes) in base_objects {
merged.insert(uri, bytes);
}
saw_base = true;
}
}
}
let mut saw_overlay = false;
if let Some(delta_module) = self.delta_index.rsync_module(&module_uri) {
for (uri, path) in &delta_module.overlay_files {
let bytes = fs::read(path).map_err(|e| {
RsyncFetchError::Fetch(format!(
"read delta rsync overlay failed: {}: {e}",
path.display()
))
})?;
merged.insert(uri.clone(), bytes);
saw_overlay = true;
}
}
if !saw_base && !saw_overlay {
return Err(RsyncFetchError::Fetch(format!(
"delta replay base rsync module not found and no delta overlay exists for module: {module_uri}"
)));
}
let filtered: Vec<(String, Vec<u8>)> = merged
.into_iter()
.filter(|(uri, _)| uri.starts_with(&normalized_base))
.collect();
if filtered.is_empty() {
return Err(RsyncFetchError::Fetch(format!(
"delta replay rsync subtree not found: {normalized_base}"
)));
}
Ok(filtered)
}
}
impl RsyncFetcher for PayloadDeltaReplayCurrentStoreRsyncFetcher<'_> {
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
let module_uri = canonical_rsync_module(rsync_base_uri)
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
let normalized_base = if rsync_base_uri.ends_with('/') {
rsync_base_uri.to_string()
} else {
format!("{rsync_base_uri}/")
};
let mut merged: BTreeMap<String, Vec<u8>> = BTreeMap::new();
let mut saw_base = false;
let overlay_only = self
.delta_index
.rsync_module(&module_uri)
.map(|module| module.overlay_only)
.unwrap_or(false);
if !overlay_only {
let entries = self
.store
.list_repository_view_entries_with_prefix(&module_uri)
.map_err(|e| RsyncFetchError::Fetch(format!("list repository view failed: {e}")))?;
for entry in entries {
if entry.state != RepositoryViewState::Present {
continue;
}
let bytes = self
.store
.load_current_object_bytes_by_uri(&entry.rsync_uri)
.map_err(|e| {
RsyncFetchError::Fetch(format!(
"load current object failed for {}: {e}",
entry.rsync_uri
))
})?
.ok_or_else(|| {
RsyncFetchError::Fetch(format!(
"current object missing for {}",
entry.rsync_uri
))
})?;
merged.insert(entry.rsync_uri, bytes);
saw_base = true;
}
}
let mut saw_overlay = false;
if let Some(delta_module) = self.delta_index.rsync_module(&module_uri) {
for (uri, path) in &delta_module.overlay_files {
let bytes = fs::read(path).map_err(|e| {
RsyncFetchError::Fetch(format!(
"read delta rsync overlay failed: {}: {e}",
path.display()
))
})?;
merged.insert(uri.clone(), bytes);
saw_overlay = true;
}
}
if !saw_base && !saw_overlay {
return Err(RsyncFetchError::Fetch(format!(
"delta replay current-store base not found and no delta overlay exists for module: {module_uri}"
)));
}
let filtered: Vec<(String, Vec<u8>)> = merged
.into_iter()
.filter(|(uri, _)| uri.starts_with(&normalized_base))
.collect();
if filtered.is_empty() {
return Err(RsyncFetchError::Fetch(format!(
"delta replay rsync subtree not found: {normalized_base}"
)));
}
Ok(filtered)
}
}
fn module_tree_root(module_uri: &str, tree_dir: &Path) -> Result<PathBuf, String> {
let rest = module_uri
.strip_prefix("rsync://")
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
let mut parts = rest.trim_end_matches('/').split('/');
let authority = parts
.next()
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
let module = parts
.next()
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
Ok(tree_dir.join(authority).join(module))
}
fn walk_dir_collect(
root: &Path,
current: &Path,
rsync_base_uri: &str,
out: &mut Vec<(String, Vec<u8>)>,
) -> Result<(), String> {
let rd = fs::read_dir(current).map_err(|e| e.to_string())?;
for entry in rd {
let entry = entry.map_err(|e| e.to_string())?;
let path = entry.path();
let meta = entry.metadata().map_err(|e| e.to_string())?;
if meta.is_dir() {
walk_dir_collect(root, &path, rsync_base_uri, out)?;
continue;
}
if !meta.is_file() {
continue;
}
let rel = path
.strip_prefix(root)
.map_err(|e| e.to_string())?
.to_string_lossy()
.replace('\\', "/");
let base = if rsync_base_uri.ends_with('/') {
rsync_base_uri.to_string()
} else {
format!("{rsync_base_uri}/")
};
let uri = format!("{base}{rel}");
let bytes = fs::read(&path).map_err(|e| e.to_string())?;
out.push((uri, bytes));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::replay::archive::ReplayArchiveIndex;
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
use crate::storage::{RawByHashEntry, RepositoryViewEntry, RepositoryViewState, RocksStore};
fn build_base_and_delta_rsync_fixture()
-> (tempfile::TempDir, PathBuf, PathBuf, PathBuf, PathBuf) {
let temp = tempfile::tempdir().expect("tempdir");
let base_archive = temp.path().join("payload-archive");
let base_capture_root = base_archive.join("v1/captures/base-cap");
std::fs::create_dir_all(&base_capture_root).expect("mkdir base capture");
std::fs::write(
base_capture_root.join("capture.json"),
r#"{"version":1,"captureId":"base-cap","createdAt":"2026-03-16T00:00:00Z","notes":""}"#,
)
.expect("write base capture meta");
let module_uri = "rsync://rsync.example.test/repo/";
let module_hash = crate::replay::archive::sha256_hex(module_uri.as_bytes());
let base_bucket = base_capture_root.join("rsync/modules").join(&module_hash);
let base_tree = base_bucket.join("tree/rsync.example.test/repo");
std::fs::create_dir_all(base_tree.join("sub")).expect("mkdir base tree");
std::fs::write(base_bucket.join("meta.json"), format!(r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#)).expect("write base meta");
std::fs::write(base_tree.join("a.roa"), b"base-a").expect("write base a");
std::fs::write(base_tree.join("sub").join("b.cer"), b"base-b").expect("write base b");
let base_locks = temp.path().join("base-locks.json");
std::fs::write(&base_locks, format!(r#"{{"version":1,"capture":"base-cap","rrdp":{{}},"rsync":{{"{module_uri}":{{"transport":"rsync"}}}}}}"#)).expect("write base locks");
let delta_archive = temp.path().join("payload-delta-archive");
let delta_capture_root = delta_archive.join("v1/captures/delta-cap");
std::fs::create_dir_all(&delta_capture_root).expect("mkdir delta capture");
std::fs::write(delta_capture_root.join("capture.json"), r#"{"version":1,"captureId":"delta-cap","createdAt":"2026-03-16T00:00:00Z","notes":""}"#).expect("write delta capture meta");
std::fs::write(delta_capture_root.join("base.json"), r#"{"version":1,"baseCapture":"base-cap","baseLocksSha256":"deadbeef","createdAt":"2026-03-16T00:00:00Z"}"#).expect("write delta base meta");
let delta_bucket = delta_capture_root.join("rsync/modules").join(&module_hash);
let delta_tree = delta_bucket.join("tree/rsync.example.test/repo");
std::fs::create_dir_all(delta_tree.join("sub")).expect("mkdir delta tree");
std::fs::write(delta_bucket.join("meta.json"), format!(r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#)).expect("write delta meta");
std::fs::write(delta_bucket.join("files.json"), format!(r#"{{"version":1,"module":"{module_uri}","fileCount":1,"files":["{module_uri}sub/b.cer"]}}"#)).expect("write files json");
std::fs::write(delta_tree.join("sub").join("b.cer"), b"delta-b")
.expect("write delta overlay");
let delta_locks = temp.path().join("locks-delta.json");
std::fs::write(&delta_locks, format!(r#"{{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{{}},"rsync":{{"{module_uri}":{{"file_count":1,"overlay_only":false}}}}}}"#)).expect("write delta locks");
(temp, base_archive, base_locks, delta_archive, delta_locks)
}
#[test]
fn delta_rsync_current_store_fetcher_merges_current_store_and_overlay() {
let (_temp, _base_archive, _base_locks, delta_archive, delta_locks) =
build_base_and_delta_rsync_fixture();
let temp = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(temp.path()).expect("open rocksdb");
store
.put_raw_by_hash_entry(&RawByHashEntry {
sha256_hex: crate::replay::archive::sha256_hex(b"base-a"),
bytes: b"base-a".to_vec(),
origin_uris: vec!["rsync://rsync.example.test/repo/a.roa".to_string()],
object_type: Some("roa".to_string()),
encoding: None,
})
.expect("put raw by hash");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://rsync.example.test/repo/a.roa".to_string(),
current_hash: Some(crate::replay::archive::sha256_hex(b"base-a")),
repository_source: Some("rsync".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put repository view");
let delta = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayCurrentStoreRsyncFetcher::new(&store, delta);
let mut objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/")
.expect("fetch current-store objects");
objects.sort_by(|a, b| a.0.cmp(&b.0));
assert_eq!(objects.len(), 2);
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/a.roa");
assert_eq!(objects[0].1, b"base-a");
assert_eq!(objects[1].0, "rsync://rsync.example.test/repo/sub/b.cer");
assert_eq!(objects[1].1, b"delta-b");
}
#[test]
fn delta_rsync_fetcher_uses_base_only_when_delta_has_no_module_entry() {
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
build_base_and_delta_rsync_fixture();
std::fs::write(
&delta_locks,
r#"{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{},"rsync":{}}"#,
)
.expect("rewrite delta locks no rsync");
let base = Arc::new(
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
);
let delta = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
let mut objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/")
.expect("fetch base only objects");
objects.sort_by(|a, b| a.0.cmp(&b.0));
assert_eq!(objects.len(), 2);
assert_eq!(objects[0].1, b"base-a");
assert_eq!(objects[1].1, b"base-b");
}
#[test]
fn delta_rsync_fetcher_reports_missing_base_subtree_and_exposes_indexes() {
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
build_base_and_delta_rsync_fixture();
let base = Arc::new(
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
);
let delta = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base.clone(), delta.clone());
assert_eq!(fetcher.base_index().rsync_modules.len(), 1);
assert_eq!(fetcher.delta_index().rsync_modules.len(), 1);
let err = fetcher
.fetch_objects("rsync://rsync.example.test/repo/missing/")
.unwrap_err();
match err {
RsyncFetchError::Fetch(msg) => {
assert!(
msg.contains("delta replay rsync subtree not found"),
"{msg}"
)
}
}
}
#[test]
fn delta_rsync_fetcher_can_use_overlay_without_base_module() {
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
build_base_and_delta_rsync_fixture();
let module_hash =
crate::replay::archive::sha256_hex("rsync://rsync.example.test/repo/".as_bytes());
std::fs::remove_dir_all(
base_archive
.join("v1/captures/base-cap/rsync/modules")
.join(&module_hash),
)
.expect("remove base module dir");
let base = Arc::new(
ReplayArchiveIndex::load_allow_missing_rsync_modules(&base_archive, &base_locks)
.expect("load lenient base index"),
);
let delta = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
let objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/sub/")
.expect("fetch subtree from overlay only");
assert_eq!(objects.len(), 1);
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/sub/b.cer");
assert_eq!(objects[0].1, b"delta-b");
}
#[test]
fn delta_rsync_fetcher_merges_base_and_overlay() {
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
build_base_and_delta_rsync_fixture();
let base = Arc::new(
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
);
let delta = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
let mut objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/")
.expect("fetch merged objects");
objects.sort_by(|a, b| a.0.cmp(&b.0));
assert_eq!(objects.len(), 2);
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/a.roa");
assert_eq!(objects[0].1, b"base-a");
assert_eq!(objects[1].0, "rsync://rsync.example.test/repo/sub/b.cer");
assert_eq!(objects[1].1, b"delta-b");
}
#[test]
fn delta_rsync_fetcher_reads_subtree_and_rejects_missing_base_module() {
let (_temp, base_archive, base_locks, delta_archive, delta_locks) =
build_base_and_delta_rsync_fixture();
let base = Arc::new(
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
);
let delta = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let fetcher = PayloadDeltaReplayRsyncFetcher::new(base, delta);
let objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/sub/")
.expect("fetch subtree");
assert_eq!(objects.len(), 1);
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/sub/b.cer");
assert_eq!(objects[0].1, b"delta-b");
let err = fetcher
.fetch_objects("rsync://missing.example/repo/")
.unwrap_err();
match err {
RsyncFetchError::Fetch(msg) => assert!(
msg.contains("delta replay base rsync module not found")
|| msg.contains("no replay lock found for rsync module"),
"{msg}"
),
}
}
}

View File

@ -0,0 +1,399 @@
use std::collections::BTreeMap;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::replay::archive::{ReplayArchiveError, ReplayArchiveIndex, ReplayTransport};
use crate::sync::rrdp::{Fetcher, parse_notification};
#[derive(Debug, thiserror::Error)]
pub enum ReplayHttpFetcherError {
#[error(transparent)]
Archive(#[from] ReplayArchiveError),
#[error("read replay RRDP file failed: {path}: {detail}")]
ReadFile { path: String, detail: String },
#[error("parse locked notification failed for {notify_uri}: {detail}")]
ParseNotification { notify_uri: String, detail: String },
#[error(
"locked notification session/serial mismatch for {notify_uri}: expected session={expected_session} serial={expected_serial}, got session={actual_session} serial={actual_serial}"
)]
NotificationLockMismatch {
notify_uri: String,
expected_session: String,
expected_serial: u64,
actual_session: String,
actual_serial: u64,
},
#[error("locked delta file not found for {notify_uri} at serial {serial}: {path}")]
MissingLockedDelta {
notify_uri: String,
serial: u64,
path: String,
},
#[error("duplicate replay HTTP URI mapping for {uri}: {first_path} vs {second_path}")]
DuplicateUriMapping {
uri: String,
first_path: String,
second_path: String,
},
#[error("RRDP notification URI is locked to rsync transport in replay: {0}")]
LockedToRsync(String),
#[error("replay HTTP URI not found in archive: {0}")]
MissingUri(String),
}
#[derive(Clone, Debug)]
pub struct PayloadReplayHttpFetcher {
index: Arc<ReplayArchiveIndex>,
routes: BTreeMap<String, PathBuf>,
}
impl PayloadReplayHttpFetcher {
pub fn new(index: Arc<ReplayArchiveIndex>) -> Result<Self, ReplayHttpFetcherError> {
let mut routes = BTreeMap::new();
for (notify_uri, lock) in &index.locks.rrdp {
if lock.transport != ReplayTransport::Rrdp {
continue;
}
let repo = index.require_rrdp_repo(notify_uri)?;
insert_unique_route(&mut routes, notify_uri, &repo.locked_notification_path)?;
let notification_xml = fs::read(&repo.locked_notification_path).map_err(|e| {
ReplayHttpFetcherError::ReadFile {
path: repo.locked_notification_path.display().to_string(),
detail: e.to_string(),
}
})?;
let notification = parse_notification(&notification_xml).map_err(|e| {
ReplayHttpFetcherError::ParseNotification {
notify_uri: notify_uri.clone(),
detail: e.to_string(),
}
})?;
let actual_session = notification.session_id.to_string();
if actual_session != repo.locked_session || notification.serial != repo.locked_serial {
return Err(ReplayHttpFetcherError::NotificationLockMismatch {
notify_uri: notify_uri.clone(),
expected_session: repo.locked_session.clone(),
expected_serial: repo.locked_serial,
actual_session,
actual_serial: notification.serial,
});
}
insert_unique_route(
&mut routes,
&notification.snapshot_uri,
&repo.locked_snapshot_path,
)?;
for delta in notification.deltas {
let expected = repo.session_dir.join(format!(
"delta-{}-{}.xml",
delta.serial,
hex::encode(delta.hash_sha256)
));
if expected.is_file() {
insert_unique_route(&mut routes, &delta.uri, &expected)?;
}
}
}
Ok(Self { index, routes })
}
pub fn from_paths(
archive_root: impl AsRef<Path>,
locks_path: impl AsRef<Path>,
) -> Result<Self, ReplayHttpFetcherError> {
let index = Arc::new(ReplayArchiveIndex::load(archive_root, locks_path)?);
Self::new(index)
}
pub fn archive_index(&self) -> &ReplayArchiveIndex {
self.index.as_ref()
}
pub fn mapped_route(&self, uri: &str) -> Option<&Path> {
self.routes.get(uri).map(PathBuf::as_path)
}
}
impl Fetcher for PayloadReplayHttpFetcher {
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
if let Some(path) = self.routes.get(uri) {
return fs::read(path).map_err(|e| {
ReplayHttpFetcherError::ReadFile {
path: path.display().to_string(),
detail: e.to_string(),
}
.to_string()
});
}
if self
.index
.rrdp_lock(uri)
.is_some_and(|lock| lock.transport == ReplayTransport::Rsync)
{
return Err(ReplayHttpFetcherError::LockedToRsync(uri.to_string()).to_string());
}
Err(ReplayHttpFetcherError::MissingUri(uri.to_string()).to_string())
}
}
fn insert_unique_route(
routes: &mut BTreeMap<String, PathBuf>,
uri: &str,
path: &Path,
) -> Result<(), ReplayHttpFetcherError> {
if let Some(existing) = routes.get(uri) {
if existing != path {
return Err(ReplayHttpFetcherError::DuplicateUriMapping {
uri: uri.to_string(),
first_path: existing.display().to_string(),
second_path: path.display().to_string(),
});
}
return Ok(());
}
routes.insert(uri.to_string(), path.to_path_buf());
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::replay::archive::sha256_hex;
fn build_archive_with_rrdp_and_rsync(
with_delta: bool,
) -> (
tempfile::TempDir,
PathBuf,
PathBuf,
String,
String,
String,
Option<String>,
) {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-archive");
let capture = "capture-http";
let capture_root = archive_root.join("v1").join("captures").join(capture);
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
std::fs::write(
capture_root.join("capture.json"),
format!(
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
),
)
.expect("write capture meta");
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
let snapshot_uri = "https://rrdp.example.test/snapshot.xml".to_string();
let delta_uri = "https://rrdp.example.test/delta.xml".to_string();
let rsync_locked_notify = "https://rrdp-fallback.example.test/notification.xml".to_string();
let session = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string();
let serial = 99u64;
let bucket_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = capture_root
.join("rrdp/repos")
.join(bucket_hash)
.join(&session);
std::fs::create_dir_all(&session_dir).expect("mkdir session dir");
std::fs::write(
session_dir.parent().unwrap().join("meta.json"),
format!(
r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write repo meta");
let snapshot_hash = "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff";
let delta_hash = "ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100";
let notification_xml = if with_delta {
format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="{serial}">
<snapshot uri="{snapshot_uri}" hash="{snapshot_hash}" />
<delta serial="99" uri="{delta_uri}" hash="{delta_hash}" />
</notification>"#
)
} else {
format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="{serial}">
<snapshot uri="{snapshot_uri}" hash="{snapshot_hash}" />
</notification>"#
)
};
std::fs::write(session_dir.join("notification-99.xml"), notification_xml)
.expect("write notification");
std::fs::write(
session_dir.join(format!("snapshot-99-{snapshot_hash}.xml")),
b"<snapshot/>",
)
.expect("write snapshot");
if with_delta {
std::fs::write(
session_dir.join(format!("delta-99-{delta_hash}.xml")),
b"<delta/>",
)
.expect("write delta");
}
let module_uri = "rsync://rsync.example.test/repo/".to_string();
let module_hash = sha256_hex(module_uri.as_bytes());
let tree_root = capture_root
.join("rsync/modules")
.join(&module_hash)
.join("tree")
.join("rsync.example.test")
.join("repo");
std::fs::create_dir_all(&tree_root).expect("mkdir tree root");
let module_bucket_dir = capture_root.join("rsync/modules").join(&module_hash);
std::fs::write(
module_bucket_dir.join("meta.json"),
format!(
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write rsync meta");
std::fs::write(tree_root.join("x.cer"), b"cer").expect("write tree file");
let locks_path = temp.path().join("locks.json");
std::fs::write(
&locks_path,
format!(
r#"{{
"version":1,
"capture":"{capture}",
"rrdp":{{
"{notify_uri}":{{"transport":"rrdp","session":"{session}","serial":{serial}}},
"{rsync_locked_notify}":{{"transport":"rsync","session":null,"serial":null}}
}},
"rsync":{{
"{module_uri}":{{"transport":"rsync"}}
}}
}}"#
),
)
.expect("write locks");
(
temp,
archive_root,
locks_path,
notify_uri,
snapshot_uri,
rsync_locked_notify,
with_delta.then_some(delta_uri),
)
}
#[test]
fn payload_replay_http_fetcher_reads_locked_notification_and_snapshot_and_delta() {
let (_temp, archive_root, locks_path, notify_uri, snapshot_uri, _rsync_locked, delta_uri) =
build_archive_with_rrdp_and_rsync(true);
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher");
let notification = fetcher.fetch(&notify_uri).expect("fetch notification");
assert!(
std::str::from_utf8(&notification)
.expect("utf8")
.contains("<notification")
);
let snapshot = fetcher.fetch(&snapshot_uri).expect("fetch snapshot");
assert_eq!(snapshot, b"<snapshot/>");
let delta = fetcher
.fetch(delta_uri.as_deref().expect("delta uri"))
.expect("fetch delta");
assert_eq!(delta, b"<delta/>");
}
#[test]
fn payload_replay_http_fetcher_rejects_notification_locked_to_rsync() {
let (_temp, archive_root, locks_path, _notify_uri, _snapshot_uri, rsync_locked, _delta_uri) =
build_archive_with_rrdp_and_rsync(false);
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher");
let err = fetcher.fetch(&rsync_locked).unwrap_err();
assert!(err.contains("locked to rsync transport"), "{err}");
}
#[test]
fn payload_replay_http_fetcher_rejects_unknown_uri() {
let (
_temp,
archive_root,
locks_path,
_notify_uri,
_snapshot_uri,
_rsync_locked,
_delta_uri,
) = build_archive_with_rrdp_and_rsync(false);
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher");
let err = fetcher
.fetch("https://unknown.example/test.xml")
.unwrap_err();
assert!(err.contains("not found in archive"), "{err}");
}
#[test]
fn payload_replay_http_fetcher_rejects_notification_lock_mismatch() {
let (_temp, archive_root, locks_path, notify_uri, _snapshot_uri, _rsync_locked, _delta_uri) =
build_archive_with_rrdp_and_rsync(false);
let bucket_hash = sha256_hex(notify_uri.as_bytes());
let notification = archive_root
.join("v1/captures/capture-http/rrdp/repos")
.join(bucket_hash)
.join("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
.join("notification-99.xml");
std::fs::write(
&notification,
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb" serial="99">
<snapshot uri="https://rrdp.example.test/snapshot.xml" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
</notification>"#,
)
.expect("rewrite notification");
let err = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayHttpFetcherError::NotificationLockMismatch { .. }),
"{err}"
);
}
#[test]
fn payload_replay_http_fetcher_skips_missing_locked_delta_route() {
let (_temp, archive_root, locks_path, notify_uri, _snapshot_uri, _rsync_locked, _delta_uri) =
build_archive_with_rrdp_and_rsync(true);
let bucket_hash = sha256_hex(notify_uri.as_bytes());
let delta = archive_root
.join("v1/captures/capture-http/rrdp/repos")
.join(bucket_hash)
.join("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
.join("delta-99-ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100.xml");
std::fs::remove_file(delta).expect("remove delta");
let fetcher = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher without delta route");
let err = fetcher
.fetch("https://rrdp.example.test/delta.xml")
.unwrap_err();
assert!(err.contains("not found in archive"), "{err}");
}
}

View File

@ -0,0 +1,253 @@
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::fetch::rsync::{RsyncFetchError, RsyncFetchResult, RsyncFetcher};
use crate::replay::archive::{ReplayArchiveIndex, canonical_rsync_module};
#[derive(Clone, Debug)]
pub struct PayloadReplayRsyncFetcher {
index: Arc<ReplayArchiveIndex>,
}
impl PayloadReplayRsyncFetcher {
pub fn new(index: Arc<ReplayArchiveIndex>) -> Self {
Self { index }
}
pub fn from_paths(
archive_root: impl AsRef<Path>,
locks_path: impl AsRef<Path>,
) -> Result<Self, String> {
let index = Arc::new(
ReplayArchiveIndex::load(archive_root, locks_path).map_err(|e| e.to_string())?,
);
Ok(Self::new(index))
}
pub fn archive_index(&self) -> &ReplayArchiveIndex {
self.index.as_ref()
}
}
impl RsyncFetcher for PayloadReplayRsyncFetcher {
fn fetch_objects(&self, rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
let (module_uri, relative_path) = split_rsync_base_uri(rsync_base_uri)
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
let module = self
.index
.resolve_rsync_module_for_base_uri(rsync_base_uri)
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
let module_root = module_tree_root(&module_uri, &module.tree_dir)
.map_err(|e| RsyncFetchError::Fetch(e.to_string()))?;
let start = if relative_path.as_os_str().is_empty() {
module_root
} else {
module_root.join(&relative_path)
};
if !start.is_dir() {
return Err(RsyncFetchError::Fetch(format!(
"replay rsync subtree not found: {}",
start.display()
)));
}
let mut out = Vec::new();
walk_dir_collect(&start, &start, rsync_base_uri, &mut out)
.map_err(RsyncFetchError::Fetch)?;
Ok(out)
}
}
fn split_rsync_base_uri(rsync_base_uri: &str) -> Result<(String, PathBuf), String> {
let module_uri = canonical_rsync_module(rsync_base_uri).map_err(|e| e.to_string())?;
let rest = rsync_base_uri
.strip_prefix(&module_uri)
.unwrap_or_default()
.trim_matches('/');
let relative_path = if rest.is_empty() {
PathBuf::new()
} else {
let mut path = PathBuf::new();
for segment in rest.split('/') {
if !segment.is_empty() {
path.push(segment);
}
}
path
};
Ok((module_uri, relative_path))
}
fn module_tree_root(module_uri: &str, tree_dir: &Path) -> Result<PathBuf, String> {
let rest = module_uri
.strip_prefix("rsync://")
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
let mut parts = rest.trim_end_matches('/').split('/');
let authority = parts
.next()
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
let module = parts
.next()
.ok_or_else(|| format!("invalid rsync module URI: {module_uri}"))?;
Ok(tree_dir.join(authority).join(module))
}
fn walk_dir_collect(
root: &Path,
current: &Path,
rsync_base_uri: &str,
out: &mut Vec<(String, Vec<u8>)>,
) -> Result<(), String> {
let rd = fs::read_dir(current).map_err(|e| e.to_string())?;
for entry in rd {
let entry = entry.map_err(|e| e.to_string())?;
let path = entry.path();
let meta = entry.metadata().map_err(|e| e.to_string())?;
if meta.is_dir() {
walk_dir_collect(root, &path, rsync_base_uri, out)?;
continue;
}
if !meta.is_file() {
continue;
}
let rel = path
.strip_prefix(root)
.map_err(|e| e.to_string())?
.to_string_lossy()
.replace('\\', "/");
let base = if rsync_base_uri.ends_with('/') {
rsync_base_uri.to_string()
} else {
format!("{rsync_base_uri}/")
};
let uri = format!("{base}{rel}");
let bytes = fs::read(&path).map_err(|e| e.to_string())?;
out.push((uri, bytes));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn build_rsync_archive() -> (tempfile::TempDir, PathBuf, PathBuf) {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-archive");
let capture = "capture-rsync";
let capture_root = archive_root.join("v1").join("captures").join(capture);
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
std::fs::write(
capture_root.join("capture.json"),
format!(
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
),
)
.expect("write capture meta");
let module_uri = "rsync://rsync.example.test/repo/";
let mod_hash = crate::replay::archive::sha256_hex(module_uri.as_bytes());
let module_root = capture_root
.join("rsync/modules")
.join(&mod_hash)
.join("tree")
.join("rsync.example.test")
.join("repo");
std::fs::create_dir_all(module_root.join("child")).expect("mkdir module tree");
let module_bucket_dir = capture_root.join("rsync/modules").join(&mod_hash);
std::fs::write(
module_bucket_dir.join("meta.json"),
format!(
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write module meta");
std::fs::write(module_root.join("a.roa"), b"a").expect("write a.roa");
std::fs::write(module_root.join("child").join("b.cer"), b"b").expect("write b.cer");
let locks_path = temp.path().join("locks.json");
std::fs::write(
&locks_path,
format!(
r#"{{"version":1,"capture":"{capture}","rrdp":{{}},"rsync":{{"{module_uri}":{{"transport":"rsync"}}}}}}"#
),
)
.expect("write locks");
(temp, archive_root, locks_path)
}
#[test]
fn payload_replay_rsync_fetcher_reads_module_root() {
let (_temp, archive_root, locks_path) = build_rsync_archive();
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
let mut objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/")
.expect("fetch root objects");
objects.sort_by(|a, b| a.0.cmp(&b.0));
assert_eq!(objects.len(), 2);
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/a.roa");
assert_eq!(objects[0].1, b"a");
assert_eq!(objects[1].0, "rsync://rsync.example.test/repo/child/b.cer");
assert_eq!(objects[1].1, b"b");
}
#[test]
fn payload_replay_rsync_fetcher_reads_subtree_only() {
let (_temp, archive_root, locks_path) = build_rsync_archive();
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
let objects = fetcher
.fetch_objects("rsync://rsync.example.test/repo/child/")
.expect("fetch child subtree");
assert_eq!(objects.len(), 1);
assert_eq!(objects[0].0, "rsync://rsync.example.test/repo/child/b.cer");
assert_eq!(objects[0].1, b"b");
}
#[test]
fn payload_replay_rsync_fetcher_rejects_missing_module_and_subtree() {
let (_temp, archive_root, locks_path) = build_rsync_archive();
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
let err = fetcher
.fetch_objects("rsync://missing.example/repo/")
.unwrap_err();
match err {
RsyncFetchError::Fetch(msg) => assert!(
msg.contains("no replay lock found for rsync module"),
"{msg}"
),
}
let err = fetcher
.fetch_objects("rsync://rsync.example.test/repo/missing/")
.unwrap_err();
match err {
RsyncFetchError::Fetch(msg) => {
assert!(msg.contains("replay rsync subtree not found"), "{msg}")
}
}
}
#[test]
fn payload_replay_rsync_fetcher_rejects_invalid_uri_and_exposes_index() {
let (_temp, archive_root, locks_path) = build_rsync_archive();
let fetcher = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
assert_eq!(fetcher.archive_index().rsync_modules.len(), 1);
let err = fetcher
.fetch_objects("https://not-rsync.example/repo/")
.unwrap_err();
match err {
RsyncFetchError::Fetch(msg) => {
assert!(msg.contains("URI must start with rsync://"), "{msg}")
}
}
}
}

View File

@ -0,0 +1,6 @@
pub mod archive;
pub mod delta_archive;
pub mod delta_fetch_http;
pub mod delta_fetch_rsync;
pub mod fetch_http;
pub mod fetch_rsync;

View File

@ -0,0 +1,57 @@
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct RfcRef(pub &'static str);
/// Stable categories for warnings written to audit reports and exported metrics.
///
/// New warning sites should use a specific category when one is available. The
/// default preserves the behaviour and schema of existing callers while making
/// their metric label explicit.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum WarningCategory {
#[default]
Unclassified,
BerCompatibleCmsEncoding,
}
impl WarningCategory {
pub const fn as_str(self) -> &'static str {
match self {
Self::Unclassified => "unclassified",
Self::BerCompatibleCmsEncoding => "ber_compatible_cms_encoding",
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Warning {
pub message: String,
pub category: WarningCategory,
pub rfc_refs: Vec<RfcRef>,
pub context: Option<String>,
}
impl Warning {
pub fn new(message: impl Into<String>) -> Self {
Self {
message: message.into(),
category: WarningCategory::Unclassified,
rfc_refs: Vec::new(),
context: None,
}
}
pub fn with_category(mut self, category: WarningCategory) -> Self {
self.category = category;
self
}
pub fn with_rfc_refs(mut self, refs: &[RfcRef]) -> Self {
self.rfc_refs.extend_from_slice(refs);
self
}
pub fn with_context(mut self, context: impl Into<String>) -> Self {
self.context = Some(context.into());
self
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,178 @@
use rocksdb::{ColumnFamilyDescriptor, DBCompressionType, Options};
pub const CF_REPOSITORY_VIEW: &str = "repository_view";
pub const CF_RAW_BY_HASH: &str = "raw_by_hash";
pub const CF_RAW_BLOB: &str = "raw_blob";
pub const CF_VCIR: &str = "vcir";
pub const CF_VCIR_FAILED_FETCH_REUSE_IDENTITY: &str = "vcir_failed_fetch_reuse_identity";
pub const CF_MANIFEST_REPLAY_META: &str = "manifest_replay_meta";
pub const CF_ROA_CACHE_PROJECTION: &str = "roa_cache_projection";
pub const CF_PUBLICATION_POINT_CACHE_PROJECTION: &str = "publication_point_cache_projection";
pub const CF_CHILD_CERTIFICATE_CACHE_PROJECTION: &str = "child_certificate_cache_projection";
pub const CF_RRDP_SOURCE: &str = "rrdp_source";
pub const CF_RRDP_SOURCE_MEMBER: &str = "rrdp_source_member";
pub const CF_RRDP_URI_OWNER: &str = "rrdp_uri_owner";
pub const CF_TRANSPORT_PREFETCH: &str = "transport_prefetch";
pub const ALL_COLUMN_FAMILY_NAMES: &[&str] = &[
CF_REPOSITORY_VIEW,
CF_RAW_BY_HASH,
CF_RAW_BLOB,
CF_VCIR,
CF_VCIR_FAILED_FETCH_REUSE_IDENTITY,
CF_MANIFEST_REPLAY_META,
CF_ROA_CACHE_PROJECTION,
CF_PUBLICATION_POINT_CACHE_PROJECTION,
CF_CHILD_CERTIFICATE_CACHE_PROJECTION,
CF_RRDP_SOURCE,
CF_RRDP_SOURCE_MEMBER,
CF_RRDP_URI_OWNER,
CF_TRANSPORT_PREFETCH,
];
pub(super) const REPOSITORY_VIEW_KEY_PREFIX: &str = "repo_view:";
pub(super) const RAW_BY_HASH_KEY_PREFIX: &str = "rawbyhash:";
pub(super) const RAW_BLOB_KEY_PREFIX: &str = "rawblob:";
pub(super) const VCIR_KEY_PREFIX: &str = "vcir:";
pub(super) const VCIR_FAILED_FETCH_REUSE_IDENTITY_KEY_PREFIX: &str =
"vcir_failed_fetch_reuse_identity:";
pub(super) const MANIFEST_REPLAY_META_KEY_PREFIX: &str = "manifest_replay_meta:";
pub(super) const ROA_CACHE_PROJECTION_KEY_PREFIX: &str = "roa_cache_projection:";
pub(super) const PUBLICATION_POINT_CACHE_PROJECTION_KEY_PREFIX: &str =
"publication_point_cache_projection:";
pub(super) const CHILD_CERTIFICATE_CACHE_PROJECTION_KEY_PREFIX: &str =
"child_certificate_cache_projection:";
pub(super) const RRDP_SOURCE_KEY_PREFIX: &str = "rrdp_source:";
pub(super) const RRDP_SOURCE_MEMBER_KEY_PREFIX: &str = "rrdp_source_member:";
pub(super) const RRDP_URI_OWNER_KEY_PREFIX: &str = "rrdp_uri_owner:";
pub(super) const TRANSPORT_PREFETCH_LAST_SNAPSHOT_KEY: &str = "transport_prefetch:last_snapshot";
const WORK_DB_BLOB_MODE_ENV: &str = "RPKI_WORK_DB_BLOB_MODE";
const WORK_DB_MEMORY_PROFILE_ENV: &str = "RPKI_WORK_DB_MEMORY_PROFILE";
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) enum WorkDbBlobMode {
Current,
Disabled,
Lz4,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) enum WorkDbMemoryProfile {
Default,
Compact,
}
pub(super) fn parse_work_db_blob_mode(raw: &str) -> Option<WorkDbBlobMode> {
match raw.trim().to_ascii_lowercase().as_str() {
"" | "default" => Some(default_work_db_blob_mode()),
"current" | "legacy" => Some(WorkDbBlobMode::Current),
"disabled" | "disable" | "off" | "none" | "no_blob" | "no-blob" => {
Some(WorkDbBlobMode::Disabled)
}
"lz4" | "blob_lz4" | "blob-lz4" => Some(WorkDbBlobMode::Lz4),
_ => None,
}
}
pub(super) fn default_work_db_blob_mode() -> WorkDbBlobMode {
WorkDbBlobMode::Disabled
}
pub(super) fn work_db_blob_mode_from_env() -> WorkDbBlobMode {
let Ok(raw) = std::env::var(WORK_DB_BLOB_MODE_ENV) else {
return default_work_db_blob_mode();
};
match parse_work_db_blob_mode(&raw) {
Some(mode) => mode,
None => {
eprintln!(
"warning: unsupported {WORK_DB_BLOB_MODE_ENV}={raw:?}; using default work-db blobdb mode"
);
default_work_db_blob_mode()
}
}
}
pub(super) fn parse_work_db_memory_profile(raw: &str) -> Option<WorkDbMemoryProfile> {
match raw.trim().to_ascii_lowercase().as_str() {
"" | "default" | "off" | "none" => Some(WorkDbMemoryProfile::Default),
"compact" | "low" | "low_memory" | "low-memory" => Some(WorkDbMemoryProfile::Compact),
_ => None,
}
}
pub(super) fn work_db_memory_profile_from_env() -> WorkDbMemoryProfile {
let Ok(raw) = std::env::var(WORK_DB_MEMORY_PROFILE_ENV) else {
return WorkDbMemoryProfile::Default;
};
match parse_work_db_memory_profile(&raw) {
Some(profile) => profile,
None => {
eprintln!(
"warning: unsupported {WORK_DB_MEMORY_PROFILE_ENV}={raw:?}; using default work-db memory profile"
);
WorkDbMemoryProfile::Default
}
}
}
pub(super) fn configure_work_db_options(
opts: &mut Options,
blob_mode: WorkDbBlobMode,
memory_profile: WorkDbMemoryProfile,
) {
opts.set_compression_type(DBCompressionType::Lz4);
apply_work_db_memory_profile(opts, memory_profile);
match blob_mode {
WorkDbBlobMode::Current => enable_blobdb_current(opts),
WorkDbBlobMode::Disabled => {}
WorkDbBlobMode::Lz4 => {
enable_blobdb_current(opts);
opts.set_blob_compression_type(DBCompressionType::Lz4);
}
}
}
pub(super) fn cf_opts(blob_mode: WorkDbBlobMode, memory_profile: WorkDbMemoryProfile) -> Options {
let mut opts = Options::default();
configure_work_db_options(&mut opts, blob_mode, memory_profile);
opts
}
pub fn column_family_descriptors() -> Vec<ColumnFamilyDescriptor> {
column_family_descriptors_for_blob_mode(work_db_blob_mode_from_env())
}
pub(super) fn column_family_descriptors_for_blob_mode(
blob_mode: WorkDbBlobMode,
) -> Vec<ColumnFamilyDescriptor> {
let memory_profile = work_db_memory_profile_from_env();
ALL_COLUMN_FAMILY_NAMES
.iter()
.map(|name| ColumnFamilyDescriptor::new(*name, cf_opts(blob_mode, memory_profile)))
.collect()
}
fn apply_work_db_memory_profile(opts: &mut Options, memory_profile: WorkDbMemoryProfile) {
match memory_profile {
WorkDbMemoryProfile::Default => {}
WorkDbMemoryProfile::Compact => {
opts.set_write_buffer_size(16 * 1024 * 1024);
opts.set_max_write_buffer_number(1);
}
}
}
pub(super) fn enable_blobdb_current(opts: &mut Options) {
#[allow(unused_mut)]
let mut _enabled = false;
#[allow(dead_code)]
fn _set(opts: &mut Options) {
opts.set_enable_blob_files(true);
opts.set_min_blob_size(1024);
}
_set(opts);
}

View File

@ -0,0 +1,217 @@
use serde::{Serialize, de::DeserializeOwned};
use crate::data_model::common::der_take_tlv;
use super::config::*;
use super::pack::PackTime;
use super::{StorageError, StorageResult};
pub(super) fn repository_view_key(rsync_uri: &str) -> String {
format!("{REPOSITORY_VIEW_KEY_PREFIX}{rsync_uri}")
}
pub(super) fn repository_view_prefix(rsync_uri_prefix: &str) -> String {
format!("{REPOSITORY_VIEW_KEY_PREFIX}{rsync_uri_prefix}")
}
pub(super) fn raw_by_hash_key(sha256_hex: &str) -> String {
format!("{RAW_BY_HASH_KEY_PREFIX}{sha256_hex}")
}
pub(super) fn raw_blob_key(sha256_hex: &str) -> String {
format!("{RAW_BLOB_KEY_PREFIX}{sha256_hex}")
}
pub(super) fn vcir_key(manifest_rsync_uri: &str) -> String {
format!("{VCIR_KEY_PREFIX}{manifest_rsync_uri}")
}
pub(super) fn vcir_failed_fetch_reuse_identity_key(manifest_rsync_uri: &str) -> String {
format!("{VCIR_FAILED_FETCH_REUSE_IDENTITY_KEY_PREFIX}{manifest_rsync_uri}")
}
pub(super) fn manifest_replay_meta_key(manifest_rsync_uri: &str) -> String {
format!("{MANIFEST_REPLAY_META_KEY_PREFIX}{manifest_rsync_uri}")
}
pub(super) fn roa_cache_projection_key(manifest_rsync_uri: &str) -> String {
format!("{ROA_CACHE_PROJECTION_KEY_PREFIX}{manifest_rsync_uri}")
}
pub(super) fn publication_point_cache_projection_key(manifest_rsync_uri: &str) -> String {
format!("{PUBLICATION_POINT_CACHE_PROJECTION_KEY_PREFIX}{manifest_rsync_uri}")
}
pub(super) fn child_certificate_cache_projection_key(cache_key_sha256_hex: &str) -> String {
format!("{CHILD_CERTIFICATE_CACHE_PROJECTION_KEY_PREFIX}{cache_key_sha256_hex}")
}
pub(super) fn publication_point_cache_projection_key_manifest_uri(key: &[u8]) -> Option<String> {
let key = std::str::from_utf8(key).ok()?;
key.strip_prefix(PUBLICATION_POINT_CACHE_PROJECTION_KEY_PREFIX)
.map(str::to_string)
}
pub(super) fn rrdp_source_key(notify_uri: &str) -> String {
format!("{RRDP_SOURCE_KEY_PREFIX}{notify_uri}")
}
pub(super) fn rrdp_source_member_key(notify_uri: &str, rsync_uri: &str) -> String {
format!("{RRDP_SOURCE_MEMBER_KEY_PREFIX}{notify_uri}:{rsync_uri}")
}
pub(super) fn rrdp_source_member_prefix(notify_uri: &str) -> String {
format!("{RRDP_SOURCE_MEMBER_KEY_PREFIX}{notify_uri}:")
}
pub(super) fn rrdp_uri_owner_key(rsync_uri: &str) -> String {
format!("{RRDP_URI_OWNER_KEY_PREFIX}{rsync_uri}")
}
pub(super) fn encode_cbor<T: Serialize>(value: &T, entity: &'static str) -> StorageResult<Vec<u8>> {
serde_cbor::to_vec(value).map_err(|e| StorageError::Codec {
entity,
detail: e.to_string(),
})
}
pub(super) fn decode_cbor<T: DeserializeOwned>(
bytes: &[u8],
entity: &'static str,
) -> StorageResult<T> {
serde_cbor::from_slice(bytes).map_err(|e| StorageError::Codec {
entity,
detail: e.to_string(),
})
}
pub(super) fn validate_non_empty(field: &'static str, value: &str) -> StorageResult<()> {
if value.is_empty() {
return Err(StorageError::InvalidData {
entity: field,
detail: "must not be empty".to_string(),
});
}
Ok(())
}
pub(super) fn validate_sha256_hex(field: &'static str, value: &str) -> StorageResult<()> {
if value.len() != 64 || !value.as_bytes().iter().all(u8::is_ascii_hexdigit) {
return Err(StorageError::InvalidData {
entity: field,
detail: "must be a 64-character lowercase or uppercase SHA-256 hex string".to_string(),
});
}
Ok(())
}
pub(super) fn decode_sha256_hex_32(field: &'static str, value: &str) -> StorageResult<[u8; 32]> {
validate_sha256_hex(field, value)?;
let mut out = [0u8; 32];
hex::decode_to_slice(value, &mut out).map_err(|e| StorageError::InvalidData {
entity: field,
detail: format!("hex decode failed: {e}"),
})?;
Ok(out)
}
pub(super) fn validate_manifest_number_be(field: &'static str, value: &[u8]) -> StorageResult<()> {
if value.is_empty() {
return Err(StorageError::InvalidData {
entity: field,
detail: "must not be empty".to_string(),
});
}
if value.len() > 20 {
return Err(StorageError::InvalidData {
entity: field,
detail: "must be at most 20 octets".to_string(),
});
}
if value.len() > 1 && value[0] == 0 {
return Err(StorageError::InvalidData {
entity: field,
detail: "must be minimal big-endian without leading zeros".to_string(),
});
}
Ok(())
}
pub(super) fn validate_sha256_digest_bytes(field: &'static str, value: &[u8]) -> StorageResult<()> {
if value.len() != 32 {
return Err(StorageError::InvalidData {
entity: field,
detail: format!("must be 32 bytes, got {}", value.len()),
});
}
Ok(())
}
pub(super) fn validate_fixed_len_bytes(
field: &'static str,
value: &[u8],
expected_len: usize,
) -> StorageResult<()> {
if value.len() != expected_len {
return Err(StorageError::InvalidData {
entity: field,
detail: format!("must be {expected_len} bytes, got {}", value.len()),
});
}
Ok(())
}
pub(super) fn validate_sorted_unique_fixed_len_bytes(
field: &'static str,
values: &[Vec<u8>],
expected_len: usize,
) -> StorageResult<()> {
for value in values {
validate_fixed_len_bytes(field, value, expected_len)?;
}
for window in values.windows(2) {
if window[0] >= window[1] {
return Err(StorageError::InvalidData {
entity: field,
detail: "must be strictly sorted and unique".to_string(),
});
}
}
Ok(())
}
pub(super) fn validate_full_der_with_tag(
field: &'static str,
der: &[u8],
expected_tag: Option<u8>,
) -> StorageResult<()> {
let (tag, _value, rem) = der_take_tlv(der).map_err(|detail| StorageError::InvalidData {
entity: field,
detail,
})?;
if !rem.is_empty() {
return Err(StorageError::InvalidData {
entity: field,
detail: "trailing bytes after DER object".to_string(),
});
}
if let Some(expected_tag) = expected_tag {
if tag != expected_tag {
return Err(StorageError::InvalidData {
entity: field,
detail: format!("unexpected tag 0x{tag:02X}, expected 0x{expected_tag:02X}"),
});
}
}
Ok(())
}
pub(super) fn parse_time(
field: &'static str,
value: &PackTime,
) -> StorageResult<time::OffsetDateTime> {
value.parse().map_err(|detail| StorageError::InvalidData {
entity: field,
detail,
})
}

View File

@ -0,0 +1,200 @@
use serde::{Deserialize, Serialize};
use sha2::Digest;
use crate::blob_store::{ExternalRawStoreDb, ExternalRepoBytesDb, RawObjectStore};
#[derive(Clone, Debug)]
pub enum PackBytes {
Eager(std::sync::Arc<[u8]>),
LazyExternal {
sha256_hex: String,
store: std::sync::Arc<ExternalRawStoreDb>,
cache: std::sync::Arc<std::sync::OnceLock<std::sync::Arc<[u8]>>>,
},
LazyRepoBytes {
sha256_hex: String,
store: std::sync::Arc<ExternalRepoBytesDb>,
cache: std::sync::Arc<std::sync::OnceLock<std::sync::Arc<[u8]>>>,
},
}
impl PackBytes {
pub fn eager(bytes: Vec<u8>) -> Self {
Self::Eager(std::sync::Arc::from(bytes))
}
pub fn lazy_external(sha256_hex: String, store: std::sync::Arc<ExternalRawStoreDb>) -> Self {
Self::LazyExternal {
sha256_hex,
store,
cache: std::sync::Arc::new(std::sync::OnceLock::new()),
}
}
pub fn lazy_repo_bytes(sha256_hex: String, store: std::sync::Arc<ExternalRepoBytesDb>) -> Self {
Self::LazyRepoBytes {
sha256_hex,
store,
cache: std::sync::Arc::new(std::sync::OnceLock::new()),
}
}
pub fn as_slice(&self) -> Result<&[u8], String> {
match self {
Self::Eager(bytes) => Ok(bytes.as_ref()),
Self::LazyExternal {
sha256_hex,
store,
cache,
} => {
if cache.get().is_none() {
let bytes = store
.get_blob_bytes(sha256_hex)
.map_err(|e| e.to_string())?
.ok_or_else(|| format!("missing raw blob for sha256={sha256_hex}"))?;
let _ = cache.set(std::sync::Arc::from(bytes));
}
let bytes = cache
.get()
.ok_or_else(|| format!("missing raw blob cache for sha256={sha256_hex}"))?;
Ok(bytes.as_ref())
}
Self::LazyRepoBytes {
sha256_hex,
store,
cache,
} => {
if cache.get().is_none() {
let bytes = store
.get_blob_bytes(sha256_hex)
.map_err(|e| e.to_string())?
.ok_or_else(|| format!("missing repo bytes for sha256={sha256_hex}"))?;
let _ = cache.set(std::sync::Arc::from(bytes));
}
let bytes = cache
.get()
.ok_or_else(|| format!("missing repo bytes cache for sha256={sha256_hex}"))?;
Ok(bytes.as_ref())
}
}
}
pub fn to_vec(&self) -> Result<Vec<u8>, String> {
Ok(self.as_slice()?.to_vec())
}
}
impl PartialEq for PackBytes {
fn eq(&self, other: &Self) -> bool {
match (self.as_slice(), other.as_slice()) {
(Ok(a), Ok(b)) => a == b,
_ => false,
}
}
}
impl Eq for PackBytes {}
#[derive(Clone, Debug)]
pub struct PackFile {
pub rsync_uri: String,
pub bytes: PackBytes,
pub sha256: [u8; 32],
}
impl PackFile {
pub fn new(rsync_uri: impl Into<String>, bytes: PackBytes, sha256: [u8; 32]) -> Self {
Self {
rsync_uri: rsync_uri.into(),
bytes,
sha256,
}
}
pub fn from_bytes_with_sha256(
rsync_uri: impl Into<String>,
bytes: Vec<u8>,
sha256: [u8; 32],
) -> Self {
Self::new(rsync_uri, PackBytes::eager(bytes), sha256)
}
pub fn from_lazy_external_raw_store(
rsync_uri: impl Into<String>,
sha256_hex: String,
sha256: [u8; 32],
store: std::sync::Arc<ExternalRawStoreDb>,
) -> Self {
Self::new(
rsync_uri,
PackBytes::lazy_external(sha256_hex, store),
sha256,
)
}
pub fn from_lazy_repo_bytes(
rsync_uri: impl Into<String>,
sha256_hex: String,
sha256: [u8; 32],
store: std::sync::Arc<ExternalRepoBytesDb>,
) -> Self {
Self::new(
rsync_uri,
PackBytes::lazy_repo_bytes(sha256_hex, store),
sha256,
)
}
pub fn from_bytes_compute_sha256(rsync_uri: impl Into<String>, bytes: Vec<u8>) -> Self {
let sha256 = compute_sha256_32(&bytes);
Self::new(rsync_uri, PackBytes::eager(bytes), sha256)
}
pub fn bytes(&self) -> Result<&[u8], String> {
self.bytes.as_slice()
}
pub fn bytes_cloned(&self) -> Result<Vec<u8>, String> {
self.bytes.to_vec()
}
pub fn compute_sha256(&self) -> Result<[u8; 32], String> {
Ok(compute_sha256_32(self.bytes()?))
}
}
impl PartialEq for PackFile {
fn eq(&self, other: &Self) -> bool {
self.rsync_uri == other.rsync_uri
&& self.sha256 == other.sha256
&& self.bytes == other.bytes
}
}
impl Eq for PackFile {}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct PackTime {
pub rfc3339_utc: String,
}
impl PackTime {
pub fn from_utc_offset_datetime(t: time::OffsetDateTime) -> Self {
use time::format_description::well_known::Rfc3339;
let utc = t.to_offset(time::UtcOffset::UTC);
let s = utc.format(&Rfc3339).expect("format RFC 3339 UTC time");
Self { rfc3339_utc: s }
}
pub fn parse(&self) -> Result<time::OffsetDateTime, String> {
use time::format_description::well_known::Rfc3339;
time::OffsetDateTime::parse(&self.rfc3339_utc, &Rfc3339).map_err(|e| e.to_string())
}
}
pub(super) fn compute_sha256_32(bytes: &[u8]) -> [u8; 32] {
let digest = sha2::Sha256::digest(bytes);
let mut out = [0u8; 32];
out.copy_from_slice(&digest);
out
}

View File

@ -0,0 +1,671 @@
use std::collections::{BTreeMap, HashMap};
use std::fs::{self, File};
use std::io::Write;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Instant, SystemTime, UNIX_EPOCH};
use memmap2::Mmap;
use serde::Serialize;
use super::{StorageError, StorageResult};
const MAGIC: &[u8; 12] = b"RPKIPPIDX\0\0\0";
const VERSION: u32 = 1;
const HEADER_LEN: usize = 64;
const ENTRY_LEN: usize = 24;
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct PpCacheIndexLoadStats {
pub source: String,
pub entries: usize,
pub bytes: usize,
pub file_bytes: u64,
pub load_ms: u64,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct PpCacheIndexRefreshStats {
pub state: String,
pub old_entries: usize,
pub dirty_entries: usize,
pub new_entries: usize,
pub file_bytes: u64,
pub write_ms: u64,
pub compaction_triggered: bool,
pub compaction_reason: Option<String>,
pub compaction_segments_before: usize,
pub compaction_total_file_bytes_before: u64,
pub compaction_live_entries: usize,
pub compaction_file_bytes: u64,
pub compaction_reclaimed_bytes: u64,
pub compaction_ms: u64,
pub compaction_deleted_segments: usize,
pub compaction_error: Option<String>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct PpCacheIndexDirectoryStats {
pub segment_count: usize,
pub total_file_bytes: u64,
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct PpCacheIndexEntry {
value_offset: usize,
value_len: usize,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum PpCacheIndexLookup<'a> {
Hit(&'a [u8]),
Deleted,
}
#[derive(Debug)]
pub struct PpCacheMmapIndex {
mmap: Arc<Mmap>,
entries: HashMap<String, PpCacheIndexEntry>,
bytes: usize,
file_bytes: u64,
}
impl PpCacheMmapIndex {
pub fn open(path: &Path) -> StorageResult<Self> {
let file = File::open(path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
let file_len = file
.metadata()
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.len() as usize;
if file_len < HEADER_LEN {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_mmap_index",
detail: "file too small".to_string(),
});
}
// SAFETY: The mmap is read-only, held by Arc for all returned slices, and the file
// format is bounds-checked before any slice is exposed.
let mmap = unsafe { Mmap::map(&file) }.map_err(|e| StorageError::RocksDb(e.to_string()))?;
if &mmap[0..MAGIC.len()] != MAGIC {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_mmap_index",
detail: "invalid magic".to_string(),
});
}
let version = read_u32(&mmap, 12)?;
if version != VERSION {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_mmap_index",
detail: format!("unsupported version {version}"),
});
}
let entry_count = read_u64(&mmap, 16)? as usize;
let entry_table_offset = read_u64(&mmap, 24)? as usize;
let entry_table_len = read_u64(&mmap, 32)? as usize;
let blob_offset = read_u64(&mmap, 40)? as usize;
let blob_len = read_u64(&mmap, 48)? as usize;
checked_range(file_len, entry_table_offset, entry_table_len)?;
checked_range(file_len, blob_offset, blob_len)?;
if entry_table_len != entry_count.saturating_mul(ENTRY_LEN) {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_mmap_index",
detail: "entry table length mismatch".to_string(),
});
}
let mut entries = HashMap::with_capacity(entry_count);
let mut bytes = 0usize;
for offset in (entry_table_offset..entry_table_offset + entry_table_len).step_by(ENTRY_LEN)
{
let key_offset = read_u64(&mmap, offset)? as usize;
let key_len = read_u32(&mmap, offset + 8)? as usize;
let value_offset = read_u64(&mmap, offset + 12)? as usize;
let value_len = read_u32(&mmap, offset + 20)? as usize;
checked_range(blob_len, key_offset, key_len)?;
checked_range(blob_len, value_offset, value_len)?;
let key_start = blob_offset + key_offset;
let key_end = key_start + key_len;
let key = std::str::from_utf8(&mmap[key_start..key_end])
.map_err(|e| StorageError::InvalidData {
entity: "publication_point_cache_mmap_index.key",
detail: e.to_string(),
})?
.to_string();
bytes = bytes.saturating_add(value_len);
entries.insert(
key,
PpCacheIndexEntry {
value_offset,
value_len,
},
);
}
Ok(Self {
mmap: Arc::new(mmap),
entries,
bytes,
file_bytes: file_len as u64,
})
}
pub fn lookup(&self, manifest_rsync_uri: &str) -> Option<PpCacheIndexLookup<'_>> {
let entry = self.entries.get(manifest_rsync_uri)?;
if entry.value_len == 0 {
return Some(PpCacheIndexLookup::Deleted);
}
let blob_offset = read_u64(self.mmap.as_ref(), 40).ok()? as usize;
let start = blob_offset + entry.value_offset;
let end = start + entry.value_len;
Some(PpCacheIndexLookup::Hit(&self.mmap[start..end]))
}
#[cfg(test)]
pub fn get(&self, manifest_rsync_uri: &str) -> Option<&[u8]> {
match self.lookup(manifest_rsync_uri)? {
PpCacheIndexLookup::Hit(bytes) => Some(bytes),
PpCacheIndexLookup::Deleted => None,
}
}
pub fn entries(&self) -> usize {
self.entries.len()
}
pub fn bytes(&self) -> usize {
self.bytes
}
pub fn file_bytes(&self) -> u64 {
self.file_bytes
}
}
#[derive(Debug)]
pub struct PpCacheMmapIndexSet {
indexes: Vec<PpCacheMmapIndex>,
entries: usize,
bytes: usize,
file_bytes: u64,
}
impl PpCacheMmapIndexSet {
pub fn lookup(&self, manifest_rsync_uri: &str) -> Option<PpCacheIndexLookup<'_>> {
self.indexes
.iter()
.find_map(|index| index.lookup(manifest_rsync_uri))
}
#[cfg(test)]
pub fn get(&self, manifest_rsync_uri: &str) -> Option<&[u8]> {
match self.lookup(manifest_rsync_uri)? {
PpCacheIndexLookup::Hit(bytes) => Some(bytes),
PpCacheIndexLookup::Deleted => None,
}
}
pub fn entries(&self) -> usize {
self.entries
}
pub fn bytes(&self) -> usize {
self.bytes
}
pub fn file_bytes(&self) -> u64 {
self.file_bytes
}
}
pub fn default_pp_cache_index_dir(db_path: &Path) -> PathBuf {
let file_name = db_path
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("work-db");
db_path.with_file_name(format!("{file_name}.pp-cache-index"))
}
pub fn load_pp_cache_mmap_index(
path: &Path,
) -> StorageResult<(PpCacheMmapIndex, PpCacheIndexLoadStats)> {
let started = Instant::now();
let index = PpCacheMmapIndex::open(path)?;
let file_bytes = fs::metadata(path)
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.len();
let stats = PpCacheIndexLoadStats {
source: "mmap".to_string(),
entries: index.entries(),
bytes: index.bytes(),
file_bytes,
load_ms: started.elapsed().as_millis() as u64,
};
Ok((index, stats))
}
pub fn load_pp_cache_mmap_index_set(
dir: &Path,
) -> StorageResult<(PpCacheMmapIndexSet, PpCacheIndexLoadStats)> {
let started = Instant::now();
let mut paths = Vec::new();
if dir.exists() {
let mut segments = fs::read_dir(dir)
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.filter_map(Result::ok)
.map(|entry| entry.path())
.filter(|path| {
path.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| name.starts_with("segment-") && name.ends_with(".idx"))
})
.collect::<Vec<_>>();
segments.sort();
segments.reverse();
paths.extend(segments);
let current = dir.join("current.idx");
if current.exists() {
paths.push(current);
}
}
if paths.is_empty() {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_mmap_index",
detail: "index file missing".to_string(),
});
}
let mut indexes = Vec::new();
let mut entries = 0usize;
let mut bytes = 0usize;
let mut file_bytes = 0u64;
for path in paths {
let (index, _) = load_pp_cache_mmap_index(&path)?;
entries = entries.saturating_add(index.entries());
bytes = bytes.saturating_add(index.bytes());
file_bytes = file_bytes.saturating_add(index.file_bytes());
indexes.push(index);
}
let set = PpCacheMmapIndexSet {
indexes,
entries,
bytes,
file_bytes,
};
let stats = PpCacheIndexLoadStats {
source: "mmap".to_string(),
entries: set.entries(),
bytes: set.bytes(),
file_bytes: set.file_bytes(),
load_ms: started.elapsed().as_millis() as u64,
};
Ok((set, stats))
}
pub fn pp_cache_index_directory_stats(dir: &Path) -> StorageResult<PpCacheIndexDirectoryStats> {
let mut stats = PpCacheIndexDirectoryStats::default();
if !dir.exists() {
return Ok(stats);
}
for entry in fs::read_dir(dir).map_err(|e| StorageError::RocksDb(e.to_string()))? {
let path = entry
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.path();
let Some(name) = path.file_name().and_then(|name| name.to_str()) else {
continue;
};
let is_index =
name == "current.idx" || (name.starts_with("segment-") && name.ends_with(".idx"));
if !is_index {
continue;
}
let len = fs::metadata(&path)
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.len();
stats.total_file_bytes = stats.total_file_bytes.saturating_add(len);
if name.starts_with("segment-") && name.ends_with(".idx") {
stats.segment_count += 1;
}
}
Ok(stats)
}
pub fn compact_pp_cache_index(dir: &Path) -> StorageResult<PpCacheIndexRefreshStats> {
let started = Instant::now();
fs::create_dir_all(dir).map_err(|e| StorageError::RocksDb(e.to_string()))?;
let before = pp_cache_index_directory_stats(dir)?;
let (index_set, _) = load_pp_cache_mmap_index_set(dir)?;
let mut compact_entries = Vec::with_capacity(index_set.entries());
let mut seen = HashMap::<String, ()>::with_capacity(index_set.entries());
for index in &index_set.indexes {
for (key, entry) in &index.entries {
if seen.insert(key.clone(), ()).is_some() {
continue;
}
if entry.value_len == 0 {
continue;
}
let blob_offset = read_u64(index.mmap.as_ref(), 40)? as usize;
let start = blob_offset + entry.value_offset;
let end = start + entry.value_len;
compact_entries.push((key.clone(), index.mmap[start..end].to_vec()));
}
}
let current = dir.join("current.idx");
let mut stats = write_pp_cache_index_atomic(&current, compact_entries)?;
stats.state = "compacted".to_string();
stats.compaction_triggered = true;
stats.compaction_segments_before = before.segment_count;
stats.compaction_total_file_bytes_before = before.total_file_bytes;
stats.compaction_live_entries = stats.new_entries;
stats.compaction_file_bytes = stats.file_bytes;
stats.compaction_reclaimed_bytes = before.total_file_bytes.saturating_sub(stats.file_bytes);
let mut deleted_segments = 0usize;
for entry in fs::read_dir(dir).map_err(|e| StorageError::RocksDb(e.to_string()))? {
let path = entry
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.path();
let Some(name) = path.file_name().and_then(|name| name.to_str()) else {
continue;
};
if name.starts_with("segment-") && name.ends_with(".idx") {
fs::remove_file(&path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
deleted_segments += 1;
}
}
stats.compaction_deleted_segments = deleted_segments;
stats.compaction_ms = started.elapsed().as_millis() as u64;
Ok(stats)
}
pub fn write_pp_cache_index_segment<I>(
dir: &Path,
entries: I,
) -> StorageResult<PpCacheIndexRefreshStats>
where
I: IntoIterator<Item = (String, Vec<u8>)>,
{
fs::create_dir_all(dir).map_err(|e| StorageError::RocksDb(e.to_string()))?;
let current = dir.join("current.idx");
if !current.exists() {
return write_pp_cache_index_atomic(&current, entries);
}
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.as_nanos();
let segment = dir.join(format!("segment-{now:020}-{}.idx", std::process::id()));
let mut stats = write_pp_cache_index_atomic(&segment, entries)?;
stats.state = "segment_written".to_string();
Ok(stats)
}
pub fn write_pp_cache_index_atomic<I>(
path: &Path,
entries: I,
) -> StorageResult<PpCacheIndexRefreshStats>
where
I: IntoIterator<Item = (String, Vec<u8>)>,
{
let started = Instant::now();
let parent = path.parent().ok_or_else(|| StorageError::InvalidData {
entity: "publication_point_cache_mmap_index.path",
detail: "missing parent".to_string(),
})?;
fs::create_dir_all(parent).map_err(|e| StorageError::RocksDb(e.to_string()))?;
let mut ordered = BTreeMap::<String, Vec<u8>>::new();
for (key, value) in entries {
ordered.insert(key, value);
}
let tmp_path = parent.join("next.tmp");
write_pp_cache_index_file(
&tmp_path,
ordered.iter().map(|(k, v)| (k.as_str(), v.as_slice())),
)?;
let file_bytes = fs::metadata(&tmp_path)
.map_err(|e| StorageError::RocksDb(e.to_string()))?
.len();
fs::rename(&tmp_path, path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(PpCacheIndexRefreshStats {
state: "written".to_string(),
old_entries: 0,
dirty_entries: 0,
new_entries: ordered.len(),
file_bytes,
write_ms: started.elapsed().as_millis() as u64,
..PpCacheIndexRefreshStats::default()
})
}
fn write_pp_cache_index_file<'a, I>(path: &Path, entries: I) -> StorageResult<()>
where
I: IntoIterator<Item = (&'a str, &'a [u8])>,
{
let entries = entries.into_iter().collect::<Vec<_>>();
let entry_table_offset = HEADER_LEN;
let entry_table_len = entries.len() * ENTRY_LEN;
let blob_offset = entry_table_offset + entry_table_len;
let mut table = Vec::with_capacity(entry_table_len);
let mut blob = Vec::new();
for (key, value) in entries.iter() {
let key_offset = blob.len();
blob.extend_from_slice(key.as_bytes());
let value_offset = blob.len();
blob.extend_from_slice(value);
write_u64(&mut table, key_offset as u64);
write_u32(&mut table, key.len() as u32);
write_u64(&mut table, value_offset as u64);
write_u32(&mut table, value.len() as u32);
}
let mut header = Vec::with_capacity(HEADER_LEN);
header.extend_from_slice(MAGIC);
write_u32(&mut header, VERSION);
write_u64(&mut header, entries.len() as u64);
write_u64(&mut header, entry_table_offset as u64);
write_u64(&mut header, entry_table_len as u64);
write_u64(&mut header, blob_offset as u64);
write_u64(&mut header, blob.len() as u64);
header.resize(HEADER_LEN, 0);
let mut file = File::create(path).map_err(|e| StorageError::RocksDb(e.to_string()))?;
file.write_all(&header)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
file.write_all(&table)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
file.write_all(&blob)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
file.sync_all()
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
fn checked_range(total: usize, offset: usize, len: usize) -> StorageResult<()> {
if offset.checked_add(len).is_none_or(|end| end > total) {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_mmap_index.range",
detail: "out of bounds".to_string(),
});
}
Ok(())
}
fn read_u32(bytes: &[u8], offset: usize) -> StorageResult<u32> {
checked_range(bytes.len(), offset, 4)?;
Ok(u32::from_le_bytes(
bytes[offset..offset + 4].try_into().unwrap(),
))
}
fn read_u64(bytes: &[u8], offset: usize) -> StorageResult<u64> {
checked_range(bytes.len(), offset, 8)?;
Ok(u64::from_le_bytes(
bytes[offset..offset + 8].try_into().unwrap(),
))
}
fn write_u32(out: &mut Vec<u8>, value: u32) {
out.extend_from_slice(&value.to_le_bytes());
}
fn write_u64(out: &mut Vec<u8>, value: u64) {
out.extend_from_slice(&value.to_le_bytes());
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn pp_cache_index_roundtrips_multiple_entries() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("current.idx");
let stats = write_pp_cache_index_atomic(
&path,
vec![
("rsync://example.test/a.mft".to_string(), b"aaa".to_vec()),
("rsync://example.test/b.mft".to_string(), b"bbbb".to_vec()),
],
)
.expect("write index");
assert_eq!(stats.new_entries, 2);
let (index, load) = load_pp_cache_mmap_index(&path).expect("load index");
assert_eq!(load.entries, 2);
assert_eq!(index.get("rsync://example.test/a.mft"), Some(&b"aaa"[..]));
assert_eq!(index.get("rsync://example.test/b.mft"), Some(&b"bbbb"[..]));
assert_eq!(index.get("rsync://example.test/missing.mft"), None);
}
#[test]
fn pp_cache_index_rejects_bad_magic() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("current.idx");
fs::write(&path, b"bad").expect("write bad");
let err = load_pp_cache_mmap_index(&path).expect_err("bad index rejected");
assert!(err.to_string().contains("file too small"));
}
#[test]
fn pp_cache_index_last_duplicate_wins() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("current.idx");
write_pp_cache_index_atomic(
&path,
vec![
("rsync://example.test/a.mft".to_string(), b"old".to_vec()),
("rsync://example.test/a.mft".to_string(), b"new".to_vec()),
],
)
.expect("write index");
let (index, _) = load_pp_cache_mmap_index(&path).expect("load index");
assert_eq!(index.entries(), 1);
assert_eq!(index.get("rsync://example.test/a.mft"), Some(&b"new"[..]));
}
#[test]
fn pp_cache_index_set_prefers_newest_segment() {
let dir = tempfile::tempdir().expect("tempdir");
let current = dir.path().join("current.idx");
write_pp_cache_index_atomic(
&current,
vec![("rsync://example.test/a.mft".to_string(), b"old".to_vec())],
)
.expect("write current index");
write_pp_cache_index_segment(
dir.path(),
vec![("rsync://example.test/a.mft".to_string(), b"new".to_vec())],
)
.expect("write segment index");
let (set, stats) = load_pp_cache_mmap_index_set(dir.path()).expect("load index set");
assert_eq!(stats.entries, 2);
assert_eq!(set.get("rsync://example.test/a.mft"), Some(&b"new"[..]));
}
#[test]
fn pp_cache_index_set_tombstone_shadows_older_entry() {
let dir = tempfile::tempdir().expect("tempdir");
let current = dir.path().join("current.idx");
write_pp_cache_index_atomic(
&current,
vec![("rsync://example.test/a.mft".to_string(), b"old".to_vec())],
)
.expect("write current index");
write_pp_cache_index_segment(
dir.path(),
vec![("rsync://example.test/a.mft".to_string(), Vec::new())],
)
.expect("write tombstone segment index");
let (set, stats) = load_pp_cache_mmap_index_set(dir.path()).expect("load index set");
assert_eq!(stats.entries, 2);
assert_eq!(
set.lookup("rsync://example.test/a.mft"),
Some(PpCacheIndexLookup::Deleted)
);
assert_eq!(set.get("rsync://example.test/a.mft"), None);
}
#[test]
fn pp_cache_index_compaction_keeps_latest_values_and_removes_segments() {
let dir = tempfile::tempdir().expect("tempdir");
let current = dir.path().join("current.idx");
write_pp_cache_index_atomic(
&current,
vec![
("rsync://example.test/a.mft".to_string(), b"old-a".to_vec()),
("rsync://example.test/b.mft".to_string(), b"old-b".to_vec()),
("rsync://example.test/c.mft".to_string(), b"old-c".to_vec()),
],
)
.expect("write current index");
write_pp_cache_index_segment(
dir.path(),
vec![
("rsync://example.test/a.mft".to_string(), b"new-a".to_vec()),
("rsync://example.test/b.mft".to_string(), Vec::new()),
],
)
.expect("write first segment");
write_pp_cache_index_segment(
dir.path(),
vec![("rsync://example.test/d.mft".to_string(), b"new-d".to_vec())],
)
.expect("write second segment");
let before = pp_cache_index_directory_stats(dir.path()).expect("stats before");
assert_eq!(before.segment_count, 2);
let stats = compact_pp_cache_index(dir.path()).expect("compact");
assert!(stats.compaction_triggered);
assert_eq!(stats.compaction_segments_before, 2);
assert_eq!(stats.compaction_live_entries, 3);
assert_eq!(stats.compaction_deleted_segments, 2);
assert!(stats.compaction_reclaimed_bytes > 0);
let after = pp_cache_index_directory_stats(dir.path()).expect("stats after");
assert_eq!(after.segment_count, 0);
let segments = fs::read_dir(dir.path())
.expect("read dir")
.filter_map(Result::ok)
.filter(|entry| {
entry
.file_name()
.to_str()
.is_some_and(|name| name.starts_with("segment-"))
})
.count();
assert_eq!(segments, 0);
let (index, _) = load_pp_cache_mmap_index(&current).expect("load compacted current");
assert_eq!(index.entries(), 3);
assert_eq!(index.get("rsync://example.test/a.mft"), Some(&b"new-a"[..]));
assert_eq!(index.get("rsync://example.test/b.mft"), None);
assert_eq!(index.get("rsync://example.test/c.mft"), Some(&b"old-c"[..]));
assert_eq!(index.get("rsync://example.test/d.mft"), Some(&b"new-d"[..]));
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,3 @@
pub mod repo;
pub mod rrdp;
pub(crate) mod store_projection;

View File

@ -0,0 +1,728 @@
use crate::analysis::timing::TimingHandle;
use crate::audit::AuditDownloadKind;
use crate::audit_downloads::DownloadLogHandle;
use crate::current_repo_index::CurrentRepoIndexHandle;
use crate::fetch::rsync::{RsyncFetchError, RsyncFetcher};
use crate::policy::{Policy, SyncPreference};
use crate::replay::archive::{ReplayArchiveIndex, ReplayTransport};
use crate::replay::delta_archive::{ReplayDeltaArchiveIndex, ReplayDeltaRrdpKind};
use crate::report::{RfcRef, Warning};
use crate::storage::RocksStore;
use crate::sync::rrdp::sync_from_notification_with_timing_and_download_log;
use crate::sync::rrdp::{Fetcher as HttpFetcher, RrdpSyncError, load_rrdp_local_state};
use crate::sync::store_projection::{
build_repository_view_present_entry, build_repository_view_withdrawn_entry,
prepare_repo_bytes_batch_owned,
};
use std::collections::HashSet;
#[cfg(test)]
use crate::storage::RrdpSourceSyncState;
#[cfg(test)]
use crate::sync::rrdp::persist_rrdp_local_state;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RepoSyncSource {
Rrdp,
Rsync,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RepoSyncPhase {
RrdpOk,
RrdpFailedRsyncOk,
RsyncOnlyOk,
ReplayRrdpOk,
ReplayRsyncOk,
ReplayNoopRrdp,
ReplayNoopRsync,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncResult {
pub source: RepoSyncSource,
pub phase: RepoSyncPhase,
pub objects_written: usize,
pub warnings: Vec<Warning>,
}
#[derive(Debug, thiserror::Error)]
pub enum RepoSyncError {
#[error("RRDP sync failed: {0}")]
Rrdp(#[from] RrdpSyncError),
#[error("rsync fallback failed: {0}")]
Rsync(#[from] RsyncFetchError),
#[error("replay sync error: {0}")]
Replay(String),
#[error("storage error: {0}")]
Storage(String),
}
/// Sync a publication point into the current repository view.
///
/// v1 behavior:
/// - If `rrdp_notification_uri` is present and `policy.sync_preference` is `rrdp_then_rsync`,
/// try RRDP snapshot sync first (RFC 8182 §3.4.1-§3.4.3).
/// - On RRDP failure, fall back to rsync (RFC 8182 §3.4.5).
/// - If `sync_preference` is `rsync_only` or there is no RRDP URI, use rsync.
pub fn sync_publication_point(
store: &RocksStore,
policy: &Policy,
rrdp_notification_uri: Option<&str>,
rsync_base_uri: &str,
http_fetcher: &dyn HttpFetcher,
rsync_fetcher: &dyn RsyncFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<RepoSyncResult, RepoSyncError> {
match (policy.sync_preference, rrdp_notification_uri) {
(SyncPreference::RrdpThenRsync, Some(notification_uri)) => {
match try_rrdp_sync_with_retry(
store,
notification_uri,
None,
http_fetcher,
timing,
download_log,
) {
Ok(written) => {
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rrdp_ok_total", 1);
t.record_count("repo_sync_rrdp_objects_written_total", written as u64);
}
crate::progress_log::emit(
"repo_sync_rrdp_ok",
serde_json::json!({
"notify_uri": notification_uri,
"objects_written": written,
}),
);
Ok(RepoSyncResult {
source: RepoSyncSource::Rrdp,
phase: RepoSyncPhase::RrdpOk,
objects_written: written,
warnings: Vec::new(),
})
}
Err(err) => {
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rrdp_failed_total", 1);
}
crate::progress_log::emit(
"rrdp_fallback_rsync",
serde_json::json!({
"notify_uri": notification_uri,
"rsync_base_uri": rsync_base_uri,
"rrdp_error": err.to_string(),
}),
);
crate::progress_log::emit(
"rrdp_failed_fallback_rsync",
serde_json::json!({
"notify_uri": notification_uri,
"rsync_base_uri": rsync_base_uri,
"rrdp_error": err.to_string(),
}),
);
let warnings = vec![
Warning::new(format!("RRDP failed; falling back to rsync: {err}"))
.with_rfc_refs(&[RfcRef("RFC 8182 §3.4.5")])
.with_context(notification_uri),
];
let written = rsync_sync_into_current_store(
store,
rsync_base_uri,
None,
rsync_fetcher,
timing,
download_log,
)?;
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rsync_peer_aligned_profile_total", 1);
}
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rsync_fallback_ok_total", 1);
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
}
Ok(RepoSyncResult {
source: RepoSyncSource::Rsync,
phase: RepoSyncPhase::RrdpFailedRsyncOk,
objects_written: written,
warnings,
})
}
}
}
_ => {
let written = rsync_sync_into_current_store(
store,
rsync_base_uri,
None,
rsync_fetcher,
timing,
download_log,
)?;
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rsync_peer_aligned_profile_total", 1);
}
crate::progress_log::emit(
"repo_sync_rsync_direct",
serde_json::json!({
"rsync_base_uri": rsync_base_uri,
"objects_written": written,
}),
);
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rsync_direct_total", 1);
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
}
Ok(RepoSyncResult {
source: RepoSyncSource::Rsync,
phase: RepoSyncPhase::RsyncOnlyOk,
objects_written: written,
warnings: Vec::new(),
})
}
}
}
pub fn sync_publication_point_replay(
store: &RocksStore,
replay_index: &ReplayArchiveIndex,
rrdp_notification_uri: Option<&str>,
rsync_base_uri: &str,
http_fetcher: &dyn HttpFetcher,
rsync_fetcher: &dyn RsyncFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<RepoSyncResult, RepoSyncError> {
match resolve_replay_transport(replay_index, rrdp_notification_uri, rsync_base_uri)? {
ReplayResolvedTransport::Rrdp(notification_uri) => {
let written = try_rrdp_sync_with_retry(
store,
notification_uri,
None,
http_fetcher,
timing,
download_log,
)?;
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rrdp_ok_total", 1);
t.record_count("repo_sync_rrdp_objects_written_total", written as u64);
}
Ok(RepoSyncResult {
source: RepoSyncSource::Rrdp,
phase: RepoSyncPhase::ReplayRrdpOk,
objects_written: written,
warnings: Vec::new(),
})
}
ReplayResolvedTransport::Rsync => {
let written = rsync_sync_into_current_store(
store,
rsync_base_uri,
None,
rsync_fetcher,
timing,
download_log,
)?;
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rsync_direct_total", 1);
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
}
Ok(RepoSyncResult {
source: RepoSyncSource::Rsync,
phase: RepoSyncPhase::ReplayRsyncOk,
objects_written: written,
warnings: Vec::new(),
})
}
}
}
pub fn sync_publication_point_replay_delta(
store: &RocksStore,
delta_index: &ReplayDeltaArchiveIndex,
rrdp_notification_uri: Option<&str>,
rsync_base_uri: &str,
http_fetcher: &dyn HttpFetcher,
rsync_fetcher: &dyn RsyncFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<RepoSyncResult, RepoSyncError> {
match resolve_replay_delta_transport(store, delta_index, rrdp_notification_uri, rsync_base_uri)?
{
ReplayDeltaResolvedTransport::Rrdp(notification_uri) => {
let written = try_rrdp_sync_with_retry(
store,
notification_uri,
None,
http_fetcher,
timing,
download_log,
)?;
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rrdp_ok_total", 1);
t.record_count("repo_sync_rrdp_objects_written_total", written as u64);
}
Ok(RepoSyncResult {
source: RepoSyncSource::Rrdp,
phase: RepoSyncPhase::ReplayRrdpOk,
objects_written: written,
warnings: Vec::new(),
})
}
ReplayDeltaResolvedTransport::Rsync => {
let written = rsync_sync_into_current_store(
store,
rsync_base_uri,
None,
rsync_fetcher,
timing,
download_log,
)?;
if let Some(t) = timing.as_ref() {
t.record_count("repo_sync_rsync_direct_total", 1);
t.record_count("repo_sync_rsync_objects_written_total", written as u64);
}
Ok(RepoSyncResult {
source: RepoSyncSource::Rsync,
phase: RepoSyncPhase::ReplayRsyncOk,
objects_written: written,
warnings: Vec::new(),
})
}
ReplayDeltaResolvedTransport::Noop(source) => Ok(RepoSyncResult {
source,
phase: match source {
RepoSyncSource::Rrdp => RepoSyncPhase::ReplayNoopRrdp,
RepoSyncSource::Rsync => RepoSyncPhase::ReplayNoopRsync,
},
objects_written: 0,
warnings: Vec::new(),
}),
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum ReplayResolvedTransport<'a> {
Rrdp(&'a str),
Rsync,
}
enum ReplayDeltaResolvedTransport<'a> {
Rrdp(&'a str),
Rsync,
Noop(RepoSyncSource),
}
fn resolve_replay_transport<'a>(
replay_index: &'a ReplayArchiveIndex,
rrdp_notification_uri: Option<&'a str>,
rsync_base_uri: &str,
) -> Result<ReplayResolvedTransport<'a>, RepoSyncError> {
if let Some(notification_uri) = rrdp_notification_uri {
let lock = replay_index.rrdp_lock(notification_uri).ok_or_else(|| {
RepoSyncError::Replay(format!(
"replay RRDP lock missing for notification URI: {notification_uri}"
))
})?;
return Ok(match lock.transport {
ReplayTransport::Rrdp => ReplayResolvedTransport::Rrdp(notification_uri),
ReplayTransport::Rsync => ReplayResolvedTransport::Rsync,
});
}
replay_index
.resolve_rsync_module_for_base_uri(rsync_base_uri)
.map_err(|e| RepoSyncError::Replay(e.to_string()))?;
Ok(ReplayResolvedTransport::Rsync)
}
fn resolve_replay_delta_transport<'a>(
store: &RocksStore,
delta_index: &'a ReplayDeltaArchiveIndex,
rrdp_notification_uri: Option<&'a str>,
rsync_base_uri: &str,
) -> Result<ReplayDeltaResolvedTransport<'a>, RepoSyncError> {
if let Some(notification_uri) = rrdp_notification_uri {
let repo = delta_index.rrdp_repo(notification_uri).ok_or_else(|| {
RepoSyncError::Replay(format!(
"delta replay RRDP entry missing for notification URI: {notification_uri}"
))
})?;
validate_delta_replay_base_state_for_repo(store, notification_uri, &repo.transition.base)?;
return match repo.transition.kind {
ReplayDeltaRrdpKind::Delta => Ok(ReplayDeltaResolvedTransport::Rrdp(notification_uri)),
ReplayDeltaRrdpKind::Unchanged => Ok(ReplayDeltaResolvedTransport::Noop(
match repo.transition.target.transport {
ReplayTransport::Rrdp => RepoSyncSource::Rrdp,
ReplayTransport::Rsync => RepoSyncSource::Rsync,
},
)),
ReplayDeltaRrdpKind::FallbackRsync => Ok(ReplayDeltaResolvedTransport::Rsync),
ReplayDeltaRrdpKind::SessionReset => Err(RepoSyncError::Replay(format!(
"delta replay kind session-reset requires fresh full replay for {notification_uri}"
))),
ReplayDeltaRrdpKind::Gap => Err(RepoSyncError::Replay(format!(
"delta replay kind gap requires fresh full replay for {notification_uri}"
))),
};
}
delta_index
.resolve_rsync_module_for_base_uri(rsync_base_uri)
.map_err(|e| RepoSyncError::Replay(e.to_string()))?;
Ok(ReplayDeltaResolvedTransport::Rsync)
}
fn validate_delta_replay_base_state_for_repo(
store: &RocksStore,
notification_uri: &str,
base: &crate::replay::delta_archive::ReplayDeltaRrdpState,
) -> Result<(), RepoSyncError> {
match base.transport {
ReplayTransport::Rrdp => {
let state = load_rrdp_local_state(store, notification_uri)
.map_err(RepoSyncError::Storage)?
.ok_or_else(|| {
RepoSyncError::Replay(format!(
"delta replay base state missing for {notification_uri}: expected RRDP session={} serial={}",
base.session.as_deref().unwrap_or("<none>"),
base.serial
.map(|v| v.to_string())
.unwrap_or_else(|| "<none>".to_string())
))
})?;
let expected_session = base.session.as_deref().unwrap_or("");
let expected_serial = base.serial.unwrap_or_default();
if state.session_id != expected_session || state.serial != expected_serial {
return Err(RepoSyncError::Replay(format!(
"delta replay base state mismatch for {notification_uri}: expected session={} serial={}, actual session={} serial={}",
expected_session, expected_serial, state.session_id, state.serial
)));
}
}
ReplayTransport::Rsync => {}
}
Ok(())
}
fn try_rrdp_sync(
store: &RocksStore,
notification_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
http_fetcher: &dyn HttpFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<usize, RrdpSyncError> {
let notification_xml = {
let _step = timing
.as_ref()
.map(|t| t.span_rrdp_repo_step(notification_uri, "fetch_notification"));
let _total = timing
.as_ref()
.map(|t| t.span_phase("rrdp_fetch_notification_total"));
let mut dl_span = download_log
.map(|dl| dl.span_download(AuditDownloadKind::RrdpNotification, notification_uri));
match http_fetcher.fetch(notification_uri) {
Ok(v) => {
if let Some(t) = timing.as_ref() {
t.record_count("rrdp_notification_fetch_ok_total", 1);
}
if let Some(s) = dl_span.as_mut() {
s.set_bytes(v.len() as u64);
s.set_ok();
}
v
}
Err(e) => {
if let Some(t) = timing.as_ref() {
t.record_count("rrdp_notification_fetch_fail_total", 1);
}
if let Some(s) = dl_span.as_mut() {
s.set_err(e.clone());
}
return Err(RrdpSyncError::Fetch(e));
}
}
};
if let Some(t) = timing.as_ref() {
t.record_count(
"rrdp_notification_bytes_total",
notification_xml.len() as u64,
);
}
sync_from_notification_with_timing_and_download_log(
store,
notification_uri,
current_repo_index,
&notification_xml,
http_fetcher,
timing,
download_log,
)
}
fn is_retryable_http_fetch_error(msg: &str) -> bool {
if msg.contains("http request failed:") || msg.contains("http read body failed:") {
return true;
}
let Some(rest) = msg.strip_prefix("http status ") else {
return false;
};
let code = rest
.trim()
.split_whitespace()
.next()
.and_then(|s| s.parse::<u16>().ok())
.unwrap_or(0);
code == 408 || code == 429 || (500..600).contains(&code)
}
fn try_rrdp_sync_with_retry(
store: &RocksStore,
notification_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
http_fetcher: &dyn HttpFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<usize, RrdpSyncError> {
let attempt = 1usize;
crate::progress_log::emit(
"rrdp_sync_attempt",
serde_json::json!({
"notify_uri": notification_uri,
"attempt": attempt,
}),
);
if let Some(t) = timing.as_ref() {
t.record_count("rrdp_retry_attempt_total", 1);
}
match try_rrdp_sync(
store,
notification_uri,
current_repo_index,
http_fetcher,
timing,
download_log,
) {
Ok(written) => {
crate::progress_log::emit(
"rrdp_sync_success",
serde_json::json!({
"notify_uri": notification_uri,
"attempt": attempt,
"objects_written": written,
}),
);
Ok(written)
}
Err(err) => {
let retryable = match &err {
RrdpSyncError::Fetch(msg) => is_retryable_http_fetch_error(msg),
_ => false,
};
crate::progress_log::emit(
"rrdp_sync_failed",
serde_json::json!({
"notify_uri": notification_uri,
"attempt": attempt,
"retryable": retryable,
"error": err.to_string(),
}),
);
if let Some(t) = timing.as_ref() {
match &err {
RrdpSyncError::Fetch(_) => t.record_count("rrdp_failed_fetch_total", 1),
RrdpSyncError::Rrdp(_) => t.record_count("rrdp_failed_protocol_total", 1),
RrdpSyncError::Storage(_) => t.record_count("rrdp_failed_storage_total", 1),
}
}
Err(err)
}
}
}
pub(crate) fn run_rrdp_transport(
store: &RocksStore,
notification_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
http_fetcher: &dyn HttpFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<usize, RrdpSyncError> {
try_rrdp_sync_with_retry(
store,
notification_uri,
current_repo_index,
http_fetcher,
timing,
download_log,
)
}
fn rsync_sync_into_current_store(
store: &RocksStore,
rsync_base_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
rsync_fetcher: &dyn RsyncFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<usize, RepoSyncError> {
let started = std::time::Instant::now();
let sync_scope_uri = rsync_fetcher.dedup_key(rsync_base_uri);
crate::progress_log::emit(
"rsync_sync_start",
serde_json::json!({
"rsync_base_uri": rsync_base_uri,
"sync_scope_uri": &sync_scope_uri,
}),
);
let _s = timing
.as_ref()
.map(|t| t.span_rrdp_repo_step(rsync_base_uri, "rsync_fetch_objects"));
let _p = timing.as_ref().map(|t| t.span_phase("rsync_fetch_total"));
let mut dl_span =
download_log.map(|dl| dl.span_download(AuditDownloadKind::Rsync, rsync_base_uri));
let mut new_set: HashSet<String> = HashSet::new();
let mut fetched_objects: Vec<(String, Vec<u8>)> = Vec::new();
let (object_count, bytes_total) =
match rsync_fetcher.visit_objects(rsync_base_uri, &mut |uri, bytes| {
new_set.insert(uri.clone());
fetched_objects.push((uri, bytes));
Ok(())
}) {
Ok(v) => {
if let Some(s) = dl_span.as_mut() {
s.set_objects(v.0 as u64, v.1);
s.set_bytes(v.1);
s.set_ok();
}
v
}
Err(e) => {
if let Some(s) = dl_span.as_mut() {
s.set_err(e.to_string());
}
return Err(e.into());
}
};
crate::progress_log::emit(
"rsync_sync_fetch_done",
serde_json::json!({
"rsync_base_uri": rsync_base_uri,
"sync_scope_uri": &sync_scope_uri,
"object_count": object_count,
"bytes_total": bytes_total,
"duration_ms": started.elapsed().as_millis() as u64,
}),
);
if let Some(t) = timing.as_ref() {
t.record_count("rsync_objects_fetched_total", object_count as u64);
t.record_count("rsync_objects_bytes_total", bytes_total);
}
drop(_p);
let existing_view = store
.list_repository_view_entries_with_prefix(&sync_scope_uri)
.map_err(|e| RepoSyncError::Storage(e.to_string()))?;
let _proj = timing
.as_ref()
.map(|t| t.span_phase("rsync_write_current_store_total"));
let prepared_bytes =
prepare_repo_bytes_batch_owned(fetched_objects).map_err(RepoSyncError::Storage)?;
let mut repository_view_entries = Vec::new();
for entry in existing_view {
if !new_set.contains(&entry.rsync_uri) {
repository_view_entries.push(build_repository_view_withdrawn_entry(
&sync_scope_uri,
&entry.rsync_uri,
entry.current_hash,
));
}
}
for uri in &new_set {
let current_hash = prepared_bytes
.uri_to_hash
.get(uri)
.cloned()
.ok_or_else(|| {
RepoSyncError::Storage(format!("missing raw_by_hash mapping for {uri}"))
})?;
repository_view_entries.push(build_repository_view_present_entry(
&sync_scope_uri,
uri,
&current_hash,
));
}
store
.put_blob_bytes_batch(&prepared_bytes.blobs_to_write)
.map_err(|e| RepoSyncError::Storage(e.to_string()))?;
store
.put_projection_batch(&repository_view_entries, &[], &[])
.map_err(|e| RepoSyncError::Storage(e.to_string()))?;
if let Some(index) = current_repo_index {
index
.write()
.map_err(|_| RepoSyncError::Storage("current repo index lock poisoned".to_string()))?
.apply_repository_view_entries(&repository_view_entries)
.map_err(RepoSyncError::Storage)?;
}
let total_duration_ms = started.elapsed().as_millis() as u64;
crate::progress_log::emit(
"rsync_sync_done",
serde_json::json!({
"rsync_base_uri": rsync_base_uri,
"sync_scope_uri": &sync_scope_uri,
"object_count": object_count,
"bytes_total": bytes_total,
"duration_ms": total_duration_ms,
}),
);
if (total_duration_ms as f64) / 1000.0 >= crate::progress_log::slow_threshold_secs() {
crate::progress_log::emit(
"rsync_sync_slow",
serde_json::json!({
"rsync_base_uri": rsync_base_uri,
"sync_scope_uri": &sync_scope_uri,
"object_count": object_count,
"bytes_total": bytes_total,
"duration_ms": total_duration_ms,
}),
);
}
Ok(object_count)
}
pub(crate) fn run_rsync_transport(
store: &RocksStore,
rsync_base_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
rsync_fetcher: &dyn RsyncFetcher,
timing: Option<&TimingHandle>,
download_log: Option<&DownloadLogHandle>,
) -> Result<usize, RepoSyncError> {
rsync_sync_into_current_store(
store,
rsync_base_uri,
current_repo_index,
rsync_fetcher,
timing,
download_log,
)
}
#[cfg(test)]
#[path = "repo/tests.rs"]
mod tests;

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,445 @@
use base64::Engine;
use quick_xml::Reader;
use quick_xml::events::Event;
use sha2::Digest;
use std::io::{BufRead, Seek, SeekFrom, Write};
use uuid::Uuid;
use crate::current_repo_index::CurrentRepoIndexHandle;
use crate::storage::RocksStore;
use crate::sync::store_projection::{
build_repository_view_present_entry, build_repository_view_withdrawn_entry,
build_rrdp_source_member_present_record, build_rrdp_source_member_withdrawn_record,
build_rrdp_uri_owner_active_record, build_rrdp_uri_owner_withdrawn_record, compute_sha256_hex,
current_rrdp_owner_is, ensure_rrdp_uri_can_be_owned_by, prepare_repo_bytes_batch,
};
use super::{
Fetcher, RRDP_SNAPSHOT_APPLY_BATCH_SIZE, RRDP_XMLNS, RrdpError, RrdpSyncError, parse_u64_str,
strip_all_ascii_whitespace,
};
#[cfg(test)]
pub(super) fn apply_snapshot(
store: &RocksStore,
notification_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
snapshot_xml: &[u8],
expected_session_id: Uuid,
expected_serial: u64,
) -> Result<usize, RrdpSyncError> {
if snapshot_xml.iter().any(|&b| b > 0x7F) {
return Err(RrdpError::NotAscii.into());
}
apply_snapshot_from_bufread(
store,
notification_uri,
current_repo_index,
std::io::Cursor::new(snapshot_xml),
expected_session_id,
expected_serial,
)
}
pub(super) fn apply_snapshot_from_bufread<R: BufRead>(
store: &RocksStore,
notification_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
input: R,
expected_session_id: Uuid,
expected_serial: u64,
) -> Result<usize, RrdpSyncError> {
let previous_members: Vec<String> = store
.list_current_rrdp_source_members(notification_uri)
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?
.into_iter()
.map(|record| record.rsync_uri)
.collect();
let session_id = expected_session_id.to_string();
let mut new_set: std::collections::HashSet<String> = std::collections::HashSet::new();
let mut batch_published: Vec<(String, Vec<u8>)> =
Vec::with_capacity(RRDP_SNAPSHOT_APPLY_BATCH_SIZE);
let mut published_count = 0usize;
let mut reader = Reader::from_reader(input);
reader.config_mut().trim_text(false);
let mut buf = Vec::new();
let mut root_seen = false;
let mut in_publish = false;
let mut publish_nested_depth = 0usize;
let mut current_publish_uri: Option<String> = None;
let mut current_publish_text = String::new();
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(e)) => {
let local_name = e.local_name();
let local_name = local_name.as_ref();
if !root_seen {
root_seen = true;
if local_name != b"snapshot" {
let got = String::from_utf8_lossy(local_name).to_string();
return Err(RrdpError::UnexpectedRoot(got).into());
}
let mut xmlns = String::new();
let mut version = String::new();
let mut session_id_attr = String::new();
let mut serial_attr = String::new();
for attr in e.attributes().with_checks(false) {
let attr = match attr {
Ok(attr) => attr,
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
};
let key = attr.key.as_ref();
let value = attr
.decode_and_unescape_value(reader.decoder())
.map_err(|e| RrdpError::Xml(e.to_string()))?
.into_owned();
match key {
b"xmlns" => xmlns = value,
b"version" => version = value,
b"session_id" => session_id_attr = value,
b"serial" => serial_attr = value,
_ => {}
}
}
if xmlns != RRDP_XMLNS {
return Err(RrdpError::InvalidNamespace(xmlns).into());
}
if version != "1" {
return Err(RrdpError::InvalidVersion(version).into());
}
let got_session_id = Uuid::parse_str(&session_id_attr)
.map_err(|_| RrdpError::InvalidSessionId(session_id_attr.clone()))?;
if got_session_id != expected_session_id {
return Err(RrdpError::SnapshotSessionIdMismatch {
expected: expected_session_id.to_string(),
got: got_session_id.to_string(),
}
.into());
}
let got_serial = parse_u64_str(&serial_attr)?;
if got_serial != expected_serial {
return Err(RrdpError::SnapshotSerialMismatch {
expected: expected_serial,
got: got_serial,
}
.into());
}
} else if in_publish {
publish_nested_depth += 1;
} else if local_name == b"publish" {
let mut uri = None;
for attr in e.attributes().with_checks(false) {
let attr = match attr {
Ok(attr) => attr,
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
};
if attr.key.as_ref() == b"uri" {
uri = Some(
attr.decode_and_unescape_value(reader.decoder())
.map_err(|e| RrdpError::Xml(e.to_string()))?
.into_owned(),
);
}
}
let uri = uri.ok_or(RrdpError::PublishUriMissing)?;
ensure_rrdp_uri_can_be_owned_by(store, notification_uri, &uri)
.map_err(RrdpSyncError::Storage)?;
in_publish = true;
publish_nested_depth = 0;
current_publish_uri = Some(uri);
current_publish_text.clear();
}
}
Ok(Event::Empty(e)) => {
let local_name = e.local_name();
let local_name = local_name.as_ref();
if !root_seen {
let got = String::from_utf8_lossy(local_name).to_string();
return Err(RrdpError::UnexpectedRoot(got).into());
}
if local_name == b"publish" {
let mut has_uri = false;
for attr in e.attributes().with_checks(false) {
let attr = match attr {
Ok(attr) => attr,
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
};
if attr.key.as_ref() == b"uri" {
has_uri = true;
break;
}
}
if !has_uri {
return Err(RrdpError::PublishUriMissing.into());
}
return Err(RrdpError::PublishContentMissing.into());
}
}
Ok(Event::Text(e)) => {
if in_publish && publish_nested_depth == 0 {
let text = reader
.decoder()
.decode(e.as_ref())
.map_err(|e| RrdpError::Xml(e.to_string()))?;
current_publish_text.push_str(&text);
}
}
Ok(Event::CData(e)) => {
if in_publish && publish_nested_depth == 0 {
let text = reader
.decoder()
.decode(e.as_ref())
.map_err(|e| RrdpError::Xml(e.to_string()))?;
current_publish_text.push_str(&text);
}
}
Ok(Event::End(e)) => {
let local_name = e.local_name();
let local_name = local_name.as_ref();
if in_publish {
if publish_nested_depth > 0 {
publish_nested_depth -= 1;
} else if local_name == b"publish" {
let uri = current_publish_uri
.take()
.ok_or_else(|| RrdpError::Xml("publish uri missing in state".into()))?;
let content_b64 = strip_all_ascii_whitespace(&current_publish_text);
current_publish_text.clear();
if content_b64.is_empty() {
return Err(RrdpError::PublishContentMissing.into());
}
let bytes = base64::engine::general_purpose::STANDARD
.decode(content_b64.as_bytes())
.map_err(|e| RrdpError::PublishBase64(e.to_string()))?;
new_set.insert(uri.clone());
batch_published.push((uri, bytes));
published_count += 1;
if batch_published.len() >= RRDP_SNAPSHOT_APPLY_BATCH_SIZE {
flush_snapshot_publish_batch(
store,
notification_uri,
current_repo_index,
&session_id,
expected_serial,
&batch_published,
)?;
batch_published.clear();
}
in_publish = false;
}
}
}
Ok(Event::Eof) => break,
Ok(Event::Decl(_) | Event::PI(_) | Event::Comment(_) | Event::DocType(_)) => {}
Err(e) => return Err(RrdpError::Xml(e.to_string()).into()),
}
buf.clear();
}
if !root_seen {
return Err(RrdpError::Xml("missing root element".to_string()).into());
}
if in_publish {
return Err(RrdpError::PublishContentMissing.into());
}
if !batch_published.is_empty() {
flush_snapshot_publish_batch(
store,
notification_uri,
current_repo_index,
&session_id,
expected_serial,
&batch_published,
)?;
batch_published.clear();
}
let mut withdrawn: Vec<(String, Option<String>)> = Vec::new();
for old_uri in &previous_members {
if new_set.contains(old_uri) {
continue;
}
let previous_hash = store
.get_repository_view_entry(old_uri)
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?
.and_then(|entry| entry.current_hash)
.or_else(|| {
store
.load_current_object_bytes_by_uri(old_uri)
.ok()
.flatten()
.map(|bytes| compute_sha256_hex(&bytes))
});
withdrawn.push((old_uri.clone(), previous_hash));
}
let mut repository_view_entries = Vec::with_capacity(withdrawn.len());
let mut member_records = Vec::with_capacity(withdrawn.len());
let mut owner_records = Vec::with_capacity(withdrawn.len());
for (uri, previous_hash) in withdrawn {
member_records.push(build_rrdp_source_member_withdrawn_record(
notification_uri,
&session_id,
expected_serial,
&uri,
previous_hash.clone(),
));
if current_rrdp_owner_is(store, notification_uri, &uri).map_err(RrdpSyncError::Storage)? {
repository_view_entries.push(build_repository_view_withdrawn_entry(
notification_uri,
&uri,
previous_hash.clone(),
));
owner_records.push(build_rrdp_uri_owner_withdrawn_record(
notification_uri,
&session_id,
expected_serial,
&uri,
previous_hash,
));
}
}
store
.put_projection_batch(&repository_view_entries, &member_records, &owner_records)
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?;
if let Some(index) = current_repo_index {
index
.write()
.map_err(|_| RrdpSyncError::Storage("current repo index lock poisoned".to_string()))?
.apply_repository_view_entries(&repository_view_entries)
.map_err(RrdpSyncError::Storage)?;
}
Ok(published_count)
}
fn flush_snapshot_publish_batch(
store: &RocksStore,
notification_uri: &str,
current_repo_index: Option<&CurrentRepoIndexHandle>,
session_id: &str,
serial: u64,
published: &[(String, Vec<u8>)],
) -> Result<(), RrdpSyncError> {
let prepared_bytes = prepare_repo_bytes_batch(published).map_err(RrdpSyncError::Storage)?;
let mut repository_view_entries = Vec::with_capacity(published.len());
let mut member_records = Vec::with_capacity(published.len());
let mut owner_records = Vec::with_capacity(published.len());
for (uri, _bytes) in published {
let current_hash = prepared_bytes
.uri_to_hash
.get(uri)
.cloned()
.ok_or_else(|| {
RrdpSyncError::Storage(format!("missing raw_by_hash mapping for {uri}"))
})?;
repository_view_entries.push(build_repository_view_present_entry(
notification_uri,
uri,
&current_hash,
));
member_records.push(build_rrdp_source_member_present_record(
notification_uri,
session_id,
serial,
uri,
&current_hash,
));
owner_records.push(build_rrdp_uri_owner_active_record(
notification_uri,
session_id,
serial,
uri,
&current_hash,
));
}
store
.put_blob_bytes_batch(&prepared_bytes.blobs_to_write)
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?;
store
.put_projection_batch(&repository_view_entries, &member_records, &owner_records)
.map_err(|e| RrdpSyncError::Storage(e.to_string()))?;
if let Some(index) = current_repo_index {
index
.write()
.map_err(|_| RrdpSyncError::Storage("current repo index lock poisoned".to_string()))?
.apply_repository_view_entries(&repository_view_entries)
.map_err(RrdpSyncError::Storage)?;
}
Ok(())
}
const SNAPSHOT_NON_ASCII_ERROR: &str = "snapshot body contains non-ASCII bytes";
struct SnapshotSpoolWriter<'a, W: Write> {
inner: &'a mut W,
hasher: sha2::Sha256,
bytes: u64,
}
impl<'a, W: Write> SnapshotSpoolWriter<'a, W> {
fn new(inner: &'a mut W) -> Self {
Self {
inner,
hasher: sha2::Sha256::new(),
bytes: 0,
}
}
fn finalize_hash(self) -> [u8; 32] {
let digest = self.hasher.finalize();
let mut out = [0u8; 32];
out.copy_from_slice(&digest);
out
}
}
impl<W: Write> Write for SnapshotSpoolWriter<'_, W> {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
if buf.iter().any(|&b| b > 0x7F) {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
SNAPSHOT_NON_ASCII_ERROR,
));
}
let n = self.inner.write(buf)?;
self.hasher.update(&buf[..n]);
self.bytes += n as u64;
Ok(n)
}
fn flush(&mut self) -> std::io::Result<()> {
self.inner.flush()
}
}
pub(super) fn fetch_snapshot_into_tempfile(
fetcher: &dyn Fetcher,
snapshot_uri: &str,
expected_hash_sha256: &[u8; 32],
) -> Result<(tempfile::NamedTempFile, u64), RrdpSyncError> {
let mut tmp = tempfile::NamedTempFile::new()
.map_err(|e| RrdpSyncError::Fetch(format!("tempfile create failed: {e}")))?;
let mut spool = SnapshotSpoolWriter::new(tmp.as_file_mut());
let bytes_written = match fetcher.fetch_to_writer(snapshot_uri, &mut spool) {
Ok(bytes) => bytes,
Err(e) if e.contains(SNAPSHOT_NON_ASCII_ERROR) => return Err(RrdpError::NotAscii.into()),
Err(e) => return Err(RrdpSyncError::Fetch(e)),
};
let computed = spool.finalize_hash();
if computed.as_slice() != expected_hash_sha256.as_slice() {
return Err(RrdpError::SnapshotHashMismatch.into());
}
tmp.as_file_mut()
.flush()
.map_err(|e| RrdpSyncError::Fetch(format!("tempfile flush failed: {e}")))?;
tmp.as_file_mut()
.seek(SeekFrom::Start(0))
.map_err(|e| RrdpSyncError::Fetch(format!("tempfile rewind failed: {e}")))?;
Ok((tmp, bytes_written))
}

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More