20260902 完成公开树清理与热点拆分
Some checks failed
ci / public-tree (push) Has been cancelled
ci / docs (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / docker-runtime (push) Has been cancelled

This commit is contained in:
2026-09-02 11:48:36 +08:00
parent 15c8de82ba
commit 2a0b5067bd
209 changed files with 59875 additions and 59423 deletions

View File

@@ -5,6 +5,23 @@ on:
pull_request:
jobs:
public-tree:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: bash scripts/ci/check_public_tree.sh
docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python -m pip install -r docs/requirements.txt
- run: sphinx-build -W --keep-going -b html docs/source docs/_build/html
- run: sphinx-build -W --keep-going -b linkcheck docs/source docs/_build/linkcheck
rust:
runs-on: ubuntu-latest
steps:

1
.gitignore vendored
View File

@@ -5,6 +5,7 @@
/logs/
/tmp/
/docker-out/
/docs/_build/
/provenance/generated/
.env
.env.*

14
.readthedocs.yaml Normal file
View File

@@ -0,0 +1,14 @@
version: 2
build:
os: ubuntu-24.04
tools:
python: "3.12"
sphinx:
configuration: docs/source/conf.py
fail_on_warning: true
python:
install:
- requirements: docs/requirements.txt

View File

@@ -1,10 +1,16 @@
# Contributing to panda-rpki
This repository is private staging for the `panda-rpki-validator` extraction.
External contributions are not enabled while ownership, license and contribution
governance are pending.
panda-rpki is preparing its contributor process. The source is licensed under
the BSD 3-Clause License in [LICENSE](LICENSE); maintainers, code-of-conduct
policy, and public security-reporting channel must be published before
external contributions are accepted.
During M3–M6, changes must preserve `docs/output-abi.md`, update the relevant
allowlist/provenance entry, and pass the locked Rust build and Docker smoke
checks. Do not add live RIR data, private state, credentials, internal hosts,
or copied history.
For local development, follow the workflow in docs/source/development.rst:
format Rust code, run the locked test suite and Clippy, build the documentation,
and add focused tests for every behaviour change. Do not submit credentials,
private state, live customer data, or material whose redistribution terms are
unknown.
The output contract is part of the runtime interface. Changes to run artifacts,
container entrypoints, persistent-volume layout, or CSV/CIR/CCR semantics
require compatibility tests and a documented migration decision.

28
LICENSE Normal file
View File

@@ -0,0 +1,28 @@
BSD 3-Clause License
Copyright (c) 2026, panda-rpki contributors
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

123
README.md
View File

@@ -1,100 +1,57 @@
# panda-rpki
`panda-rpki` is the staging workspace for the independently distributable RPKI
synchronization and validation component. The first component is
`panda-rpki-validator` (Cargo package and CLI); its Rust import path is
`panda_rpki_validator`.
panda-rpki is an RPKI synchronization and validation runtime written in Rust.
The first component, panda-rpki-validator, fetches RPKI repositories through
RRDP or rsync, validates the resulting object graph, and writes a structured
run directory containing reports, payload CSV files, and canonical artifacts.
This directory is currently a **private staging repository**. It has no copied
history from the existing private `rpki` repository. The normal synchronization
and validation path has now been extracted, together with its daemon and
Docker lifecycle wrapper. The first M5 canonical snapshot/delta baseline and
amd64/arm64 staging image checks are complete; remaining profile/performance
gates still block any public release. Do not use this staging build as a
production validator.
The project is currently a pre-release. Its command-line interface, output
directory contract, container runtime, and supported deployment practices are
documented in the Sphinx source tree at docs/source. The source is provided
under the BSD 3-Clause License; public support and release policies will be
published before a production release.
## Current milestone
## Quick start
M4/M5 contain the first functional extraction from source commit
`74cbebbd3334ac0063761c1a97a88ee000cc2a57`: normal RRDP/rsync synchronization,
RPKI validation, RocksDB state, CIR/CCR/report/CSV outputs, the run daemon, and
the snapshot/delta lifecycle wrapper. The `verification-only` mode is
intentionally deferred. The project license is intentionally **TBD**; no
external contributions or public release are accepted until ownership and
licensing are approved.
Build and test the validator from a checkout:
## Build and test the staging component
cargo build --locked -p panda-rpki-validator
cargo test --locked -p panda-rpki-validator
cargo fmt --all --check
cargo clippy --locked -p panda-rpki-validator --all-targets -- -D warnings
```bash
cargo build --locked -p panda-rpki-validator
cargo test --locked -p panda-rpki-validator
cargo fmt --all --check
cargo clippy --locked -p panda-rpki-validator --all-targets -- -D warnings
./scripts/docker/build_image.sh --arch amd64 --allow-dirty --no-save
```
Build a local amd64 runtime image:
The extracted test suite currently passes `735` tests with `1` ignored test in
the normal profile. The APNIC offline snapshot/delta profile has also passed
the M5 canonical comparator and a five-run pinned release smoke baseline; the
full cache/fallback/failure/profile matrix is still pending.
./scripts/docker/build_image.sh --arch amd64 --allow-dirty
./scripts/docker/verify_image.sh \
--image panda-rpki-validator:0.1.0-dirty-amd64
For a deterministic single-RIR run against the checked-in test repository,
build the binaries first and invoke the lifecycle wrapper with
`RPKI_EXTRA_ARGS=--disable-rrdp --rsync-local-dir ...`; the wrapper writes the
normal run ABI under `RUN_ROOT/runs/run_0001/`.
The checked-in TAL and trust-anchor files are bootstrap inputs, not a bundled
offline repository snapshot. A deterministic offline run requires a reviewed
replay archive supplied by the operator. The runtime wrapper writes normal run
artifacts below RUN_ROOT/runs/run_0001.
## Docker smoke
## Documentation
```bash
./scripts/docker/build_image.sh --arch amd64 --allow-dirty
./scripts/docker/verify_image.sh \
--image panda-rpki-validator:0.1.0-dirty-amd64
```
The documentation website is built with Sphinx and the Read the Docs Sphinx
Theme. Before the hosted site is configured, build it locally:
The runtime image contains both `panda-rpki-validator` and
`panda-rpki-validator-daemon`, the complete normal-run wrapper, and the
redistributable TAL/TA fixtures. It uses one persistent data root mounted at
`/var/lib/panda-rpki-validator`; `run`, `run-validator`, and `daemon` are
entrypoint subcommands. The wrapper preserves the output contract documented
in
[`docs/output-abi.md`](docs/output-abi.md).
python -m pip install -r docs/requirements.txt
sphinx-build -W --keep-going -b html docs/source docs/_build/html
For native or container A/B output checks, run the ABI verifier and canonical
comparator against two retained run directories:
Start with docs/source/getting-started.rst. It links to configuration, CLI,
operations, testing, development, architecture, and output-contract material.
```bash
tests/compat/verify_run_abi.sh /path/to/run_0001
tests/compat/compare_runs.py \
/path/to/original/runs/run_0001 \
/path/to/panda/runs/run_0001
```
This project is licensed under the [BSD 3-Clause License](LICENSE).
The comparator has matched the original runtime for the APNIC offline
snapshot/delta baseline. Five serial release samples on that small single-RIR
profile stay within the current wall-time gate under a pinned CPU. A live APNIC
run also produced an identical decoded CCR state (MFT/VRP/VAP/TA/RK); only the
time-bearing `producedAt` byte differed in the raw DER. A concurrent all-RIR
(`all5`) run is retained as network/fallback evidence, but its source was not an
atomic snapshot (the original image timed out on the ARIN RRDP notification),
so its different CCR state is not a code-parity verdict. See the detailed
[`M5 live RIR comparison report`](../specs/develop/20260901/m5_live_rir_image_artifact_comparison_milestone_report.md).
The follow-up remote-231 serial all5 run (old image first, then this image;
one snapshot plus three warm deltas per image with cache/prefetch/parallel
flags) also completed 4/4 runs on each side and retained full artifacts. Its
live CCR/CIR and timing differences are documented separately and are not a
frozen-input parity result: [`remote-231 all5 cache/prefetch report`](../specs/develop/20260901_2/m5_live_all5_cache_prefetch_serial_4run_milestone_report.md).
The new image also completed a separate remote-231 all5 long sequence of one
snapshot plus ten deltas; the timing and cache counters are retained as live
health evidence, not as a frozen-input parity gate: [`one snapshot + ten delta
timing report`](../specs/develop/20260901_2/m5_new_image_all5_snapshot_10delta_timing_report.md).
Cache, RRDP/fallback, constraints, replay and failure-path profiles remain in
M5.
## Safety and support
## Repository status
RPKI inputs are network- and attacker-controlled data. Run the validator with
a dedicated writable data directory and follow the resource and retention
guidance in the operations documentation. Do not place credentials, production
state, or private trust-anchor material in this repository or a container
image.
- `verification-only`: deferred to a separate backlog item.
- License and copyright owner: TBD; do not add a speculative `LICENSE` file.
- Staging Git remote: `https://git.nasp.fit/yuyr/panda-rpki.git` (private
staging target selected); public visibility, registry prefix, signing identity,
and release tags remain unselected until release governance is complete.
- Source provenance baseline: `provenance/source-baseline.toml`.
The public security-reporting and support channels are not yet assigned.
Until they are published, do not disclose a suspected vulnerability in a public
issue or commit.

View File

@@ -1,12 +1,13 @@
# Security policy
This project is not public yet and does not have a public vulnerability intake
address. Do not publish vulnerability details in an issue or commit. Report
urgent findings to the project owner through the private project channel and
include the affected commit, image tag/digest, reproduction steps, and whether
state or output data is exposed.
panda-rpki processes untrusted network data and is not yet a supported
production release. A public security-reporting channel, response targets, and
supported release branches have not been assigned.
The public contact, response SLA, supported release branches, and disclosure
process will be added after ownership and license approval. Container images
must never contain credentials, production state, private registry settings,
or internal hostnames.
Do not disclose suspected vulnerabilities in a public issue, pull request, or
commit until a public reporting address is published. Release artifacts and
container images must never include credentials, production state, private
registry configuration, or internal host names.
The first supported release will publish a security contact, disclosure
process, update policy, and supported-version table in the documentation site.

View File

@@ -4,6 +4,7 @@ version = "0.1.0"
edition = "2024"
rust-version = "1.92"
description = "Panda RPKI synchronization and validation runtime"
license = "BSD-3-Clause"
publish = false
[lib]

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,224 @@
// CIR construction, serialization, and export entry points.
pub fn build_cir_from_run(
store: &RocksStore,
trust_anchor: &TrustAnchor,
tal_uri: &str,
validation_time: time::OffsetDateTime,
publication_points: &[PublicationPointAudit],
) -> Result<CanonicalInputRepresentation, CirExportError> {
build_cir_from_run_multi(
store,
&[CirTrustAnchorBinding {
trust_anchor,
tal_uri,
}],
validation_time,
publication_points,
None,
)
}
pub fn build_cir_from_run_multi(
_store: &RocksStore,
tal_bindings: &[CirTrustAnchorBinding<'_>],
validation_time: time::OffsetDateTime,
publication_points: &[PublicationPointAudit],
_current_repo_objects: Option<&[CurrentRepoObject]>,
) -> Result<CanonicalInputRepresentation, CirExportError> {
let fresh_objects =
collect_cir_objects_from_validation_audit(publication_points, CirObjectSection::Fresh)?;
let cached_objects =
collect_cir_objects_from_validation_audit(publication_points, CirObjectSection::Cached)?;
let trust_anchors = build_cir_trust_anchors(tal_bindings)?;
let fresh_rejected_objects =
collect_rejected_objects_from_validation_audit(publication_points, CirObjectSection::Fresh);
let cached_rejected_objects = collect_rejected_objects_from_validation_audit(
publication_points,
CirObjectSection::Cached,
);
let cir = CanonicalInputRepresentation::new_v4(
validation_time,
cir_objects_from_hash_map(fresh_objects),
cir_objects_from_hash_map(cached_objects),
trust_anchors,
fresh_rejected_objects,
cached_rejected_objects,
);
cir.validate().map_err(CirExportError::Validate)?;
Ok(cir)
}
pub fn build_cir_from_input_snapshot_multi(
tal_bindings: &[CirTrustAnchorBinding<'_>],
validation_time: time::OffsetDateTime,
input: CirInputSnapshot,
) -> Result<CanonicalInputRepresentation, CirExportError> {
let trust_anchors = build_cir_trust_anchors(tal_bindings)?;
Ok(CanonicalInputRepresentation::new_v4(
validation_time,
input.fresh_validated_objects,
input.cached_validated_objects,
trust_anchors,
input.fresh_rejected_objects,
input.cached_rejected_objects,
))
}
pub fn write_cir_file(
path: &Path,
cir: &CanonicalInputRepresentation,
) -> Result<(), CirExportError> {
let der = encode_cir(cir)?;
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| CirExportError::Write(path.display().to_string(), e.to_string()))?;
}
std::fs::write(path, der)
.map_err(|e| CirExportError::Write(path.display().to_string(), e.to_string()))
}
pub fn export_cir_static_pool(
store: &RocksStore,
static_root: &Path,
capture_date_utc: time::Date,
cir: &CanonicalInputRepresentation,
trust_anchors: &[&TrustAnchor],
) -> Result<CirStaticPoolExportSummary, CirExportError> {
let _ = trust_anchors;
let hashes = cir
.validated_objects()
.map(|item| hex::encode(&item.sha256))
.collect::<Vec<_>>();
export_hashes_from_store(store, static_root, capture_date_utc, &hashes).map_err(Into::into)
}
pub fn export_cir_raw_store(
store: &RocksStore,
raw_store_path: &Path,
cir: &CanonicalInputRepresentation,
trust_anchors: &[&TrustAnchor],
) -> Result<CirRawStoreExportSummary, CirExportError> {
let _ = trust_anchors;
let unique: BTreeSet<String> = cir
.validated_objects()
.map(|item| hex::encode(&item.sha256))
.collect();
let written_entries = 0usize;
let mut reused_entries = 0usize;
for sha256_hex in &unique {
if store
.get_blob_bytes(sha256_hex)
.map_err(|e| {
CirExportError::Write(raw_store_path.display().to_string(), e.to_string())
})?
.is_some()
{
reused_entries += 1;
continue;
}
return Err(CirExportError::Write(
raw_store_path.display().to_string(),
format!("raw store missing object for sha256={sha256_hex}"),
));
}
Ok(CirRawStoreExportSummary {
unique_hashes: unique.len(),
written_entries,
reused_entries,
})
}
pub fn export_cir_from_run(
store: &RocksStore,
trust_anchor: &TrustAnchor,
tal_uri: &str,
validation_time: time::OffsetDateTime,
publication_points: &[PublicationPointAudit],
cir_out: &Path,
capture_date_utc: time::Date,
) -> Result<CirExportSummary, CirExportError> {
export_cir_from_run_multi(
store,
&[CirTrustAnchorBinding {
trust_anchor,
tal_uri,
}],
validation_time,
publication_points,
cir_out,
capture_date_utc,
None,
)
}
pub fn export_cir_from_run_multi(
store: &RocksStore,
tal_bindings: &[CirTrustAnchorBinding<'_>],
validation_time: time::OffsetDateTime,
publication_points: &[PublicationPointAudit],
cir_out: &Path,
capture_date_utc: time::Date,
current_repo_objects: Option<&[CurrentRepoObject]>,
) -> Result<CirExportSummary, CirExportError> {
let _ = capture_date_utc;
let total_started = std::time::Instant::now();
let started = std::time::Instant::now();
let cir = build_cir_from_run_multi(
store,
tal_bindings,
validation_time,
publication_points,
current_repo_objects,
)?;
let build_cir_ms = started.elapsed().as_millis() as u64;
let _ = store;
let started = std::time::Instant::now();
write_cir_file(cir_out, &cir)?;
let write_cir_ms = started.elapsed().as_millis() as u64;
Ok(CirExportSummary {
object_count: cir.validated_object_count(),
trust_anchor_count: cir.trust_anchors.len(),
timing: CirExportTiming {
build_cir_ms,
write_cir_ms,
total_ms: total_started.elapsed().as_millis() as u64,
},
})
}
pub fn export_cir_from_input_snapshot_multi(
tal_bindings: &[CirTrustAnchorBinding<'_>],
validation_time: time::OffsetDateTime,
input: CirInputSnapshot,
cir_out: &Path,
) -> Result<CirExportSummary, CirExportError> {
let total_started = std::time::Instant::now();
let started = std::time::Instant::now();
let cir = build_cir_from_input_snapshot_multi(tal_bindings, validation_time, input)?;
let build_cir_ms = started.elapsed().as_millis() as u64;
let started = std::time::Instant::now();
write_cir_file(cir_out, &cir)?;
let write_cir_ms = started.elapsed().as_millis() as u64;
Ok(CirExportSummary {
object_count: cir.validated_object_count(),
trust_anchor_count: cir.trust_anchors.len(),
timing: CirExportTiming {
build_cir_ms,
write_cir_ms,
total_ms: total_started.elapsed().as_millis() as u64,
},
})
}

View File

@@ -0,0 +1,192 @@
// CIR export data types and audit-object collection helpers.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirExportTiming {
pub build_cir_ms: u64,
pub write_cir_ms: u64,
pub total_ms: u64,
}
#[derive(Debug, thiserror::Error)]
pub enum CirExportError {
#[error("CIR TAL URI must be http(s), got: {0}")]
InvalidTalUri(String),
#[error("TAL does not contain any rsync TA URI; CIR replay scheme A requires one")]
MissingTaRsyncUri,
#[error("CIR model validation failed: {0}")]
Validate(String),
#[error("CIR consumed audit has conflicting hashes for {rsync_uri}: {first} vs {second}")]
ConflictingObjectHash {
rsync_uri: String,
first: String,
second: String,
},
#[error("encode CIR failed: {0}")]
Encode(#[from] CirEncodeError),
#[error("static pool export failed: {0}")]
StaticPool(#[from] CirStaticPoolError),
#[error("write CIR file failed: {0}: {1}")]
Write(String, String),
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirRawStoreExportSummary {
pub unique_hashes: usize,
pub written_entries: usize,
pub reused_entries: usize,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CirExportSummary {
pub object_count: usize,
pub trust_anchor_count: usize,
pub timing: CirExportTiming,
}
#[derive(Clone, Copy, Debug)]
pub struct CirTrustAnchorBinding<'a> {
pub trust_anchor: &'a TrustAnchor,
pub tal_uri: &'a str,
}
fn is_sha256_hex(value: &str) -> bool {
value.len() == 64 && value.as_bytes().iter().all(u8::is_ascii_hexdigit)
}
fn insert_consumed_object_hash(
objects: &mut BTreeMap<String, String>,
rsync_uri: &str,
sha256_hex: &str,
) -> Result<(), CirExportError> {
if !rsync_uri.starts_with("rsync://") || !is_sha256_hex(sha256_hex) {
return Ok(());
}
let normalized = sha256_hex.to_ascii_lowercase();
if let Some(existing) = objects.get(rsync_uri) {
if existing != &normalized {
return Err(CirExportError::ConflictingObjectHash {
rsync_uri: rsync_uri.to_string(),
first: existing.clone(),
second: normalized,
});
}
return Ok(());
}
objects.insert(rsync_uri.to_string(), normalized);
Ok(())
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum CirObjectSection {
Fresh,
Cached,
}
fn publication_point_is_cached_fallback(pp: &PublicationPointAudit) -> bool {
pp.source == "vcir_current_instance" || pp.repo_terminal_state == "fallback_current_instance"
}
fn publication_point_cir_entries<'a>(
pp: &'a PublicationPointAudit,
section: CirObjectSection,
) -> &'a [ObjectAuditEntry] {
match (section, publication_point_is_cached_fallback(pp)) {
(CirObjectSection::Fresh, false) => &pp.objects,
(CirObjectSection::Cached, true) => &pp.objects,
_ => &[],
}
}
fn collect_cir_objects_from_validation_audit(
publication_points: &[PublicationPointAudit],
section: CirObjectSection,
) -> Result<BTreeMap<String, String>, CirExportError> {
let mut objects = BTreeMap::new();
for pp in publication_points {
for obj in publication_point_cir_entries(pp, section) {
if !matches!(obj.result, AuditObjectResult::Ok | AuditObjectResult::Error) {
continue;
}
insert_consumed_object_hash(&mut objects, &obj.rsync_uri, &obj.sha256_hex)?;
}
}
Ok(objects)
}
fn collect_rejected_objects_from_validation_audit(
publication_points: &[PublicationPointAudit],
section: CirObjectSection,
) -> Vec<CirRejectedObject> {
let mut rejected_objects = publication_points
.iter()
.flat_map(|pp| publication_point_cir_entries(pp, section).iter())
.filter(|item| item.result == AuditObjectResult::Error)
.filter(|item| item.rsync_uri.starts_with("rsync://"))
.map(|item| CirRejectedObject {
object_uri: item.rsync_uri.clone(),
reason: item.detail.clone(),
})
.collect::<Vec<_>>();
rejected_objects.sort_by(|a, b| a.object_uri.cmp(&b.object_uri));
rejected_objects.dedup_by(|a, b| a.object_uri == b.object_uri);
rejected_objects
}
fn cir_objects_from_hash_map(objects: BTreeMap<String, String>) -> Vec<CirObject> {
objects
.into_iter()
.map(|(rsync_uri, sha256_hex)| CirObject {
rsync_uri,
sha256: hex::decode(sha256_hex).expect("validated hex"),
})
.collect()
}
fn canonical_ta_rsync_uri(trust_anchor: &TrustAnchor) -> Result<String, CirExportError> {
if let Some(uri) = &trust_anchor.resolved_ta_uri
&& uri.scheme() == "rsync"
{
return Ok(uri.as_str().to_string());
}
trust_anchor
.tal
.ta_uris
.iter()
.filter(|uri| uri.scheme() == "rsync")
.map(|uri| uri.as_str().to_string())
.min()
.ok_or(CirExportError::MissingTaRsyncUri)
}
fn build_cir_trust_anchors(
tal_bindings: &[CirTrustAnchorBinding<'_>],
) -> Result<Vec<CirTrustAnchor>, CirExportError> {
for binding in tal_bindings {
if !(binding.tal_uri.starts_with("https://") || binding.tal_uri.starts_with("http://")) {
return Err(CirExportError::InvalidTalUri(binding.tal_uri.to_string()));
}
}
let mut trust_anchors = Vec::with_capacity(tal_bindings.len());
for binding in tal_bindings {
let ta_rsync_uri = canonical_ta_rsync_uri(binding.trust_anchor)?;
let ta_certificate_der = binding.trust_anchor.ta_certificate.raw_der.clone();
trust_anchors.push(CirTrustAnchor {
ta_rsync_uri,
tal_uri: binding.tal_uri.to_string(),
tal_bytes: binding.trust_anchor.tal.raw.clone(),
ta_certificate_sha256: crate::cir::model::sha256(&ta_certificate_der),
ta_certificate_der,
});
}
trust_anchors.sort_by(|a, b| a.ta_rsync_uri.cmp(&b.ta_rsync_uri));
Ok(trust_anchors)
}

View File

@@ -0,0 +1,780 @@
// CIR export and raw-store integration tests.
use super::*;
use crate::cir::decode::decode_cir;
use crate::data_model::ta::TrustAnchor;
use crate::data_model::tal::Tal;
use crate::storage::{RawByHashEntry, RocksStore};
fn sample_time() -> time::OffsetDateTime {
time::OffsetDateTime::parse(
"2026-04-07T12:34:56Z",
&time::format_description::well_known::Rfc3339,
)
.unwrap()
}
fn sample_date() -> time::Date {
time::Date::from_calendar_date(2026, time::Month::April, 7).unwrap()
}
fn sample_trust_anchor() -> TrustAnchor {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let tal_bytes = std::fs::read(base.join("tests/fixtures/tal/apnic-rfc7730-https.tal")).unwrap();
let ta_der = std::fs::read(base.join("tests/fixtures/ta/apnic-ta.cer")).unwrap();
let tal = Tal::decode_bytes(&tal_bytes).unwrap();
TrustAnchor::bind_der(tal, &ta_der, None).unwrap()
}
fn sample_arin_trust_anchor() -> TrustAnchor {
let base = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let tal_bytes = std::fs::read(base.join("tests/fixtures/tal/arin.tal")).unwrap();
let ta_der = std::fs::read(base.join("tests/fixtures/ta/arin-ta.cer")).unwrap();
let tal = Tal::decode_bytes(&tal_bytes).unwrap();
TrustAnchor::bind_der(tal, &ta_der, None).unwrap()
}
fn sample_trust_anchor_without_rsync_uri() -> TrustAnchor {
let mut ta = sample_trust_anchor();
ta.tal.ta_uris.retain(|uri| uri.scheme() != "rsync");
ta
}
fn sha256_hex(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
hex::encode(Sha256::digest(bytes))
}
fn audit_entry(
uri: &str,
hash: &str,
kind: crate::audit::AuditObjectKind,
result: crate::audit::AuditObjectResult,
detail: Option<&str>,
) -> crate::audit::ObjectAuditEntry {
crate::audit::ObjectAuditEntry {
rsync_uri: uri.to_string(),
sha256_hex: hash.to_string(),
kind,
result,
detail: detail.map(ToString::to_string),
}
}
#[test]
fn build_cir_from_run_collects_consumed_audit_objects_and_tal() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let bytes = b"object-a".to_vec();
let hash = sha256_hex(&bytes);
let publication_points = vec![PublicationPointAudit {
objects: vec![audit_entry(
"rsync://example.test/repo/a.cer",
&hash,
crate::audit::AuditObjectKind::Certificate,
crate::audit::AuditObjectResult::Ok,
None,
)],
..PublicationPointAudit::default()
}];
let ta = sample_trust_anchor();
let cir = build_cir_from_run(
&store,
&ta,
"https://example.test/root.tal",
sample_time(),
&publication_points,
)
.expect("build cir");
assert_eq!(cir.version, crate::cir::model::CIR_VERSION_V4);
assert_eq!(cir.trust_anchors.len(), 1);
assert_eq!(
cir.trust_anchors[0].tal_uri,
"https://example.test/root.tal"
);
assert!(
cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/a.cer")
);
assert!(
!cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == cir.trust_anchors[0].ta_rsync_uri)
);
assert!(!cir.trust_anchors[0].ta_certificate_der.is_empty());
}
#[test]
fn export_cir_from_run_writes_der_and_static_pool() {
let td = tempfile::tempdir().unwrap();
let store_dir = td.path().join("db");
let out_dir = td.path().join("out");
let _static_root = td.path().join("static");
let store = RocksStore::open(&store_dir).unwrap();
let bytes = b"object-b".to_vec();
let hash = sha256_hex(&bytes);
let mut raw = RawByHashEntry::from_bytes(hash.clone(), bytes.clone());
raw.origin_uris
.push("rsync://example.test/repo/b.roa".into());
store.put_raw_by_hash_entry(&raw).unwrap();
let publication_points = vec![PublicationPointAudit {
objects: vec![audit_entry(
"rsync://example.test/repo/b.roa",
&hash,
crate::audit::AuditObjectKind::Roa,
crate::audit::AuditObjectResult::Ok,
None,
)],
..PublicationPointAudit::default()
}];
let ta = sample_trust_anchor();
let cir_path = out_dir.join("example.cir");
let summary = export_cir_from_run(
&store,
&ta,
"https://example.test/root.tal",
sample_time(),
&publication_points,
&cir_path,
sample_date(),
)
.expect("export cir");
assert_eq!(summary.trust_anchor_count, 1);
assert_eq!(summary.object_count, 1);
assert!(summary.timing.total_ms >= summary.timing.build_cir_ms);
let der = std::fs::read(&cir_path).unwrap();
let cir = decode_cir(&der).unwrap();
assert_eq!(
cir.trust_anchors[0].tal_uri,
"https://example.test/root.tal"
);
}
#[test]
fn export_cir_from_run_uses_raw_store_backend_without_pool_export() {
let td = tempfile::tempdir().unwrap();
let store_dir = td.path().join("db");
let raw_store = td.path().join("raw-store.db");
let out_dir = td.path().join("out");
let store = RocksStore::open_with_external_raw_store(&store_dir, &raw_store).unwrap();
let bytes = b"object-d".to_vec();
let hash = sha256_hex(&bytes);
let mut raw = RawByHashEntry::from_bytes(hash.clone(), bytes.clone());
raw.origin_uris
.push("rsync://example.test/repo/d.roa".into());
store.put_raw_by_hash_entry(&raw).unwrap();
let publication_points = vec![PublicationPointAudit {
objects: vec![audit_entry(
"rsync://example.test/repo/d.roa",
&hash,
crate::audit::AuditObjectKind::Roa,
crate::audit::AuditObjectResult::Ok,
None,
)],
..PublicationPointAudit::default()
}];
let ta = sample_trust_anchor();
let cir_path = out_dir.join("example.cir");
let summary = export_cir_from_run(
&store,
&ta,
"https://example.test/root.tal",
sample_time(),
&publication_points,
&cir_path,
sample_date(),
)
.expect("export cir");
assert_eq!(summary.object_count, 1);
assert!(raw_store.exists());
assert!(cir_path.exists());
}
#[test]
fn export_cir_from_run_does_not_write_ta_bytes_to_repo_bytes_store() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open_with_external_repo_bytes(
&td.path().join("db"),
&td.path().join("repo-bytes.db"),
)
.unwrap();
let ta = sample_trust_anchor();
let ta_hash = sha256_hex(&ta.ta_certificate.raw_der);
let cir_path = td.path().join("out").join("example.cir");
export_cir_from_run(
&store,
&ta,
"https://example.test/root.tal",
sample_time(),
&[],
&cir_path,
sample_date(),
)
.expect("export cir");
assert_eq!(store.get_blob_bytes(&ta_hash).unwrap(), None);
}
#[test]
fn build_cir_from_run_includes_consumed_vcir_current_instance_objects_from_audit() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let ta = sample_trust_anchor();
let mut pp = PublicationPointAudit {
source: "vcir_current_instance".to_string(),
..PublicationPointAudit::default()
};
pp.objects.push(crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/fallback.mft".to_string(),
sha256_hex: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".to_string(),
kind: crate::audit::AuditObjectKind::Manifest,
result: crate::audit::AuditObjectResult::Ok,
detail: None,
});
pp.objects.push(crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/fallback.roa".to_string(),
sha256_hex: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb".to_string(),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Ok,
detail: None,
});
let cir = build_cir_from_run(
&store,
&ta,
"https://example.test/root.tal",
sample_time(),
&[pp],
)
.expect("build cir");
assert!(
cir.cached_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/fallback.mft")
);
assert!(
cir.cached_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/fallback.roa")
);
}
#[test]
fn build_cir_from_run_multi_ignores_current_repo_superfluous_objects() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let ta1 = sample_trust_anchor();
let ta2 = sample_arin_trust_anchor();
let current_repo_objects = vec![crate::current_repo_index::CurrentRepoObject {
rsync_uri: "rsync://example.test/repo/superfluous.roa".to_string(),
current_hash_hex: "11".repeat(32),
repository_source: "https://rrdp.example.test/notification.xml".to_string(),
object_type: Some("roa".to_string()),
}];
let publication_points = vec![PublicationPointAudit {
objects: vec![audit_entry(
"rsync://example.test/repo/consumed.roa",
&"22".repeat(32),
crate::audit::AuditObjectKind::Roa,
crate::audit::AuditObjectResult::Ok,
None,
)],
..PublicationPointAudit::default()
}];
let cir = build_cir_from_run_multi(
&store,
&[
CirTrustAnchorBinding {
trust_anchor: &ta1,
tal_uri: "https://example.test/apnic.tal",
},
CirTrustAnchorBinding {
trust_anchor: &ta2,
tal_uri: "https://example.test/arin.tal",
},
],
sample_time(),
&publication_points,
Some(&current_repo_objects),
)
.expect("build cir from consumed audit objects");
assert_eq!(cir.trust_anchors.len(), 2);
assert!(
cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/consumed.roa")
);
assert!(
!cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/superfluous.roa"),
"current repo objects must not be included unless validation consumed them",
);
for trust_anchor in &cir.trust_anchors {
assert!(
!cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == trust_anchor.ta_rsync_uri),
"trust anchor rsync objects must not be included in CIR.objects",
);
}
}
#[test]
fn build_cir_from_run_multi_sorts_trust_anchors_by_ta_rsync_uri() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let apnic = sample_trust_anchor();
let arin = sample_arin_trust_anchor();
let cir = build_cir_from_run_multi(
&store,
&[
CirTrustAnchorBinding {
trust_anchor: &apnic,
tal_uri: "https://rpki.apnic.net/repository/apnic-rpki-root-iana-origin.cer",
},
CirTrustAnchorBinding {
trust_anchor: &arin,
tal_uri: "https://rrdp.arin.net/arin-rpki-ta.cer",
},
],
sample_time(),
&[],
Some(&[]),
)
.expect("build cir with unsorted input bindings");
assert_eq!(
cir.trust_anchors
.iter()
.map(|trust_anchor| trust_anchor.ta_rsync_uri.as_str())
.collect::<Vec<_>>(),
vec![
"rsync://rpki.apnic.net/repository/apnic-rpki-root-iana-origin.cer",
"rsync://rpki.arin.net/repository/arin-rpki-ta.cer",
]
);
}
#[test]
fn build_cir_from_run_multi_exports_rejected_objects_from_error_audit_only() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let ta = sample_trust_anchor();
let publication_points = vec![PublicationPointAudit {
objects: vec![
crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
sha256_hex: "11".repeat(32),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Error,
detail: Some("invalid roa".to_string()),
},
crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/b.asa".to_string(),
sha256_hex: "22".repeat(32),
kind: crate::audit::AuditObjectKind::Aspa,
result: crate::audit::AuditObjectResult::Skipped,
detail: Some("skipped".to_string()),
},
crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/c.roa".to_string(),
sha256_hex: "33".repeat(32),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Error,
detail: Some("second rejected roa".to_string()),
},
],
..PublicationPointAudit::default()
}];
let cir = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &ta,
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
&publication_points,
None,
)
.expect("build cir");
assert_eq!(cir.rejected_object_count(), 2);
assert_eq!(
cir.fresh_rejected_objects[0].object_uri,
"rsync://example.test/repo/a.roa"
);
assert_eq!(
cir.fresh_rejected_objects[0].reason.as_deref(),
Some("invalid roa")
);
assert_eq!(
cir.fresh_rejected_objects[1].object_uri,
"rsync://example.test/repo/c.roa"
);
assert!(
cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/a.roa"),
"rejected audit objects were still consumed as validation input",
);
assert!(
cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/c.roa"),
"rejected audit objects were still consumed as validation input",
);
assert!(
!cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == "rsync://example.test/repo/b.asa"),
"skipped audit objects are not considered consumed input",
);
}
#[test]
fn build_cir_from_run_multi_records_crl_expired_manifest_in_objects_and_rejects() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let ta = sample_trust_anchor();
let manifest_uri = "rsync://example.test/repo/expired-crl.mft";
let reject_reason = "manifest embedded EE certificate path validation failed: CRL not valid at validation_time (RFC 5280 §6.3.3(g); RFC 5280 §5.1.2.4-§5.1.2.5; RFC 6487 §5)";
let publication_points = vec![PublicationPointAudit {
objects: vec![crate::audit::ObjectAuditEntry {
rsync_uri: manifest_uri.to_string(),
sha256_hex: "44".repeat(32),
kind: crate::audit::AuditObjectKind::Manifest,
result: crate::audit::AuditObjectResult::Error,
detail: Some(reject_reason.to_string()),
}],
..PublicationPointAudit::default()
}];
let cir = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &ta,
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
&publication_points,
None,
)
.expect("build cir");
assert!(
cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == manifest_uri),
"manifest rejected because issuer CRL is expired was still read as validation input",
);
assert_eq!(cir.rejected_object_count(), 1);
assert_eq!(cir.fresh_rejected_objects[0].object_uri, manifest_uri);
assert_eq!(
cir.fresh_rejected_objects[0].reason.as_deref(),
Some(reject_reason)
);
assert_eq!(
cir.reject_list_sha256,
crate::cir::model::compute_reject_list_sha256([manifest_uri].into_iter())
);
}
#[test]
fn build_cir_from_run_multi_excludes_manifest_locked_files_when_manifest_is_rejected() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let ta = sample_trust_anchor();
let manifest_uri = "rsync://example.test/repo/rejected.mft";
let roa_uri = "rsync://example.test/repo/listed.roa";
let crl_uri = "rsync://example.test/repo/listed.crl";
let publication_points = vec![PublicationPointAudit {
objects: vec![
crate::audit::ObjectAuditEntry {
rsync_uri: manifest_uri.to_string(),
sha256_hex: "44".repeat(32),
kind: crate::audit::AuditObjectKind::Manifest,
result: crate::audit::AuditObjectResult::Error,
detail: Some("manifest EE cert path rejected".to_string()),
},
crate::audit::ObjectAuditEntry {
rsync_uri: roa_uri.to_string(),
sha256_hex: "55".repeat(32),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Skipped,
detail: Some("manifest rejected before locked object validation".to_string()),
},
crate::audit::ObjectAuditEntry {
rsync_uri: crl_uri.to_string(),
sha256_hex: "66".repeat(32),
kind: crate::audit::AuditObjectKind::Crl,
result: crate::audit::AuditObjectResult::Skipped,
detail: Some("manifest rejected before locked object validation".to_string()),
},
],
..PublicationPointAudit::default()
}];
let cir = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &ta,
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
&publication_points,
None,
)
.expect("build cir");
assert!(
cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == manifest_uri),
"rejected manifest is still a current-run validation input",
);
assert!(
!cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == roa_uri),
"ROA listed by a rejected manifest must not enter CIR objects",
);
assert!(
!cir.fresh_validated_objects
.iter()
.any(|item| item.rsync_uri == crl_uri),
"CRL listed by a rejected manifest must not enter CIR objects",
);
assert_eq!(cir.rejected_object_count(), 1);
assert_eq!(cir.fresh_rejected_objects[0].object_uri, manifest_uri);
}
#[test]
fn build_cir_from_run_multi_reject_digest_ignores_reason_text() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let ta = sample_trust_anchor();
let mk_pp = |detail: &str| PublicationPointAudit {
objects: vec![crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
sha256_hex: "11".repeat(32),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Error,
detail: Some(detail.to_string()),
}],
..PublicationPointAudit::default()
};
let cir_a = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &ta,
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
&[mk_pp("reason-a")],
None,
)
.expect("build cir a");
let cir_b = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &ta,
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
&[mk_pp("reason-b")],
None,
)
.expect("build cir b");
assert_eq!(cir_a.reject_list_sha256, cir_b.reject_list_sha256);
assert_ne!(
cir_a.fresh_rejected_objects[0].reason,
cir_b.fresh_rejected_objects[0].reason
);
}
#[test]
fn build_cir_from_run_multi_rejects_invalid_tal_uri_and_missing_rsync_ta_uri() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(td.path()).unwrap();
let err = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &sample_trust_anchor(),
tal_uri: "file:///not-supported.tal",
}],
sample_time(),
&[],
None,
)
.expect_err("non-http tal uri must fail");
assert!(matches!(err, CirExportError::InvalidTalUri(_)), "{err}");
let err = build_cir_from_run_multi(
&store,
&[CirTrustAnchorBinding {
trust_anchor: &sample_trust_anchor_without_rsync_uri(),
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
&[],
None,
)
.expect_err("missing rsync ta uri must fail");
assert!(matches!(err, CirExportError::MissingTaRsyncUri), "{err}");
}
#[test]
fn build_cir_from_input_snapshot_multi_preserves_online_sections() {
let ta = sample_trust_anchor();
let cir = build_cir_from_input_snapshot_multi(
&[CirTrustAnchorBinding {
trust_anchor: &ta,
tal_uri: "https://example.test/root.tal",
}],
sample_time(),
CirInputSnapshot {
fresh_validated_objects: vec![CirObject {
rsync_uri: "rsync://example.test/repo/fresh.roa".to_string(),
sha256: vec![0x11; 32],
}],
cached_validated_objects: vec![CirObject {
rsync_uri: "rsync://example.test/repo/cached.roa".to_string(),
sha256: vec![0x22; 32],
}],
fresh_rejected_objects: vec![CirRejectedObject {
object_uri: "rsync://example.test/repo/fresh.roa".to_string(),
reason: Some("fresh rejected".to_string()),
}],
cached_rejected_objects: vec![CirRejectedObject {
object_uri: "rsync://example.test/repo/cached.roa".to_string(),
reason: None,
}],
},
)
.expect("build CIR from online input");
cir.validate().expect("online CIR snapshot is valid");
assert_eq!(cir.fresh_validated_objects.len(), 1);
assert_eq!(cir.cached_validated_objects.len(), 1);
assert_eq!(cir.fresh_rejected_objects.len(), 1);
assert_eq!(cir.cached_rejected_objects.len(), 1);
}
#[test]
fn export_cir_static_pool_writes_repository_objects_only() {
let td = tempfile::tempdir().unwrap();
let store = RocksStore::open(&td.path().join("db")).unwrap();
let static_root = td.path().join("static");
let ta1 = sample_trust_anchor();
let ta2 = sample_arin_trust_anchor();
let object_bytes = b"object-z".to_vec();
let hash = sha256_hex(&object_bytes);
let mut raw = RawByHashEntry::from_bytes(hash.clone(), object_bytes.clone());
raw.origin_uris
.push("rsync://example.test/repo/z.roa".into());
store.put_raw_by_hash_entry(&raw).unwrap();
let publication_points = vec![PublicationPointAudit {
objects: vec![audit_entry(
"rsync://example.test/repo/z.roa",
&hash,
crate::audit::AuditObjectKind::Roa,
crate::audit::AuditObjectResult::Ok,
None,
)],
..PublicationPointAudit::default()
}];
let cir = build_cir_from_run_multi(
&store,
&[
CirTrustAnchorBinding {
trust_anchor: &ta1,
tal_uri: "https://example.test/apnic.tal",
},
CirTrustAnchorBinding {
trust_anchor: &ta2,
tal_uri: "https://example.test/arin.tal",
},
],
sample_time(),
&publication_points,
None,
)
.expect("build cir");
let summary = export_cir_static_pool(&store, &static_root, sample_date(), &cir, &[&ta1, &ta2])
.expect("export static pool");
assert_eq!(summary.unique_hashes, 1);
assert_eq!(summary.written_files, 1);
for trust_anchor in &cir.trust_anchors {
let ta_hash = hex::encode(&trust_anchor.ta_certificate_sha256);
assert!(
!crate::cir::static_pool::static_pool_path(&static_root, sample_date(), &ta_hash)
.expect("static pool ta path")
.exists()
);
}
}
#[test]
fn export_cir_raw_store_reports_missing_non_ta_object_only() {
let td = tempfile::tempdir().unwrap();
let raw_store_path = td.path().join("raw-store.db");
let store =
RocksStore::open_with_external_raw_store(&td.path().join("db"), &raw_store_path).unwrap();
let ta1 = sample_trust_anchor();
let ta2 = sample_arin_trust_anchor();
let cir_only_tas = build_cir_from_run_multi(
&store,
&[
CirTrustAnchorBinding {
trust_anchor: &ta1,
tal_uri: "https://example.test/apnic.tal",
},
CirTrustAnchorBinding {
trust_anchor: &ta2,
tal_uri: "https://example.test/arin.tal",
},
],
sample_time(),
&[],
Some(&[]),
)
.expect("build cir with tas only");
let summary = export_cir_raw_store(&store, &raw_store_path, &cir_only_tas, &[&ta1, &ta2])
.expect("export raw store");
assert_eq!(summary.unique_hashes, 0);
assert_eq!(summary.written_entries, 0);
assert_eq!(summary.reused_entries, 0);
let mut cir_missing_object = cir_only_tas.clone();
cir_missing_object.fresh_validated_objects.push(CirObject {
rsync_uri: "rsync://example.test/repo/missing.roa".to_string(),
sha256: vec![0x44; 32],
});
let err = export_cir_raw_store(&store, &raw_store_path, &cir_missing_object, &[&ta1, &ta2])
.expect_err("missing non-ta object must fail");
assert!(matches!(err, CirExportError::Write(_, _)), "{err}");
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,819 @@
// Command-line argument parsing and validation.
pub fn parse_args(argv: &[String]) -> Result<CliArgs, String> {
let mut validation_contract_out_path: Option<PathBuf> = None;
let mut tal_urls: Vec<String> = Vec::new();
let mut tal_paths: Vec<PathBuf> = Vec::new();
let mut ta_paths: Vec<PathBuf> = Vec::new();
let mut ta_constraint_specs: Vec<String> = Vec::new();
let mut parallel_phase1_cfg = ParallelPhase1Config::default();
let mut parallel_phase2_cfg = ParallelPhase2Config::default();
let mut dead_repo_blacklist_path: Option<PathBuf> = None;
let mut dead_repo_blacklist_fail_threshold: Option<u32> = None;
let mut db_path: Option<PathBuf> = None;
let mut raw_store_db: Option<PathBuf> = None;
let mut repo_bytes_db: Option<PathBuf> = None;
let mut policy_path: Option<PathBuf> = None;
let mut strict_policy: Option<StrictPolicy> = None;
let mut resource_validation_mode: Option<ResourceValidationMode> = None;
let mut report_json_path: Option<PathBuf> = None;
let mut report_json_compact: bool = false;
let mut skip_report_build: bool = false;
let mut skip_vcir_persist: bool = false;
let mut enable_roa_validation_cache: bool = false;
let mut enable_child_certificate_validation_cache: bool = false;
let mut publication_point_cache_observe_only: bool = false;
let mut crypto_signature_cache_observe_only: bool = false;
let mut enable_crypto_signature_cache: bool = false;
let mut enable_publication_point_validation_cache: bool = false;
let mut enable_transport_request_prefetch: bool = false;
let mut ccr_out_path: Option<PathBuf> = None;
let mut vrps_csv_out_path: Option<PathBuf> = None;
let mut vaps_csv_out_path: Option<PathBuf> = None;
let mut compare_view_trust_anchor: Option<String> = None;
let mut cir_enabled: bool = false;
let mut cir_out_path: Option<PathBuf> = None;
let mut cir_static_root: Option<PathBuf> = None;
let mut cir_tal_uris: Vec<String> = Vec::new();
let mut cir_tal_uri: Option<String> = None;
let mut payload_replay_archive: Option<PathBuf> = None;
let mut payload_replay_locks: Option<PathBuf> = None;
let mut payload_base_archive: Option<PathBuf> = None;
let mut payload_base_locks: Option<PathBuf> = None;
let mut payload_base_validation_time: Option<time::OffsetDateTime> = None;
let mut payload_delta_archive: Option<PathBuf> = None;
let mut payload_delta_locks: Option<PathBuf> = None;
let mut memory_trim_after_validation = false;
let mut rsync_local_dir: Option<PathBuf> = None;
let mut disable_rrdp: bool = false;
let mut rsync_command: Option<PathBuf> = None;
let mut http_timeout_secs: u64 = 30;
let mut http_root_cert_paths: Vec<PathBuf> = Vec::new();
let mut rsync_timeout_secs: u64 = 30;
let mut rsync_mirror_root: Option<PathBuf> = None;
let mut rsync_scope_policy = RsyncScopePolicy::default();
let mut max_ca_depth: Option<usize> = None;
let mut max_ca_depth_option: Option<&'static str> = None;
let mut max_instances: Option<usize> = None;
let mut validation_time: Option<time::OffsetDateTime> = None;
let mut analyze: bool = false;
let mut analysis_out_path: Option<PathBuf> = None;
let mut profile_cpu: bool = false;
let mut i = 1usize;
while i < argv.len() {
let arg = argv[i].as_str();
match arg {
"--help" | "-h" => return Err(usage()),
"--validation-contract-out" => {
i += 1;
validation_contract_out_path = Some(PathBuf::from(
argv.get(i)
.ok_or("--validation-contract-out requires a value")?,
));
}
"--tal-url" => {
i += 1;
let v = argv.get(i).ok_or("--tal-url requires a value")?;
tal_urls.push(v.clone());
}
"--tal-path" => {
i += 1;
let v = argv.get(i).ok_or("--tal-path requires a value")?;
tal_paths.push(PathBuf::from(v));
}
"--ta-path" => {
i += 1;
let v = argv.get(i).ok_or("--ta-path requires a value")?;
ta_paths.push(PathBuf::from(v));
}
"--ta-constraints" => {
i += 1;
let v = argv
.get(i)
.ok_or("--ta-constraints requires <tal-id>=<path>")?;
ta_constraint_specs.push(v.clone());
}
"--parallel-max-repo-sync-workers-global" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-max-repo-sync-workers-global requires a value")?;
parallel_phase1_cfg.max_repo_sync_workers_global = v
.parse::<usize>()
.map_err(|_| format!("invalid --parallel-max-repo-sync-workers-global: {v}"))?;
}
"--parallel-max-inflight-snapshot-bytes-global" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-max-inflight-snapshot-bytes-global requires a value")?;
parallel_phase1_cfg.max_inflight_snapshot_bytes_global =
v.parse::<usize>().map_err(|_| {
format!("invalid --parallel-max-inflight-snapshot-bytes-global: {v}")
})?;
}
"--parallel-max-pending-repo-results" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-max-pending-repo-results requires a value")?;
parallel_phase1_cfg.max_pending_repo_results = v
.parse::<usize>()
.map_err(|_| format!("invalid --parallel-max-pending-repo-results: {v}"))?;
}
"--parallel-phase2-object-workers" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-object-workers requires a value")?;
parallel_phase2_cfg.object_workers = v
.parse::<usize>()
.map_err(|_| format!("invalid --parallel-phase2-object-workers: {v}"))?;
}
"--parallel-phase2-worker-queue-capacity" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-worker-queue-capacity requires a value")?;
parallel_phase2_cfg.worker_queue_capacity = v
.parse::<usize>()
.map_err(|_| format!("invalid --parallel-phase2-worker-queue-capacity: {v}"))?;
}
"--parallel-phase2-ready-batch-size" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-ready-batch-size requires a value")?;
parallel_phase2_cfg.ready_batch_size = v
.parse::<usize>()
.map_err(|_| format!("invalid --parallel-phase2-ready-batch-size: {v}"))?;
}
"--parallel-phase2-ready-batch-wall-time-budget-ms" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-ready-batch-wall-time-budget-ms requires a value")?;
parallel_phase2_cfg.ready_batch_wall_time_budget_ms =
v.parse::<u64>().map_err(|_| {
format!("invalid --parallel-phase2-ready-batch-wall-time-budget-ms: {v}")
})?;
}
"--parallel-phase2-result-drain-batch-size" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-result-drain-batch-size requires a value")?;
parallel_phase2_cfg.object_result_drain_batch_size =
v.parse::<usize>().map_err(|_| {
format!("invalid --parallel-phase2-result-drain-batch-size: {v}")
})?;
}
"--parallel-phase2-finalize-batch-size" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-finalize-batch-size requires a value")?;
parallel_phase2_cfg.publication_point_finalize_batch_size = v
.parse::<usize>()
.map_err(|_| format!("invalid --parallel-phase2-finalize-batch-size: {v}"))?;
}
"--parallel-phase2-finalize-batch-wall-time-budget-ms" => {
i += 1;
let v = argv.get(i).ok_or(
"--parallel-phase2-finalize-batch-wall-time-budget-ms requires a value",
)?;
parallel_phase2_cfg.publication_point_finalize_wall_time_budget_ms =
v.parse::<u64>().map_err(|_| {
format!("invalid --parallel-phase2-finalize-batch-wall-time-budget-ms: {v}")
})?;
}
"--parallel-phase2-finalize-queue-capacity" => {
i += 1;
let v = argv
.get(i)
.ok_or("--parallel-phase2-finalize-queue-capacity requires a value")?;
parallel_phase2_cfg.publication_point_finalize_queue_capacity =
v.parse::<usize>().map_err(|_| {
format!("invalid --parallel-phase2-finalize-queue-capacity: {v}")
})?;
}
"--control-plane-stage-workers" => {
i += 1;
let v = argv
.get(i)
.ok_or("--control-plane-stage-workers requires a value")?;
parallel_phase2_cfg.stage_workers = v
.parse::<usize>()
.map_err(|_| format!("invalid --control-plane-stage-workers: {v}"))?;
}
"--dead-repo-blacklist" => {
i += 1;
let v = argv
.get(i)
.ok_or("--dead-repo-blacklist requires a value")?;
dead_repo_blacklist_path = Some(PathBuf::from(v));
}
"--dead-repo-blacklist-fail-threshold" => {
i += 1;
let v = argv
.get(i)
.ok_or("--dead-repo-blacklist-fail-threshold requires a value")?;
dead_repo_blacklist_fail_threshold =
Some(v.parse::<u32>().map_err(|_| {
format!("invalid --dead-repo-blacklist-fail-threshold: {v}")
})?);
}
"--db" => {
i += 1;
let v = argv.get(i).ok_or("--db requires a value")?;
db_path = Some(PathBuf::from(v));
}
"--raw-store-db" => {
i += 1;
let v = argv.get(i).ok_or("--raw-store-db requires a value")?;
raw_store_db = Some(PathBuf::from(v));
}
"--repo-bytes-db" => {
i += 1;
let v = argv.get(i).ok_or("--repo-bytes-db requires a value")?;
repo_bytes_db = Some(PathBuf::from(v));
}
"--policy" => {
i += 1;
let v = argv.get(i).ok_or("--policy requires a value")?;
policy_path = Some(PathBuf::from(v));
}
"--strict" => {
let next = argv.get(i + 1).map(String::as_str);
let spec = next.filter(|v| !v.starts_with("--"));
if spec.is_some() {
i += 1;
}
strict_policy = Some(StrictPolicy::parse_cli_spec(spec)?);
}
_ if arg.starts_with("--strict=") => {
let spec = arg.strip_prefix("--strict=").expect("prefix checked");
strict_policy = Some(StrictPolicy::parse_cli_spec(Some(spec))?);
}
"--resource-validation-mode" => {
i += 1;
let v = argv
.get(i)
.ok_or("--resource-validation-mode requires a value")?;
resource_validation_mode = Some(ResourceValidationMode::parse_cli_value(v)?);
}
_ if arg.starts_with("--resource-validation-mode=") => {
let spec = arg
.strip_prefix("--resource-validation-mode=")
.expect("prefix checked");
resource_validation_mode = Some(ResourceValidationMode::parse_cli_value(spec)?);
}
"--report-json" => {
i += 1;
let v = argv.get(i).ok_or("--report-json requires a value")?;
report_json_path = Some(PathBuf::from(v));
}
"--report-json-compact" => {
report_json_compact = true;
}
"--skip-report-build" => {
skip_report_build = true;
}
"--skip-vcir-persist" => {
skip_vcir_persist = true;
}
"--enable-roa-validation-cache" => {
enable_roa_validation_cache = true;
}
"--enable-child-certificate-validation-cache" => {
enable_child_certificate_validation_cache = true;
}
"--publication-point-cache-observe-only" => {
publication_point_cache_observe_only = true;
}
"--enable-publication-point-validation-cache" => {
enable_publication_point_validation_cache = true;
}
"--crypto-signature-cache-observe-only" => {
crypto_signature_cache_observe_only = true;
}
"--enable-crypto-signature-cache" => {
enable_crypto_signature_cache = true;
}
"--enable-transport-request-prefetch" => {
enable_transport_request_prefetch = true;
}
"--ccr-out" => {
i += 1;
let v = argv.get(i).ok_or("--ccr-out requires a value")?;
ccr_out_path = Some(PathBuf::from(v));
}
"--vrps-csv-out" => {
i += 1;
let v = argv.get(i).ok_or("--vrps-csv-out requires a value")?;
vrps_csv_out_path = Some(PathBuf::from(v));
}
"--vaps-csv-out" => {
i += 1;
let v = argv.get(i).ok_or("--vaps-csv-out requires a value")?;
vaps_csv_out_path = Some(PathBuf::from(v));
}
"--compare-view-trust-anchor" => {
i += 1;
let v = argv
.get(i)
.ok_or("--compare-view-trust-anchor requires a value")?;
compare_view_trust_anchor = Some(v.clone());
}
"--cir-enable" => {
cir_enabled = true;
}
"--cir-out" => {
i += 1;
let v = argv.get(i).ok_or("--cir-out requires a value")?;
cir_out_path = Some(PathBuf::from(v));
}
"--cir-static-root" => {
i += 1;
let v = argv.get(i).ok_or("--cir-static-root requires a value")?;
cir_static_root = Some(PathBuf::from(v));
}
"--cir-tal-uri" => {
i += 1;
let v = argv.get(i).ok_or("--cir-tal-uri requires a value")?;
cir_tal_uris.push(v.clone());
cir_tal_uri = cir_tal_uris.first().cloned();
}
"--payload-replay-archive" => {
i += 1;
let v = argv
.get(i)
.ok_or("--payload-replay-archive requires a value")?;
payload_replay_archive = Some(PathBuf::from(v));
}
"--payload-replay-locks" => {
i += 1;
let v = argv
.get(i)
.ok_or("--payload-replay-locks requires a value")?;
payload_replay_locks = Some(PathBuf::from(v));
}
"--payload-base-archive" => {
i += 1;
let v = argv
.get(i)
.ok_or("--payload-base-archive requires a value")?;
payload_base_archive = Some(PathBuf::from(v));
}
"--payload-base-locks" => {
i += 1;
let v = argv.get(i).ok_or("--payload-base-locks requires a value")?;
payload_base_locks = Some(PathBuf::from(v));
}
"--payload-base-validation-time" => {
i += 1;
let v = argv
.get(i)
.ok_or("--payload-base-validation-time requires a value")?;
use time::format_description::well_known::Rfc3339;
let t = time::OffsetDateTime::parse(v, &Rfc3339).map_err(|e| {
format!("invalid --payload-base-validation-time (RFC3339 expected): {e}")
})?;
payload_base_validation_time = Some(t);
}
"--payload-delta-archive" => {
i += 1;
let v = argv
.get(i)
.ok_or("--payload-delta-archive requires a value")?;
payload_delta_archive = Some(PathBuf::from(v));
}
"--payload-delta-locks" => {
i += 1;
let v = argv
.get(i)
.ok_or("--payload-delta-locks requires a value")?;
payload_delta_locks = Some(PathBuf::from(v));
}
"--memory-trim-after-validation" => {
memory_trim_after_validation = true;
}
"--rsync-local-dir" => {
i += 1;
let v = argv.get(i).ok_or("--rsync-local-dir requires a value")?;
rsync_local_dir = Some(PathBuf::from(v));
}
"--disable-rrdp" => {
disable_rrdp = true;
}
"--rsync-command" => {
i += 1;
let v = argv.get(i).ok_or("--rsync-command requires a value")?;
rsync_command = Some(PathBuf::from(v));
}
"--http-timeout-secs" => {
i += 1;
let v = argv.get(i).ok_or("--http-timeout-secs requires a value")?;
http_timeout_secs = v
.parse::<u64>()
.map_err(|_| format!("invalid --http-timeout-secs: {v}"))?;
}
"--http-root-cert" => {
i += 1;
let v = argv.get(i).ok_or("--http-root-cert requires a value")?;
http_root_cert_paths.push(PathBuf::from(v));
}
"--rsync-timeout-secs" => {
i += 1;
let v = argv.get(i).ok_or("--rsync-timeout-secs requires a value")?;
rsync_timeout_secs = v
.parse::<u64>()
.map_err(|_| format!("invalid --rsync-timeout-secs: {v}"))?;
}
"--rsync-mirror-root" => {
i += 1;
let v = argv.get(i).ok_or("--rsync-mirror-root requires a value")?;
rsync_mirror_root = Some(PathBuf::from(v));
}
"--rsync-scope" => {
i += 1;
let v = argv.get(i).ok_or("--rsync-scope requires a value")?;
rsync_scope_policy = RsyncScopePolicy::parse_cli_value(v)?;
}
"--max-ca-depth" | "--max-depth" => {
i += 1;
let option = arg;
let v = argv
.get(i)
.ok_or_else(|| format!("{option} requires a value"))?;
if let Some(previous_option) = max_ca_depth_option {
return Err(format!(
"{option} cannot be combined with {previous_option}; use only --max-ca-depth"
));
}
max_ca_depth = Some(
v.parse::<usize>()
.map_err(|_| format!("invalid {option}: {v}"))?,
);
max_ca_depth_option = Some(match option {
"--max-ca-depth" => "--max-ca-depth",
"--max-depth" => "--max-depth",
_ => unreachable!("matched CA depth option"),
});
}
"--max-instances" => {
i += 1;
let v = argv.get(i).ok_or("--max-instances requires a value")?;
max_instances = Some(
v.parse::<usize>()
.map_err(|_| format!("invalid --max-instances: {v}"))?,
);
}
"--validation-time" => {
i += 1;
let v = argv.get(i).ok_or("--validation-time requires a value")?;
use time::format_description::well_known::Rfc3339;
let t = time::OffsetDateTime::parse(v, &Rfc3339)
.map_err(|e| format!("invalid --validation-time (RFC3339 expected): {e}"))?;
validation_time = Some(t);
}
"--analyze" => {
analyze = true;
}
"--analysis-out" => {
i += 1;
let v = argv.get(i).ok_or("--analysis-out requires a value")?;
analyze = true;
analysis_out_path = Some(PathBuf::from(v));
}
"--profile-cpu" => {
profile_cpu = true;
}
_ => return Err(format!("unknown argument: {arg}\n\n{}", usage())),
}
i += 1;
}
let db_path = db_path.ok_or_else(|| format!("--db is required\n\n{}", usage()))?;
let tal_mode_count = (!tal_urls.is_empty()) as u8 + (!tal_paths.is_empty()) as u8;
if tal_mode_count != 1 {
return Err(format!(
"must specify either one-or-more --tal-url or one-or-more --tal-path/--ta-path pairs\n\n{}",
usage()
));
}
if parallel_phase2_cfg.object_workers == 0 {
return Err(format!(
"--parallel-phase2-object-workers must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.worker_queue_capacity == 0 {
return Err(format!(
"--parallel-phase2-worker-queue-capacity must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.ready_batch_size == 0 {
return Err(format!(
"--parallel-phase2-ready-batch-size must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.ready_batch_wall_time_budget_ms == 0 {
return Err(format!(
"--parallel-phase2-ready-batch-wall-time-budget-ms must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.object_result_drain_batch_size == 0 {
return Err(format!(
"--parallel-phase2-result-drain-batch-size must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.publication_point_finalize_batch_size == 0 {
return Err(format!(
"--parallel-phase2-finalize-batch-size must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.publication_point_finalize_wall_time_budget_ms == 0 {
return Err(format!(
"--parallel-phase2-finalize-batch-wall-time-budget-ms must be > 0\n\n{}",
usage()
));
}
if parallel_phase2_cfg.publication_point_finalize_queue_capacity == 0 {
return Err(format!(
"--parallel-phase2-finalize-queue-capacity must be > 0\n\n{}",
usage()
));
}
if !tal_urls.is_empty() && !ta_paths.is_empty() {
return Err(format!(
"--ta-path cannot be used with --tal-url mode\n\n{}",
usage()
));
}
if !tal_paths.is_empty() {
if !ta_paths.is_empty() {
if ta_paths.len() != tal_paths.len() {
return Err(format!(
"--tal-path and --ta-path counts must match in file mode\n\n{}",
usage()
));
}
} else if ta_paths.is_empty() && !disable_rrdp {
return Err(format!(
"--tal-path requires --ta-path unless --disable-rrdp is set\n\n{}",
usage()
));
}
}
let tal_url = tal_urls.first().cloned();
let tal_path = tal_paths.first().cloned();
let ta_path = ta_paths.first().cloned();
if cir_enabled && cir_out_path.is_none() {
return Err(format!("--cir-enable requires --cir-out\n\n{}", usage()));
}
if report_json_compact && report_json_path.is_none() {
return Err(format!(
"--report-json-compact requires --report-json\n\n{}",
usage()
));
}
if skip_report_build && report_json_path.is_some() {
return Err(format!(
"--skip-report-build cannot be combined with --report-json\n\n{}",
usage()
));
}
if vrps_csv_out_path.is_some() != vaps_csv_out_path.is_some() {
return Err(format!(
"--vrps-csv-out and --vaps-csv-out must be provided together\n\n{}",
usage()
));
}
if compare_view_trust_anchor.is_some() && vrps_csv_out_path.is_none() {
return Err(format!(
"--compare-view-trust-anchor requires --vrps-csv-out/--vaps-csv-out\n\n{}",
usage()
));
}
if cir_static_root.is_some() {
return Err(format!(
"--cir-static-root is no longer supported; CIR export now writes only .cir files\n\n{}",
usage()
));
}
if !cir_enabled && (cir_out_path.is_some() || !cir_tal_uris.is_empty()) {
return Err(format!(
"--cir-out/--cir-tal-uri require --cir-enable\n\n{}",
usage()
));
}
if cir_enabled && !cir_tal_uris.is_empty() {
let expected = if !tal_paths.is_empty() {
tal_paths.len()
} else {
tal_urls.len()
};
if cir_tal_uris.len() != expected {
return Err(format!(
"--cir-tal-uri count must match TAL input count when provided\n\n{}",
usage()
));
}
}
if cir_enabled && !tal_paths.is_empty() && cir_tal_uris.is_empty() {
return Err(format!(
"CIR export in --tal-path mode requires --cir-tal-uri for each TAL\n\n{}",
usage()
));
}
let replay_mode_count =
payload_replay_archive.is_some() as u8 + payload_replay_locks.is_some() as u8;
if replay_mode_count == 1 {
return Err(format!(
"--payload-replay-archive and --payload-replay-locks must be provided together
{}",
usage()
));
}
let delta_mode_count = payload_base_archive.is_some() as u8
+ payload_base_locks.is_some() as u8
+ payload_delta_archive.is_some() as u8
+ payload_delta_locks.is_some() as u8;
if delta_mode_count > 0 && delta_mode_count < 4 {
return Err(format!(
"--payload-base-archive, --payload-base-locks, --payload-delta-archive and --payload-delta-locks must be provided together
{}",
usage()
));
}
if replay_mode_count == 2 && delta_mode_count == 4 {
return Err(format!(
"snapshot replay mode and delta replay mode are mutually exclusive
{}",
usage()
));
}
if replay_mode_count == 2 {
if tal_url.is_some() {
return Err(format!(
"payload replay mode requires --tal-path and --ta-path; --tal-url is not supported
{}",
usage()
));
}
if tal_path.is_none() || ta_path.is_none() {
return Err(format!(
"payload replay mode requires --tal-path and --ta-path
{}",
usage()
));
}
if rsync_local_dir.is_some() {
return Err(format!(
"payload replay mode cannot be combined with --rsync-local-dir
{}",
usage()
));
}
}
if delta_mode_count == 4 {
if tal_url.is_some() {
return Err(format!(
"payload delta replay mode requires --tal-path and --ta-path; --tal-url is not supported
{}",
usage()
));
}
if tal_path.is_none() || ta_path.is_none() {
return Err(format!(
"payload delta replay mode requires --tal-path and --ta-path
{}",
usage()
));
}
if rsync_local_dir.is_some() {
return Err(format!(
"payload delta replay mode cannot be combined with --rsync-local-dir
{}",
usage()
));
}
}
let mut tal_inputs = Vec::new();
if !tal_urls.is_empty() {
tal_inputs.extend(tal_urls.iter().cloned().map(TalInputSpec::from_url));
} else if !tal_paths.is_empty() {
if ta_paths.len() == tal_paths.len() {
tal_inputs.extend(tal_paths.iter().cloned().zip(ta_paths.iter().cloned()).map(
|(tal_path, ta_path)| TalInputSpec::from_file_path_with_ta(tal_path, ta_path),
));
} else {
tal_inputs.extend(tal_paths.iter().cloned().map(TalInputSpec::from_file_path));
}
}
let ta_constraints = TaConstraintsByTal::load_for_tals(&tal_inputs, &ta_constraint_specs)?;
if dead_repo_blacklist_fail_threshold.is_some() && dead_repo_blacklist_path.is_none() {
return Err(
"--dead-repo-blacklist-fail-threshold requires --dead-repo-blacklist".to_string(),
);
}
if let Some(path) = dead_repo_blacklist_path {
let fail_threshold = dead_repo_blacklist_fail_threshold.unwrap_or(3);
if fail_threshold == 0 {
return Err("--dead-repo-blacklist-fail-threshold must be >= 1".to_string());
}
parallel_phase1_cfg.dead_repo_blacklist = Some(
crate::parallel::dead_repo_blacklist::DeadRepoBlacklistConfig {
path,
fail_threshold,
capacity:
crate::parallel::dead_repo_blacklist::DEAD_REPO_BLACKLIST_DEFAULT_CAPACITY,
},
);
}
Ok(CliArgs {
validation_contract_out_path,
tal_urls,
tal_paths,
ta_paths,
tal_url,
tal_path,
ta_path,
parallel_phase1_config: parallel_phase1_cfg,
parallel_phase2_config: parallel_phase2_cfg,
tal_inputs,
ta_constraints,
db_path,
raw_store_db,
repo_bytes_db,
policy_path,
strict_policy,
resource_validation_mode,
report_json_path,
report_json_compact,
skip_report_build,
skip_vcir_persist,
enable_roa_validation_cache,
enable_child_certificate_validation_cache,
publication_point_cache_observe_only,
enable_publication_point_validation_cache,
crypto_signature_cache_observe_only,
enable_crypto_signature_cache,
enable_transport_request_prefetch,
ccr_out_path,
vrps_csv_out_path,
vaps_csv_out_path,
compare_view_trust_anchor,
cir_enabled,
cir_out_path,
cir_static_root,
cir_tal_uris,
cir_tal_uri,
payload_replay_archive,
payload_replay_locks,
payload_base_archive,
payload_base_locks,
payload_base_validation_time,
payload_delta_archive,
payload_delta_locks,
memory_trim_after_validation,
rsync_local_dir,
disable_rrdp,
rsync_command,
http_timeout_secs,
http_root_cert_paths,
rsync_timeout_secs,
rsync_mirror_root,
rsync_scope_policy,
max_ca_depth: max_ca_depth.unwrap_or(DEFAULT_MAX_CA_DEPTH),
max_instances,
validation_time,
analyze,
analysis_out_path,
profile_cpu,
})
}

View File

@@ -0,0 +1,619 @@
// Shared post-validation state and memory/report graph estimation.
fn read_policy(path: Option<&Path>) -> Result<Policy, String> {
match path {
None => Ok(Policy::default()),
Some(p) => {
let s = std::fs::read_to_string(p)
.map_err(|e| format!("read policy file failed: {}: {e}", p.display()))?;
Policy::from_toml_str(&s).map_err(|e| e.to_string())
}
}
}
fn unique_rrdp_repos_from_publication_points(
publication_points: &[crate::audit::PublicationPointAudit],
) -> usize {
use std::collections::HashSet;
let mut set: HashSet<&str> = HashSet::new();
for pp in publication_points {
if let Some(u) = pp.rrdp_notification_uri.as_deref() {
set.insert(u);
}
}
set.len()
}
#[cfg(test)]
fn unique_rrdp_repos(report: &AuditReportV2) -> usize {
unique_rrdp_repos_from_publication_points(&report.publication_points)
}
#[cfg(test)]
fn print_summary(report: &AuditReportV2) {
let rrdp_repos = unique_rrdp_repos(report);
println!("RPKI validation run summary");
println!(
"validation_time={}",
report.meta.validation_time_rfc3339_utc
);
println!(
"publication_points_processed={} publication_points_failed={}",
report.tree.instances_processed, report.tree.instances_failed
);
println!("rrdp_repos_unique={rrdp_repos}");
println!("vrps={}", report.vrps.len());
println!("aspas={}", report.aspas.len());
println!(
"audit_publication_points={}",
report.publication_points.len()
);
println!(
"warnings_total={}",
report.tree.warnings.len()
+ report
.publication_points
.iter()
.map(|pp| pp.warnings.len())
.sum::<usize>()
);
}
fn print_summary_from_shared(validation_time: time::OffsetDateTime, shared: &PostValidationShared) {
use time::format_description::well_known::Rfc3339;
let validation_time_rfc3339_utc = validation_time
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.expect("format validation_time");
let rrdp_repos = unique_rrdp_repos_from_publication_points(shared.publication_points.as_ref());
println!("RPKI validation run summary");
println!("validation_time={validation_time_rfc3339_utc}");
println!(
"publication_points_processed={} publication_points_failed={}",
shared.instances_processed, shared.instances_failed
);
println!("rrdp_repos_unique={rrdp_repos}");
println!("vrps={}", shared.vrps.len());
println!("aspas={}", shared.aspas.len());
println!(
"audit_publication_points={}",
shared.publication_points.len()
);
println!(
"warnings_total={}",
shared.tree_warnings.len()
+ shared
.publication_points
.iter()
.map(|pp| pp.warnings.len())
.sum::<usize>()
);
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct PostValidationShared {
discovery: crate::validation::from_tal::DiscoveredRootCaInstance,
discoveries: Arc<[crate::validation::from_tal::DiscoveredRootCaInstance]>,
successful_tal_inputs: Arc<[TalInputSpec]>,
instances_processed: usize,
instances_failed: usize,
tree_warnings: Arc<[crate::report::Warning]>,
vrps: Arc<[crate::validation::objects::Vrp]>,
aspas: Arc<[crate::validation::objects::AspaAttestation]>,
router_keys: Arc<[crate::validation::objects::RouterKeyPayload]>,
publication_points: Arc<[crate::audit::PublicationPointAudit]>,
roa_cache_stats: crate::validation::objects::RoaValidationCacheStats,
downloads: Arc<[crate::audit::AuditDownloadEvent]>,
download_stats: crate::audit::AuditDownloadStats,
current_repo_objects: Arc<[crate::current_repo_index::CurrentRepoObject]>,
ccr_accumulator: Option<CcrAccumulator>,
cir_input: crate::cir::CirInputSnapshot,
}
impl PostValidationShared {
fn from_run_output(out: RunTreeFromTalAuditOutput) -> Self {
let RunTreeFromTalAuditOutput {
discovery,
discoveries,
successful_tal_inputs,
tree,
publication_points,
roa_cache_stats,
downloads,
download_stats,
current_repo_objects,
ccr_accumulator,
cir_input,
} = out;
let crate::validation::tree::TreeRunOutput {
instances_processed,
instances_failed,
warnings,
vrps,
aspas,
router_keys,
} = tree;
Self {
discovery,
discoveries: discoveries.into(),
successful_tal_inputs: successful_tal_inputs.into(),
instances_processed,
instances_failed,
tree_warnings: warnings.into(),
vrps: vrps.into(),
aspas: aspas.into(),
router_keys: router_keys.into(),
publication_points: publication_points.into(),
roa_cache_stats,
downloads: downloads.into(),
download_stats,
current_repo_objects: current_repo_objects.into(),
ccr_accumulator,
cir_input,
}
}
fn trust_anchors(&self) -> Vec<crate::data_model::ta::TrustAnchor> {
if self.discoveries.is_empty() {
vec![self.discovery.trust_anchor.clone()]
} else {
self.discoveries
.iter()
.map(|item| item.trust_anchor.clone())
.collect()
}
}
}
#[derive(Default)]
struct ObjectGraphSectionBuilder {
name: String,
item_count: u64,
shallow_bytes: u64,
heap_bytes: u64,
string_count: u64,
string_bytes: u64,
string_capacity_bytes: u64,
vec_count: u64,
vec_heap_bytes: u64,
vec_capacity_bytes: u64,
details: Vec<ObjectGraphMemoryMetric>,
}
impl ObjectGraphSectionBuilder {
fn new(name: impl Into<String>) -> Self {
Self {
name: name.into(),
..Self::default()
}
}
fn items(&mut self, count: usize, item_size: usize) {
self.item_count += count as u64;
self.shallow_bytes += (count as u64) * (item_size as u64);
}
fn heap_bytes(&mut self, value: usize) {
self.heap_bytes += value as u64;
}
fn string(&mut self, value: &str) {
self.string_count += 1;
self.string_bytes += value.len() as u64;
self.string_capacity_bytes += value.len() as u64;
self.heap_bytes += value.len() as u64;
}
fn owned_string(&mut self, value: &String) {
self.string_count += 1;
self.string_bytes += value.len() as u64;
self.string_capacity_bytes += value.capacity() as u64;
self.heap_bytes += value.capacity() as u64;
}
fn optional_string(&mut self, value: Option<&String>) {
if let Some(value) = value {
self.owned_string(value);
}
}
fn vec_header_with_capacity(&mut self, len: usize, capacity: usize, element_size: usize) {
self.vec_count += 1;
let payload_bytes = len * element_size;
let capacity_bytes = capacity * element_size;
self.vec_heap_bytes += payload_bytes as u64;
self.vec_capacity_bytes += capacity_bytes as u64;
self.heap_bytes += capacity_bytes as u64;
}
fn byte_vec_owned(&mut self, value: &Vec<u8>) {
self.vec_header_with_capacity(value.len(), value.capacity(), std::mem::size_of::<u8>());
}
fn string_vec_owned(&mut self, values: &Vec<String>) {
self.vec_header_with_capacity(
values.len(),
values.capacity(),
std::mem::size_of::<String>(),
);
for value in values {
self.owned_string(value);
}
}
fn metric(&mut self, name: impl Into<String>, value: u64) {
self.details.push(ObjectGraphMemoryMetric {
name: name.into(),
value,
});
}
fn finish(self) -> ObjectGraphMemorySection {
let estimated_bytes = self.shallow_bytes + self.heap_bytes;
ObjectGraphMemorySection {
name: self.name,
item_count: self.item_count,
shallow_bytes: self.shallow_bytes,
heap_bytes: self.heap_bytes,
estimated_bytes,
string_count: self.string_count,
string_bytes: self.string_bytes,
string_capacity_bytes: self.string_capacity_bytes,
vec_count: self.vec_count,
vec_heap_bytes: self.vec_heap_bytes,
vec_capacity_bytes: self.vec_capacity_bytes,
details: self.details,
}
}
}
fn estimate_shared_object_graph(shared: &PostValidationShared) -> ObjectGraphMemorySummary {
let mut sections = Vec::new();
sections.push(estimate_publication_points_graph(
shared.publication_points.as_ref(),
));
sections.push(estimate_vrps_graph(shared.vrps.as_ref()));
sections.push(estimate_aspas_graph(shared.aspas.as_ref()));
sections.push(estimate_router_keys_graph(shared.router_keys.as_ref()));
sections.push(estimate_warnings_graph(
"tree_warnings",
shared.tree_warnings.as_ref(),
));
sections.push(estimate_downloads_graph(shared.downloads.as_ref()));
sections.push(estimate_current_repo_objects_graph(
shared.current_repo_objects.as_ref(),
));
sections.push(estimate_trust_anchor_graph(shared));
sections.push(estimate_ccr_accumulator_graph(
shared.ccr_accumulator.as_ref(),
));
let total_estimated_bytes = sections
.iter()
.map(|section| section.estimated_bytes)
.sum::<u64>();
ObjectGraphMemorySummary {
captured_at_label: "after_validation".to_string(),
total_estimated_bytes,
sections,
notes: vec![
"Estimated bytes are Rust object graph approximations based on struct sizes and owned String/Vec payload lengths.".to_string(),
"The estimate intentionally excludes allocator metadata, fragmentation, freed-but-retained arenas, RocksDB C++ heap, and transient worker allocations.".to_string(),
"Large RSS minus this estimate points to allocator retention or structures not yet modeled by this telemetry.".to_string(),
],
}
}
fn estimate_publication_points_graph(
publication_points: &[crate::audit::PublicationPointAudit],
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("publication_points");
builder.items(
publication_points.len(),
std::mem::size_of::<crate::audit::PublicationPointAudit>(),
);
builder.metric("publication_point_count", publication_points.len() as u64);
let mut object_count = 0u64;
let mut pp_warning_count = 0u64;
let mut pp_discovered_from_count = 0u64;
let mut object_detail_count = 0u64;
for pp in publication_points {
builder.owned_string(&pp.rsync_base_uri);
builder.owned_string(&pp.manifest_rsync_uri);
builder.owned_string(&pp.publication_point_rsync_uri);
builder.optional_string(pp.rrdp_notification_uri.as_ref());
builder.owned_string(&pp.source);
builder.optional_string(pp.repo_sync_source.as_ref());
builder.optional_string(pp.repo_sync_phase.as_ref());
builder.optional_string(pp.repo_sync_error.as_ref());
builder.owned_string(&pp.repo_terminal_state);
builder.owned_string(&pp.this_update_rfc3339_utc);
builder.owned_string(&pp.next_update_rfc3339_utc);
builder.owned_string(&pp.verified_at_rfc3339_utc);
if let Some(discovered_from) = &pp.discovered_from {
pp_discovered_from_count += 1;
builder.heap_bytes(std::mem::size_of::<crate::audit::DiscoveredFrom>());
builder.owned_string(&discovered_from.parent_manifest_rsync_uri);
builder.owned_string(&discovered_from.child_ca_certificate_rsync_uri);
builder.owned_string(&discovered_from.child_ca_certificate_sha256_hex);
}
pp_warning_count += pp.warnings.len() as u64;
builder.vec_header_with_capacity(
pp.warnings.len(),
pp.warnings.capacity(),
std::mem::size_of::<crate::audit::AuditWarning>(),
);
for warning in &pp.warnings {
builder.owned_string(&warning.message);
builder.string_vec_owned(&warning.rfc_refs);
builder.optional_string(warning.context.as_ref());
}
object_count += pp.objects.len() as u64;
builder.vec_header_with_capacity(
pp.objects.len(),
pp.objects.capacity(),
std::mem::size_of::<crate::audit::ObjectAuditEntry>(),
);
for object in &pp.objects {
builder.owned_string(&object.rsync_uri);
builder.owned_string(&object.sha256_hex);
if object.detail.is_some() {
object_detail_count += 1;
}
builder.optional_string(object.detail.as_ref());
}
}
builder.metric("object_audit_entry_count", object_count);
builder.metric("publication_point_warning_count", pp_warning_count);
builder.metric(
"publication_point_discovered_from_count",
pp_discovered_from_count,
);
builder.metric("object_detail_count", object_detail_count);
builder.finish()
}
fn estimate_vrps_graph(vrps: &[crate::validation::objects::Vrp]) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("vrps");
builder.items(
vrps.len(),
std::mem::size_of::<crate::validation::objects::Vrp>(),
);
builder.metric("vrp_count", vrps.len() as u64);
builder.finish()
}
fn estimate_aspas_graph(
aspas: &[crate::validation::objects::AspaAttestation],
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("aspas");
builder.items(
aspas.len(),
std::mem::size_of::<crate::validation::objects::AspaAttestation>(),
);
let mut providers_total = 0u64;
for aspa in aspas {
providers_total += aspa.provider_as_ids.len() as u64;
builder.vec_header_with_capacity(
aspa.provider_as_ids.len(),
aspa.provider_as_ids.capacity(),
std::mem::size_of::<u32>(),
);
}
builder.metric("aspa_count", aspas.len() as u64);
builder.metric("provider_asn_count", providers_total);
builder.finish()
}
fn estimate_router_keys_graph(
router_keys: &[crate::validation::objects::RouterKeyPayload],
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("router_keys");
builder.items(
router_keys.len(),
std::mem::size_of::<crate::validation::objects::RouterKeyPayload>(),
);
for router_key in router_keys {
builder.byte_vec_owned(&router_key.ski);
builder.byte_vec_owned(&router_key.spki_der);
builder.owned_string(&router_key.source_object_uri);
builder.owned_string(&router_key.source_object_hash);
builder.owned_string(&router_key.source_ee_cert_hash);
}
builder.metric("router_key_count", router_keys.len() as u64);
builder.finish()
}
fn estimate_warnings_graph(
name: &str,
warnings: &[crate::report::Warning],
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new(name);
builder.items(
warnings.len(),
std::mem::size_of::<crate::report::Warning>(),
);
for warning in warnings {
builder.owned_string(&warning.message);
builder.vec_header_with_capacity(
warning.rfc_refs.len(),
warning.rfc_refs.capacity(),
std::mem::size_of::<crate::report::RfcRef>(),
);
builder.optional_string(warning.context.as_ref());
}
builder.metric("warning_count", warnings.len() as u64);
builder.finish()
}
fn estimate_downloads_graph(
downloads: &[crate::audit::AuditDownloadEvent],
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("downloads");
builder.items(
downloads.len(),
std::mem::size_of::<crate::audit::AuditDownloadEvent>(),
);
let mut error_count = 0u64;
let mut bytes_count = 0u64;
let mut objects_stat_count = 0u64;
for event in downloads {
builder.owned_string(&event.uri);
builder.owned_string(&event.started_at_rfc3339_utc);
builder.owned_string(&event.finished_at_rfc3339_utc);
if event.error.is_some() {
error_count += 1;
}
if event.bytes.is_some() {
bytes_count += 1;
}
if event.objects.is_some() {
objects_stat_count += 1;
}
builder.optional_string(event.error.as_ref());
}
builder.metric("download_event_count", downloads.len() as u64);
builder.metric("download_error_count", error_count);
builder.metric("download_bytes_field_count", bytes_count);
builder.metric("download_objects_stat_count", objects_stat_count);
builder.finish()
}
fn estimate_current_repo_objects_graph(
objects: &[crate::current_repo_index::CurrentRepoObject],
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("current_repo_objects");
builder.items(
objects.len(),
std::mem::size_of::<crate::current_repo_index::CurrentRepoObject>(),
);
let mut object_type_count = 0u64;
for object in objects {
builder.owned_string(&object.rsync_uri);
builder.owned_string(&object.current_hash_hex);
builder.owned_string(&object.repository_source);
if object.object_type.is_some() {
object_type_count += 1;
}
builder.optional_string(object.object_type.as_ref());
}
builder.metric("current_repo_object_count", objects.len() as u64);
builder.metric("current_repo_object_type_count", object_type_count);
builder.finish()
}
fn estimate_trust_anchor_graph(shared: &PostValidationShared) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("trust_anchors_and_tal_inputs");
builder.items(
1,
std::mem::size_of::<crate::validation::from_tal::DiscoveredRootCaInstance>(),
);
estimate_discovered_root(&mut builder, &shared.discovery);
builder.items(
shared.discoveries.len(),
std::mem::size_of::<crate::validation::from_tal::DiscoveredRootCaInstance>(),
);
for discovery in shared.discoveries.iter() {
estimate_discovered_root(&mut builder, discovery);
}
builder.items(
shared.successful_tal_inputs.len(),
std::mem::size_of::<TalInputSpec>(),
);
for tal_input in shared.successful_tal_inputs.iter() {
estimate_tal_input(&mut builder, tal_input);
}
builder.metric("discoveries_count", shared.discoveries.len() as u64);
builder.metric(
"successful_tal_inputs_count",
shared.successful_tal_inputs.len() as u64,
);
builder.finish()
}
fn estimate_discovered_root(
builder: &mut ObjectGraphSectionBuilder,
discovery: &crate::validation::from_tal::DiscoveredRootCaInstance,
) {
builder.optional_string(discovery.tal_url.as_ref());
estimate_trust_anchor(builder, &discovery.trust_anchor);
builder.owned_string(&discovery.ca_instance.rsync_base_uri);
builder.owned_string(&discovery.ca_instance.manifest_rsync_uri);
builder.owned_string(&discovery.ca_instance.publication_point_rsync_uri);
builder.optional_string(discovery.ca_instance.rrdp_notification_uri.as_ref());
}
fn estimate_trust_anchor(
builder: &mut ObjectGraphSectionBuilder,
trust_anchor: &crate::data_model::ta::TrustAnchor,
) {
builder.byte_vec_owned(&trust_anchor.tal.raw);
builder.string_vec_owned(&trust_anchor.tal.comments);
builder.vec_header_with_capacity(
trust_anchor.tal.ta_uris.len(),
trust_anchor.tal.ta_uris.capacity(),
std::mem::size_of::<url::Url>(),
);
for uri in &trust_anchor.tal.ta_uris {
builder.string(uri.as_str());
}
builder.byte_vec_owned(&trust_anchor.tal.subject_public_key_info_der);
builder.byte_vec_owned(&trust_anchor.ta_certificate.raw_der);
if let Some(uri) = &trust_anchor.resolved_ta_uri {
builder.string(uri.as_str());
}
}
fn estimate_tal_input(builder: &mut ObjectGraphSectionBuilder, tal_input: &TalInputSpec) {
builder.owned_string(&tal_input.tal_id);
builder.owned_string(&tal_input.rir_id);
match &tal_input.source {
crate::parallel::types::TalSource::Url(url) => builder.owned_string(url),
crate::parallel::types::TalSource::DerBytes {
tal_url,
tal_bytes,
ta_der,
} => {
builder.owned_string(tal_url);
builder.byte_vec_owned(tal_bytes);
builder.byte_vec_owned(ta_der);
}
crate::parallel::types::TalSource::FilePath(path) => {
builder.string(&path.to_string_lossy());
}
crate::parallel::types::TalSource::FilePathWithTa { tal_path, ta_path } => {
builder.string(&tal_path.to_string_lossy());
builder.string(&ta_path.to_string_lossy());
}
}
}
fn estimate_ccr_accumulator_graph(
accumulator: Option<&CcrAccumulator>,
) -> ObjectGraphMemorySection {
let mut builder = ObjectGraphSectionBuilder::new("ccr_accumulator");
if let Some(accumulator) = accumulator {
builder.items(1, std::mem::size_of::<CcrAccumulator>());
let stats = accumulator.memory_stats();
builder.heap_bytes(stats.estimated_heap_bytes as usize);
builder.metric("trust_anchor_count", stats.trust_anchor_count);
builder.metric("manifest_count", stats.manifest_count);
builder.metric("string_bytes", stats.string_bytes);
builder.metric("string_capacity_bytes", stats.string_capacity_bytes);
builder.metric("vec_payload_bytes", stats.vec_payload_bytes);
builder.metric("vec_capacity_bytes", stats.vec_capacity_bytes);
builder.metric("locations_der_count", stats.locations_der_count);
builder.metric("subordinate_ski_count", stats.subordinate_ski_count);
builder.metric("btree_key_capacity_bytes", stats.btree_key_capacity_bytes);
builder.metric("btree_entry_shallow_bytes", stats.btree_entry_shallow_bytes);
} else {
builder.metric("manifest_count", 0);
}
builder.finish()
}

View File

@@ -0,0 +1,218 @@
// Audit report construction and online validation orchestration.
#[cfg(test)]
fn build_report(
policy: &Policy,
validation_time: time::OffsetDateTime,
shared: &PostValidationShared,
) -> AuditReportV2 {
use time::format_description::well_known::Rfc3339;
let validation_time_rfc3339_utc = validation_time
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.expect("format validation_time");
let vrps = shared
.vrps
.iter()
.map(|v| VrpOutput {
asn: v.asn,
prefix: format_roa_ip_prefix(&v.prefix),
max_length: v.max_length,
})
.collect::<Vec<_>>();
let aspas = shared
.aspas
.iter()
.map(|a| AspaOutput {
customer_as_id: a.customer_as_id,
provider_as_ids: a.provider_as_ids.clone(),
})
.collect::<Vec<_>>();
let repo_sync_stats = build_repo_sync_stats(shared.publication_points.as_ref());
AuditReportV2 {
format_version: 2,
meta: AuditRunMeta {
validation_time_rfc3339_utc,
},
policy: policy.clone(),
tree: TreeSummary {
instances_processed: shared.instances_processed,
instances_failed: shared.instances_failed,
warnings: shared
.tree_warnings
.iter()
.map(AuditWarning::from)
.collect(),
},
publication_points: shared.publication_points.iter().cloned().collect(),
vrps,
aspas,
downloads: shared.downloads.iter().cloned().collect(),
download_stats: shared.download_stats.clone(),
repo_sync_stats,
query_audit: None,
}
}
fn run_online_validation_with_fetchers<H, R>(
store: Arc<RocksStore>,
policy: &Policy,
args: &CliArgs,
http: &H,
rsync: &R,
validation_time: time::OffsetDateTime,
config: &TreeRunConfig,
collect_current_repo_objects: bool,
timing: Option<&TimingHandle>,
) -> Result<RunTreeFromTalAuditOutput, String>
where
H: crate::sync::rrdp::Fetcher + Clone + 'static,
R: crate::fetch::rsync::RsyncFetcher + Clone + 'static,
{
// The multi-TAL entry point preserves the TAL id supplied by the CLI.
// A single-file TAL may otherwise derive its id from the embedded TA URI,
// which is intentionally different from the local filename used for
// adjacent <tal>.constraints discovery.
if args.tal_inputs.len() > 1 || !policy.ta_constraints.is_empty() {
return if let Some(t) = timing {
run_tree_from_multiple_tals_parallel_phase2_audit_with_timing(
store,
policy,
args.tal_inputs.clone(),
http,
rsync,
validation_time,
config,
args.parallel_phase1_config.clone(),
args.parallel_phase2_config.clone(),
collect_current_repo_objects,
t,
)
} else {
run_tree_from_multiple_tals_parallel_phase2_audit(
store,
policy,
args.tal_inputs.clone(),
http,
rsync,
validation_time,
config,
args.parallel_phase1_config.clone(),
args.parallel_phase2_config.clone(),
collect_current_repo_objects,
)
}
.map_err(|e| e.to_string());
}
match (
args.tal_url.as_ref(),
args.tal_path.as_ref(),
args.ta_path.as_ref(),
) {
(Some(url), _, _) => if let Some(t) = timing {
run_tree_from_tal_url_parallel_phase2_audit_with_timing(
store,
policy,
url,
http,
rsync,
validation_time,
config,
args.parallel_phase1_config.clone(),
args.parallel_phase2_config.clone(),
collect_current_repo_objects,
t,
)
} else {
run_tree_from_tal_url_parallel_phase2_audit(
store,
policy,
url,
http,
rsync,
validation_time,
config,
args.parallel_phase1_config.clone(),
args.parallel_phase2_config.clone(),
collect_current_repo_objects,
)
}
.map_err(|e| e.to_string()),
(None, Some(tal_path), Some(ta_path)) => {
let tal_bytes = std::fs::read(tal_path)
.map_err(|e| format!("read tal failed: {}: {e}", tal_path.display()))?;
let ta_der = std::fs::read(ta_path)
.map_err(|e| format!("read ta failed: {}: {e}", ta_path.display()))?;
if let Some(t) = timing {
run_tree_from_tal_and_ta_der_parallel_phase2_audit_with_timing(
store,
policy,
&tal_bytes,
&ta_der,
None,
http,
rsync,
validation_time,
config,
args.parallel_phase1_config.clone(),
args.parallel_phase2_config.clone(),
collect_current_repo_objects,
t,
)
} else {
run_tree_from_tal_and_ta_der_parallel_phase2_audit(
store,
policy,
&tal_bytes,
&ta_der,
None,
http,
rsync,
validation_time,
config,
args.parallel_phase1_config.clone(),
args.parallel_phase2_config.clone(),
collect_current_repo_objects,
)
}
.map_err(|e| e.to_string())
}
(None, Some(tal_path), None) => {
let tal_bytes = std::fs::read(tal_path)
.map_err(|e| format!("read tal failed: {}: {e}", tal_path.display()))?;
let tal_uri = args.cir_tal_uri.clone();
if let Some(t) = timing {
crate::validation::run_tree_from_tal::run_tree_from_tal_bytes_serial_audit_with_timing(
store.as_ref(),
policy,
&tal_bytes,
tal_uri,
http,
rsync,
validation_time,
config,
t,
)
.map_err(|e| e.to_string())
} else {
crate::validation::run_tree_from_tal::run_tree_from_tal_bytes_serial_audit(
store.as_ref(),
policy,
&tal_bytes,
tal_uri,
http,
rsync,
validation_time,
config,
)
.map_err(|e| e.to_string())
}
}
_ => unreachable!("validated by parse_args"),
}
}

View File

@@ -0,0 +1,180 @@
//! Isolated CLI output tasks.
//!
//! Report, CCR, CIR input selection, and repository-sync aggregation have no
//! dependency on network execution. Keeping them here lets the CLI
//! orchestrator own scheduling while these functions retain focused tests.
use std::path::Path;
use crate::audit::AuditRepoSyncStats;
use crate::ccr::{CcrBuildBreakdown, build_ccr_from_run_with_breakdown, write_ccr_file};
use crate::policy::Policy;
use crate::storage::RocksStore;
use super::output::{ReportJsonFormat, write_report_json_from_shared};
use super::{CliArgs, PostValidationShared};
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct ReportTaskOutput {
pub(super) report_build_ms: u64,
pub(super) report_write_ms: Option<u64>,
}
impl ReportTaskOutput {
pub(super) fn skipped() -> Self {
Self {
report_build_ms: 0,
report_write_ms: None,
}
}
}
pub(super) fn run_report_task(
policy: &Policy,
validation_time: time::OffsetDateTime,
shared: &PostValidationShared,
report_json_path: Option<&Path>,
report_json_format: ReportJsonFormat,
) -> Result<ReportTaskOutput, String> {
if let Some(path) = report_json_path {
let timing = write_report_json_from_shared(
path,
policy,
validation_time,
shared,
report_json_format,
)?;
Ok(ReportTaskOutput {
report_build_ms: timing.build_ms,
report_write_ms: Some(timing.write_ms),
})
} else {
Ok(ReportTaskOutput::skipped())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct CcrTaskOutput {
pub(super) ccr_build_ms: Option<u64>,
pub(super) ccr_build_breakdown: Option<CcrBuildBreakdown>,
pub(super) ccr_write_ms: Option<u64>,
}
pub(super) fn run_ccr_task(
store: &RocksStore,
shared: &PostValidationShared,
ccr_out_path: Option<&Path>,
produced_at: time::OffsetDateTime,
) -> Result<CcrTaskOutput, String> {
let mut ccr_build_ms = None;
let mut ccr_build_breakdown = None;
let mut ccr_write_ms = None;
if let Some(path) = ccr_out_path {
let started = std::time::Instant::now();
let (ccr, build_breakdown) = if let Some(accumulator) = shared.ccr_accumulator.as_ref() {
(
accumulator
.finish(
produced_at,
shared.vrps.as_ref(),
shared.aspas.as_ref(),
shared.router_keys.as_ref(),
)
.map_err(|e| e.to_string())?,
None,
)
} else {
let trust_anchors = shared.trust_anchors();
let (ccr, build_breakdown) = build_ccr_from_run_with_breakdown(
store,
&trust_anchors,
shared.vrps.as_ref(),
shared.aspas.as_ref(),
shared.router_keys.as_ref(),
produced_at,
)
.map_err(|e| e.to_string())?;
(ccr, Some(build_breakdown))
};
ccr_build_ms = Some(started.elapsed().as_millis() as u64);
ccr_build_breakdown = build_breakdown;
let started = std::time::Instant::now();
write_ccr_file(path, &ccr).map_err(|e| e.to_string())?;
ccr_write_ms = Some(started.elapsed().as_millis() as u64);
eprintln!("wrote CCR: {}", path.display());
}
Ok(CcrTaskOutput {
ccr_build_ms,
ccr_build_breakdown,
ccr_write_ms,
})
}
pub(super) fn resolve_cir_export_tal_uris(args: &CliArgs) -> Result<Vec<String>, String> {
if !args.cir_tal_uris.is_empty() {
return Ok(args.cir_tal_uris.clone());
}
if !args.tal_urls.is_empty() {
return Ok(args.tal_urls.clone());
}
Err("CIR export requires TAL URI source(s)".to_string())
}
pub(super) fn effective_cir_tal_uris_for_discoveries(
args: &CliArgs,
shared: &PostValidationShared,
cir_tal_uris: Vec<String>,
) -> Result<Vec<String>, String> {
if shared.successful_tal_inputs.is_empty() {
return Ok(cir_tal_uris);
}
if cir_tal_uris.len() == shared.discoveries.len() {
return Ok(cir_tal_uris);
}
if cir_tal_uris.len() != args.tal_inputs.len() {
return Ok(cir_tal_uris);
}
let mut mapped = Vec::with_capacity(shared.successful_tal_inputs.len());
for successful in shared.successful_tal_inputs.iter() {
let input_index = args
.tal_inputs
.iter()
.position(|candidate| candidate == successful)
.ok_or_else(|| {
format!(
"successful TAL '{}' was not found in original TAL input list",
successful.tal_id
)
})?;
mapped.push(cir_tal_uris[input_index].clone());
}
Ok(mapped)
}
pub(super) fn build_repo_sync_stats(
publication_points: &[crate::audit::PublicationPointAudit],
) -> AuditRepoSyncStats {
let mut stats = AuditRepoSyncStats {
publication_points_total: publication_points.len() as u64,
..AuditRepoSyncStats::default()
};
for pp in publication_points {
let duration = pp.repo_sync_duration_ms.unwrap_or(0);
if let Some(phase) = pp.repo_sync_phase.as_ref() {
let entry = stats.by_phase.entry(phase.clone()).or_default();
entry.count += 1;
entry.duration_ms_total += duration;
}
let entry = stats
.by_terminal_state
.entry(pp.repo_terminal_state.clone())
.or_default();
entry.count += 1;
entry.duration_ms_total += duration;
}
stats
}

View File

@@ -0,0 +1,777 @@
// Top-level CLI execution pipeline.
pub fn run(argv: &[String]) -> Result<(), String> {
let mut args = parse_args(argv)?;
let mut policy = read_policy(args.policy_path.as_deref())?;
if let Some(strict_policy) = args.strict_policy {
policy.strict = strict_policy;
}
if let Some(resource_validation_mode) = args.resource_validation_mode {
policy.resource_validation_mode = resource_validation_mode;
}
if args.disable_rrdp {
policy.sync_preference = crate::policy::SyncPreference::RsyncOnly;
}
policy.ta_constraints = args.ta_constraints.clone();
for warning in policy.ta_constraints.configuration_warnings() {
eprintln!("warning: {warning}");
}
let validation_time = args
.validation_time
.unwrap_or_else(time::OffsetDateTime::now_utc);
let validation_time =
time::OffsetDateTime::from_unix_timestamp(validation_time.unix_timestamp())
.map_err(|error| format!("normalize validation time failed: {error}"))?;
let http_root_certificates_pem = args
.http_root_cert_paths
.iter()
.map(|path| {
std::fs::read(path)
.map_err(|e| format!("read HTTP root certificate failed: {}: {e}", path.display()))
})
.collect::<Result<Vec<_>, _>>()?;
let store = if args.raw_store_db.is_some() || args.repo_bytes_db.is_some() {
Arc::new(
RocksStore::open_with_external_stores(
&args.db_path,
args.raw_store_db.as_deref(),
args.repo_bytes_db.as_deref(),
)
.map_err(|e| e.to_string())?,
)
} else {
Arc::new(RocksStore::open(&args.db_path).map_err(|e| e.to_string())?)
};
let config = TreeRunConfig {
max_depth: Some(args.max_ca_depth),
max_instances: args.max_instances,
compact_audit: args.skip_report_build
&& args.report_json_path.is_none()
&& !args.cir_enabled,
persist_vcir: !args.skip_vcir_persist,
build_ccr_accumulator: args.ccr_out_path.is_some(),
enable_roa_validation_cache: args.enable_roa_validation_cache,
enable_child_certificate_validation_cache: args.enable_child_certificate_validation_cache,
publication_point_cache_observe_only: args.publication_point_cache_observe_only,
enable_publication_point_validation_cache: args.enable_publication_point_validation_cache,
enable_transport_request_prefetch: args.enable_transport_request_prefetch,
};
let replay_mode = args.payload_replay_archive.is_some();
let delta_replay_mode = args.payload_base_archive.is_some();
use time::format_description::well_known::Rfc3339;
let mut timing: Option<(std::path::PathBuf, TimingHandle)> = None;
if args.analyze {
let recorded_at_utc_rfc3339 = time::OffsetDateTime::now_utc()
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.map_err(|e| format!("format recorded_at_utc failed: {e}"))?;
let validation_time_utc_rfc3339 = validation_time
.to_offset(time::UtcOffset::UTC)
.format(&Rfc3339)
.map_err(|e| format!("format validation_time failed: {e}"))?;
let ts_compact = {
let fmt = time::format_description::parse("[year][month][day]T[hour][minute][second]Z")
.map_err(|e| format!("format description parse failed: {e}"))?;
time::OffsetDateTime::now_utc()
.format(&fmt)
.map_err(|e| format!("format timestamp failed: {e}"))?
};
let out_dir = args.analysis_out_path.clone().unwrap_or_else(|| {
std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("target")
.join("live")
.join("analyze")
.join(ts_compact)
});
std::fs::create_dir_all(&out_dir)
.map_err(|e| format!("create analyze out dir failed: {}: {e}", out_dir.display()))?;
let handle = TimingHandle::new(TimingMeta {
recorded_at_utc_rfc3339,
validation_time_utc_rfc3339,
tal_url: None,
db_path: None,
});
handle.set_meta(TimingMetaUpdate {
tal_url: args.tal_url.as_deref(),
db_path: Some(args.db_path.to_string_lossy().as_ref()),
});
timing = Some((out_dir, handle));
}
if args.profile_cpu && !args.analyze {
return Err("--profile-cpu requires --analyze".to_string());
}
#[cfg(not(feature = "profile"))]
if args.profile_cpu {
return Err("CPU profiling requires building with: --features profile".to_string());
}
#[cfg(feature = "profile")]
let mut profiler_guard: Option<pprof::ProfilerGuard<'static>> = if args.profile_cpu {
Some(
pprof::ProfilerGuard::new(100)
.map_err(|e| format!("pprof ProfilerGuard init failed: {e}"))?,
)
} else {
None
};
let total_started = std::time::Instant::now();
let mut memory_checkpoints: Vec<MemoryTelemetryCheckpoint> = Vec::new();
let mut malloc_trim_probes: Vec<MallocTrimProbe> = Vec::new();
let enable_memory_trim_probe = memory_trim_probe_enabled() || args.memory_trim_after_validation;
record_memory_checkpoint(
&mut memory_checkpoints,
"after_store_open",
&total_started,
store.as_ref(),
);
let validation_started = std::time::Instant::now();
let crypto_sig_cache =
if args.crypto_signature_cache_observe_only || args.enable_crypto_signature_cache {
let cache_file = crate::crypto_sig_cache::default_cache_file_path(&args.db_path);
let cache = Arc::new(crate::crypto_sig_cache::CryptoSigCache::load_or_rebuild(
cache_file,
args.enable_crypto_signature_cache,
));
crate::crypto_sig_cache::install_global(Arc::clone(&cache));
Some(cache)
} else {
None
};
let collect_current_repo_objects = false;
let out = if delta_replay_mode {
let tal_path = args
.tal_path
.as_ref()
.expect("validated by parse_args for delta replay mode");
let ta_path = args
.ta_path
.as_ref()
.expect("validated by parse_args for delta replay mode");
let base_archive = args
.payload_base_archive
.as_ref()
.expect("validated by parse_args for delta replay mode");
let base_locks = args
.payload_base_locks
.as_ref()
.expect("validated by parse_args for delta replay mode");
let base_validation_time = args.payload_base_validation_time.unwrap_or(validation_time);
let delta_archive = args
.payload_delta_archive
.as_ref()
.expect("validated by parse_args for delta replay mode");
let delta_locks = args
.payload_delta_locks
.as_ref()
.expect("validated by parse_args for delta replay mode");
let tal_bytes = std::fs::read(tal_path)
.map_err(|e| format!("read tal failed: {}: {e}", tal_path.display()))?;
let ta_der = std::fs::read(ta_path)
.map_err(|e| format!("read ta failed: {}: {e}", ta_path.display()))?;
if let Some((_, t)) = timing.as_ref() {
run_tree_from_tal_and_ta_der_payload_delta_replay_serial_audit_with_timing(
store.as_ref(),
&policy,
&tal_bytes,
&ta_der,
None,
base_archive,
base_locks,
delta_archive,
delta_locks,
base_validation_time,
validation_time,
&config,
t,
)
.map_err(|e| e.to_string())?
} else {
run_tree_from_tal_and_ta_der_payload_delta_replay_serial_audit(
store.as_ref(),
&policy,
&tal_bytes,
&ta_der,
None,
base_archive,
base_locks,
delta_archive,
delta_locks,
base_validation_time,
validation_time,
&config,
)
.map_err(|e| e.to_string())?
}
} else if replay_mode {
let tal_path = args
.tal_path
.as_ref()
.expect("validated by parse_args for replay mode");
let ta_path = args
.ta_path
.as_ref()
.expect("validated by parse_args for replay mode");
let archive_root = args
.payload_replay_archive
.as_ref()
.expect("validated by parse_args for replay mode");
let locks_path = args
.payload_replay_locks
.as_ref()
.expect("validated by parse_args for replay mode");
let tal_bytes = std::fs::read(tal_path)
.map_err(|e| format!("read tal failed: {}: {e}", tal_path.display()))?;
let ta_der = std::fs::read(ta_path)
.map_err(|e| format!("read ta failed: {}: {e}", ta_path.display()))?;
if let Some((_, t)) = timing.as_ref() {
run_tree_from_tal_and_ta_der_payload_replay_serial_audit_with_timing(
store.as_ref(),
&policy,
&tal_bytes,
&ta_der,
None,
archive_root,
locks_path,
validation_time,
&config,
t,
)
.map_err(|e| e.to_string())?
} else {
run_tree_from_tal_and_ta_der_payload_replay_serial_audit(
store.as_ref(),
&policy,
&tal_bytes,
&ta_der,
None,
archive_root,
locks_path,
validation_time,
&config,
)
.map_err(|e| e.to_string())?
}
} else if let Some(dir) = args.rsync_local_dir.as_ref() {
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: std::time::Duration::from_secs(args.http_timeout_secs.max(1)),
extra_root_certificates_pem: http_root_certificates_pem.clone(),
..HttpFetcherConfig::default()
})
.map_err(|e| e.to_string())?;
let rsync = LocalDirRsyncFetcher::new(dir);
run_online_validation_with_fetchers(
Arc::clone(&store),
&policy,
&args,
&http,
&rsync,
validation_time,
&config,
collect_current_repo_objects,
timing.as_ref().map(|(_, t)| t),
)?
} else {
let http = BlockingHttpFetcher::new(HttpFetcherConfig {
timeout: std::time::Duration::from_secs(args.http_timeout_secs.max(1)),
extra_root_certificates_pem: http_root_certificates_pem.clone(),
..HttpFetcherConfig::default()
})
.map_err(|e| e.to_string())?;
let rsync = SystemRsyncFetcher::new(SystemRsyncConfig {
rsync_bin: args
.rsync_command
.clone()
.unwrap_or_else(|| PathBuf::from("rsync")),
timeout: std::time::Duration::from_secs(args.rsync_timeout_secs.max(1)),
mirror_root: args.rsync_mirror_root.clone(),
scope_policy: args.rsync_scope_policy,
..SystemRsyncConfig::default()
});
run_online_validation_with_fetchers(
Arc::clone(&store),
&policy,
&args,
&http,
&rsync,
validation_time,
&config,
collect_current_repo_objects,
timing.as_ref().map(|(_, t)| t),
)?
};
let validation_ms = validation_started.elapsed().as_millis() as u64;
let mut shared = PostValidationShared::from_run_output(out);
let vcir_storage_summary_enabled = vcir_storage_summary_enabled();
let vcir_storage_summary_started = std::time::Instant::now();
let vcir_storage = if config.persist_vcir && vcir_storage_summary_enabled {
Some(
store
.summarize_vcir_storage()
.map_err(|e| format!("summarize VCIR storage failed: {e}"))?,
)
} else {
None
};
let vcir_storage_summary_ms = (config.persist_vcir && vcir_storage_summary_enabled)
.then(|| vcir_storage_summary_started.elapsed().as_millis() as u64);
record_memory_checkpoint(
&mut memory_checkpoints,
"after_validation",
&total_started,
store.as_ref(),
);
if enable_memory_trim_probe {
malloc_trim_probes.push(crate::memory_telemetry::malloc_trim_probe());
record_memory_checkpoint(
&mut memory_checkpoints,
"after_validation_malloc_trim",
&total_started,
store.as_ref(),
);
}
if let Some((_out_dir, t)) = timing.as_ref() {
t.record_count("instances_processed", shared.instances_processed as u64);
t.record_count("instances_failed", shared.instances_failed as u64);
}
let publication_points = shared.publication_points.len();
let publication_point_repo_sync_ms_total: u64 = shared
.publication_points
.iter()
.map(|pp| pp.repo_sync_duration_ms.unwrap_or(0))
.sum();
let download_event_count = shared.download_stats.events_total;
let rrdp_download_ms_total: u64 = ["rrdp_notification", "rrdp_snapshot", "rrdp_delta"]
.iter()
.map(|key| {
shared
.download_stats
.by_kind
.get(*key)
.map(|item| item.duration_ms_total)
.unwrap_or(0)
})
.sum();
let rsync_download_ms_total = shared
.download_stats
.by_kind
.get("rsync")
.map(|item| item.duration_ms_total)
.unwrap_or(0);
let repo_sync_ms_total = rrdp_download_ms_total + rsync_download_ms_total;
let download_bytes_total: u64 = shared
.download_stats
.by_kind
.values()
.map(|item| item.bytes_total.unwrap_or(0))
.sum();
#[cfg(feature = "profile")]
let profiler_report = if let Some(guard) = profiler_guard.take() {
Some(
guard
.report()
.build()
.map_err(|e| format!("pprof report build failed: {e}"))?,
)
} else {
None
};
let report_json_format = if args.report_json_compact {
ReportJsonFormat::Compact
} else {
ReportJsonFormat::Pretty
};
let ccr_produced_at = time::OffsetDateTime::now_utc();
let compare_view_trust_anchor = args
.compare_view_trust_anchor
.as_deref()
.unwrap_or("unknown");
let cir_tal_uris = if args.cir_enabled {
Some(effective_cir_tal_uris_for_discoveries(
&args,
&shared,
resolve_cir_export_tal_uris(&args)?,
)?)
} else {
None
};
let cir_out_path = if args.cir_enabled {
Some(
args.cir_out_path
.as_deref()
.expect("validated by parse_args for cir"),
)
} else {
None
};
// Take the CIR input snapshot before the output stage so the CIR export can
// run inside the same scoped-thread group as report/ccr/compare_view; no
// other output task reads `shared.cir_input`.
let cir_input_owned = args
.cir_enabled
.then(|| std::mem::take(&mut shared.cir_input));
let (report_result, ccr_result, compare_view_result, cir_result) =
std::thread::scope(|scope| {
// Reborrow `shared` as a plain reference so the scoped output tasks
// (incl. the `move` CIR task) capture the reference instead of
// moving fields out of the owned value.
let shared = &shared;
let report_handle = if args.skip_report_build {
None
} else {
Some(scope.spawn(|| {
run_report_task(
&policy,
validation_time,
shared,
args.report_json_path.as_deref(),
report_json_format,
)
}))
};
let ccr_handle = scope.spawn(|| {
run_ccr_task(
store.as_ref(),
shared,
args.ccr_out_path.as_deref(),
ccr_produced_at,
)
});
let compare_view_handle = scope.spawn(|| {
run_compare_view_task(
shared,
args.vrps_csv_out_path.as_deref(),
args.vaps_csv_out_path.as_deref(),
compare_view_trust_anchor,
)
});
let cir_handle = match (cir_tal_uris.as_ref(), cir_out_path, cir_input_owned) {
(Some(cir_tal_uris), Some(cir_out_path), Some(cir_input)) => {
Some(scope.spawn(move || {
if cir_tal_uris.len() != shared.discoveries.len() {
return Err(format!(
"CIR export TAL URI count ({}) does not match discovery count ({})",
cir_tal_uris.len(),
shared.discoveries.len()
));
}
let tal_bindings = shared
.discoveries
.iter()
.zip(cir_tal_uris.iter())
.map(|(discovery, tal_uri)| CirTrustAnchorBinding {
trust_anchor: &discovery.trust_anchor,
tal_uri: tal_uri.as_str(),
})
.collect::<Vec<_>>();
export_cir_from_input_snapshot_multi(
&tal_bindings,
validation_time,
cir_input,
cir_out_path,
)
.map_err(|e| e.to_string())
}))
}
_ => None,
};
let report_result = match report_handle {
Some(handle) => handle
.join()
.map_err(|_| "report task panicked".to_string())
.and_then(|result| result),
None => Ok(ReportTaskOutput::skipped()),
};
let ccr_result = ccr_handle
.join()
.map_err(|_| "ccr task panicked".to_string())
.and_then(|result| result);
let compare_view_result = compare_view_handle
.join()
.map_err(|_| "compare view task panicked".to_string())
.and_then(|result| result);
let cir_result = cir_handle.map(|handle| {
handle
.join()
.map_err(|_| "cir task panicked".to_string())
.and_then(|result| result)
});
(report_result, ccr_result, compare_view_result, cir_result)
});
let report_output = report_result?;
let ccr_output = ccr_result?;
let compare_view_output = compare_view_result?;
let cir_summary = match cir_result {
Some(result) => Some(result?),
None => None,
};
record_memory_checkpoint(
&mut memory_checkpoints,
"after_report_and_ccr",
&total_started,
store.as_ref(),
);
if enable_memory_trim_probe {
malloc_trim_probes.push(crate::memory_telemetry::malloc_trim_probe());
record_memory_checkpoint(
&mut memory_checkpoints,
"after_report_and_ccr_malloc_trim",
&total_started,
store.as_ref(),
);
}
let report_build_ms = report_output.report_build_ms;
let report_write_ms = report_output.report_write_ms;
let ccr_build_ms = ccr_output.ccr_build_ms;
let ccr_build_breakdown = ccr_output.ccr_build_breakdown;
let ccr_write_ms = ccr_output.ccr_write_ms;
let compare_view_build_ms = compare_view_output.build_ms;
let compare_view_write_ms = compare_view_output.write_ms;
record_memory_checkpoint(
&mut memory_checkpoints,
"after_compare_view",
&total_started,
store.as_ref(),
);
let mut cir_build_cir_ms = None;
let mut cir_write_cir_ms = None;
let mut cir_total_ms = None;
if let Some(summary) = cir_summary {
cir_build_cir_ms = Some(summary.timing.build_cir_ms);
cir_write_cir_ms = Some(summary.timing.write_cir_ms);
cir_total_ms = Some(summary.timing.total_ms);
eprintln!(
"wrote CIR: {} (objects={}, trust_anchors={}, build_cir_ms={}, write_cir_ms={}, total_ms={})",
cir_out_path
.expect("cir path present when cir enabled")
.display(),
summary.object_count,
summary.trust_anchor_count,
summary.timing.build_cir_ms,
summary.timing.write_cir_ms,
summary.timing.total_ms
);
record_memory_checkpoint(
&mut memory_checkpoints,
"after_cir",
&total_started,
store.as_ref(),
);
}
record_memory_checkpoint(
&mut memory_checkpoints,
"before_stage_timing",
&total_started,
store.as_ref(),
);
let publication_point_cache_index_refresh = if args.enable_publication_point_validation_cache
|| args.publication_point_cache_observe_only
{
match store.refresh_publication_point_cache_mmap_index() {
Ok(stats) => stats,
Err(e) => {
crate::progress_log::emit(
"publication_point_cache_mmap_index_refresh",
serde_json::json!({
"state": "failed",
"error": e.to_string(),
}),
);
None
}
}
} else {
None
};
let publication_point_cache_index_load = store.publication_point_cache_mmap_index_load_stats();
let crypto_signature_cache_observe = crypto_sig_cache.as_ref().map(|cache| {
if let Err(e) = cache.persist() {
crate::progress_log::emit(
"crypto_signature_cache_persist",
serde_json::json!({
"state": "failed",
"error": e,
}),
);
}
crate::crypto_sig_cache::clear_global();
cache.summary()
});
if let (Some((_, t)), Some(summary)) =
(timing.as_ref(), crypto_signature_cache_observe.as_ref())
{
let (mut calls, mut would_hit, mut new_keys, mut executed, mut skipped) =
(0u64, 0u64, 0u64, 0u64, 0u64);
for stats in summary.per_point.values() {
calls += stats.calls;
would_hit += stats.would_hit;
new_keys += stats.new_keys;
executed += stats.verify_executed;
skipped += stats.verify_skipped;
}
t.record_count("crypto_signature_cache_observe_calls", calls);
t.record_count("crypto_signature_cache_observe_would_hit", would_hit);
t.record_count("crypto_signature_cache_observe_new_keys", new_keys);
t.record_count("crypto_signature_cache_verify_executed", executed);
t.record_count("crypto_signature_cache_verify_skipped", skipped);
}
let timing_report_snapshot = timing
.as_ref()
.map(|(_, handle)| handle.report_snapshot(50));
let stage_timing = RunStageTiming {
validation_ms,
enable_roa_validation_cache: args.enable_roa_validation_cache,
enable_child_certificate_validation_cache: args.enable_child_certificate_validation_cache,
publication_point_cache_observe_only: args.publication_point_cache_observe_only,
enable_publication_point_validation_cache: args.enable_publication_point_validation_cache,
crypto_signature_cache_observe,
enable_transport_request_prefetch: args.enable_transport_request_prefetch,
report_build_ms,
report_write_ms,
ccr_build_ms,
ccr_build_breakdown,
ccr_write_ms,
compare_view_build_ms,
compare_view_write_ms,
cir_build_cir_ms,
cir_write_cir_ms,
cir_total_ms,
total_ms: total_started.elapsed().as_millis() as u64,
publication_points,
repo_sync_ms_total,
publication_point_repo_sync_ms_total,
download_event_count,
rrdp_download_ms_total,
rsync_download_ms_total,
download_bytes_total,
roa_validation_cache: shared.roa_cache_stats.clone(),
analysis_counts: timing
.as_ref()
.map(|(_, handle)| handle.counts_snapshot())
.unwrap_or_default(),
analysis_phases: timing_report_snapshot
.as_ref()
.map(|report| report.phases.clone())
.unwrap_or_default(),
analysis_top_publication_points: timing_report_snapshot
.as_ref()
.map(|report| report.top_publication_points.clone())
.unwrap_or_default(),
analysis_top_publication_point_steps: timing_report_snapshot
.as_ref()
.map(|report| report.top_publication_point_steps.clone())
.unwrap_or_default(),
analysis_top_publication_point_cache_steps: timing_report_snapshot
.as_ref()
.map(|report| {
report
.top_publication_point_steps
.iter()
.filter(|entry| entry.key.contains("::publication_point_cache_"))
.cloned()
.collect()
})
.unwrap_or_default(),
vcir_storage_summary_ms,
vcir_storage,
publication_point_cache_index_load,
publication_point_cache_index_refresh,
memory_telemetry: Some(MemoryTelemetrySummary {
checkpoints: memory_checkpoints,
object_graph: Some(estimate_shared_object_graph(&shared)),
malloc_trim_probes,
}),
};
let stage_timing_anchor_path = args
.report_json_path
.as_deref()
.or(args.ccr_out_path.as_deref())
.or(args.vrps_csv_out_path.as_deref());
write_stage_timing(stage_timing_anchor_path, &stage_timing)?;
// Finalize the normal-run contract only after all validation outputs have
// been written successfully. This prevents incomplete runs from leaving a
// contract that looks usable to downstream replay tooling.
if let Some(path) = args.validation_contract_out_path.as_deref() {
let mut contract = crate::contract::ValidationContract::for_current_binary(
validation_time,
policy.clone(),
args.max_ca_depth,
args.max_instances,
crate::contract::ValidationCacheContract {
publication_point: args.enable_publication_point_validation_cache,
roa: args.enable_roa_validation_cache,
child_certificate: args.enable_child_certificate_validation_cache,
transport_prefetch: args.enable_transport_request_prefetch,
crypto_signature: args.enable_crypto_signature_cache,
},
args.rsync_scope_policy,
)?;
if !policy.ta_constraints.is_empty() {
contract.ta_constraints_fingerprint =
Some(policy.ta_constraints.fingerprint_sha256_hex());
}
crate::contract::write_validation_contract(path, &contract)?;
}
if let Some((out_dir, t)) = timing.as_ref() {
t.record_count("vrps", shared.vrps.len() as u64);
t.record_count("aspas", shared.aspas.len() as u64);
t.record_count(
"audit_publication_points",
shared.publication_points.len() as u64,
);
let timing_json_path = out_dir.join("timing.json");
t.write_json(&timing_json_path, 20)?;
eprintln!("analysis: wrote {}", timing_json_path.display());
}
#[cfg(feature = "profile")]
if let (Some((out_dir, _)), Some(report)) = (timing.as_ref(), profiler_report) {
let svg_path = out_dir.join("flamegraph.svg");
let svg_file = std::fs::File::create(&svg_path)
.map_err(|e| format!("create flamegraph failed: {}: {e}", svg_path.display()))?;
report
.flamegraph(svg_file)
.map_err(|e| format!("write flamegraph failed: {e}"))?;
eprintln!("analysis: wrote {}", svg_path.display());
let pb_path = out_dir.join("pprof.pb.gz");
let pprof_profile = report
.pprof()
.map_err(|e| format!("pprof export failed: {e}"))?;
use pprof::protos::Message;
let mut body = Vec::with_capacity(pprof_profile.encoded_len());
pprof_profile
.encode(&mut body)
.map_err(|e| format!("pprof encode failed: {e}"))?;
let gz = flate2::write::GzEncoder::new(
std::fs::File::create(&pb_path)
.map_err(|e| format!("create pprof.pb.gz failed: {}: {e}", pb_path.display()))?,
flate2::Compression::default(),
);
let mut gz = gz;
use std::io::Write;
gz.write_all(&body)
.map_err(|e| format!("write pprof.pb.gz failed: {e}"))?;
gz.finish()
.map_err(|e| format!("finish pprof.pb.gz failed: {e}"))?;
eprintln!("analysis: wrote {}", pb_path.display());
}
print_summary_from_shared(validation_time, &shared);
Ok(())
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,977 @@
// CLI test group: parse core.
use super::*;
use crate::data_model::oid::OID_AD_SIGNED_OBJECT;
use crate::data_model::rc::AccessDescription;
use crate::memory_telemetry::{
MemoryTelemetryCheckpoint, MemoryTelemetrySummary, ProcessMemorySnapshot,
};
use crate::policy::ResourceValidationMode;
use crate::storage::{
RocksDbMemorySnapshot, RocksDbMemoryTotals, VcirCcrProjectionSizeBreakdown,
VcirChildResourceSizeBreakdown, VcirCoreFieldSizeBreakdown, VcirFieldSizeBreakdown,
VcirStorageEntrySummary, VcirStorageSummary,
};
#[test]
fn parse_help_returns_usage() {
let argv = vec!["rpki".to_string(), "--help".to_string()];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("Usage:"), "{err}");
assert!(err.contains("--db"), "{err}");
assert!(err.contains("--rsync-mirror-root"), "{err}");
assert!(err.contains("--rsync-scope"), "{err}");
assert!(err.contains("--parallel-phase2-object-workers"), "{err}");
assert!(err.contains("--memory-trim-after-validation"), "{err}");
assert!(err.contains("--enable-roa-validation-cache"), "{err}");
assert!(err.contains("--resource-validation-mode"), "{err}");
assert!(err.contains("--ta-constraints"), "{err}");
assert!(err.contains("--max-ca-depth"), "{err}");
assert!(err.contains("default: 32"), "{err}");
assert!(
err.contains("--publication-point-cache-observe-only"),
"{err}"
);
assert!(
err.contains("--enable-publication-point-validation-cache"),
"{err}"
);
assert!(!err.contains("--parallel-phase1"), "{err}");
assert!(!err.contains("--parallel-phase2 "), "{err}");
}
#[test]
fn parse_accepts_explicit_ta_constraints_for_known_tal() {
let dir = tempfile::tempdir().expect("tmpdir");
let constraints_path = dir.path().join("example.constraints");
std::fs::write(&constraints_path, "allow 192.0.2.0/24\n").expect("write constraints");
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"example.tal".to_string(),
"--ta-path".to_string(),
"example.cer".to_string(),
"--ta-constraints".to_string(),
format!("example={}", constraints_path.display()),
];
let args = parse_args(&argv).expect("parse args");
assert!(!args.ta_constraints.is_empty());
}
#[test]
fn parse_rejects_unknown_argument() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--nope".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("unknown argument"), "{err}");
}
#[test]
fn parse_accepts_normal_validation_contract_output() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--validation-contract-out".to_string(),
"out/validation-contract.json".to_string(),
];
let args = parse_args(&argv).expect("parse normal contract output");
assert_eq!(
args.validation_contract_out_path,
Some(PathBuf::from("out/validation-contract.json"))
);
}
#[test]
fn parse_rejects_both_tal_url_and_tal_path() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(
err.contains("one-or-more --tal-url or one-or-more --tal-path/--ta-path pairs"),
"{err}"
);
}
#[test]
fn parse_rejects_invalid_max_depth() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--max-depth".to_string(),
"nope".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("invalid --max-depth"), "{err}");
}
#[test]
fn parse_rejects_invalid_max_ca_depth() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--max-ca-depth".to_string(),
"nope".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("invalid --max-ca-depth"), "{err}");
}
#[test]
fn parse_rejects_both_ca_depth_options() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--max-ca-depth".to_string(),
"32".to_string(),
"--max-depth".to_string(),
"12".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("cannot be combined"), "{err}");
}
#[test]
fn parse_defaults_max_ca_depth_to_32() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.max_ca_depth, 32);
}
#[test]
fn parse_accepts_ccr_out_path() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
"--ccr-out".to_string(),
"out/example.ccr".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.ccr_out_path.as_deref(),
Some(std::path::Path::new("out/example.ccr"))
);
}
#[test]
fn parse_accepts_memory_trim_after_validation() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--memory-trim-after-validation".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.memory_trim_after_validation);
}
#[test]
fn parse_disables_memory_trim_after_validation_by_default() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(!args.memory_trim_after_validation);
}
#[test]
fn parse_accepts_enable_roa_validation_cache() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--enable-roa-validation-cache".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.enable_roa_validation_cache);
}
#[test]
fn parse_accepts_enable_child_certificate_validation_cache() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--enable-child-certificate-validation-cache".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.enable_child_certificate_validation_cache);
}
#[test]
fn parse_accepts_publication_point_cache_flags() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--publication-point-cache-observe-only".to_string(),
"--enable-publication-point-validation-cache".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.publication_point_cache_observe_only);
assert!(args.enable_publication_point_validation_cache);
}
#[test]
fn parse_disables_crypto_signature_cache_observe_only_by_default() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(!args.crypto_signature_cache_observe_only);
assert!(!args.enable_crypto_signature_cache);
}
#[test]
fn parse_accepts_crypto_signature_cache_observe_only() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--crypto-signature-cache-observe-only".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.crypto_signature_cache_observe_only);
assert!(!args.enable_crypto_signature_cache);
}
#[test]
fn parse_accepts_enable_crypto_signature_cache() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--enable-crypto-signature-cache".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.enable_crypto_signature_cache);
assert!(!args.crypto_signature_cache_observe_only);
}
#[test]
fn parse_accepts_enable_transport_request_prefetch() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--enable-transport-request-prefetch".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.enable_transport_request_prefetch);
}
#[test]
fn parse_disables_roa_validation_cache_by_default() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(!args.enable_roa_validation_cache);
assert!(!args.enable_child_certificate_validation_cache);
assert!(!args.publication_point_cache_observe_only);
assert!(!args.enable_publication_point_validation_cache);
assert!(!args.enable_transport_request_prefetch);
}
#[test]
fn parse_accepts_analysis_out_and_implies_analyze() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--analysis-out".to_string(),
"run/analyze".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.analyze);
assert_eq!(
args.analysis_out_path.as_deref(),
Some(std::path::Path::new("run/analyze"))
);
}
#[test]
fn parse_accepts_report_json_compact_when_report_json_is_set() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--report-json".to_string(),
"out/report.json".to_string(),
"--report-json-compact".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.report_json_path.as_deref(),
Some(std::path::Path::new("out/report.json"))
);
assert!(args.report_json_compact);
}
#[test]
fn parse_accepts_rsync_scope_policy() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--rsync-scope".to_string(),
"module-root".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(args.rsync_scope_policy, RsyncScopePolicy::ModuleRoot);
}
#[test]
fn parse_accepts_host_rsync_scope_policy() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--rsync-scope".to_string(),
"host".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(args.rsync_scope_policy, RsyncScopePolicy::Host);
}
#[test]
fn parse_accepts_repeatable_http_root_cert_paths() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--http-root-cert".to_string(),
"local-ca-1.pem".to_string(),
"--http-root-cert".to_string(),
"local-ca-2.pem".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.http_root_cert_paths,
vec![
PathBuf::from("local-ca-1.pem"),
PathBuf::from("local-ca-2.pem")
]
);
}
#[test]
fn parse_rejects_invalid_rsync_scope_policy() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--rsync-scope".to_string(),
"wide".to_string(),
];
let err = parse_args(&argv).expect_err("invalid rsync scope should fail");
assert!(err.contains("invalid --rsync-scope"), "{err}");
}
#[test]
fn parse_accepts_strict_policy_list() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--strict".to_string(),
"name,cms-der".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.strict_policy,
Some(StrictPolicy {
name: true,
cms_der: true,
signed_attrs: false,
})
);
}
#[test]
fn parse_accepts_strict_without_value_as_all() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--strict".to_string(),
"--report-json".to_string(),
"out/report.json".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(args.strict_policy, Some(StrictPolicy::all()));
}
#[test]
fn parse_accepts_resource_validation_mode() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--resource-validation-mode".to_string(),
"rfc6487".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.resource_validation_mode,
Some(ResourceValidationMode::Rfc6487)
);
}
#[test]
fn parse_rejects_unknown_resource_validation_mode() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--resource-validation-mode".to_string(),
"bogus".to_string(),
];
let err = parse_args(&argv).expect_err("unknown mode should fail");
assert!(err.contains("unknown resource validation mode"), "{err}");
}
#[test]
fn parse_rejects_unknown_strict_policy() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--strict=unknown".to_string(),
];
let err = parse_args(&argv).expect_err("unknown strict policy should fail");
assert!(err.contains("unknown strict policy"), "{err}");
}
#[test]
fn effective_cir_tal_uris_filters_skipped_multi_tal_inputs() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"afrinic.tal".to_string(),
"--ta-path".to_string(),
"afrinic.cer".to_string(),
"--tal-path".to_string(),
"apnic.tal".to_string(),
"--ta-path".to_string(),
"apnic.cer".to_string(),
"--tal-path".to_string(),
"arin.tal".to_string(),
"--ta-path".to_string(),
"arin.cer".to_string(),
"--cir-enable".to_string(),
"--cir-out".to_string(),
"out.cir".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/afrinic.cer".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/apnic.cer".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/arin.cer".to_string(),
];
let args = parse_args(&argv).expect("parse args");
let mut shared = synthetic_post_validation_shared();
shared.discoveries = vec![shared.discovery.clone(), shared.discovery.clone()].into();
shared.successful_tal_inputs =
vec![args.tal_inputs[0].clone(), args.tal_inputs[2].clone()].into();
let effective = effective_cir_tal_uris_for_discoveries(
&args,
&shared,
resolve_cir_export_tal_uris(&args).expect("resolve cir tal uris"),
)
.expect("map effective cir tal uris");
assert_eq!(
effective,
vec![
"https://example.test/afrinic.cer".to_string(),
"https://example.test/arin.cer".to_string(),
]
);
}
#[test]
fn parse_rejects_report_json_compact_without_report_json() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--report-json-compact".to_string(),
];
let err = parse_args(&argv).expect_err("compact flag without report path should fail");
assert!(
err.contains("--report-json-compact requires --report-json"),
"{err}"
);
}
#[test]
fn parse_accepts_skip_report_build_without_report_json() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--ccr-out".to_string(),
"out/result.ccr".to_string(),
"--skip-report-build".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.skip_report_build);
assert_eq!(
args.ccr_out_path.as_deref(),
Some(std::path::Path::new("out/result.ccr"))
);
}
#[test]
fn parse_accepts_skip_vcir_persist() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--skip-vcir-persist".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.skip_vcir_persist);
}
#[test]
fn parse_rejects_skip_report_build_with_report_json() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--report-json".to_string(),
"out/report.json".to_string(),
"--skip-report-build".to_string(),
];
let err = parse_args(&argv).expect_err("skip report build with report path should fail");
assert!(
err.contains("--skip-report-build cannot be combined with --report-json"),
"{err}"
);
}
#[test]
fn parse_accepts_direct_compare_view_csv_outputs() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--vrps-csv-out".to_string(),
"out/vrps.csv".to_string(),
"--vaps-csv-out".to_string(),
"out/vaps.csv".to_string(),
"--compare-view-trust-anchor".to_string(),
"unknown".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.vrps_csv_out_path.as_deref(),
Some(std::path::Path::new("out/vrps.csv"))
);
assert_eq!(
args.vaps_csv_out_path.as_deref(),
Some(std::path::Path::new("out/vaps.csv"))
);
assert_eq!(args.compare_view_trust_anchor.as_deref(), Some("unknown"));
}
#[test]
fn parse_rejects_partial_direct_compare_view_csv_outputs() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--vrps-csv-out".to_string(),
"out/vrps.csv".to_string(),
];
let err = parse_args(&argv).expect_err("partial direct compare view output should fail");
assert!(
err.contains("--vrps-csv-out and --vaps-csv-out must be provided together"),
"{err}"
);
}
#[test]
fn parse_accepts_external_raw_store_db() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--raw-store-db".to_string(),
"raw-store.db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.raw_store_db.as_deref(),
Some(std::path::Path::new("raw-store.db"))
);
}
#[test]
fn parse_accepts_external_repo_bytes_db() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--repo-bytes-db".to_string(),
"repo-bytes.db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.repo_bytes_db.as_deref(),
Some(std::path::Path::new("repo-bytes.db"))
);
}
#[test]
fn parse_accepts_cir_enable_with_raw_store_backend() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--raw-store-db".to_string(),
"raw-store.db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
"--cir-enable".to_string(),
"--cir-out".to_string(),
"out/example.cir".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/root.tal".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.cir_enabled);
assert_eq!(
args.raw_store_db.as_deref(),
Some(std::path::Path::new("raw-store.db"))
);
assert_eq!(args.cir_static_root, None);
}
#[test]
fn parse_accepts_cir_enable_with_required_paths_and_tal_override() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
"--cir-enable".to_string(),
"--cir-out".to_string(),
"out/example.cir".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/root.tal".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert!(args.cir_enabled);
assert_eq!(
args.cir_out_path.as_deref(),
Some(std::path::Path::new("out/example.cir"))
);
assert_eq!(
args.cir_tal_uri.as_deref(),
Some("https://example.test/root.tal")
);
assert_eq!(
args.cir_tal_uris,
vec!["https://example.test/root.tal".to_string()]
);
}
#[test]
fn parse_rejects_deprecated_cir_static_root() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--cir-enable".to_string(),
"--cir-out".to_string(),
"out/example.cir".to_string(),
"--cir-static-root".to_string(),
"out/static".to_string(),
];
let err = parse_args(&argv).expect_err("cir-static-root should be rejected");
assert!(err.contains("no longer supported"), "{err}");
}
#[test]
fn parse_accepts_default_parallel_config_and_phase2_overrides() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-object-workers".to_string(),
"3".to_string(),
"--parallel-phase2-worker-queue-capacity".to_string(),
"17".to_string(),
"--parallel-phase2-ready-batch-size".to_string(),
"31".to_string(),
"--parallel-phase2-ready-batch-wall-time-budget-ms".to_string(),
"43".to_string(),
"--parallel-phase2-result-drain-batch-size".to_string(),
"37".to_string(),
"--parallel-phase2-finalize-batch-size".to_string(),
"41".to_string(),
"--parallel-phase2-finalize-batch-wall-time-budget-ms".to_string(),
"47".to_string(),
"--parallel-phase2-finalize-queue-capacity".to_string(),
"8192".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(args.parallel_phase2_config.object_workers, 3);
assert_eq!(args.parallel_phase2_config.worker_queue_capacity, 17);
assert_eq!(args.parallel_phase2_config.ready_batch_size, 31);
assert_eq!(
args.parallel_phase2_config.ready_batch_wall_time_budget_ms,
43
);
assert_eq!(
args.parallel_phase2_config.object_result_drain_batch_size,
37
);
assert_eq!(
args.parallel_phase2_config
.publication_point_finalize_batch_size,
41
);
assert_eq!(
args.parallel_phase2_config
.publication_point_finalize_wall_time_budget_ms,
47
);
assert_eq!(
args.parallel_phase2_config
.publication_point_finalize_queue_capacity,
8192
);
assert_eq!(args.parallel_phase1_config, ParallelPhase1Config::default());
}
#[test]
fn parse_rejects_zero_phase2_ready_batch_size() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-ready-batch-size".to_string(),
"0".to_string(),
];
let err = parse_args(&argv).expect_err("zero ready batch must fail");
assert!(err.contains("--parallel-phase2-ready-batch-size"), "{err}");
}
#[test]
fn parse_rejects_zero_phase2_result_drain_batch_size() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-result-drain-batch-size".to_string(),
"0".to_string(),
];
let err = parse_args(&argv).expect_err("zero result drain batch must fail");
assert!(
err.contains("--parallel-phase2-result-drain-batch-size"),
"{err}"
);
}
#[test]
fn parse_rejects_zero_phase2_ready_batch_wall_time_budget_ms() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-ready-batch-wall-time-budget-ms".to_string(),
"0".to_string(),
];
let err = parse_args(&argv).expect_err("zero ready time budget must fail");
assert!(
err.contains("--parallel-phase2-ready-batch-wall-time-budget-ms"),
"{err}"
);
}
#[test]
fn parse_rejects_zero_phase2_finalize_batch_size() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-finalize-batch-size".to_string(),
"0".to_string(),
];
let err = parse_args(&argv).expect_err("zero finalize batch must fail");
assert!(
err.contains("--parallel-phase2-finalize-batch-size"),
"{err}"
);
}
#[test]
fn parse_rejects_zero_phase2_finalize_batch_wall_time_budget_ms() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-finalize-batch-wall-time-budget-ms".to_string(),
"0".to_string(),
];
let err = parse_args(&argv).expect_err("zero finalize time budget must fail");
assert!(
err.contains("--parallel-phase2-finalize-batch-wall-time-budget-ms"),
"{err}"
);
}

View File

@@ -0,0 +1,683 @@
// CLI test group: parse options.
#[test]
fn parse_rejects_zero_phase2_finalize_queue_capacity() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2-finalize-queue-capacity".to_string(),
"0".to_string(),
];
let err = parse_args(&argv).expect_err("zero finalize queue capacity must fail");
assert!(
err.contains("--parallel-phase2-finalize-queue-capacity"),
"{err}"
);
}
#[test]
fn parse_rejects_removed_parallel_enable_flags() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase1".to_string(),
];
let err = parse_args(&argv).expect_err("removed phase flag should fail");
assert!(err.contains("unknown argument: --parallel-phase1"), "{err}");
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--parallel-phase2".to_string(),
];
let err = parse_args(&argv).expect_err("removed phase flag should fail");
assert!(err.contains("unknown argument: --parallel-phase2"), "{err}");
}
#[test]
fn parse_accepts_multi_tal_cir_overrides_in_file_mode() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"apnic.tal".to_string(),
"--ta-path".to_string(),
"apnic.cer".to_string(),
"--tal-path".to_string(),
"arin.tal".to_string(),
"--ta-path".to_string(),
"arin.cer".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
"--cir-enable".to_string(),
"--cir-out".to_string(),
"out/example.cir".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/apnic.tal".to_string(),
"--cir-tal-uri".to_string(),
"https://example.test/arin.tal".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(
args.cir_tal_uris,
vec![
"https://example.test/apnic.tal".to_string(),
"https://example.test/arin.tal".to_string()
]
);
}
#[test]
fn parse_rejects_incomplete_or_invalid_cir_flags() {
let argv_missing = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--cir-enable".to_string(),
];
let err = parse_args(&argv_missing).unwrap_err();
assert!(err.contains("--cir-enable requires --cir-out"), "{err}");
let argv_needs_enable = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/root.tal".to_string(),
"--cir-out".to_string(),
"out/example.cir".to_string(),
];
let err = parse_args(&argv_needs_enable).unwrap_err();
assert!(err.contains("require --cir-enable"), "{err}");
let argv_offline_missing_uri = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"x.tal".to_string(),
"--ta-path".to_string(),
"x.cer".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
"--cir-enable".to_string(),
"--cir-out".to_string(),
"out/example.cir".to_string(),
];
let err = parse_args(&argv_offline_missing_uri).unwrap_err();
assert!(err.contains("requires --cir-tal-uri"), "{err}");
}
#[test]
fn parse_rejects_invalid_validation_time() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--validation-time".to_string(),
"not-a-time".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("invalid --validation-time"), "{err}");
}
#[test]
fn parse_rejects_invalid_max_instances() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--max-instances".to_string(),
"nope".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("invalid --max-instances"), "{err}");
}
#[test]
fn parse_rejects_missing_value_for_db() {
let argv = vec!["rpki".to_string(), "--db".to_string()];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("--db requires a value"), "{err}");
}
#[test]
fn parse_rejects_missing_value_for_tal_url() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("--tal-url requires a value"), "{err}");
}
#[test]
fn parse_rejects_missing_db() {
let argv = vec!["rpki".to_string(), "--tal-url".to_string(), "x".to_string()];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("--db is required"), "{err}");
}
#[test]
fn parse_rejects_missing_tal_mode() {
let argv = vec!["rpki".to_string(), "--db".to_string(), "db".to_string()];
let err = parse_args(&argv).unwrap_err();
assert!(
err.contains("--tal-url") || err.contains("--tal-path"),
"{err}"
);
}
#[test]
fn parse_accepts_tal_url_mode() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.tal_url.as_deref(), Some("https://example.test/x.tal"));
assert_eq!(
args.tal_urls,
vec!["https://example.test/x.tal".to_string()]
);
assert!(args.tal_path.is_none());
assert!(args.ta_path.is_none());
assert_eq!(args.tal_inputs.len(), 1);
assert_eq!(args.tal_inputs[0].tal_id, "x");
assert_eq!(args.parallel_phase1_config, ParallelPhase1Config::default());
assert_eq!(args.parallel_phase2_config, ParallelPhase2Config::default());
}
#[test]
fn parse_accepts_multi_tal_without_parallel_flags() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/arin.tal".to_string(),
"--tal-url".to_string(),
"https://example.test/apnic.tal".to_string(),
"--tal-url".to_string(),
"https://example.test/ripe.tal".to_string(),
"--parallel-max-repo-sync-workers-global".to_string(),
"8".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.tal_urls.len(), 3);
assert_eq!(args.tal_inputs.len(), 3);
assert_eq!(args.tal_inputs[0].tal_id, "arin");
assert_eq!(args.tal_inputs[1].tal_id, "apnic");
assert_eq!(args.tal_inputs[2].tal_id, "ripe");
assert_eq!(args.parallel_phase1_config.max_repo_sync_workers_global, 8);
}
#[test]
fn parse_accepts_multi_tal_urls_by_default() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/arin.tal".to_string(),
"--tal-url".to_string(),
"https://example.test/apnic.tal".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.tal_urls.len(), 2);
assert_eq!(args.tal_inputs.len(), 2);
}
#[test]
fn parse_accepts_offline_mode_requires_ta() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--max-depth".to_string(),
"0".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.tal_paths, vec![PathBuf::from("a.tal")]);
assert_eq!(args.ta_paths, vec![PathBuf::from("ta.cer")]);
assert_eq!(args.tal_path.as_deref(), Some(Path::new("a.tal")));
assert_eq!(args.ta_path.as_deref(), Some(Path::new("ta.cer")));
assert_eq!(args.max_ca_depth, 0);
}
#[test]
fn parse_accepts_multiple_tal_path_pairs_by_default() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"apnic.tal".to_string(),
"--ta-path".to_string(),
"apnic-ta.cer".to_string(),
"--tal-path".to_string(),
"arin.tal".to_string(),
"--ta-path".to_string(),
"arin-ta.cer".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.tal_paths.len(), 2);
assert_eq!(args.ta_paths.len(), 2);
assert_eq!(args.tal_inputs.len(), 2);
}
#[test]
fn parse_rejects_mixed_tal_url_and_tal_path_modes() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/arin.tal".to_string(),
"--tal-path".to_string(),
"apnic.tal".to_string(),
"--ta-path".to_string(),
"apnic-ta.cer".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(
err.contains(
"must specify either one-or-more --tal-url or one-or-more --tal-path/--ta-path pairs"
),
"{err}"
);
}
#[test]
fn parse_rejects_mismatched_tal_path_and_ta_path_counts() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"apnic.tal".to_string(),
"--tal-path".to_string(),
"arin.tal".to_string(),
"--ta-path".to_string(),
"apnic-ta.cer".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(
err.contains("--tal-path and --ta-path counts must match"),
"{err}"
);
}
#[test]
fn parse_accepts_tal_path_without_ta_when_disable_rrdp_is_set() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--disable-rrdp".to_string(),
"--rsync-command".to_string(),
"/tmp/fake-rsync".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(args.tal_path.as_deref(), Some(Path::new("a.tal")));
assert!(args.ta_path.is_none());
assert!(args.disable_rrdp);
assert_eq!(
args.rsync_command.as_deref(),
Some(Path::new("/tmp/fake-rsync"))
);
}
#[test]
fn parse_accepts_multiple_tal_paths_without_ta_when_disable_rrdp() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--tal-path".to_string(),
"b.tal".to_string(),
"--disable-rrdp".to_string(),
"--rsync-command".to_string(),
"/tmp/fake-rsync".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(
args.tal_paths,
vec![PathBuf::from("a.tal"), PathBuf::from("b.tal")]
);
assert!(args.ta_paths.is_empty());
assert_eq!(args.tal_inputs.len(), 2);
assert!(args.disable_rrdp);
}
#[test]
fn parse_accepts_payload_delta_replay_mode_with_offline_tal_and_ta() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-base-archive".to_string(),
"base-archive".to_string(),
"--payload-base-locks".to_string(),
"base-locks.json".to_string(),
"--payload-delta-archive".to_string(),
"delta-archive".to_string(),
"--payload-delta-locks".to_string(),
"delta-locks.json".to_string(),
];
let args = parse_args(&argv).expect("parse delta replay mode");
assert_eq!(
args.payload_base_archive.as_deref(),
Some(Path::new("base-archive"))
);
assert_eq!(
args.payload_base_locks.as_deref(),
Some(Path::new("base-locks.json"))
);
assert_eq!(
args.payload_delta_archive.as_deref(),
Some(Path::new("delta-archive"))
);
assert_eq!(
args.payload_delta_locks.as_deref(),
Some(Path::new("delta-locks.json"))
);
}
#[test]
fn parse_rejects_partial_payload_delta_arguments_and_mutual_exclusion() {
let argv_partial = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-base-archive".to_string(),
"base-archive".to_string(),
];
let err = parse_args(&argv_partial).unwrap_err();
assert!(err.contains("must be provided together"), "{err}");
let argv_both = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-replay-archive".to_string(),
"archive".to_string(),
"--payload-replay-locks".to_string(),
"locks.json".to_string(),
"--payload-base-archive".to_string(),
"base-archive".to_string(),
"--payload-base-locks".to_string(),
"base-locks.json".to_string(),
"--payload-delta-archive".to_string(),
"delta-archive".to_string(),
"--payload-delta-locks".to_string(),
"delta-locks.json".to_string(),
];
let err = parse_args(&argv_both).unwrap_err();
assert!(err.contains("mutually exclusive"), "{err}");
}
#[test]
fn parse_rejects_payload_delta_with_tal_url_or_rsync_local_dir() {
let argv_url = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--payload-base-archive".to_string(),
"base-archive".to_string(),
"--payload-base-locks".to_string(),
"base-locks.json".to_string(),
"--payload-delta-archive".to_string(),
"delta-archive".to_string(),
"--payload-delta-locks".to_string(),
"delta-locks.json".to_string(),
];
let err = parse_args(&argv_url).unwrap_err();
assert!(err.contains("--tal-url is not supported"), "{err}");
let argv_rsync = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-base-archive".to_string(),
"base-archive".to_string(),
"--payload-base-locks".to_string(),
"base-locks.json".to_string(),
"--payload-delta-archive".to_string(),
"delta-archive".to_string(),
"--payload-delta-locks".to_string(),
"delta-locks.json".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
];
let err = parse_args(&argv_rsync).unwrap_err();
assert!(
err.contains("payload delta replay mode cannot be combined with --rsync-local-dir"),
"{err}"
);
}
#[test]
fn parse_accepts_payload_replay_mode_with_offline_tal_and_ta() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-replay-archive".to_string(),
"archive".to_string(),
"--payload-replay-locks".to_string(),
"locks.json".to_string(),
];
let args = parse_args(&argv).expect("parse replay mode");
assert_eq!(
args.payload_replay_archive.as_deref(),
Some(Path::new("archive"))
);
assert_eq!(
args.payload_replay_locks.as_deref(),
Some(Path::new("locks.json"))
);
}
#[test]
fn parse_rejects_partial_payload_replay_arguments() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-replay-archive".to_string(),
"archive".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("must be provided together"), "{err}");
}
#[test]
fn parse_rejects_payload_replay_with_tal_url_or_rsync_local_dir() {
let argv_url = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--payload-replay-archive".to_string(),
"archive".to_string(),
"--payload-replay-locks".to_string(),
"locks.json".to_string(),
];
let err = parse_args(&argv_url).unwrap_err();
assert!(err.contains("--tal-url is not supported"), "{err}");
let argv_rsync = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-path".to_string(),
"a.tal".to_string(),
"--ta-path".to_string(),
"ta.cer".to_string(),
"--payload-replay-archive".to_string(),
"archive".to_string(),
"--payload-replay-locks".to_string(),
"locks.json".to_string(),
"--rsync-local-dir".to_string(),
"repo".to_string(),
];
let err = parse_args(&argv_rsync).unwrap_err();
assert!(
err.contains("cannot be combined with --rsync-local-dir"),
"{err}"
);
}
#[test]
fn parse_accepts_validation_time_rfc3339() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--validation-time".to_string(),
"2026-01-01T00:00:00Z".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert!(args.validation_time.is_some());
}
#[test]
fn parse_rejects_removed_revalidate_only_flag() {
let argv = vec![
"rpki".to_string(),
"--db".to_string(),
"db".to_string(),
"--tal-url".to_string(),
"https://example.test/x.tal".to_string(),
"--revalidate-only".to_string(),
];
let err = parse_args(&argv).unwrap_err();
assert!(err.contains("unknown argument: --revalidate-only"), "{err}");
}
#[test]
fn read_policy_accepts_valid_toml() {
let dir = tempfile::tempdir().expect("tmpdir");
let p = dir.path().join("policy.toml");
std::fs::write(
&p,
"signed_object_failure_policy = \"drop_publication_point\"\n",
)
.expect("write policy");
let policy = read_policy(Some(&p)).expect("parse policy");
assert_eq!(
policy.signed_object_failure_policy,
crate::policy::SignedObjectFailurePolicy::DropPublicationPoint
);
assert_eq!(
policy.resource_validation_mode,
ResourceValidationMode::ValidationUpdate03
);
assert_eq!(policy.strict, StrictPolicy::default());
}
#[test]
fn read_policy_accepts_strict_table() {
let dir = tempfile::tempdir().expect("tmpdir");
let p = dir.path().join("policy.toml");
std::fs::write(
&p,
r#"
[strict]
name = true
cms_der = true
"#,
)
.expect("write policy");
let policy = read_policy(Some(&p)).expect("parse policy");
assert_eq!(
policy.strict,
StrictPolicy {
name: true,
cms_der: true,
signed_attrs: false,
}
);
}
#[test]
fn read_policy_accepts_resource_validation_mode() {
let dir = tempfile::tempdir().expect("tmpdir");
let p = dir.path().join("policy.toml");
std::fs::write(&p, "resource_validation_mode = \"rfc6487\"\n").expect("write policy");
let policy = read_policy(Some(&p)).expect("parse policy");
assert_eq!(
policy.resource_validation_mode,
ResourceValidationMode::Rfc6487
);
}
#[test]
fn read_policy_reports_missing_file() {
let dir = tempfile::tempdir().expect("tmpdir");
let p = dir.path().join("missing.toml");
let err = read_policy(Some(&p)).unwrap_err();
assert!(err.contains("read policy file failed"), "{err}");
}

View File

@@ -0,0 +1,281 @@
// CLI test group: report helpers.
fn synthetic_post_validation_shared() -> PostValidationShared {
let tal_bytes = std::fs::read(
std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/tal/apnic-rfc7730-https.tal"),
)
.expect("read tal fixture");
let ta_der = std::fs::read(
std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/ta/apnic-ta.cer"),
)
.expect("read ta fixture");
let discovery = crate::validation::from_tal::discover_root_ca_instance_from_tal_and_ta_der(
&tal_bytes, &ta_der, None,
)
.expect("discover root");
let tree = crate::validation::tree::TreeRunOutput {
instances_processed: 1,
instances_failed: 0,
warnings: vec![
crate::report::Warning::new("synthetic warning")
.with_rfc_refs(&[crate::report::RfcRef("RFC 6487 §4.8.8.1")])
.with_context("rsync://example.test/repo/pp/"),
],
vrps: vec![
crate::validation::objects::Vrp {
asn: 64496,
prefix: crate::data_model::roa::IpPrefix {
afi: crate::data_model::roa::RoaAfi::Ipv4,
prefix_len: 24,
addr: [192, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length: 24,
},
crate::validation::objects::Vrp {
asn: 64497,
prefix: crate::data_model::roa::IpPrefix {
afi: crate::data_model::roa::RoaAfi::Ipv6,
prefix_len: 48,
addr: [0x20, 0x01, 0x0d, 0xb8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
},
max_length: 64,
},
],
aspas: vec![crate::validation::objects::AspaAttestation {
customer_as_id: 64496,
provider_as_ids: vec![64497, 64498],
}],
router_keys: Vec::new(),
};
let mut pp1 = crate::audit::PublicationPointAudit::default();
pp1.source = "fresh".to_string();
pp1.rrdp_notification_uri = Some("https://example.test/n1.xml".to_string());
pp1.manifest_rsync_uri = "rsync://example.test/repo/pp1/manifest.mft".to_string();
pp1.objects.push(crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/pp1/a.roa".to_string(),
sha256_hex: "11".repeat(32),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Ok,
detail: None,
});
let mut pp2 = crate::audit::PublicationPointAudit::default();
pp2.source = "fresh".to_string();
pp2.rrdp_notification_uri = Some("https://example.test/n1.xml".to_string());
let mut pp3 = crate::audit::PublicationPointAudit::default();
pp3.source = "fresh".to_string();
pp3.rrdp_notification_uri = Some("https://example.test/n2.xml".to_string());
let out = crate::validation::run_tree_from_tal::RunTreeFromTalAuditOutput {
discovery: discovery.clone(),
discoveries: vec![discovery],
successful_tal_inputs: Vec::new(),
tree,
publication_points: vec![pp1, pp2, pp3],
roa_cache_stats: crate::validation::objects::RoaValidationCacheStats::default(),
downloads: Vec::new(),
download_stats: crate::audit::AuditDownloadStats::default(),
current_repo_objects: Vec::new(),
ccr_accumulator: None,
cir_input: crate::cir::CirInputSnapshot::default(),
};
PostValidationShared::from_run_output(out)
}
fn sample_cli_ccr_accumulator() -> CcrAccumulator {
let tal_bytes = std::fs::read(
std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/tal/apnic-rfc7730-https.tal"),
)
.expect("read tal fixture");
let ta_der = std::fs::read(
std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/ta/apnic-ta.cer"),
)
.expect("read ta fixture");
let discovery = crate::validation::from_tal::discover_root_ca_instance_from_tal_and_ta_der(
&tal_bytes, &ta_der, None,
)
.expect("discover root");
let mut accumulator = CcrAccumulator::new(vec![discovery.trust_anchor.clone()]);
let manifest_uri = "rsync://example.test/repo/current.mft".to_string();
let projection = crate::storage::VcirCcrManifestProjection {
manifest_rsync_uri: manifest_uri.clone(),
manifest_sha256: vec![0x44; 32],
manifest_size: 2048,
manifest_ee_aki: vec![0x55; 20],
manifest_number_be: vec![1],
manifest_this_update: crate::storage::PackTime::from_utc_offset_datetime(
time::OffsetDateTime::now_utc(),
),
manifest_sia_locations_der: vec![
crate::ccr::manifest_location::encode_access_description_der(&AccessDescription {
access_method_oid: OID_AD_SIGNED_OBJECT.to_string(),
access_location: manifest_uri,
})
.expect("encode signedObject"),
],
subordinate_skis: vec![vec![0x33; 20]],
};
accumulator
.append_manifest_projection(&projection)
.expect("append manifest projection");
accumulator
}
#[test]
fn build_report_and_helpers_work_on_synthetic_output() {
let shared = synthetic_post_validation_shared();
let policy = Policy::default();
let validation_time = time::OffsetDateTime::now_utc();
let report = build_report(&policy, validation_time, &shared);
assert_eq!(unique_rrdp_repos(&report), 2);
assert_eq!(report.vrps.len(), 2);
assert_eq!(report.aspas.len(), 1);
print_summary(&report);
}
#[test]
fn run_report_task_and_stage_timing_work() {
let shared = synthetic_post_validation_shared();
let policy = Policy::default();
let validation_time = time::OffsetDateTime::now_utc();
let dir = tempfile::tempdir().expect("tmpdir");
let report_path = dir.path().join("report.json");
let report_output = run_report_task(
&policy,
validation_time,
&shared,
Some(&report_path),
ReportJsonFormat::Compact,
)
.expect("run report task");
assert!(report_output.report_write_ms.is_some());
let report_json = std::fs::read_to_string(&report_path).expect("read report json");
assert!(!report_json.contains('\n'), "{report_json}");
let report: serde_json::Value =
serde_json::from_str(&report_json).expect("parse compact report json");
assert_eq!(report["vrps"].as_array().unwrap().len(), 2);
assert_eq!(report["aspas"].as_array().unwrap().len(), 1);
assert_eq!(report["queryAudit"]["status"].as_str(), Some("complete"));
assert!(report["queryAudit"]["eventsCount"].as_u64().unwrap() > 0);
let events_path = dir.path().join(
report["queryAudit"]["eventsPath"]
.as_str()
.expect("events path"),
);
let events = std::fs::read_to_string(events_path).expect("read validation events");
assert!(
events
.lines()
.any(|line| line.contains("\"eventType\":\"object\""))
);
let stage_timing = RunStageTiming {
validation_ms: 1,
enable_roa_validation_cache: false,
enable_child_certificate_validation_cache: false,
publication_point_cache_observe_only: false,
enable_publication_point_validation_cache: false,
crypto_signature_cache_observe: None,
enable_transport_request_prefetch: false,
report_build_ms: report_output.report_build_ms,
report_write_ms: report_output.report_write_ms,
ccr_build_ms: Some(2),
ccr_build_breakdown: None,
ccr_write_ms: Some(3),
compare_view_build_ms: Some(4),
compare_view_write_ms: Some(5),
cir_build_cir_ms: Some(6),
cir_write_cir_ms: Some(7),
cir_total_ms: Some(8),
total_ms: 9,
publication_points: shared.publication_points.len(),
repo_sync_ms_total: 10,
publication_point_repo_sync_ms_total: 11,
download_event_count: 12,
rrdp_download_ms_total: 13,
rsync_download_ms_total: 14,
download_bytes_total: 15,
roa_validation_cache: crate::validation::objects::RoaValidationCacheStats::default(),
analysis_counts: std::collections::HashMap::new(),
analysis_phases: std::collections::HashMap::new(),
analysis_top_publication_points: Vec::new(),
analysis_top_publication_point_steps: Vec::new(),
analysis_top_publication_point_cache_steps: Vec::new(),
vcir_storage_summary_ms: Some(16),
vcir_storage: Some(VcirStorageSummary {
entry_count: 2,
vcir_value_bytes: 100,
vcir_value_bytes_max: 60,
vcir_value_bytes_max_manifest_rsync_uri: Some(
"rsync://example.test/repo/max.mft".to_string(),
),
core_fields: VcirCoreFieldSizeBreakdown {
manifest_rsync_uri_bytes: 10,
..VcirCoreFieldSizeBreakdown::default()
},
ccr_projection: VcirCcrProjectionSizeBreakdown {
manifest_sha256_bytes: 32,
..VcirCcrProjectionSizeBreakdown::default()
},
child_resources: VcirChildResourceSizeBreakdown {
effective_ip_resource_cbor_bytes: 12,
effective_as_resource_cbor_bytes: 6,
},
field_sizes: VcirFieldSizeBreakdown {
local_output_count: 1,
local_output_payload_json_bytes: 70,
local_output_payload_typed_body_bytes: 20,
..VcirFieldSizeBreakdown::default()
},
local_output_old_projection_bytes: 80,
local_output_typed_projection_bytes: 30,
local_output_projection_saved_bytes: 50,
top_entries_by_vcir_value_bytes: vec![VcirStorageEntrySummary {
manifest_rsync_uri: "rsync://example.test/repo/max.mft".to_string(),
vcir_value_bytes: 60,
local_vrp_count: 1,
local_aspa_count: 0,
local_router_key_count: 0,
accepted_object_count: 1,
rejected_object_count: 0,
child_count: 0,
core_fields: VcirCoreFieldSizeBreakdown::default(),
ccr_projection: VcirCcrProjectionSizeBreakdown::default(),
child_resources: VcirChildResourceSizeBreakdown::default(),
field_sizes: VcirFieldSizeBreakdown::default(),
local_output_old_projection_bytes: 1,
local_output_typed_projection_bytes: 1,
local_output_projection_saved_bytes: 0,
}],
}),
publication_point_cache_index_load: None,
publication_point_cache_index_refresh: None,
memory_telemetry: None,
};
write_stage_timing(Some(&report_path), &stage_timing).expect("write stage timing");
let stage_timing_json =
std::fs::read_to_string(dir.path().join("stage-timing.json")).expect("read timing");
assert!(stage_timing_json.contains("\"validation_ms\""));
assert!(stage_timing_json.contains("\"ccr_build_ms\""));
assert!(stage_timing_json.contains("\"vcir_storage\""));
assert!(stage_timing_json.contains("\"local_output_projection_saved_bytes\""));
let ccr_path = dir.path().join("result.ccr");
write_stage_timing(Some(&ccr_path), &stage_timing).expect("write stage timing via ccr path");
assert!(
dir.path().join("stage-timing.json").exists(),
"stage timing should use parent directory of the anchor path"
);
let skipped = ReportTaskOutput::skipped();
assert_eq!(skipped.report_build_ms, 0);
assert!(skipped.report_write_ms.is_none());
}

View File

@@ -0,0 +1,281 @@
// CLI test group: report tasks.
#[test]
fn stage_timing_serializes_memory_telemetry() {
let dir = tempfile::tempdir().expect("tmpdir");
let report_path = dir.path().join("report.json");
let stage_timing = RunStageTiming {
validation_ms: 1,
enable_roa_validation_cache: true,
enable_child_certificate_validation_cache: true,
publication_point_cache_observe_only: false,
enable_publication_point_validation_cache: false,
crypto_signature_cache_observe: None,
enable_transport_request_prefetch: true,
report_build_ms: 2,
report_write_ms: None,
ccr_build_ms: None,
ccr_build_breakdown: None,
ccr_write_ms: None,
compare_view_build_ms: None,
compare_view_write_ms: None,
cir_build_cir_ms: None,
cir_write_cir_ms: None,
cir_total_ms: None,
total_ms: 3,
publication_points: 4,
repo_sync_ms_total: 5,
publication_point_repo_sync_ms_total: 6,
download_event_count: 7,
rrdp_download_ms_total: 8,
rsync_download_ms_total: 9,
download_bytes_total: 10,
roa_validation_cache: crate::validation::objects::RoaValidationCacheStats {
hit_roas: 2,
..crate::validation::objects::RoaValidationCacheStats::default()
},
analysis_counts: std::collections::HashMap::from([(
"roa_validation_cache_hit_roas".to_string(),
2,
)]),
analysis_phases: std::collections::HashMap::new(),
analysis_top_publication_points: Vec::new(),
analysis_top_publication_point_steps: Vec::new(),
analysis_top_publication_point_cache_steps: Vec::new(),
vcir_storage_summary_ms: None,
vcir_storage: None,
publication_point_cache_index_load: None,
publication_point_cache_index_refresh: None,
memory_telemetry: Some(MemoryTelemetrySummary {
checkpoints: vec![MemoryTelemetryCheckpoint {
label: "after_validation".to_string(),
elapsed_ms: 11,
process: ProcessMemorySnapshot {
label: "after_validation".to_string(),
vm_rss_kb: Some(12),
vm_size_kb: None,
vm_data_kb: None,
vm_swap_kb: None,
rss_anon_kb: Some(13),
rss_file_kb: None,
rss_shmem_kb: None,
threads: Some(14),
fd_count: Some(15),
smaps_rollup: None,
smaps_mapping_summary: None,
errors: Vec::new(),
},
rocksdb: RocksDbMemorySnapshot {
databases: Vec::new(),
totals: RocksDbMemoryTotals {
cur_size_all_mem_tables: 16,
size_all_mem_tables: 17,
estimate_table_readers_mem: 18,
block_cache_capacity: 19,
block_cache_usage: 20,
block_cache_pinned_usage: 21,
},
},
}],
object_graph: None,
malloc_trim_probes: Vec::new(),
}),
};
write_stage_timing(Some(&report_path), &stage_timing).expect("write stage timing");
let value: serde_json::Value = serde_json::from_str(
&std::fs::read_to_string(dir.path().join("stage-timing.json")).unwrap(),
)
.expect("parse stage timing json");
let checkpoint = &value["memory_telemetry"]["checkpoints"][0];
assert_eq!(checkpoint["label"], "after_validation");
assert_eq!(checkpoint["process"]["vm_rss_kb"], 12);
assert_eq!(
checkpoint["rocksdb"]["totals"]["cur_size_all_mem_tables"],
16
);
assert_eq!(value["analysis_counts"]["roa_validation_cache_hit_roas"], 2);
assert_eq!(value["roa_validation_cache"]["hit_roas"], 2);
assert_eq!(value["publication_point_cache_observe_only"], false);
assert_eq!(value["enable_publication_point_validation_cache"], false);
assert!(
value["memory_telemetry"]
.as_object()
.expect("memory telemetry object")
.get("malloc_trim_probes")
.is_none()
);
}
#[test]
fn shared_object_graph_estimate_counts_audit_and_outputs() {
let mut shared = synthetic_post_validation_shared();
let mut publication_points = shared
.publication_points
.iter()
.cloned()
.collect::<Vec<_>>();
publication_points[0].rsync_base_uri = "rsync://example.test/repo/".to_string();
publication_points[0].manifest_rsync_uri = "rsync://example.test/repo/a.mft".to_string();
publication_points[0].publication_point_rsync_uri = "rsync://example.test/repo/".to_string();
publication_points[0].objects = vec![crate::audit::ObjectAuditEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
sha256_hex: "11".repeat(32),
kind: crate::audit::AuditObjectKind::Roa,
result: crate::audit::AuditObjectResult::Ok,
detail: None,
}];
shared.publication_points = publication_points.into();
let graph = estimate_shared_object_graph(&shared);
let publication_points_section = graph
.sections
.iter()
.find(|section| section.name == "publication_points")
.expect("publication points section");
let object_count = publication_points_section
.details
.iter()
.find(|metric| metric.name == "object_audit_entry_count")
.expect("object count metric");
assert_eq!(object_count.value, 1);
assert!(publication_points_section.estimated_bytes > 0);
let vrps_section = graph
.sections
.iter()
.find(|section| section.name == "vrps")
.expect("vrps section");
assert_eq!(vrps_section.item_count, 2);
assert!(graph.total_estimated_bytes >= publication_points_section.estimated_bytes);
}
#[test]
fn run_compare_view_task_writes_csv_from_shared_output() {
let shared = synthetic_post_validation_shared();
let dir = tempfile::tempdir().expect("tmpdir");
let vrps_path = dir.path().join("vrps.csv");
let vaps_path = dir.path().join("vaps.csv");
let output = run_compare_view_task(&shared, Some(&vrps_path), Some(&vaps_path), "unknown")
.expect("write direct compare views");
assert!(output.build_ms.is_some());
assert!(output.write_ms.is_some());
let vrps_csv = std::fs::read_to_string(vrps_path).expect("read vrps csv");
let vaps_csv = std::fs::read_to_string(vaps_path).expect("read vaps csv");
assert!(vrps_csv.contains("ASN,IP Prefix,Max Length,Trust Anchor"));
assert!(vrps_csv.contains("AS64496,192.0.2.0/24,24,unknown"));
assert!(vrps_csv.contains("AS64497,2001:db8::/48,64,unknown"));
assert!(vaps_csv.contains("Customer ASN,Providers,Trust Anchor"));
assert!(vaps_csv.contains("AS64496,AS64497;AS64498,unknown"));
}
#[test]
fn run_ccr_task_uses_accumulator_when_phase2_output_contains_reuse_sources() {
let mut shared = synthetic_post_validation_shared();
shared.ccr_accumulator = Some(sample_cli_ccr_accumulator());
let mut publication_points = shared
.publication_points
.iter()
.cloned()
.collect::<Vec<_>>();
publication_points[1].source = "vcir_current_instance".to_string();
publication_points[2].source = "failed_no_cache".to_string();
shared.publication_points = publication_points.into();
let dir = tempfile::tempdir().expect("tmpdir");
let ccr_path = dir.path().join("result.ccr");
let store = RocksStore::open(&dir.path().join("db")).expect("open empty store");
let output = run_ccr_task(
&store,
&shared,
Some(&ccr_path),
time::OffsetDateTime::now_utc(),
)
.expect("run ccr task");
assert!(output.ccr_build_ms.is_some());
assert!(output.ccr_build_breakdown.is_none());
let der = std::fs::read(&ccr_path).expect("read ccr");
let ci = crate::ccr::decode_content_info(&der).expect("decode ccr");
assert_eq!(
ci.content
.mfts
.as_ref()
.map(|manifest_state| manifest_state.mis.len()),
Some(1)
);
}
#[test]
fn write_json_writes_report() {
let report = AuditReportV2 {
format_version: 2,
meta: AuditRunMeta {
validation_time_rfc3339_utc: "2026-01-01T00:00:00Z".to_string(),
},
policy: Policy::default(),
tree: TreeSummary {
instances_processed: 0,
instances_failed: 0,
warnings: Vec::new(),
},
publication_points: Vec::new(),
vrps: Vec::new(),
aspas: Vec::new(),
downloads: Vec::new(),
download_stats: crate::audit::AuditDownloadStats::default(),
repo_sync_stats: crate::audit::AuditRepoSyncStats::default(),
query_audit: None,
};
let dir = tempfile::tempdir().expect("tmpdir");
let pretty_path = dir.path().join("report-pretty.json");
write_json(&pretty_path, &report, ReportJsonFormat::Pretty).expect("write pretty json");
let pretty = std::fs::read_to_string(&pretty_path).expect("read pretty report");
assert!(pretty.contains("\"format_version\""));
assert!(pretty.contains("\"policy\""));
assert!(pretty.contains("\n \"format_version\""), "{pretty}");
let compact_path = dir.path().join("report-compact.json");
write_json(&compact_path, &report, ReportJsonFormat::Compact).expect("write compact json");
let compact = std::fs::read_to_string(&compact_path).expect("read compact report");
assert!(compact.contains("\"format_version\""));
assert!(compact.contains("\"policy\""));
assert!(!compact.contains('\n'), "{compact}");
}
#[test]
fn build_repo_sync_stats_aggregates_phase_and_terminal_state() {
let mut pp1 = crate::audit::PublicationPointAudit::default();
pp1.repo_sync_phase = Some("rrdp_ok".to_string());
pp1.repo_sync_duration_ms = Some(10);
pp1.repo_terminal_state = "fresh".to_string();
let mut pp2 = crate::audit::PublicationPointAudit::default();
pp2.repo_sync_phase = Some("rrdp_failed_rsync_failed".to_string());
pp2.repo_sync_duration_ms = Some(20);
pp2.repo_terminal_state = "failed_no_cache".to_string();
let mut pp3 = crate::audit::PublicationPointAudit::default();
pp3.repo_sync_phase = Some("rrdp_failed_rsync_failed".to_string());
pp3.repo_sync_duration_ms = Some(30);
pp3.repo_terminal_state = "failed_no_cache".to_string();
let stats = build_repo_sync_stats(&[pp1, pp2, pp3]);
assert_eq!(stats.publication_points_total, 3);
assert_eq!(stats.by_phase["rrdp_ok"].count, 1);
assert_eq!(stats.by_phase["rrdp_ok"].duration_ms_total, 10);
assert_eq!(stats.by_phase["rrdp_failed_rsync_failed"].count, 2);
assert_eq!(
stats.by_phase["rrdp_failed_rsync_failed"].duration_ms_total,
50
);
assert_eq!(stats.by_terminal_state["fresh"].count, 1);
assert_eq!(stats.by_terminal_state["failed_no_cache"].count, 2);
assert_eq!(
stats.by_terminal_state["failed_no_cache"].duration_ms_total,
50
);
}

View File

@@ -0,0 +1,135 @@
// CLI timing and argument data types.
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
struct RunStageTiming {
validation_ms: u64,
enable_roa_validation_cache: bool,
enable_child_certificate_validation_cache: bool,
publication_point_cache_observe_only: bool,
enable_publication_point_validation_cache: bool,
crypto_signature_cache_observe: Option<crate::crypto_sig_cache::CryptoSigCacheSummary>,
enable_transport_request_prefetch: bool,
report_build_ms: u64,
report_write_ms: Option<u64>,
ccr_build_ms: Option<u64>,
ccr_build_breakdown: Option<CcrBuildBreakdown>,
ccr_write_ms: Option<u64>,
compare_view_build_ms: Option<u64>,
compare_view_write_ms: Option<u64>,
cir_build_cir_ms: Option<u64>,
cir_write_cir_ms: Option<u64>,
cir_total_ms: Option<u64>,
total_ms: u64,
publication_points: usize,
repo_sync_ms_total: u64,
publication_point_repo_sync_ms_total: u64,
download_event_count: u64,
rrdp_download_ms_total: u64,
rsync_download_ms_total: u64,
download_bytes_total: u64,
roa_validation_cache: crate::validation::objects::RoaValidationCacheStats,
analysis_counts: HashMap<String, u64>,
analysis_phases: HashMap<String, DurationStats>,
analysis_top_publication_points: Vec<TopDurationEntry>,
analysis_top_publication_point_steps: Vec<TopDurationEntry>,
analysis_top_publication_point_cache_steps: Vec<TopDurationEntry>,
vcir_storage_summary_ms: Option<u64>,
vcir_storage: Option<VcirStorageSummary>,
publication_point_cache_index_load: Option<crate::storage::PpCacheIndexLoadStats>,
publication_point_cache_index_refresh: Option<crate::storage::PpCacheIndexRefreshStats>,
memory_telemetry: Option<MemoryTelemetrySummary>,
}
fn record_memory_checkpoint(
checkpoints: &mut Vec<MemoryTelemetryCheckpoint>,
label: &str,
total_started: &std::time::Instant,
store: &RocksStore,
) {
checkpoints.push(MemoryTelemetryCheckpoint {
label: label.to_string(),
elapsed_ms: total_started.elapsed().as_millis() as u64,
process: crate::memory_telemetry::process_memory_snapshot(label),
rocksdb: store.memory_snapshot(),
});
}
fn memory_trim_probe_enabled() -> bool {
std::env::var("RPKI_MEMORY_TRIM_PROBE")
.map(|value| matches!(value.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"))
.unwrap_or(false)
}
fn vcir_storage_summary_enabled() -> bool {
std::env::var("RPKI_VCIR_STORAGE_SUMMARY")
.map(|value| matches!(value.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"))
.unwrap_or(false)
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CliArgs {
pub validation_contract_out_path: Option<PathBuf>,
pub tal_urls: Vec<String>,
pub tal_paths: Vec<PathBuf>,
pub ta_paths: Vec<PathBuf>,
pub tal_url: Option<String>,
pub tal_path: Option<PathBuf>,
pub ta_path: Option<PathBuf>,
pub parallel_phase1_config: ParallelPhase1Config,
pub parallel_phase2_config: ParallelPhase2Config,
pub tal_inputs: Vec<TalInputSpec>,
pub ta_constraints: TaConstraintsByTal,
pub db_path: PathBuf,
pub raw_store_db: Option<PathBuf>,
pub repo_bytes_db: Option<PathBuf>,
pub policy_path: Option<PathBuf>,
pub strict_policy: Option<StrictPolicy>,
pub resource_validation_mode: Option<ResourceValidationMode>,
pub report_json_path: Option<PathBuf>,
pub report_json_compact: bool,
pub skip_report_build: bool,
pub skip_vcir_persist: bool,
pub enable_roa_validation_cache: bool,
pub enable_child_certificate_validation_cache: bool,
pub publication_point_cache_observe_only: bool,
pub enable_publication_point_validation_cache: bool,
pub crypto_signature_cache_observe_only: bool,
pub enable_crypto_signature_cache: bool,
pub enable_transport_request_prefetch: bool,
pub ccr_out_path: Option<PathBuf>,
pub vrps_csv_out_path: Option<PathBuf>,
pub vaps_csv_out_path: Option<PathBuf>,
pub compare_view_trust_anchor: Option<String>,
pub cir_enabled: bool,
pub cir_out_path: Option<PathBuf>,
pub cir_static_root: Option<PathBuf>,
pub cir_tal_uris: Vec<String>,
pub cir_tal_uri: Option<String>,
pub payload_replay_archive: Option<PathBuf>,
pub payload_replay_locks: Option<PathBuf>,
pub payload_base_archive: Option<PathBuf>,
pub payload_base_locks: Option<PathBuf>,
pub payload_base_validation_time: Option<time::OffsetDateTime>,
pub payload_delta_archive: Option<PathBuf>,
pub payload_delta_locks: Option<PathBuf>,
pub memory_trim_after_validation: bool,
pub rsync_local_dir: Option<PathBuf>,
pub disable_rrdp: bool,
pub rsync_command: Option<PathBuf>,
pub http_timeout_secs: u64,
pub http_root_cert_paths: Vec<PathBuf>,
pub rsync_timeout_secs: u64,
pub rsync_mirror_root: Option<PathBuf>,
pub rsync_scope_policy: RsyncScopePolicy,
pub max_ca_depth: usize,
pub max_instances: Option<usize>,
pub validation_time: Option<time::OffsetDateTime>,
pub analyze: bool,
pub analysis_out_path: Option<PathBuf>,
pub profile_cpu: bool,
}

View File

@@ -0,0 +1,116 @@
// Command-line usage text.
fn usage() -> String {
let bin = "panda-rpki-validator";
format!(
"\
Usage:
{bin} --db <path> --tal-url <url> [--tal-url <url> ...] [options]
{bin} --db <path> --tal-path <path> --ta-path <path> [--tal-path <path> --ta-path <path> ...] [options]
Options:
--validation-contract-out <path>
Write the effective normal-run validation contract
--db <path> RocksDB directory path (required)
--raw-store-db <path> External raw-by-hash store DB path (optional)
--repo-bytes-db <path> External repo object bytes DB path (optional)
--policy <path> Policy TOML path (optional)
--ta-constraints <tal-id>=<path>
Apply local EE-resource constraints to one TAL (repeatable); adjacent <tal>.constraints files are auto-discovered
--strict [policies] Enable strict policies (default all; comma list: name,cms-der,signed-attrs; none disables)
--resource-validation-mode <validation-update-03|rfc6487>
Resource certificate validation mode (default: validation-update-03)
--report-json <path> Write full audit report as JSON (optional)
--report-json-compact Write report JSON without pretty-printing (requires --report-json)
--skip-report-build Skip full audit report construction when --report-json is not requested
--skip-vcir-persist Skip VCIR persistence/projection building for compare-only runs
--enable-roa-validation-cache
Reuse accepted ROA validation outputs from previous VCIR records (default: off)
--enable-child-certificate-validation-cache
Experimental: reuse validated child certificate discovery results
--publication-point-cache-observe-only
Evaluate publication-point cache eligibility without changing results
--enable-publication-point-validation-cache
Experimental: reuse complete publication-point validation projections
--crypto-signature-cache-observe-only
Measure crypto signature cache key hit rates and verify durations
without changing validation behavior (default: off)
--enable-crypto-signature-cache
Experimental: skip cryptographic signature verification on cache
hits (positive conclusions only; default: off)
--enable-transport-request-prefetch
Experimental: prefetch previous run transport repo requests before tree traversal
--ccr-out <path> Write CCR DER ContentInfo to this path (optional)
--vrps-csv-out <path> Write VRP compare-view CSV directly from validation output (optional; requires --vaps-csv-out)
--vaps-csv-out <path> Write VAP compare-view CSV directly from validation output (optional; requires --vrps-csv-out)
--compare-view-trust-anchor <name>
Trust-anchor label used by direct compare-view CSV output (default: unknown)
--cir-enable Export CIR after the run completes
--cir-out <path> Write CIR DER to this path (requires --cir-enable)
--cir-static-root <path> Deprecated; CIR export no longer exports object pools
--cir-tal-uri <url> Override TAL URI for CIR export (repeatable in multi-TAL mode)
--payload-replay-archive <path> Use local payload replay archive root (offline replay mode)
--payload-replay-locks <path> Use local payload replay locks.json (offline replay mode)
--payload-base-archive <path> Use local base payload archive root (offline delta replay)
--payload-base-locks <path> Use local base locks.json (offline delta replay)
--payload-base-validation-time <rfc3339> Validation time for the base bootstrap inside offline delta replay
--payload-delta-archive <path> Use local delta payload archive root (offline delta replay)
--payload-delta-locks <path> Use local locks-delta.json (offline delta replay)
--memory-trim-after-validation Call malloc_trim(0) after validation/report memory checkpoints (Linux glibc only; default off)
--tal-url <url> TAL URL (repeatable; URL mode)
--tal-path <path> TAL file path (repeatable; file mode)
--ta-path <path> TA certificate DER file path (repeatable in file mode; pairs with --tal-path by position)
--parallel-max-repo-sync-workers-global <n>
Phase 1 global repo sync worker budget (default: 4)
--parallel-max-inflight-snapshot-bytes-global <n>
Phase 1 inflight snapshot byte budget (default: 512MiB)
--parallel-max-pending-repo-results <n>
Phase 1 pending repo result budget (default: 1024)
--parallel-phase2-object-workers <n>
Phase 2 object worker count (default: 8)
--parallel-phase2-worker-queue-capacity <n>
Phase 2 per-worker object queue capacity (default: 256)
--parallel-phase2-ready-batch-size <n>
Phase 2 ready publication points processed per scheduler turn (default: 256)
--parallel-phase2-ready-batch-wall-time-budget-ms <n>
Phase 2 ready staging wall-time budget per scheduler turn (default: 100)
--parallel-phase2-result-drain-batch-size <n>
Phase 2 object results drained per scheduler turn (default: 2048)
--parallel-phase2-finalize-batch-size <n>
Legacy Phase 2 scheduler finalize budget; dedicated finalize worker ignores it (default: 256)
--parallel-phase2-finalize-batch-wall-time-budget-ms <n>
Legacy Phase 2 scheduler finalize time budget; dedicated finalize worker ignores it (default: 100)
--parallel-phase2-finalize-queue-capacity <n>
Phase 2 dedicated finalize worker queue capacity (default: 32768)
--control-plane-stage-workers <n>
Experimental: Phase 2 ready publication point stage worker count;
0 disables the stage pool and keeps inline staging (default: 0)
--dead-repo-blacklist <path>
Enable the dead-repo transport blacklist persisted at this JSON
path (default: disabled). Blacklisted rrdp repos skip straight to
rsync; dual-blacklisted repos terminate instantly.
--dead-repo-blacklist-fail-threshold <n>
Consecutive runs with transport-class fetch failure before a
(repo, transport) entry is blacklisted (default: 3)
--rsync-local-dir <path> Use LocalDirRsyncFetcher rooted at this directory (offline tests)
--disable-rrdp Disable RRDP and synchronize only via rsync
--rsync-command <path> Use this rsync command instead of the default rsync binary
--http-timeout-secs <n> HTTP fetch timeout seconds (default: 20)
--http-root-cert <path> Extra PEM root certificate trusted by HTTPS fetches (repeatable)
--rsync-timeout-secs <n> rsync I/O timeout seconds (default: 60)
--rsync-mirror-root <path> Persist rsync mirrors under this directory (default: disabled)
--rsync-scope <policy> rsync scope policy: host, publication-point, or module-root (default: module-root)
--max-ca-depth <n> Maximum CA depth from a trust anchor (root = 0, default: {DEFAULT_MAX_CA_DEPTH})
--max-depth <n> Deprecated alias for --max-ca-depth
--max-instances <n> Max number of CA instances to process
--validation-time <rfc3339> Validation time in RFC3339 (default: now UTC)
--analyze Write timing analysis JSON under target/live/analyze/<timestamp>/
--analysis-out <path> Write timing analysis JSON under this directory (implies --analyze)
--profile-cpu (Requires build feature 'profile') Write CPU flamegraph under analyze dir
--help Show this help
"
)
}

View File

@@ -1,4 +1,4 @@
//! Crypto signature verification cache (feature #126).
//! Crypto signature verification cache.
//!
//! Caches the context-free cryptographic fact "this signature over these exact bytes verifies
//! under this public key" so that later runs can skip the RSA verification entirely. Only
@@ -8,14 +8,14 @@
//!
//! Modes (CLI):
//!
//! - `--crypto-signature-cache-observe-only` (M1): compute keys, maintain the persistent set,
//! - `--crypto-signature-cache-observe-only`: compute keys, maintain the persistent set,
//! record hit/miss/timing statistics, always run the real verification.
//! - `--enable-crypto-signature-cache` (M2): on a cache hit, skip the real verification and
//! - `--enable-crypto-signature-cache`: on a cache hit, skip the real verification and
//! return success; on a miss, run the real verification and store the positive conclusion.
//! Statistics are recorded the same way; `verify_skipped`/`verify_executed` report what
//! actually happened. Both flags may be combined (reuse + statistics).
//!
//! The five choke points (see the development plan for the full caller inventory):
//! The five verification choke points are:
//!
//! - `CmsSignedObject`: `data_model::signed_object` CMS verify (ROA / ASPA / manifest on the
//! main validation path; GBR objects are not signature-verified by the tree runner).
@@ -30,9 +30,9 @@
//! Trust-anchor self-signature verification is deliberately excluded (a handful of calls per
//! run).
//!
//! Cache key (M1 decision: verify-input bytes, not full object bytes — the key material is in
//! Cache key: verify-input bytes, not full object bytes. The key material is in
//! hand inside each choke point, while the repo-bytes content hash is only available at the
//! upper processing layer):
//! upper processing layer.
//!
//! ```text
//! key = SHA-256( "rpki-crypto-sig-cache-key-v1" | point-tag | "sha256WithRSAEncryption"
@@ -46,18 +46,18 @@
//! Storage:
//!
//! - Single small file next to the work DB (`<work-db-name>.crypto-sig-cache`), never inside
//! the work DB, not touched by periodic snapshot reset (#091/#092): the reset paths only
//! the work DB, not touched by periodic snapshot reset: the reset paths only
//! remove/replace the work DB directory and the PP/child-cert cache artifacts, not siblings
//! of their own choosing.
//! - Binary format: 8-byte magic `RPKICSC2` | u64 record-count | records of
//! key[32] | verified_at_unix_secs(i64) | crypto_impl_version(u32) | reserved(u32).
//! A missing, corrupt, or magic-mismatched file (including the M1 `RPKICSC1` seen-set
//! key\[32\] | verified_at_unix_secs(i64) | crypto_impl_version(u32) | reserved(u32).
//! A missing, corrupt, or magic-mismatched file (including the legacy `RPKICSC1` seen-set
//! format) is silently rebuilt empty. Writes are atomic (write-then-rename).
//! - The cache is persisted once at the end of the validation stage (same point as the M1
//! seen-set). If a run fails mid-validation, entries added during that run are lost;
//! - The cache is persisted once at the end of the validation stage. If a run fails
//! mid-validation, entries added during that run are lost;
//! previously persisted entries are unaffected.
//! - Size cap: approximate上限 `capacity` entries (default 2,000,000; measured M1 APNIC
//! ~148k keys/run, all5 estimated ~750k). When an insert would exceed the cap, the whole
//! - Size cap: approximately `capacity` entries (default 2,000,000). When an insert
//! would exceed the cap, the whole
//! cache is cleared and rebuilt from scratch (full-rebuild eviction — the simplest policy;
//! concurrent inserts may briefly overshoot the cap by a few entries). Each eviction is
//! counted in `evictions_total`.
@@ -222,8 +222,8 @@ pub struct CryptoSigCache {
impl CryptoSigCache {
/// Load the cache from `cache_file`; start empty when the file is missing, corrupt, or
/// has an unknown schema header. `reuse_enabled` selects the M2 hit-skips-verify
/// behavior; when false the cache is observe-only (M1 semantics).
/// has an unknown schema header. `reuse_enabled` selects hit-skips-verify
/// behavior; when false the cache is observe-only.
pub fn load_or_rebuild(cache_file: PathBuf, reuse_enabled: bool) -> Self {
Self::load_or_rebuild_with_capacity(cache_file, reuse_enabled, DEFAULT_CAPACITY)
}
@@ -760,7 +760,7 @@ mod tests {
let cache = CryptoSigCache::load_or_rebuild(path.clone(), true);
assert_eq!(cache.summary().entries_loaded, 0);
// M1 seen-set format (RPKICSC1) must not load as a v2 cache: cold start.
// Legacy seen-set format (RPKICSC1) must not load as a v2 cache: cold start.
let mut bytes = Vec::new();
bytes.extend_from_slice(b"RPKICSC1");
bytes.extend_from_slice(&1u64.to_le_bytes());

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,332 @@
// Daemon command-line parsing and JSON/status helpers.
fn usage() -> String {
let bin = "rpki_daemon";
format!(
"\
Usage:
{bin} --state-root <path> --rpki-bin <path> [options] -- <rpki args...>
Options:
--state-root <path> Persistent daemon root containing state/, runs/, status, and JSONL summary
--rpki-bin <path> rpki child binary to execute for each run
--interval-secs <n> Sleep seconds between runs (default: 60)
--max-runs <n> Stop after n runs (default: run forever)
--retain-runs <n> Keep only the latest n run directories (default: 10)
--status-json <path> Override status JSON path (default: <state-root>/daemon-status.json)
--summary-jsonl <path> Override summary JSONL path (default: <state-root>/daemon-runs.jsonl)
--work-db <path> Work DB path for metrics (default: <state-root>/state/work-db)
--repo-bytes-db <path> Repo bytes DB path for file metrics (default: <state-root>/state/repo-bytes.db)
--raw-store-db <path> Raw store DB path for file metrics (optional)
--db-stats-bin <path> db_stats binary path (default: sibling db_stats next to this executable when present)
--db-stats-exact-every <n>
Run db_stats --exact every n runs (default: disabled)
--time-bin <path> GNU time binary for child process metrics (default: /usr/bin/time when present)
--no-time-wrapper Disable GNU time wrapper
--dead-repo-blacklist <path>
Enable the dead-repository transport blacklist: daemon probes
blacklisted entries between runs and removes revived repos;
auto-injects the child flag when not already present
--dead-repo-blacklist-fail-threshold <n>
Admission threshold forwarded to the child when its args do not
already set it (default: child default 3)
--dead-repo-health-check-interval-secs <n>
Minimum seconds between health check sweeps (default: 600)
--dead-repo-probe-rsync-bin <path>
rsync binary used for health probes (default: rsync)
--help Show this help
Child argument placeholders:
{{state_root}} Daemon state root
{{run_out}} Current run output directory
{{run_id}} Current run id, e.g. 000001-20260428T090000Z
{{run_seq}} Current run sequence number
"
)
}
fn default_time_bin() -> Option<PathBuf> {
let path = PathBuf::from("/usr/bin/time");
if path.is_file() { Some(path) } else { None }
}
fn default_db_stats_bin() -> Option<PathBuf> {
let mut path = std::env::current_exe().ok()?;
path.set_file_name("db_stats");
if path.is_file() { Some(path) } else { None }
}
fn parse_args(argv: &[String]) -> Result<Args, String> {
if argv.iter().any(|arg| arg == "--help" || arg == "-h") {
return Err(usage());
}
let mut state_root: Option<PathBuf> = None;
let mut rpki_bin: Option<PathBuf> = None;
let mut interval_secs = 60u64;
let mut max_runs = None;
let mut retain_runs = 10usize;
let mut status_json: Option<PathBuf> = None;
let mut summary_jsonl: Option<PathBuf> = None;
let mut work_db: Option<PathBuf> = None;
let mut repo_bytes_db: Option<PathBuf> = None;
let mut raw_store_db: Option<PathBuf> = None;
let mut db_stats_bin: Option<PathBuf> = None;
let mut db_stats_exact_every = None;
let mut time_bin = default_time_bin();
let mut no_time_wrapper = false;
let mut dead_repo_blacklist: Option<PathBuf> = None;
let mut dead_repo_blacklist_fail_threshold: Option<u32> = None;
let mut dead_repo_health_check_interval_secs = 600u64;
let mut dead_repo_probe_rsync_bin = PathBuf::from("rsync");
let mut i = 1usize;
while i < argv.len() {
match argv[i].as_str() {
"--" => {
let mut child_args = argv[i + 1..].to_vec();
let state_root =
state_root.ok_or_else(|| format!("--state-root is required\n\n{}", usage()))?;
let work_db = work_db.unwrap_or_else(|| state_root.join("state").join("work-db"));
let repo_bytes_db =
repo_bytes_db.or_else(|| Some(state_root.join("state").join("repo-bytes.db")));
if no_time_wrapper {
time_bin = None;
}
// Forward blacklist enablement to the child process.
// unless its args already carry the flag explicitly.
if let Some(path) = dead_repo_blacklist.as_ref() {
if !child_args.iter().any(|arg| arg == "--dead-repo-blacklist") {
child_args.push("--dead-repo-blacklist".to_string());
child_args.push(path_string(path));
}
if let Some(threshold) = dead_repo_blacklist_fail_threshold {
if !child_args
.iter()
.any(|arg| arg == "--dead-repo-blacklist-fail-threshold")
{
child_args.push("--dead-repo-blacklist-fail-threshold".to_string());
child_args.push(threshold.to_string());
}
}
}
let args = Args {
state_root,
rpki_bin: rpki_bin
.ok_or_else(|| format!("--rpki-bin is required\n\n{}", usage()))?,
interval_secs,
max_runs,
retain_runs,
status_json,
summary_jsonl,
work_db,
repo_bytes_db,
raw_store_db,
db_stats_bin,
db_stats_exact_every,
time_bin,
dead_repo_blacklist,
dead_repo_blacklist_fail_threshold,
dead_repo_health_check_interval_secs,
dead_repo_probe_rsync_bin,
child_args,
};
return validate_args(args);
}
"--state-root" => {
i += 1;
state_root = Some(PathBuf::from(value_at(argv, i, "--state-root")?));
}
"--rpki-bin" => {
i += 1;
rpki_bin = Some(PathBuf::from(value_at(argv, i, "--rpki-bin")?));
}
"--interval-secs" => {
i += 1;
interval_secs =
parse_u64(value_at(argv, i, "--interval-secs")?, "--interval-secs")?;
}
"--max-runs" => {
i += 1;
let parsed = parse_u64(value_at(argv, i, "--max-runs")?, "--max-runs")?;
if parsed == 0 {
return Err("--max-runs must be > 0".to_string());
}
max_runs = Some(parsed);
}
"--retain-runs" => {
i += 1;
let parsed = parse_usize(value_at(argv, i, "--retain-runs")?, "--retain-runs")?;
if parsed == 0 {
return Err("--retain-runs must be > 0".to_string());
}
retain_runs = parsed;
}
"--status-json" => {
i += 1;
status_json = Some(PathBuf::from(value_at(argv, i, "--status-json")?));
}
"--summary-jsonl" => {
i += 1;
summary_jsonl = Some(PathBuf::from(value_at(argv, i, "--summary-jsonl")?));
}
"--work-db" => {
i += 1;
work_db = Some(PathBuf::from(value_at(argv, i, "--work-db")?));
}
"--repo-bytes-db" => {
i += 1;
repo_bytes_db = Some(PathBuf::from(value_at(argv, i, "--repo-bytes-db")?));
}
"--raw-store-db" => {
i += 1;
raw_store_db = Some(PathBuf::from(value_at(argv, i, "--raw-store-db")?));
}
"--db-stats-bin" => {
i += 1;
db_stats_bin = Some(PathBuf::from(value_at(argv, i, "--db-stats-bin")?));
}
"--db-stats-exact-every" => {
i += 1;
let parsed = parse_u64(
value_at(argv, i, "--db-stats-exact-every")?,
"--db-stats-exact-every",
)?;
if parsed == 0 {
return Err("--db-stats-exact-every must be > 0".to_string());
}
db_stats_exact_every = Some(parsed);
}
"--time-bin" => {
i += 1;
time_bin = Some(PathBuf::from(value_at(argv, i, "--time-bin")?));
}
"--no-time-wrapper" => {
no_time_wrapper = true;
}
"--dead-repo-blacklist" => {
i += 1;
dead_repo_blacklist =
Some(PathBuf::from(value_at(argv, i, "--dead-repo-blacklist")?));
}
"--dead-repo-blacklist-fail-threshold" => {
i += 1;
let parsed = parse_u64(
value_at(argv, i, "--dead-repo-blacklist-fail-threshold")?,
"--dead-repo-blacklist-fail-threshold",
)?;
if parsed == 0 || parsed > u32::MAX as u64 {
return Err(
"--dead-repo-blacklist-fail-threshold must be in 1..=u32::MAX".to_string(),
);
}
dead_repo_blacklist_fail_threshold = Some(parsed as u32);
}
"--dead-repo-health-check-interval-secs" => {
i += 1;
dead_repo_health_check_interval_secs = parse_u64(
value_at(argv, i, "--dead-repo-health-check-interval-secs")?,
"--dead-repo-health-check-interval-secs",
)?;
}
"--dead-repo-probe-rsync-bin" => {
i += 1;
dead_repo_probe_rsync_bin =
PathBuf::from(value_at(argv, i, "--dead-repo-probe-rsync-bin")?);
}
other => return Err(format!("unknown argument: {other}\n\n{}", usage())),
}
i += 1;
}
Err(format!("missing -- before child rpki args\n\n{}", usage()))
}
fn validate_args(args: Args) -> Result<Args, String> {
if args.child_args.is_empty() {
return Err(format!(
"child rpki args are required after --\n\n{}",
usage()
));
}
if args.dead_repo_blacklist_fail_threshold.is_some() && args.dead_repo_blacklist.is_none() {
return Err(
"--dead-repo-blacklist-fail-threshold requires --dead-repo-blacklist".to_string(),
);
}
Ok(args)
}
fn value_at<'a>(argv: &'a [String], index: usize, flag: &str) -> Result<&'a str, String> {
argv.get(index)
.map(String::as_str)
.ok_or_else(|| format!("{flag} requires a value"))
}
fn parse_u64(raw: &str, flag: &str) -> Result<u64, String> {
raw.parse::<u64>()
.map_err(|_| format!("invalid {flag}: {raw}"))
}
fn parse_usize(raw: &str, flag: &str) -> Result<usize, String> {
raw.parse::<usize>()
.map_err(|_| format!("invalid {flag}: {raw}"))
}
fn status_path(args: &Args) -> PathBuf {
args.status_json
.clone()
.unwrap_or_else(|| args.state_root.join("daemon-status.json"))
}
fn summary_jsonl_path(args: &Args) -> PathBuf {
args.summary_jsonl
.clone()
.unwrap_or_else(|| args.state_root.join("daemon-runs.jsonl"))
}
fn utc_now() -> time::OffsetDateTime {
time::OffsetDateTime::now_utc().to_offset(time::UtcOffset::UTC)
}
fn format_rfc3339(t: time::OffsetDateTime) -> Result<String, String> {
t.format(&time::format_description::well_known::Rfc3339)
.map_err(|e| format!("format RFC3339 failed: {e}"))
}
fn format_compact_utc(t: time::OffsetDateTime) -> String {
format!(
"{:04}{:02}{:02}T{:02}{:02}{:02}Z",
t.year(),
u8::from(t.month()),
t.day(),
t.hour(),
t.minute(),
t.second()
)
}
fn write_json_pretty<T: Serialize>(path: &Path, value: &T) -> Result<(), String> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)
.map_err(|e| format!("create parent dir failed: {}: {e}", parent.display()))?;
}
let bytes = serde_json::to_vec_pretty(value)
.map_err(|e| format!("serialize json failed: {}: {e}", path.display()))?;
// Atomic write (tmp + rename) so concurrent readers never see a torn file.
let mut tmp_name = path
.file_name()
.map(|name| name.to_os_string())
.unwrap_or_default();
tmp_name.push(".tmp");
let tmp_path = path.with_file_name(tmp_name);
fs::write(&tmp_path, &bytes)
.map_err(|e| format!("write json tmp failed: {}: {e}", tmp_path.display()))?;
fs::rename(&tmp_path, path).map_err(|e| {
format!(
"rename json tmp failed: {} -> {}: {e}",
tmp_path.display(),
path.display()
)
})
}

View File

@@ -0,0 +1,120 @@
// Daemon loop and public entry point.
fn run_daemon(args: &Args) -> Result<(), String> {
fs::create_dir_all(args.state_root.join("state")).map_err(|e| {
format!(
"create daemon state dir failed: {}: {e}",
args.state_root.join("state").display()
)
})?;
fs::create_dir_all(args.state_root.join("runs")).map_err(|e| {
format!(
"create daemon runs dir failed: {}: {e}",
args.state_root.join("runs").display()
)
})?;
let mut runs_completed = 0u64;
let mut next_seq = 1u64;
let mut last_run_id = None;
let mut health = DeadRepoHealthRuntime::default();
write_status(
args,
DaemonState::Starting,
runs_completed,
None,
last_run_id.clone(),
&health,
)?;
// Probe once at startup so revived repos are removed before the first run.
maybe_run_dead_repo_health_check(args, &mut health);
loop {
if args.max_runs.is_some_and(|max| runs_completed >= max) {
break;
}
write_status(
args,
DaemonState::Idle,
runs_completed,
None,
last_run_id.clone(),
&health,
)?;
let ctx = make_run_context(args, next_seq, utc_now());
write_status(
args,
DaemonState::Running,
runs_completed,
Some(&ctx),
last_run_id.clone(),
&health,
)?;
let mut summary = run_child_once(args, &ctx)?;
write_status(
args,
DaemonState::Collecting,
runs_completed,
Some(&ctx),
last_run_id.clone(),
&health,
)?;
collect_post_run_metrics(args, &ctx, &mut summary);
let removed = apply_retention(&args.state_root.join("runs"), args.retain_runs)?;
summary.retention_deleted_runs = removed.iter().map(|p| path_string(p)).collect();
summary.artifacts = collect_artifacts(&ctx.run_dir).unwrap_or_default();
write_json_pretty(&ctx.run_dir.join("run-summary.json"), &summary)?;
append_json_line(&summary_jsonl_path(args), &summary)?;
runs_completed += 1;
next_seq += 1;
last_run_id = Some(ctx.run_id);
if args.max_runs.is_some_and(|max| runs_completed >= max) {
break;
}
// Health probes run only while no child is active (single-writer).
maybe_run_dead_repo_health_check(args, &mut health);
write_status(
args,
DaemonState::Sleeping,
runs_completed,
None,
last_run_id.clone(),
&health,
)?;
if args.interval_secs > 0 {
std::thread::sleep(Duration::from_secs(args.interval_secs));
}
}
write_status(
args,
DaemonState::Exited,
runs_completed,
None,
last_run_id,
&health,
)
}
pub fn main_entry() -> i32 {
let argv: Vec<String> = std::env::args().collect();
match parse_args(&argv) {
Ok(args) => match run_daemon(&args) {
Ok(()) => 0,
Err(err) => {
eprintln!("{err}");
2
}
},
Err(err) => {
if argv.iter().any(|a| a == "--help" || a == "-h") {
println!("{err}");
return 0;
}
eprintln!("{err}");
2
}
}
}

View File

@@ -0,0 +1,92 @@
// One child validation run and summary assembly.
fn run_child_once(args: &Args, ctx: &RunContext) -> Result<RunSummary, String> {
fs::create_dir_all(&ctx.run_dir)
.map_err(|e| format!("create run dir failed: {}: {e}", ctx.run_dir.display()))?;
let started_at = utc_now();
let started_at_rfc3339_utc = format_rfc3339(started_at)?;
let stdout_path = ctx.run_dir.join("stdout.log");
let stderr_path = ctx.run_dir.join("stderr.log");
let stdout = File::create(&stdout_path)
.map_err(|e| format!("create stdout log failed: {}: {e}", stdout_path.display()))?;
let stderr = File::create(&stderr_path)
.map_err(|e| format!("create stderr log failed: {}: {e}", stderr_path.display()))?;
let child_args = render_child_args(&args.child_args, args, ctx);
let time_output_path = ctx.run_dir.join("process-time.txt");
let mut command = if let Some(time_bin) = args.time_bin.as_ref() {
let mut command = Command::new(time_bin);
command
.arg("-v")
.arg("-o")
.arg(&time_output_path)
.arg("--")
.arg(&args.rpki_bin)
.args(&child_args);
command
} else {
let mut command = Command::new(&args.rpki_bin);
command.args(&child_args);
command
};
command
.stdout(Stdio::from(stdout))
.stderr(Stdio::from(stderr));
let (status, exit_code, exit_status, error) = match command.status() {
Ok(status) if status.success() => (
RunStatus::Success,
status.code(),
Some(status.to_string()),
None,
),
Ok(status) => (
RunStatus::Failed,
status.code(),
Some(status.to_string()),
None,
),
Err(err) => (
RunStatus::SpawnFailed,
None,
None,
Some(format!("spawn child failed: {err}")),
),
};
let finished_at = utc_now();
let finished_at_rfc3339_utc = format_rfc3339(finished_at)?;
let wall_ms = (finished_at - started_at).whole_milliseconds().max(0) as u64;
let process_metrics = Some(collect_process_metrics(
args.time_bin.is_some(),
&time_output_path,
));
let artifacts = collect_artifacts(&ctx.run_dir)?;
let summary = RunSummary {
run_seq: ctx.seq,
run_id: ctx.run_id.clone(),
run_dir: path_string(&ctx.run_dir),
started_at_rfc3339_utc,
finished_at_rfc3339_utc,
wall_ms,
status,
exit_code,
exit_status,
error,
rpki_bin: path_string(&args.rpki_bin),
child_args,
stdout_path: path_string(&stdout_path),
stderr_path: path_string(&stderr_path),
process_metrics,
stage_timing: None,
report_counts: None,
repo_sync_stats: None,
path_stats: Vec::new(),
db_stats: Vec::new(),
retention_deleted_runs: Vec::new(),
artifacts,
dead_repo_blacklist: None,
};
Ok(summary)
}

View File

@@ -0,0 +1,404 @@
// Retention, report parsing, and post-run metric collection.
fn apply_retention(runs_root: &Path, retain_runs: usize) -> Result<Vec<PathBuf>, String> {
if !runs_root.exists() {
return Ok(Vec::new());
}
let mut dirs = Vec::new();
for entry in fs::read_dir(runs_root)
.map_err(|e| format!("read runs dir failed: {}: {e}", runs_root.display()))?
{
let entry = entry.map_err(|e| format!("read runs dir entry failed: {e}"))?;
if entry
.file_type()
.map_err(|e| format!("read file type failed: {}: {e}", entry.path().display()))?
.is_dir()
{
dirs.push(entry.path());
}
}
dirs.sort();
let remove_count = dirs.len().saturating_sub(retain_runs);
let mut removed = Vec::new();
for dir in dirs.into_iter().take(remove_count) {
fs::remove_dir_all(&dir)
.map_err(|e| format!("remove old run dir failed: {}: {e}", dir.display()))?;
removed.push(dir);
}
Ok(removed)
}
fn find_named_file(root: &Path, name: &str) -> Option<PathBuf> {
let mut stack = vec![root.to_path_buf()];
while let Some(dir) = stack.pop() {
let entries = fs::read_dir(&dir).ok()?;
for entry in entries.flatten() {
let path = entry.path();
let file_type = entry.file_type().ok()?;
if file_type.is_file() && entry.file_name().to_string_lossy() == name {
return Some(path);
}
if file_type.is_dir() {
stack.push(path);
}
}
}
None
}
fn read_json_value_if_exists(path: &Path) -> Option<serde_json::Value> {
let bytes = fs::read(path).ok()?;
serde_json::from_slice(&bytes).ok()
}
fn json_array_len(value: &serde_json::Value, key: &str) -> usize {
value
.get(key)
.and_then(serde_json::Value::as_array)
.map(Vec::len)
.unwrap_or(0)
}
fn parse_stdout_summary(run_dir: &Path) -> Option<ReportCounts> {
let stdout_path = run_dir.join("stdout.log");
let text = fs::read_to_string(stdout_path).ok()?;
let mut vrps = None;
let mut aspas = None;
let mut publication_points = None;
let mut rrdp_repos_unique = None;
let mut tree_instances_processed = None;
let mut tree_instances_failed = None;
let mut warnings = None;
for line in text.lines() {
if let Some(value) = line.strip_prefix("vrps=") {
vrps = value.trim().parse::<usize>().ok();
} else if let Some(value) = line.strip_prefix("aspas=") {
aspas = value.trim().parse::<usize>().ok();
} else if let Some(value) = line.strip_prefix("audit_publication_points=") {
publication_points = value.trim().parse::<usize>().ok();
} else if let Some(value) = line.strip_prefix("rrdp_repos_unique=") {
rrdp_repos_unique = value.trim().parse::<u64>().ok();
} else if let Some(value) = line.strip_prefix("warnings_total=") {
warnings = value.trim().parse::<usize>().ok();
} else if let Some(rest) = line.strip_prefix("publication_points_processed=") {
for token in rest.split_whitespace() {
if let Some(value) = token.strip_prefix("publication_points_failed=") {
tree_instances_failed = value.parse::<u64>().ok();
} else if tree_instances_processed.is_none() {
tree_instances_processed = token.parse::<u64>().ok();
}
}
}
}
Some(ReportCounts {
vrps: vrps?,
aspas: aspas?,
publication_points: publication_points?,
rrdp_repos_unique,
tree_instances_processed,
tree_instances_failed,
warnings: warnings.unwrap_or(0),
})
}
fn parse_report_counts_fallback(report: &serde_json::Value) -> ReportCounts {
let tree = report.get("tree");
let tree_warnings = tree
.and_then(|tree| tree.get("warnings"))
.and_then(serde_json::Value::as_array)
.map(Vec::len)
.unwrap_or(0);
let pp_warnings = report
.get("publication_points")
.and_then(serde_json::Value::as_array)
.map(|items| {
items
.iter()
.map(|pp| {
pp.get("warnings")
.and_then(serde_json::Value::as_array)
.map(Vec::len)
.unwrap_or(0)
})
.sum()
})
.unwrap_or(0);
ReportCounts {
vrps: json_array_len(report, "vrps"),
aspas: json_array_len(report, "aspas"),
publication_points: json_array_len(report, "publication_points"),
rrdp_repos_unique: None,
tree_instances_processed: tree
.and_then(|tree| tree.get("instances_processed"))
.and_then(serde_json::Value::as_u64),
tree_instances_failed: tree
.and_then(|tree| tree.get("instances_failed"))
.and_then(serde_json::Value::as_u64),
warnings: tree_warnings + pp_warnings,
}
}
fn find_subslice(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack
.windows(needle.len())
.position(|window| window == needle)
}
fn extract_json_object_field(path: &Path, field_name: &str) -> Option<serde_json::Value> {
let bytes = fs::read(path).ok()?;
let needle = format!("\"{field_name}\":");
let pos = find_subslice(&bytes, needle.as_bytes())?;
let mut i = pos + needle.len();
while i < bytes.len() && bytes[i].is_ascii_whitespace() {
i += 1;
}
if bytes.get(i).copied()? != b'{' {
return None;
}
let start = i;
let mut depth = 0u32;
let mut in_string = false;
let mut escaped = false;
for (offset, &b) in bytes[start..].iter().enumerate() {
if in_string {
if escaped {
escaped = false;
} else if b == b'\\' {
escaped = true;
} else if b == b'"' {
in_string = false;
}
continue;
}
match b {
b'"' => in_string = true,
b'{' => depth = depth.saturating_add(1),
b'}' => {
depth = depth.saturating_sub(1);
if depth == 0 {
let end = start + offset + 1;
return serde_json::from_slice(&bytes[start..end]).ok();
}
}
_ => {}
}
}
None
}
fn parse_report_metadata(run_dir: &Path) -> (Option<ReportCounts>, Option<serde_json::Value>) {
let Some(report_path) = find_named_file(run_dir, "report.json") else {
return (parse_stdout_summary(run_dir), None);
};
let counts = parse_stdout_summary(run_dir).or_else(|| {
read_json_value_if_exists(&report_path).map(|report| parse_report_counts_fallback(&report))
});
let repo_sync_stats = extract_json_object_field(&report_path, "repo_sync_stats");
(counts, repo_sync_stats)
}
fn collect_path_file_stats(label: &str, path: &Path) -> PathFileStats {
let mut stats = PathFileStats {
label: label.to_string(),
path: path_string(path),
exists: path.exists(),
is_dir: path.is_dir(),
total_size_bytes: 0,
file_count: 0,
dir_count: 0,
};
if !stats.exists {
return stats;
}
if path.is_file() {
if let Ok(metadata) = path.metadata() {
stats.total_size_bytes = metadata.len();
stats.file_count = 1;
}
return stats;
}
let mut stack = vec![path.to_path_buf()];
while let Some(dir) = stack.pop() {
let Ok(entries) = fs::read_dir(&dir) else {
continue;
};
for entry in entries.flatten() {
let Ok(file_type) = entry.file_type() else {
continue;
};
if file_type.is_dir() {
stats.dir_count = stats.dir_count.saturating_add(1);
stack.push(entry.path());
} else if file_type.is_file() {
stats.file_count = stats.file_count.saturating_add(1);
if let Ok(metadata) = entry.metadata() {
stats.total_size_bytes = stats.total_size_bytes.saturating_add(metadata.len());
}
}
}
}
stats
}
fn collect_state_path_stats(args: &Args, ctx: &RunContext) -> Vec<PathFileStats> {
let mut stats = Vec::new();
stats.push(collect_path_file_stats(
"work_db",
&render_path_template(&args.work_db, args, ctx),
));
if let Some(path) = args.repo_bytes_db.as_ref() {
stats.push(collect_path_file_stats(
"repo_bytes_db",
&render_path_template(path, args, ctx),
));
}
if let Some(path) = args.raw_store_db.as_ref() {
stats.push(collect_path_file_stats(
"raw_store_db",
&render_path_template(path, args, ctx),
));
}
stats
}
fn parse_key_value_metrics(text: &str) -> BTreeMap<String, String> {
let mut metrics = BTreeMap::new();
for line in text.lines() {
let Some((key, value)) = line.split_once('=') else {
continue;
};
metrics.insert(key.trim().to_string(), value.trim().to_string());
}
metrics
}
fn run_db_stats_command(
db_stats_bin: &Path,
db_path: &Path,
run_dir: &Path,
mode: &str,
) -> DbStatsSummary {
let output_path = run_dir.join(format!("db-stats-{mode}.txt"));
let stderr_path = run_dir.join(format!("db-stats-{mode}.stderr.txt"));
let mut summary = DbStatsSummary {
mode: mode.to_string(),
db_path: path_string(db_path),
output_path: Some(path_string(&output_path)),
stderr_path: None,
status: "success".to_string(),
exit_code: None,
error: None,
metrics: BTreeMap::new(),
};
if !db_path.exists() {
summary.status = "skipped".to_string();
summary.error = Some(format!("db path does not exist: {}", db_path.display()));
summary.output_path = None;
return summary;
}
let mut command = Command::new(db_stats_bin);
command.arg("--db").arg(db_path);
if mode == "exact" {
command.arg("--exact");
}
match command.output() {
Ok(output) => {
summary.exit_code = output.status.code();
if !output.status.success() {
summary.status = "failed".to_string();
}
let stdout_text = String::from_utf8_lossy(&output.stdout).into_owned();
if let Err(err) = fs::write(&output_path, stdout_text.as_bytes()) {
summary.status = "failed".to_string();
summary.error = Some(format!(
"write db_stats output failed: {}: {err}",
output_path.display()
));
}
summary.metrics = parse_key_value_metrics(&stdout_text);
if !output.stderr.is_empty() {
if fs::write(&stderr_path, &output.stderr).is_ok() {
summary.stderr_path = Some(path_string(&stderr_path));
}
}
if !output.status.success() && summary.error.is_none() {
summary.error = Some(String::from_utf8_lossy(&output.stderr).into_owned());
}
}
Err(err) => {
summary.status = "spawn_failed".to_string();
summary.exit_code = None;
summary.output_path = None;
summary.error = Some(format!("spawn db_stats failed: {err}"));
}
}
summary
}
fn collect_db_stats(args: &Args, ctx: &RunContext) -> Vec<DbStatsSummary> {
let work_db = render_path_template(&args.work_db, args, ctx);
let Some(db_stats_bin) = args
.db_stats_bin
.as_ref()
.cloned()
.or_else(default_db_stats_bin)
else {
return vec![DbStatsSummary {
mode: "estimate".to_string(),
db_path: path_string(&work_db),
output_path: None,
stderr_path: None,
status: "skipped".to_string(),
exit_code: None,
error: Some(
"db_stats binary not configured and sibling db_stats was not found".to_string(),
),
metrics: BTreeMap::new(),
}];
};
let mut stats = Vec::new();
stats.push(run_db_stats_command(
&db_stats_bin,
&work_db,
&ctx.run_dir,
"estimate",
));
if args
.db_stats_exact_every
.is_some_and(|every| ctx.seq % every == 0)
{
stats.push(run_db_stats_command(
&db_stats_bin,
&work_db,
&ctx.run_dir,
"exact",
));
}
stats
}
fn collect_post_run_metrics(args: &Args, ctx: &RunContext, summary: &mut RunSummary) {
if let Some(path) = find_named_file(&ctx.run_dir, "stage-timing.json") {
summary.stage_timing = read_json_value_if_exists(&path);
}
let (report_counts, repo_sync_stats) = parse_report_metadata(&ctx.run_dir);
summary.report_counts = report_counts;
summary.repo_sync_stats = repo_sync_stats;
summary.path_stats = collect_state_path_stats(args, ctx);
summary.db_stats = collect_db_stats(args, ctx);
summary.artifacts = collect_artifacts(&ctx.run_dir).unwrap_or_default();
if let Some(path) = args.dead_repo_blacklist.as_ref() {
let (blacklist, _) = DeadRepoBlacklist::load(path);
summary.dead_repo_blacklist = Some(DeadRepoBlacklistRunSummary {
path: path_string(path),
entries: blacklist.len(),
blacklisted: blacklist.blacklisted_len(),
});
}
}

View File

@@ -0,0 +1,299 @@
// Health probes, child argument rendering, and process metrics.
fn append_json_line<T: Serialize>(path: &Path, value: &T) -> Result<(), String> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)
.map_err(|e| format!("create parent dir failed: {}: {e}", parent.display()))?;
}
let mut file = OpenOptions::new()
.create(true)
.append(true)
.open(path)
.map_err(|e| format!("open jsonl failed: {}: {e}", path.display()))?;
serde_json::to_writer(&mut file, value)
.map_err(|e| format!("write jsonl failed: {}: {e}", path.display()))?;
file.write_all(b"\n")
.map_err(|e| format!("flush jsonl failed: {}: {e}", path.display()))
}
fn dead_repo_blacklist_status(
args: &Args,
health: &DeadRepoHealthRuntime,
) -> Option<DeadRepoBlacklistStatus> {
let path = args.dead_repo_blacklist.as_ref()?;
let (blacklist, _) = DeadRepoBlacklist::load(path);
let last_health_check_at_rfc3339_utc = health
.last_health_check_at
.and_then(|t| format_rfc3339(t).ok());
Some(DeadRepoBlacklistStatus {
path: path_string(path),
entries: blacklist.len(),
blacklisted: blacklist.blacklisted_len(),
last_health_check_at_rfc3339_utc,
})
}
fn write_status(
args: &Args,
state: DaemonState,
runs_completed: u64,
current: Option<&RunContext>,
last_run_id: Option<String>,
health: &DeadRepoHealthRuntime,
) -> Result<(), String> {
let updated_at_rfc3339_utc = format_rfc3339(utc_now())?;
let status = DaemonStatus {
state,
updated_at_rfc3339_utc,
runs_completed,
max_runs: args.max_runs,
current_run_seq: current.map(|ctx| ctx.seq),
current_run_id: current.map(|ctx| ctx.run_id.clone()),
last_run_id,
dead_repo_blacklist: dead_repo_blacklist_status(args, health),
};
write_json_pretty(&status_path(args), &status)
}
/// Short-timeout HTTP GET of the RRDP notification file. Any HTTP status
/// (even 4xx) proves the transport is alive; only transport errors keep the
/// entry blacklisted.
fn probe_rrdp_transport(uri: &str) -> bool {
let config = crate::fetch::http::HttpFetcherConfig {
connect_timeout: Duration::from_secs(3),
timeout: Duration::from_secs(6),
large_body_timeout: Duration::from_secs(6),
..Default::default()
};
let fetcher = match crate::fetch::http::BlockingHttpFetcher::new(config) {
Ok(fetcher) => fetcher,
Err(_) => return false,
};
match fetcher.fetch_bytes(uri) {
Ok(_) => true,
Err(err) => err.starts_with("http status"),
}
}
/// rsync list-only probe: connecting to the daemon and listing the module
/// root is enough to prove the transport is alive.
fn probe_rsync_transport(rsync_bin: &Path, base_uri: &str) -> bool {
Command::new(rsync_bin)
.arg("--contimeout=5")
.arg("--timeout=8")
.arg(base_uri)
.stdout(Stdio::null())
.stderr(Stdio::null())
.status()
.map(|status| status.success())
.unwrap_or(false)
}
/// Probe due blacklist entries between runs. Runs in the supervisor
/// loop while no child is active, preserving the single-writer assumption.
fn maybe_run_dead_repo_health_check(args: &Args, health: &mut DeadRepoHealthRuntime) {
let Some(path) = args.dead_repo_blacklist.clone() else {
return;
};
let now = utc_now();
if let Some(last) = health.last_health_check_at {
let elapsed = (now - last).whole_seconds();
if elapsed >= 0 && (elapsed as u64) < args.dead_repo_health_check_interval_secs {
return;
}
}
let (mut blacklist, warning) = DeadRepoBlacklist::load(&path);
if let Some(warning) = warning {
eprintln!("[dead-repo-health] {warning}");
}
let now_unix = now.unix_timestamp().max(0) as u64;
let due = blacklist.probe_due_entries(now_unix, args.dead_repo_health_check_interval_secs);
if due.is_empty() {
health.last_health_check_at = Some(now);
return;
}
// Probe in parallel: in the docker soak the daemon runs once per run
// (--max-runs 1), so a serial sweep of ~12 dead entries would add up to a
// minute of latency to every run.
let rsync_bin = args.dead_repo_probe_rsync_bin.clone();
let probe_results: Vec<(RepoTransportMode, String, bool)> = due
.iter()
.map(|entry| (entry.transport, entry.uri.clone()))
.collect::<Vec<_>>()
.into_iter()
.map(|(transport, uri)| {
let rsync_bin = rsync_bin.clone();
std::thread::spawn(move || {
let alive = match transport {
RepoTransportMode::Rrdp => probe_rrdp_transport(&uri),
RepoTransportMode::Rsync => probe_rsync_transport(&rsync_bin, &uri),
};
(transport, uri, alive)
})
})
.filter_map(|handle| handle.join().ok())
.collect();
let mut removed = 0usize;
let mut still_dead = 0usize;
for (transport, uri, alive) in &probe_results {
if *alive {
blacklist.record_probe_success(*transport, uri);
removed += 1;
eprintln!(
"[dead-repo-health] removed {} {} (probe ok)",
transport.as_str(),
uri
);
crate::progress_log::emit(
"dead_repo_blacklist_remove",
serde_json::json!({
"transport": transport.as_str(),
"uri": uri,
"reason": "health_probe_ok",
}),
);
} else {
blacklist.record_probe_failure(*transport, uri, now_unix);
still_dead += 1;
eprintln!(
"[dead-repo-health] still dead {} {} (probe failed)",
transport.as_str(),
uri
);
}
}
if let Err(err) = blacklist.store_atomic(&path, now_unix) {
eprintln!(
"[dead-repo-health] persist blacklist failed: {}: {err}",
path.display()
);
}
crate::progress_log::emit(
"dead_repo_health_check",
serde_json::json!({
"probed": probe_results.len(),
"removed": removed,
"still_dead": still_dead,
}),
);
health.last_health_check_at = Some(now);
}
fn render_child_args(args: &[String], daemon_args: &Args, ctx: &RunContext) -> Vec<String> {
args.iter()
.map(|arg| {
arg.replace("{state_root}", &path_string(&daemon_args.state_root))
.replace("{run_out}", &path_string(&ctx.run_dir))
.replace("{run_id}", &ctx.run_id)
.replace("{run_seq}", &ctx.seq.to_string())
})
.collect()
}
fn render_path_template(path: &Path, daemon_args: &Args, ctx: &RunContext) -> PathBuf {
PathBuf::from(
path_string(path)
.replace("{state_root}", &path_string(&daemon_args.state_root))
.replace("{run_out}", &path_string(&ctx.run_dir))
.replace("{run_id}", &ctx.run_id)
.replace("{run_seq}", &ctx.seq.to_string()),
)
}
fn path_string(path: &Path) -> String {
path.to_string_lossy().into_owned()
}
fn make_run_context(args: &Args, seq: u64, now: time::OffsetDateTime) -> RunContext {
let run_id = format!("{seq:06}-{}", format_compact_utc(now));
let run_dir = args.state_root.join("runs").join(&run_id);
RunContext {
seq,
run_id,
run_dir,
}
}
fn collect_artifacts(run_dir: &Path) -> Result<Vec<ArtifactInfo>, String> {
let mut artifacts = Vec::new();
for entry in fs::read_dir(run_dir)
.map_err(|e| format!("read run dir failed: {}: {e}", run_dir.display()))?
{
let entry = entry.map_err(|e| format!("read run dir entry failed: {e}"))?;
if !entry
.file_type()
.map_err(|e| format!("read file type failed: {}: {e}", entry.path().display()))?
.is_file()
{
continue;
}
let metadata = entry
.metadata()
.map_err(|e| format!("read metadata failed: {}: {e}", entry.path().display()))?;
artifacts.push(ArtifactInfo {
path: path_string(&entry.path()),
size_bytes: metadata.len(),
});
}
artifacts.sort_by(|a, b| a.path.cmp(&b.path));
Ok(artifacts)
}
fn collect_process_metrics(time_wrapper_used: bool, time_output_path: &Path) -> ProcessMetrics {
if !time_wrapper_used {
return ProcessMetrics {
time_wrapper_used,
time_output_path: None,
user_seconds: None,
system_seconds: None,
cpu_percent: None,
elapsed_raw: None,
max_rss_kb: None,
exit_status_from_time: None,
parse_error: None,
};
}
let mut metrics = ProcessMetrics {
time_wrapper_used,
time_output_path: Some(path_string(time_output_path)),
user_seconds: None,
system_seconds: None,
cpu_percent: None,
elapsed_raw: None,
max_rss_kb: None,
exit_status_from_time: None,
parse_error: None,
};
let text = match fs::read_to_string(time_output_path) {
Ok(text) => text,
Err(err) => {
metrics.parse_error = Some(format!(
"read process time output failed: {}: {err}",
time_output_path.display()
));
return metrics;
}
};
for line in text.lines() {
let line = line.trim();
if let Some(value) = line.strip_prefix("User time (seconds):") {
metrics.user_seconds = value.trim().parse::<f64>().ok();
} else if let Some(value) = line.strip_prefix("System time (seconds):") {
metrics.system_seconds = value.trim().parse::<f64>().ok();
} else if let Some(value) = line.strip_prefix("Percent of CPU this job got:") {
metrics.cpu_percent = value.trim().trim_end_matches('%').parse::<f64>().ok();
} else if let Some(value) =
line.strip_prefix("Elapsed (wall clock) time (h:mm:ss or m:ss):")
{
metrics.elapsed_raw = Some(value.trim().to_string());
} else if let Some(value) = line.strip_prefix("Maximum resident set size (kbytes):") {
metrics.max_rss_kb = value.trim().parse::<u64>().ok();
} else if let Some(value) = line.strip_prefix("Exit status:") {
metrics.exit_status_from_time = value.trim().parse::<i32>().ok();
}
}
metrics
}

View File

@@ -0,0 +1,785 @@
use super::*;
fn test_args(state_root: PathBuf) -> Args {
Args {
work_db: state_root.join("state/work-db"),
repo_bytes_db: Some(state_root.join("state/repo-bytes.db")),
state_root,
rpki_bin: PathBuf::from("/bin/true"),
interval_secs: 0,
max_runs: Some(1),
retain_runs: 10,
status_json: None,
summary_jsonl: None,
raw_store_db: None,
db_stats_bin: None,
db_stats_exact_every: None,
time_bin: None,
dead_repo_blacklist: None,
dead_repo_blacklist_fail_threshold: None,
dead_repo_health_check_interval_secs: 600,
dead_repo_probe_rsync_bin: PathBuf::from("rsync"),
child_args: vec!["--version".to_string()],
}
}
#[test]
fn parse_args_accepts_required_flags_and_child_args() {
let argv = vec![
"rpki_daemon".to_string(),
"--state-root".to_string(),
"/tmp/daemon".to_string(),
"--rpki-bin".to_string(),
"/bin/echo".to_string(),
"--interval-secs".to_string(),
"0".to_string(),
"--max-runs".to_string(),
"2".to_string(),
"--retain-runs".to_string(),
"3".to_string(),
"--".to_string(),
"--db".to_string(),
"{state_root}/state/work-db".to_string(),
"--report-json".to_string(),
"{run_out}/report.json".to_string(),
];
let args = parse_args(&argv).expect("parse args");
assert_eq!(args.state_root, PathBuf::from("/tmp/daemon"));
assert_eq!(args.rpki_bin, PathBuf::from("/bin/echo"));
assert_eq!(args.interval_secs, 0);
assert_eq!(args.max_runs, Some(2));
assert_eq!(args.retain_runs, 3);
assert_eq!(args.work_db, PathBuf::from("/tmp/daemon/state/work-db"));
assert_eq!(
args.repo_bytes_db,
Some(PathBuf::from("/tmp/daemon/state/repo-bytes.db"))
);
assert_eq!(
args.child_args,
vec![
"--db",
"{state_root}/state/work-db",
"--report-json",
"{run_out}/report.json"
]
);
}
#[test]
fn usage_and_parse_args_cover_optional_flags_and_errors() {
let help = usage();
assert!(help.contains("--db-stats-exact-every"));
assert!(help.contains("{run_seq}"));
let argv = vec![
"rpki_daemon".to_string(),
"--state-root".to_string(),
"/tmp/daemon".to_string(),
"--rpki-bin".to_string(),
"/bin/echo".to_string(),
"--status-json".to_string(),
"/tmp/status.json".to_string(),
"--summary-jsonl".to_string(),
"/tmp/runs.jsonl".to_string(),
"--work-db".to_string(),
"{state_root}/work".to_string(),
"--repo-bytes-db".to_string(),
"{state_root}/repo-bytes".to_string(),
"--raw-store-db".to_string(),
"{state_root}/raw".to_string(),
"--db-stats-bin".to_string(),
"/bin/echo".to_string(),
"--db-stats-exact-every".to_string(),
"2".to_string(),
"--time-bin".to_string(),
"/usr/bin/time".to_string(),
"--no-time-wrapper".to_string(),
"--".to_string(),
"child".to_string(),
];
let args = parse_args(&argv).expect("optional args");
assert_eq!(args.status_json, Some(PathBuf::from("/tmp/status.json")));
assert_eq!(args.summary_jsonl, Some(PathBuf::from("/tmp/runs.jsonl")));
assert_eq!(args.work_db, PathBuf::from("{state_root}/work"));
assert_eq!(
args.repo_bytes_db,
Some(PathBuf::from("{state_root}/repo-bytes"))
);
assert_eq!(args.raw_store_db, Some(PathBuf::from("{state_root}/raw")));
assert_eq!(args.db_stats_bin, Some(PathBuf::from("/bin/echo")));
assert_eq!(args.db_stats_exact_every, Some(2));
assert_eq!(args.time_bin, None);
for (argv, expected) in [
(vec!["rpki_daemon", "--help"], "Usage:"),
(
vec![
"rpki_daemon",
"--state-root",
"/tmp/x",
"--rpki-bin",
"/bin/echo",
"--max-runs",
"0",
"--",
"child",
],
"--max-runs must be > 0",
),
(
vec![
"rpki_daemon",
"--state-root",
"/tmp/x",
"--rpki-bin",
"/bin/echo",
"--retain-runs",
"0",
"--",
"child",
],
"--retain-runs must be > 0",
),
(
vec![
"rpki_daemon",
"--state-root",
"/tmp/x",
"--rpki-bin",
"/bin/echo",
"--db-stats-exact-every",
"0",
"--",
"child",
],
"--db-stats-exact-every must be > 0",
),
(vec!["rpki_daemon", "--unknown"], "unknown argument"),
(
vec!["rpki_daemon", "--state-root"],
"--state-root requires a value",
),
(vec!["rpki_daemon"], "missing -- before child rpki args"),
(
vec![
"rpki_daemon",
"--state-root",
"/tmp/x",
"--rpki-bin",
"/bin/echo",
"--",
],
"child rpki args are required",
),
] {
let owned: Vec<String> = argv.into_iter().map(str::to_string).collect();
let err = parse_args(&owned).expect_err("parse should fail");
assert!(err.contains(expected), "{err}");
}
}
#[test]
fn render_child_args_replaces_placeholders() {
let args = Args {
state_root: PathBuf::from("/tmp/root"),
rpki_bin: PathBuf::from("/bin/echo"),
interval_secs: 0,
max_runs: Some(1),
retain_runs: 10,
status_json: None,
summary_jsonl: None,
work_db: PathBuf::from("/tmp/root/state/work-db"),
repo_bytes_db: Some(PathBuf::from("/tmp/root/state/repo-bytes.db")),
raw_store_db: None,
db_stats_bin: None,
db_stats_exact_every: None,
time_bin: None,
dead_repo_blacklist: None,
dead_repo_blacklist_fail_threshold: None,
dead_repo_health_check_interval_secs: 600,
dead_repo_probe_rsync_bin: PathBuf::from("rsync"),
child_args: vec![
"{state_root}/state/work-db".to_string(),
"{run_out}/result.ccr".to_string(),
"{run_id}".to_string(),
"{run_seq}".to_string(),
],
};
let ctx = RunContext {
seq: 7,
run_id: "000007-20260428T090000Z".to_string(),
run_dir: PathBuf::from("/tmp/root/runs/000007-20260428T090000Z"),
};
assert_eq!(
render_child_args(&args.child_args, &args, &ctx),
vec![
"/tmp/root/state/work-db",
"/tmp/root/runs/000007-20260428T090000Z/result.ccr",
"000007-20260428T090000Z",
"7",
]
);
}
#[test]
fn path_json_and_report_helpers_cover_fallbacks_and_nested_stats() {
let td = tempfile::tempdir().expect("tempdir");
let state_root = td.path().join("daemon");
let mut args = test_args(state_root.clone());
args.work_db = PathBuf::from("{state_root}/state/work-db");
args.repo_bytes_db = Some(PathBuf::from("{state_root}/state/repo-bytes.db"));
args.raw_store_db = Some(PathBuf::from("{state_root}/state/raw-store.db"));
let now = time::Date::from_calendar_date(2026, time::Month::April, 28)
.expect("date")
.with_hms(9, 0, 0)
.expect("time")
.assume_utc();
let ctx = make_run_context(&args, 42, now);
assert_eq!(ctx.run_id, "000042-20260428T090000Z");
let rendered = render_path_template(Path::new("{run_out}/{run_id}/{run_seq}"), &args, &ctx);
assert!(rendered.ends_with("000042-20260428T090000Z/000042-20260428T090000Z/42"));
let nested_json = td.path().join("nested/out/status.json");
write_json_pretty(&nested_json, &serde_json::json!({"ok": true})).expect("write json");
append_json_line(
&td.path().join("nested/out/runs.jsonl"),
&serde_json::json!({"n": 1}),
)
.expect("append jsonl");
fs::create_dir_all(ctx.run_dir.join("subdir")).expect("subdir");
fs::write(ctx.run_dir.join("a.txt"), "aaa").expect("file");
fs::write(ctx.run_dir.join("subdir/ignored.txt"), "bbb").expect("nested file");
let artifacts = collect_artifacts(&ctx.run_dir).expect("artifacts");
assert_eq!(artifacts.len(), 1);
assert!(artifacts[0].path.ends_with("a.txt"));
fs::create_dir_all(state_root.join("state/work-db/nested")).expect("work db");
fs::write(state_root.join("state/work-db/file.sst"), "abc").expect("sst");
fs::write(state_root.join("state/work-db/nested/inner.sst"), "def").expect("inner");
fs::write(state_root.join("state/raw-store.db"), "raw").expect("raw file");
let file_stats =
collect_path_file_stats("raw_store_db", &state_root.join("state/raw-store.db"));
assert!(file_stats.exists);
assert!(!file_stats.is_dir);
assert_eq!(file_stats.file_count, 1);
let missing_stats = collect_path_file_stats("missing", &state_root.join("missing"));
assert!(!missing_stats.exists);
let state_stats = collect_state_path_stats(&args, &ctx);
assert!(
state_stats
.iter()
.any(|s| s.label == "raw_store_db" && s.exists)
);
assert!(
state_stats
.iter()
.any(|s| s.label == "work_db" && s.file_count == 2)
);
let report_path = td.path().join("report.json");
fs::write(
&report_path,
r#"{"repo_sync_stats": { "nested": {"text": "a\"b"} }, "after": 1}"#,
)
.expect("report");
assert_eq!(
extract_json_object_field(&report_path, "repo_sync_stats").expect("repo stats")["nested"]["text"],
"a\"b"
);
fs::write(&report_path, r#"{"repo_sync_stats": []}"#).expect("report");
assert!(extract_json_object_field(&report_path, "repo_sync_stats").is_none());
assert!(extract_json_object_field(&report_path, "missing").is_none());
let counts = parse_report_counts_fallback(&serde_json::json!({
"vrps": [{}, {}],
"aspas": [{}],
"publication_points": [{"warnings": [{}, {}]}, {"warnings": [{}]}],
"tree": {"instances_processed": 2, "instances_failed": 1, "warnings": [{}]}
}));
assert_eq!(counts.vrps, 2);
assert_eq!(counts.aspas, 1);
assert_eq!(counts.publication_points, 2);
assert_eq!(counts.warnings, 4);
assert_eq!(counts.tree_instances_processed, Some(2));
assert_eq!(counts.tree_instances_failed, Some(1));
}
#[test]
fn retention_removes_oldest_run_directories() {
let td = tempfile::tempdir().expect("tempdir");
let runs = td.path().join("runs");
fs::create_dir_all(&runs).expect("runs dir");
for name in [
"000001-20260428T000001Z",
"000002-20260428T000002Z",
"000003-20260428T000003Z",
] {
fs::create_dir_all(runs.join(name)).expect("run dir");
}
let removed = apply_retention(&runs, 2).expect("retention");
assert_eq!(removed.len(), 1);
assert!(!runs.join("000001-20260428T000001Z").exists());
assert!(runs.join("000002-20260428T000002Z").exists());
assert!(runs.join("000003-20260428T000003Z").exists());
}
#[test]
fn retention_empty_root_and_parse_metrics_error_paths_are_reported() {
let td = tempfile::tempdir().expect("tempdir");
let missing_runs = td.path().join("missing-runs");
assert!(
apply_retention(&missing_runs, 2)
.expect("empty retention")
.is_empty()
);
let disabled = collect_process_metrics(false, &td.path().join("missing-time.txt"));
assert!(!disabled.time_wrapper_used);
assert!(disabled.time_output_path.is_none());
let missing = collect_process_metrics(true, &td.path().join("missing-time.txt"));
assert!(missing.time_wrapper_used);
assert!(
missing
.parse_error
.expect("parse error")
.contains("read process time output failed")
);
}
#[test]
fn process_metrics_parses_gnu_time_elapsed_line() {
let td = tempfile::tempdir().expect("tempdir");
let path = td.path().join("time.txt");
fs::write(
&path,
"User time (seconds): 1.25\nSystem time (seconds): 0.50\nPercent of CPU this job got: 175%\nElapsed (wall clock) time (h:mm:ss or m:ss): 0:01.00\nMaximum resident set size (kbytes): 12345\nExit status: 0\n",
)
.expect("write time");
let metrics = collect_process_metrics(true, &path);
assert_eq!(metrics.user_seconds, Some(1.25));
assert_eq!(metrics.system_seconds, Some(0.50));
assert_eq!(metrics.cpu_percent, Some(175.0));
assert_eq!(metrics.elapsed_raw.as_deref(), Some("0:01.00"));
assert_eq!(metrics.max_rss_kb, Some(12345));
assert_eq!(metrics.exit_status_from_time, Some(0));
}
#[test]
#[cfg(unix)]
fn child_and_db_stats_error_paths_are_reported() {
use std::os::unix::fs::PermissionsExt;
let td = tempfile::tempdir().expect("tempdir");
let run_dir = td.path().join("run");
fs::create_dir_all(&run_dir).expect("run dir");
let missing_db = td.path().join("missing-db");
let skipped = run_db_stats_command(Path::new("/bin/echo"), &missing_db, &run_dir, "estimate");
assert_eq!(skipped.status, "skipped");
assert!(skipped.output_path.is_none());
let db_path = td.path().join("db");
fs::create_dir_all(&db_path).expect("db");
let failing_bin = td.path().join("failing_db_stats.sh");
fs::write(
&failing_bin,
"#!/bin/sh\necho partial=1\necho db-stats failed >&2\nexit 7\n",
)
.expect("script");
let mut permissions = fs::metadata(&failing_bin).expect("metadata").permissions();
permissions.set_mode(0o755);
fs::set_permissions(&failing_bin, permissions).expect("chmod");
let failed = run_db_stats_command(&failing_bin, &db_path, &run_dir, "exact");
assert_eq!(failed.status, "failed");
assert_eq!(failed.exit_code, Some(7));
assert_eq!(failed.metrics.get("partial").map(String::as_str), Some("1"));
assert!(failed.stderr_path.is_some());
assert!(failed.error.expect("stderr").contains("db-stats failed"));
let spawn_failed = run_db_stats_command(
Path::new("/definitely/not/db_stats"),
&db_path,
&run_dir,
"estimate",
);
assert_eq!(spawn_failed.status, "spawn_failed");
assert!(
spawn_failed
.error
.expect("spawn err")
.contains("spawn db_stats failed")
);
let metrics = parse_key_value_metrics("alpha=1\nnot-a-pair\nbeta = two\n");
assert_eq!(metrics.get("alpha").map(String::as_str), Some("1"));
assert_eq!(metrics.get("beta").map(String::as_str), Some("two"));
let mut args = test_args(td.path().join("daemon"));
args.rpki_bin = PathBuf::from("/bin/sh");
args.time_bin = Some(PathBuf::from("/usr/bin/time"));
args.child_args = vec![
"-c".to_string(),
"echo child-out; echo child-err >&2; exit 7".to_string(),
];
let ctx = RunContext {
seq: 1,
run_id: "000001-20260428T000000Z".to_string(),
run_dir: args.state_root.join("runs/000001-20260428T000000Z"),
};
let summary = run_child_once(&args, &ctx).expect("run failing child");
assert_eq!(summary.status, RunStatus::Failed);
assert_eq!(summary.exit_code, Some(7));
let process_metrics = summary.process_metrics.expect("process metrics");
assert!(process_metrics.time_wrapper_used);
assert_eq!(process_metrics.exit_status_from_time, Some(7));
let mut spawn_args = test_args(td.path().join("spawn"));
spawn_args.rpki_bin = PathBuf::from("/definitely/not/rpki");
spawn_args.child_args = vec!["--help".to_string()];
let spawn_ctx = RunContext {
seq: 1,
run_id: "000001-20260428T000001Z".to_string(),
run_dir: spawn_args.state_root.join("runs/000001-20260428T000001Z"),
};
let spawn_summary = run_child_once(&spawn_args, &spawn_ctx).expect("spawn summary");
assert_eq!(spawn_summary.status, RunStatus::SpawnFailed);
assert!(
spawn_summary
.error
.expect("spawn error")
.contains("spawn child failed")
);
}
#[test]
fn report_metadata_prefers_stdout_summary_and_extracts_repo_sync_stats() {
let td = tempfile::tempdir().expect("tempdir");
fs::write(
td.path().join("stdout.log"),
"RPKI validation run summary\npublication_points_processed=7 publication_points_failed=1\nrrdp_repos_unique=3\nvrps=11\naspas=2\naudit_publication_points=7\nwarnings_total=5\n",
)
.expect("stdout");
fs::write(
td.path().join("report.json"),
"{\"large\":[{\"ignored\":\"{}\"}],\"repo_sync_stats\":{\"by_phase\":{\"rrdp_ok\":{\"count\":7}},\"by_terminal_state\":{},\"publication_points_total\":7}}",
)
.expect("report");
let (counts, repo_sync_stats) = parse_report_metadata(td.path());
let counts = counts.expect("counts");
assert_eq!(counts.vrps, 11);
assert_eq!(counts.aspas, 2);
assert_eq!(counts.publication_points, 7);
assert_eq!(counts.rrdp_repos_unique, Some(3));
assert_eq!(counts.tree_instances_processed, Some(7));
assert_eq!(counts.tree_instances_failed, Some(1));
assert_eq!(counts.warnings, 5);
assert_eq!(
repo_sync_stats.expect("repo sync")["by_phase"]["rrdp_ok"]["count"].as_u64(),
Some(7)
);
}
#[test]
fn daemon_exits_immediately_when_max_runs_already_reached() {
let td = tempfile::tempdir().expect("tempdir");
let mut args = test_args(td.path().join("daemon"));
args.max_runs = Some(0);
run_daemon(&args).expect("run daemon");
let status_text =
fs::read_to_string(args.state_root.join("daemon-status.json")).expect("status");
assert!(status_text.contains("\"state\": \"exited\""));
assert!(status_text.contains("\"runsCompleted\": 0"));
assert!(args.state_root.join("runs").exists());
}
#[test]
fn daemon_runs_fake_child_twice_and_writes_summaries() {
let td = tempfile::tempdir().expect("tempdir");
let args = Args {
state_root: td.path().join("daemon"),
rpki_bin: PathBuf::from("/bin/sh"),
interval_secs: 0,
max_runs: Some(2),
retain_runs: 10,
status_json: None,
summary_jsonl: None,
work_db: td.path().join("daemon/state/work-db"),
repo_bytes_db: Some(td.path().join("daemon/state/repo-bytes.db")),
raw_store_db: None,
db_stats_bin: None,
db_stats_exact_every: None,
time_bin: None,
dead_repo_blacklist: None,
dead_repo_blacklist_fail_threshold: None,
dead_repo_health_check_interval_secs: 600,
dead_repo_probe_rsync_bin: PathBuf::from("rsync"),
child_args: vec![
"-c".to_string(),
"echo stdout-{run_seq}; echo stderr-{run_seq} >&2; echo marker > {run_out}/marker.txt"
.to_string(),
],
};
run_daemon(&args).expect("run daemon");
let status_text =
fs::read_to_string(args.state_root.join("daemon-status.json")).expect("status");
assert!(status_text.contains("\"state\": \"exited\""));
assert!(status_text.contains("\"runsCompleted\": 2"));
let jsonl =
fs::read_to_string(args.state_root.join("daemon-runs.jsonl")).expect("summary jsonl");
assert_eq!(jsonl.lines().count(), 2);
let run_dirs = fs::read_dir(args.state_root.join("runs"))
.expect("runs dir")
.collect::<Result<Vec<_>, _>>()
.expect("entries");
assert_eq!(run_dirs.len(), 2);
for entry in run_dirs {
assert!(entry.path().join("run-summary.json").exists());
assert!(entry.path().join("marker.txt").exists());
}
}
#[test]
fn daemon_sleep_and_retention_deletion_are_recorded() {
let td = tempfile::tempdir().expect("tempdir");
let mut args = test_args(td.path().join("daemon"));
args.rpki_bin = PathBuf::from("/bin/sh");
args.interval_secs = 1;
args.max_runs = Some(2);
args.retain_runs = 1;
args.child_args = vec![
"-c".to_string(),
"echo run-{run_seq}; printf marker > {run_out}/marker.txt".to_string(),
];
run_daemon(&args).expect("run daemon");
let jsonl = fs::read_to_string(args.state_root.join("daemon-runs.jsonl")).expect("jsonl");
assert_eq!(jsonl.lines().count(), 2);
let last: serde_json::Value =
serde_json::from_str(jsonl.lines().last().expect("last line")).expect("summary");
assert_eq!(
last["retentionDeletedRuns"]
.as_array()
.expect("deleted")
.len(),
1
);
let run_dirs = fs::read_dir(args.state_root.join("runs"))
.expect("runs")
.collect::<Result<Vec<_>, _>>()
.expect("entries");
assert_eq!(run_dirs.len(), 1);
}
#[test]
#[cfg(unix)]
fn daemon_collects_stage_report_db_and_file_metrics() {
use std::os::unix::fs::PermissionsExt;
let td = tempfile::tempdir().expect("tempdir");
let db_stats_bin = td.path().join("fake_db_stats.sh");
fs::write(
&db_stats_bin,
"#!/bin/sh\nif [ \"$3\" = \"--exact\" ]; then echo mode=exact; else echo mode=estimate; fi\necho total=42\necho db.files.total_size_bytes=123\n",
)
.expect("fake db_stats");
let mut permissions = fs::metadata(&db_stats_bin).expect("metadata").permissions();
permissions.set_mode(0o755);
fs::set_permissions(&db_stats_bin, permissions).expect("chmod");
let args = Args {
state_root: td.path().join("daemon"),
rpki_bin: PathBuf::from("/bin/sh"),
interval_secs: 0,
max_runs: Some(1),
retain_runs: 10,
status_json: None,
summary_jsonl: None,
work_db: PathBuf::from("{state_root}/state/work-db"),
repo_bytes_db: Some(PathBuf::from("{state_root}/state/repo-bytes.db")),
raw_store_db: None,
db_stats_bin: Some(db_stats_bin),
db_stats_exact_every: Some(1),
time_bin: None,
dead_repo_blacklist: None,
dead_repo_blacklist_fail_threshold: None,
dead_repo_health_check_interval_secs: 600,
dead_repo_probe_rsync_bin: PathBuf::from("rsync"),
child_args: vec![
"-c".to_string(),
"mkdir -p {state_root}/state/work-db {state_root}/state/repo-bytes.db; \
printf x > {state_root}/state/work-db/000001.sst; \
printf '{\"validation_ms\":7,\"download_event_count\":2}' > {run_out}/stage-timing.json; \
printf '{\"tree\":{\"instances_processed\":3,\"instances_failed\":1,\"warnings\":[{}]},\"publication_points\":[{},{}],\"vrps\":[{},{}],\"aspas\":[{}],\"repo_sync_stats\":{\"by_phase\":{\"snapshot\":{\"count\":1}}}}' > {run_out}/report.json"
.to_string(),
],
};
run_daemon(&args).expect("run daemon");
let run_summary =
find_named_file(&args.state_root.join("runs"), "run-summary.json").expect("run summary");
let summary: serde_json::Value =
serde_json::from_slice(&fs::read(run_summary).expect("read run summary"))
.expect("parse run summary");
assert_eq!(summary["stageTiming"]["validation_ms"].as_u64(), Some(7));
assert_eq!(summary["reportCounts"]["vrps"].as_u64(), Some(2));
assert_eq!(summary["reportCounts"]["aspas"].as_u64(), Some(1));
assert_eq!(
summary["reportCounts"]["publicationPoints"].as_u64(),
Some(2)
);
assert_eq!(
summary["repoSyncStats"]["by_phase"]["snapshot"]["count"].as_u64(),
Some(1)
);
assert_eq!(summary["dbStats"].as_array().expect("db stats").len(), 2);
assert!(
summary["pathStats"]
.as_array()
.expect("path stats")
.iter()
.any(|item| item["label"] == "work_db" && item["exists"] == true)
);
}
#[test]
fn dead_repo_blacklist_flag_is_injected_into_child_args() {
let argv = vec![
"rpki_daemon".to_string(),
"--state-root".to_string(),
"/tmp/daemon".to_string(),
"--rpki-bin".to_string(),
"/bin/echo".to_string(),
"--dead-repo-blacklist".to_string(),
"/tmp/daemon/state/dead-repo-blacklist.json".to_string(),
"--dead-repo-blacklist-fail-threshold".to_string(),
"2".to_string(),
"--".to_string(),
"--db".to_string(),
"{state_root}/state/work-db".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(
args.child_args,
vec![
"--db".to_string(),
"{state_root}/state/work-db".to_string(),
"--dead-repo-blacklist".to_string(),
"/tmp/daemon/state/dead-repo-blacklist.json".to_string(),
"--dead-repo-blacklist-fail-threshold".to_string(),
"2".to_string(),
]
);
}
#[test]
fn dead_repo_blacklist_injection_respects_existing_child_flags() {
let argv = vec![
"rpki_daemon".to_string(),
"--state-root".to_string(),
"/tmp/daemon".to_string(),
"--rpki-bin".to_string(),
"/bin/echo".to_string(),
"--dead-repo-blacklist".to_string(),
"/tmp/daemon/state/dead-repo-blacklist.json".to_string(),
"--".to_string(),
"--dead-repo-blacklist".to_string(),
"/custom/path.json".to_string(),
];
let args = parse_args(&argv).expect("parse");
assert_eq!(
args.child_args,
vec![
"--dead-repo-blacklist".to_string(),
"/custom/path.json".to_string(),
]
);
}
#[test]
fn dead_repo_threshold_without_blacklist_is_rejected() {
let argv = vec![
"rpki_daemon".to_string(),
"--state-root".to_string(),
"/tmp/daemon".to_string(),
"--rpki-bin".to_string(),
"/bin/echo".to_string(),
"--dead-repo-blacklist-fail-threshold".to_string(),
"2".to_string(),
"--".to_string(),
"--version".to_string(),
];
let err = parse_args(&argv).expect_err("parse should fail");
assert!(err.contains("requires --dead-repo-blacklist"), "{err}");
}
#[test]
fn health_check_keeps_dead_entry_and_removes_revived_entry() {
use std::io::{Read as _, Write as _};
use std::net::TcpListener;
// Revived RRDP host: one-shot HTTP server answering 200.
let listener = TcpListener::bind("127.0.0.1:0").expect("bind");
let port = listener.local_addr().expect("local addr").port();
std::thread::spawn(move || {
if let Ok((mut stream, _)) = listener.accept() {
let mut buf = [0u8; 1024];
let _ = stream.read(&mut buf);
let body = b"<notification/>";
let _ = stream.write_all(
format!(
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
)
.as_bytes(),
);
let _ = stream.write_all(body);
}
});
let td = tempfile::tempdir().expect("tempdir");
let blacklist_path = td.path().join("dead-repo-blacklist.json");
let revived_uri = format!("http://127.0.0.1:{port}/notification.xml");
let dead_uri = "http://127.0.0.1:1/notification.xml".to_string();
let mut blacklist = DeadRepoBlacklist::new();
for uri in [&revived_uri, &dead_uri] {
blacklist.record_transport_failure(RepoTransportMode::Rrdp, uri, 100, 1, 256);
}
blacklist
.store_atomic(&blacklist_path, 100)
.expect("store blacklist");
let mut args = test_args(td.path().join("daemon"));
args.dead_repo_blacklist = Some(blacklist_path.clone());
args.dead_repo_health_check_interval_secs = 0;
let mut health = DeadRepoHealthRuntime::default();
maybe_run_dead_repo_health_check(&args, &mut health);
let (after, warning) = DeadRepoBlacklist::load(&blacklist_path);
assert!(warning.is_none());
assert!(!after.is_blacklisted(RepoTransportMode::Rrdp, &revived_uri));
assert!(after.is_blacklisted(RepoTransportMode::Rrdp, &dead_uri));
assert!(health.last_health_check_at.is_some());
// Status JSON exposes the blacklist section.
let status = dead_repo_blacklist_status(&args, &health).expect("status");
assert_eq!(status.entries, 1);
assert_eq!(status.blacklisted, 1);
assert!(status.last_health_check_at_rfc3339_utc.is_some());
}

View File

@@ -0,0 +1,175 @@
// Daemon arguments, state, and summary data types.
#[derive(Clone, Debug, PartialEq, Eq)]
struct Args {
state_root: PathBuf,
rpki_bin: PathBuf,
interval_secs: u64,
max_runs: Option<u64>,
retain_runs: usize,
status_json: Option<PathBuf>,
summary_jsonl: Option<PathBuf>,
work_db: PathBuf,
repo_bytes_db: Option<PathBuf>,
raw_store_db: Option<PathBuf>,
db_stats_bin: Option<PathBuf>,
db_stats_exact_every: Option<u64>,
time_bin: Option<PathBuf>,
/// Enables daemon-side dead-repository health checks and
/// auto-injects the child `--dead-repo-blacklist` flag when absent.
dead_repo_blacklist: Option<PathBuf>,
dead_repo_blacklist_fail_threshold: Option<u32>,
dead_repo_health_check_interval_secs: u64,
dead_repo_probe_rsync_bin: PathBuf,
child_args: Vec<String>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct RunContext {
seq: u64,
run_id: String,
run_dir: PathBuf,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
enum DaemonState {
Starting,
Idle,
Running,
Collecting,
Sleeping,
Exited,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct DaemonStatus {
state: DaemonState,
updated_at_rfc3339_utc: String,
runs_completed: u64,
max_runs: Option<u64>,
current_run_seq: Option<u64>,
current_run_id: Option<String>,
last_run_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
dead_repo_blacklist: Option<DeadRepoBlacklistStatus>,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct DeadRepoBlacklistStatus {
path: String,
entries: usize,
blacklisted: usize,
last_health_check_at_rfc3339_utc: Option<String>,
}
/// Mutable daemon-side health-check bookkeeping.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
struct DeadRepoHealthRuntime {
last_health_check_at: Option<time::OffsetDateTime>,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
enum RunStatus {
Success,
Failed,
SpawnFailed,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct ArtifactInfo {
path: String,
size_bytes: u64,
}
#[derive(Clone, Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct ProcessMetrics {
time_wrapper_used: bool,
time_output_path: Option<String>,
user_seconds: Option<f64>,
system_seconds: Option<f64>,
cpu_percent: Option<f64>,
elapsed_raw: Option<String>,
max_rss_kb: Option<u64>,
exit_status_from_time: Option<i32>,
parse_error: Option<String>,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct ReportCounts {
vrps: usize,
aspas: usize,
publication_points: usize,
rrdp_repos_unique: Option<u64>,
tree_instances_processed: Option<u64>,
tree_instances_failed: Option<u64>,
warnings: usize,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct PathFileStats {
label: String,
path: String,
exists: bool,
is_dir: bool,
total_size_bytes: u64,
file_count: u64,
dir_count: u64,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct DbStatsSummary {
mode: String,
db_path: String,
output_path: Option<String>,
stderr_path: Option<String>,
status: String,
exit_code: Option<i32>,
error: Option<String>,
metrics: BTreeMap<String, String>,
}
#[derive(Clone, Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct RunSummary {
run_seq: u64,
run_id: String,
run_dir: String,
started_at_rfc3339_utc: String,
finished_at_rfc3339_utc: String,
wall_ms: u64,
status: RunStatus,
exit_code: Option<i32>,
exit_status: Option<String>,
error: Option<String>,
rpki_bin: String,
child_args: Vec<String>,
stdout_path: String,
stderr_path: String,
process_metrics: Option<ProcessMetrics>,
stage_timing: Option<serde_json::Value>,
report_counts: Option<ReportCounts>,
repo_sync_stats: Option<serde_json::Value>,
path_stats: Vec<PathFileStats>,
db_stats: Vec<DbStatsSummary>,
retention_deleted_runs: Vec<String>,
artifacts: Vec<ArtifactInfo>,
#[serde(skip_serializing_if = "Option::is_none")]
dead_repo_blacklist: Option<DeadRepoBlacklistRunSummary>,
}
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
struct DeadRepoBlacklistRunSummary {
path: String,
entries: usize,
blacklisted: usize,
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,249 @@
// Resource certificate profile validation and strict-name checks.
impl ResourceCertificate {
/// Parse step of scheme A (`parse → validate → verify`).
pub fn parse_der(
der: &[u8],
) -> Result<ResourceCertificateParsed, ResourceCertificateParseError> {
let (rem, cert) = X509Certificate::from_der(der)
.map_err(|e| ResourceCertificateParseError::Parse(e.to_string()))?;
if !rem.is_empty() {
return Err(ResourceCertificateParseError::TrailingBytes(rem.len()));
}
let validity_not_before = asn1_time_to_model(cert.validity().not_before);
let validity_not_after = asn1_time_to_model(cert.validity().not_after);
let subject_public_key_info = cert.tbs_certificate.subject_pki.raw.to_vec();
let signature_algorithm = algorithm_identifier_value(&cert.signature_algorithm);
let tbs_signature_algorithm = algorithm_identifier_value(&cert.tbs_certificate.signature);
let extensions = parse_extensions_parse(cert.extensions())?;
Ok(ResourceCertificateParsed {
raw_der: der.to_vec(),
version: cert.version(),
serial_number: cert.tbs_certificate.serial.clone(),
signature_algorithm,
tbs_signature_algorithm,
issuer_name: X509NameDer(cert.issuer().as_raw().to_vec()),
subject_name: X509NameDer(cert.subject().as_raw().to_vec()),
validity_not_before,
validity_not_after,
subject_public_key_info,
extensions,
})
}
/// Profile validate step of scheme A (`parse → validate → verify`).
///
/// `ResourceCertificate` is already profile-validated when constructed via `decode_der()` /
/// `ResourceCertificateParsed::validate_profile()`.
pub fn validate_profile(&self) -> Result<(), ResourceCertificateProfileError> {
Ok(())
}
pub fn validate_rfc6487_profile(
&self,
role: ResourceCertificateRole,
) -> Result<(), ResourceCertificateProfileError> {
let role_name = match role {
ResourceCertificateRole::TrustAnchor => "trust anchor CA",
ResourceCertificateRole::Ca => "CA",
ResourceCertificateRole::SignedObjectEe => "signed-object EE",
ResourceCertificateRole::RouterEe => "router EE",
};
let ca_role = matches!(
role,
ResourceCertificateRole::TrustAnchor | ResourceCertificateRole::Ca
);
if ca_role {
let constraints = self
.tbs
.extensions
.basic_constraints
.as_ref()
.ok_or(ResourceCertificateProfileError::BasicConstraintsMissing)?;
if !constraints.critical {
return Err(ResourceCertificateProfileError::BasicConstraintsCriticality);
}
if !constraints.ca {
return Err(ResourceCertificateProfileError::BasicConstraintsCaFalse);
}
if constraints.path_len_constraint.is_some() {
return Err(ResourceCertificateProfileError::BasicConstraintsPathLenPresent);
}
} else if self.tbs.extensions.basic_constraints.is_some() {
return Err(ResourceCertificateProfileError::BasicConstraintsEeMustOmit);
}
for oid in &self.tbs.extensions.extension_oids {
if !is_permitted_extension(oid, role) {
return Err(ResourceCertificateProfileError::DisallowedExtension {
role: role_name,
oid: oid.clone(),
});
}
}
let policies = self
.tbs
.extensions
.certificate_policies
.as_ref()
.ok_or(ResourceCertificateProfileError::CertificatePoliciesMissing)?;
if policies.policy_oid != OID_CP_IPADDR_ASNUMBER {
return Err(ResourceCertificateProfileError::InvalidCertificatePolicy(
policies.policy_oid.clone(),
));
}
if policies.qualifier_oids.len() > 1 {
return Err(ResourceCertificateProfileError::CertificatePoliciesTooManyQualifiers);
}
if let Some(qualifier_oid) = policies.qualifier_oids.first() {
if qualifier_oid != OID_QT_CPS {
return Err(
ResourceCertificateProfileError::CertificatePoliciesInvalidQualifier(
qualifier_oid.clone(),
),
);
}
}
if self
.tbs
.extensions
.as_resources
.as_ref()
.is_some_and(|resources| resources.rdi.is_some())
{
return Err(ResourceCertificateProfileError::AsResourcesRdiPresent);
}
Ok(())
}
pub fn validate_strict_name_profile(&self) -> Result<(), ResourceCertificateProfileError> {
validate_strict_rpki_name(&self.tbs.issuer_name, "issuer")?;
validate_strict_rpki_name(&self.tbs.subject_name, "subject")?;
Ok(())
}
/// Decode a resource certificate (`parse + validate`).
pub fn decode_der(der: &[u8]) -> Result<Self, ResourceCertificateDecodeError> {
Ok(Self::parse_der(der)?.validate_profile()?)
}
pub fn decode_der_with_strict_name(der: &[u8]) -> Result<Self, ResourceCertificateDecodeError> {
let cert = Self::decode_der(der)?;
cert.validate_strict_name_profile()?;
Ok(cert)
}
/// Backwards-compatible helper (historical name).
pub fn from_der(der: &[u8]) -> Result<Self, ResourceCertificateError> {
Self::decode_der(der)
}
}
fn is_permitted_extension(oid: &str, role: ResourceCertificateRole) -> bool {
matches!(
oid,
OID_BASIC_CONSTRAINTS
| OID_KEY_USAGE
| OID_SUBJECT_KEY_IDENTIFIER
| OID_AUTHORITY_KEY_IDENTIFIER
| OID_CRL_DISTRIBUTION_POINTS
| OID_AUTHORITY_INFO_ACCESS
| OID_SUBJECT_INFO_ACCESS
| OID_CERTIFICATE_POLICIES
| OID_IP_ADDR_BLOCKS
| OID_AUTONOMOUS_SYS_IDS
) || (role == ResourceCertificateRole::RouterEe && oid == OID_EXTENDED_KEY_USAGE)
}
fn validate_strict_rpki_name(
name: &X509NameDer,
role: &'static str,
) -> Result<(), ResourceCertificateProfileError> {
let mut name_seq = DerReader::new(name.as_raw())
.take_sequence()
.map_err(|e| ResourceCertificateProfileError::StrictName { role, detail: e })?;
let mut common_name_count = 0usize;
let mut serial_number_count = 0usize;
while !name_seq.is_empty() {
let set_bytes = name_seq
.take_tag(0x31)
.map_err(|e| ResourceCertificateProfileError::StrictName { role, detail: e })?;
let mut rdn_set = DerReader::new(set_bytes);
if rdn_set.is_empty() {
return Err(ResourceCertificateProfileError::StrictName {
role,
detail: "RelativeDistinguishedName SET is empty".to_string(),
});
}
while !rdn_set.is_empty() {
let mut attr = rdn_set
.take_sequence()
.map_err(|e| ResourceCertificateProfileError::StrictName { role, detail: e })?;
let oid = attr
.take_tag(0x06)
.map_err(|e| ResourceCertificateProfileError::StrictName { role, detail: e })?;
let (value_tag, _value) = attr
.take_any()
.map_err(|e| ResourceCertificateProfileError::StrictName { role, detail: e })?;
if !attr.is_empty() {
return Err(ResourceCertificateProfileError::StrictName {
role,
detail: "AttributeTypeAndValue must be SEQUENCE of 2".to_string(),
});
}
match oid {
// 2.5.4.3 commonName
&[0x55, 0x04, 0x03] => {
common_name_count += 1;
if value_tag != 0x13 {
return Err(ResourceCertificateProfileError::StrictName {
role,
detail: format!(
"commonName must be PrintableString, got tag 0x{value_tag:02X}"
),
});
}
}
// 2.5.4.5 serialNumber
&[0x55, 0x04, 0x05] => {
serial_number_count += 1;
if value_tag != 0x13 {
return Err(ResourceCertificateProfileError::StrictName {
role,
detail: format!(
"serialNumber must be PrintableString, got tag 0x{value_tag:02X}"
),
});
}
}
_ => {}
}
}
}
if common_name_count != 1 {
return Err(ResourceCertificateProfileError::StrictName {
role,
detail: format!("commonName must appear exactly once, got {common_name_count}"),
});
}
if serial_number_count > 1 {
return Err(ResourceCertificateProfileError::StrictName {
role,
detail: format!("serialNumber must appear at most once, got {serial_number_count}"),
});
}
Ok(())
}

View File

@@ -0,0 +1,348 @@
// Parsed certificate and extension profile validation.
impl ResourceCertificateParsed {
pub fn validate_profile(self) -> Result<ResourceCertificate, ResourceCertificateProfileError> {
let version = match self.version {
X509Version::V3 => 2u32,
_ => return Err(ResourceCertificateProfileError::InvalidVersion),
};
self.validity_not_before
.validate_encoding_rfc5280("notBefore")?;
self.validity_not_after
.validate_encoding_rfc5280("notAfter")?;
if self.signature_algorithm != self.tbs_signature_algorithm {
return Err(ResourceCertificateProfileError::SignatureAlgorithmMismatch);
}
if self.signature_algorithm.oid != OID_SHA256_WITH_RSA_ENCRYPTION {
return Err(ResourceCertificateProfileError::UnsupportedSignatureAlgorithm);
}
if !self.signature_algorithm.params_absent_or_null() {
return Err(ResourceCertificateProfileError::InvalidSignatureAlgorithmParameters);
}
let is_self_signed = self.issuer_name == self.subject_name;
let extensions = self.extensions.validate_profile(is_self_signed)?;
let kind = if extensions.basic_constraints_ca {
ResourceCertKind::Ca
} else {
ResourceCertKind::Ee
};
Ok(ResourceCertificate {
raw_der: self.raw_der,
tbs: RpkixTbsCertificate {
version,
serial_number: self.serial_number,
signature_algorithm: self.signature_algorithm.oid,
issuer_name: self.issuer_name,
subject_name: self.subject_name,
validity_not_before: self.validity_not_before.utc,
validity_not_after: self.validity_not_after.utc,
subject_public_key_info: self.subject_public_key_info,
extensions,
},
kind,
})
}
}
impl RcExtensionsParsed {
pub fn validate_profile(
self,
is_self_signed: bool,
) -> Result<RcExtensions, ResourceCertificateProfileError> {
// NOTE(perf): `self` is consumed. Prefer moving decoded fields out rather than cloning,
// especially for large resource sets and URI lists.
let RcExtensionsParsed {
basic_constraints,
subject_key_identifier,
authority_key_identifier,
crl_distribution_points,
authority_info_access,
subject_info_access,
certificate_policies,
extension_oids,
ip_resources,
as_resources,
} = self;
if basic_constraints.len() > 1 {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"basicConstraints",
));
}
let basic_constraints = basic_constraints.into_iter().next();
let basic_constraints_ca = basic_constraints.as_ref().is_some_and(|bc| bc.ca);
let subject_key_identifier = match subject_key_identifier.len() {
0 => None,
1 => {
let (ski, critical) = subject_key_identifier.into_iter().next().expect("len==1");
if critical {
return Err(ResourceCertificateProfileError::SkiCriticality);
}
Some(ski)
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"subjectKeyIdentifier",
));
}
};
let authority_key_identifier = match authority_key_identifier.len() {
0 => {
if is_self_signed {
None
} else {
return Err(ResourceCertificateProfileError::AkiMissing);
}
}
1 => {
let (aki, critical) = authority_key_identifier.into_iter().next().expect("len==1");
if critical {
return Err(ResourceCertificateProfileError::AkiCriticality);
}
if aki.has_authority_cert_issuer {
return Err(ResourceCertificateProfileError::AkiAuthorityCertIssuerPresent);
}
if aki.has_authority_cert_serial {
return Err(ResourceCertificateProfileError::AkiAuthorityCertSerialPresent);
}
let keyid = aki.key_identifier;
if is_self_signed {
if let (Some(keyid), Some(ski)) =
(keyid.as_ref(), subject_key_identifier.as_ref())
{
if keyid != ski {
return Err(ResourceCertificateProfileError::AkiSelfSignedNotEqualSki);
}
}
} else if keyid.is_none() {
return Err(ResourceCertificateProfileError::AkiMissing);
}
keyid
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"authorityKeyIdentifier",
));
}
};
let crl_distribution_points_uris = match crl_distribution_points.len() {
0 => {
if is_self_signed {
None
} else {
return Err(ResourceCertificateProfileError::CrlDistributionPointsMissing);
}
}
1 => {
let (crldp, critical) = crl_distribution_points.into_iter().next().expect("len==1");
if critical {
return Err(ResourceCertificateProfileError::CrlDistributionPointsCriticality);
}
if is_self_signed {
return Err(
ResourceCertificateProfileError::CrlDistributionPointsSelfSignedMustOmit,
);
}
if crldp.distribution_points.len() != 1 {
return Err(ResourceCertificateProfileError::CrlDistributionPointsNotSingle);
}
let dp = crldp
.distribution_points
.into_iter()
.next()
.expect("len==1");
if dp.reasons_present {
return Err(ResourceCertificateProfileError::CrlDistributionPointsHasReasons);
}
if dp.crl_issuer_present {
return Err(ResourceCertificateProfileError::CrlDistributionPointsHasCrlIssuer);
}
if !dp.distribution_point_present {
return Err(
ResourceCertificateProfileError::CrlDistributionPointsNoDistributionPoint,
);
}
if dp.name_relative_to_crl_issuer_present || !dp.full_name_present {
return Err(ResourceCertificateProfileError::CrlDistributionPointsInvalidName);
}
if dp.full_name_not_uri {
return Err(
ResourceCertificateProfileError::CrlDistributionPointsFullNameNotUri,
);
}
if !dp.full_name_uris.iter().any(|u| u.starts_with("rsync://")) {
return Err(ResourceCertificateProfileError::CrlDistributionPointsNoRsync);
}
Some(dp.full_name_uris)
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"cRLDistributionPoints",
));
}
};
let ca_issuers_uris = match authority_info_access.len() {
0 => {
if is_self_signed {
None
} else {
return Err(ResourceCertificateProfileError::AuthorityInfoAccessMissing);
}
}
1 => {
let (aia, critical) = authority_info_access.into_iter().next().expect("len==1");
if critical {
return Err(ResourceCertificateProfileError::AuthorityInfoAccessCriticality);
}
if is_self_signed {
return Err(
ResourceCertificateProfileError::AuthorityInfoAccessSelfSignedMustOmit,
);
}
if aia.ca_issuers_access_location_not_uri {
return Err(
ResourceCertificateProfileError::AuthorityInfoAccessCaIssuersNotUri,
);
}
if aia.ca_issuers_uris.is_empty() {
return Err(
ResourceCertificateProfileError::AuthorityInfoAccessMissingCaIssuers,
);
}
if !aia
.ca_issuers_uris
.iter()
.any(|u| u.starts_with("rsync://"))
{
return Err(ResourceCertificateProfileError::AuthorityInfoAccessNoRsync);
}
Some(aia.ca_issuers_uris)
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"authorityInfoAccess",
));
}
};
let subject_info_access = match subject_info_access.len() {
0 => None,
1 => {
let (sia, critical) = subject_info_access.into_iter().next().expect("len==1");
if critical {
return Err(ResourceCertificateProfileError::SiaCriticality);
}
if sia.signed_object_access_location_not_uri {
return Err(ResourceCertificateProfileError::SignedObjectSiaNotUri);
}
if !sia.signed_object_uris.is_empty()
&& !sia
.signed_object_uris
.iter()
.any(|u| u.starts_with("rsync://"))
{
return Err(ResourceCertificateProfileError::SignedObjectSiaNoRsync);
}
if sia.signed_object_uris.is_empty() {
Some(SubjectInfoAccess::Ca(SubjectInfoAccessCa {
access_descriptions: sia.access_descriptions,
}))
} else {
Some(SubjectInfoAccess::Ee(SubjectInfoAccessEe {
signed_object_uris: sia.signed_object_uris,
access_descriptions: sia.access_descriptions,
}))
}
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"subjectInfoAccess",
));
}
};
let certificate_policies = match certificate_policies.len() {
0 => None,
1 => {
let (policies, critical) = certificate_policies.into_iter().next().expect("len==1");
if !critical {
return Err(ResourceCertificateProfileError::CertificatePoliciesCriticality);
}
if policies.len() != 1 {
return Err(ResourceCertificateProfileError::InvalidCertificatePolicy(
"expected exactly one policy".into(),
));
}
let policy = policies.into_iter().next().expect("len==1");
if policy.policy_oid != OID_CP_IPADDR_ASNUMBER {
return Err(ResourceCertificateProfileError::InvalidCertificatePolicy(
policy.policy_oid,
));
}
Some(policy)
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"certificatePolicies",
));
}
};
let ip_resources = match ip_resources.len() {
0 => None,
1 => {
let (ip, critical) = ip_resources.into_iter().next().expect("len==1");
if !critical {
return Err(ResourceCertificateProfileError::IpResourcesCriticality);
}
Some(ip)
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"ipAddrBlocks",
));
}
};
let as_resources = match as_resources.len() {
0 => None,
1 => {
let (asn, critical) = as_resources.into_iter().next().expect("len==1");
if !critical {
return Err(ResourceCertificateProfileError::AsResourcesCriticality);
}
Some(asn)
}
_ => {
return Err(ResourceCertificateProfileError::DuplicateExtension(
"autonomousSysIds",
));
}
};
Ok(RcExtensions {
basic_constraints_ca,
basic_constraints,
subject_key_identifier,
authority_key_identifier,
crl_distribution_points_uris,
ca_issuers_uris,
subject_info_access,
certificate_policies_oid: certificate_policies
.as_ref()
.map(|_| OID_CP_IPADDR_ASNUMBER.to_string()),
certificate_policies,
extension_oids,
ip_resources,
as_resources,
})
}
}

View File

@@ -0,0 +1,586 @@
// DER parsing helpers for certificate extensions and resources.
fn algorithm_identifier_value(
ai: &x509_parser::x509::AlgorithmIdentifier<'_>,
) -> AlgorithmIdentifierValue {
let parameters = ai.parameters.as_ref().map(|p| AlgorithmParametersValue {
class: p.class(),
tag: p.tag(),
data: p.as_bytes().to_vec(),
});
// NOTE(perf): Avoid `to_id_string()` allocations for the algorithms we expect
// in RPKI resource certificates. Fall back to `to_id_string()` for unexpected
// algorithms (mostly error paths).
let oid = if ai.algorithm.as_bytes() == OID_SHA256_WITH_RSA_ENCRYPTION_RAW {
OID_SHA256_WITH_RSA_ENCRYPTION.to_string()
} else {
ai.algorithm.to_id_string()
};
AlgorithmIdentifierValue { oid, parameters }
}
fn parse_extensions_parse(
exts: &[X509Extension<'_>],
) -> Result<RcExtensionsParsed, ResourceCertificateParseError> {
let mut basic_constraints: Vec<BasicConstraintsProfile> = Vec::new();
let mut ski: Vec<(Vec<u8>, bool)> = Vec::new();
let mut aki: Vec<(AuthorityKeyIdentifierParsed, bool)> = Vec::new();
let mut crldp: Vec<(CrlDistributionPointsParsed, bool)> = Vec::new();
let mut aia: Vec<(AuthorityInfoAccessParsed, bool)> = Vec::new();
let mut sia: Vec<(SubjectInfoAccessParsed, bool)> = Vec::new();
let mut cert_policies: Vec<(Vec<CertificatePoliciesProfile>, bool)> = Vec::new();
let mut extension_oids: Vec<String> = Vec::with_capacity(exts.len());
let mut ip_resources: Vec<(IpResourceSet, bool)> = Vec::new();
let mut as_resources: Vec<(AsResourceSet, bool)> = Vec::new();
for ext in exts {
let oid = ext.oid.as_bytes();
extension_oids.push(ext.oid.to_id_string());
if oid == OID_BASIC_CONSTRAINTS_RAW {
let ParsedExtension::BasicConstraints(bc) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"basicConstraints parse failed".into(),
));
};
basic_constraints.push(BasicConstraintsProfile {
ca: bc.ca,
critical: ext.critical,
path_len_constraint: bc.path_len_constraint,
});
} else if oid == OID_SUBJECT_KEY_IDENTIFIER_RAW {
let ParsedExtension::SubjectKeyIdentifier(s) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"subjectKeyIdentifier parse failed".into(),
));
};
ski.push((s.0.to_vec(), ext.critical));
} else if oid == OID_AUTHORITY_KEY_IDENTIFIER_RAW {
let ParsedExtension::AuthorityKeyIdentifier(a) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"authorityKeyIdentifier parse failed".into(),
));
};
aki.push((
AuthorityKeyIdentifierParsed {
key_identifier: a.key_identifier.as_ref().map(|k| k.0.to_vec()),
has_authority_cert_issuer: a.authority_cert_issuer.is_some(),
has_authority_cert_serial: a.authority_cert_serial.is_some(),
},
ext.critical,
));
} else if oid == OID_CRL_DISTRIBUTION_POINTS_RAW {
let ParsedExtension::CRLDistributionPoints(p) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"cRLDistributionPoints parse failed".into(),
));
};
crldp.push((parse_crldp_parse(p)?, ext.critical));
} else if oid == OID_AUTHORITY_INFO_ACCESS_RAW {
let ParsedExtension::AuthorityInfoAccess(p) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"authorityInfoAccess parse failed".into(),
));
};
aia.push((parse_aia_parse(p.accessdescs.as_slice())?, ext.critical));
} else if oid == OID_SUBJECT_INFO_ACCESS_RAW {
let ParsedExtension::SubjectInfoAccess(s) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"subjectInfoAccess parse failed".into(),
));
};
sia.push((parse_sia_parse(s.accessdescs.as_slice())?, ext.critical));
} else if oid == OID_CERTIFICATE_POLICIES_RAW {
let ParsedExtension::CertificatePolicies(cp) = ext.parsed_extension() else {
return Err(ResourceCertificateParseError::Parse(
"certificatePolicies parse failed".into(),
));
};
let mut policies: Vec<CertificatePoliciesProfile> = Vec::with_capacity(cp.len());
for p in cp.iter() {
let b = p.policy_id.as_bytes();
let policy_oid = if b == OID_CP_IPADDR_ASNUMBER_RAW {
OID_CP_IPADDR_ASNUMBER.to_string()
} else {
p.policy_id.to_id_string()
};
let qualifier_oids = p
.policy_qualifiers
.as_ref()
.map(|qualifiers| {
qualifiers
.iter()
.map(|qualifier| qualifier.policy_qualifier_id.to_id_string())
.collect()
})
.unwrap_or_default();
policies.push(CertificatePoliciesProfile {
policy_oid,
qualifier_oids,
});
}
cert_policies.push((policies, ext.critical));
} else if oid == OID_IP_ADDR_BLOCKS_RAW {
let parsed = IpResourceSet::decode_extn_value(ext.value)
.map_err(|_e| ResourceCertificateParseError::InvalidIpResourcesEncoding)?;
ip_resources.push((parsed, ext.critical));
} else if oid == OID_AUTONOMOUS_SYS_IDS_RAW {
let parsed = AsResourceSet::decode_extn_value(ext.value)
.map_err(|_e| ResourceCertificateParseError::InvalidAsResourcesEncoding)?;
as_resources.push((parsed, ext.critical));
}
}
Ok(RcExtensionsParsed {
basic_constraints,
subject_key_identifier: ski,
authority_key_identifier: aki,
crl_distribution_points: crldp,
authority_info_access: aia,
subject_info_access: sia,
certificate_policies: cert_policies,
extension_oids,
ip_resources,
as_resources,
})
}
fn parse_aia_parse(
access: &[x509_parser::extensions::AccessDescription<'_>],
) -> Result<AuthorityInfoAccessParsed, ResourceCertificateParseError> {
let mut ca_issuers_uris: Vec<String> = Vec::new();
let mut ca_issuers_access_location_not_uri = false;
for ad in access {
if ad.access_method.as_bytes() != OID_AD_CA_ISSUERS_RAW {
continue;
}
let uri = match &ad.access_location {
x509_parser::extensions::GeneralName::URI(u) => u,
_ => {
ca_issuers_access_location_not_uri = true;
continue;
}
};
ca_issuers_uris.push(uri.to_string());
}
Ok(AuthorityInfoAccessParsed {
ca_issuers_uris,
ca_issuers_access_location_not_uri,
})
}
fn parse_crldp_parse(
crldp: &x509_parser::extensions::CRLDistributionPoints<'_>,
) -> Result<CrlDistributionPointsParsed, ResourceCertificateParseError> {
let mut out: Vec<CrlDistributionPointParsed> = Vec::new();
for p in crldp.iter() {
let mut full_name_uris: Vec<String> = Vec::new();
let mut full_name_not_uri = false;
let mut full_name_present = false;
let mut name_relative_to_crl_issuer_present = false;
let mut distribution_point_present = false;
if let Some(dp) = &p.distribution_point {
distribution_point_present = true;
match dp {
x509_parser::extensions::DistributionPointName::FullName(names) => {
full_name_present = true;
for n in names {
match n {
x509_parser::extensions::GeneralName::URI(u) => {
full_name_uris.push(u.to_string());
}
_ => {
full_name_not_uri = true;
}
}
}
}
x509_parser::extensions::DistributionPointName::NameRelativeToCRLIssuer(_) => {
name_relative_to_crl_issuer_present = true;
}
}
}
out.push(CrlDistributionPointParsed {
distribution_point_present,
reasons_present: p.reasons.is_some(),
crl_issuer_present: p.crl_issuer.is_some(),
name_relative_to_crl_issuer_present,
full_name_uris,
full_name_not_uri,
full_name_present,
});
}
Ok(CrlDistributionPointsParsed {
distribution_points: out,
})
}
fn parse_sia_parse(
access: &[x509_parser::extensions::AccessDescription<'_>],
) -> Result<SubjectInfoAccessParsed, ResourceCertificateParseError> {
let mut all = Vec::with_capacity(access.len());
let mut signed_object_uris: Vec<String> = Vec::new();
let mut signed_object_access_location_not_uri = false;
for ad in access {
let access_method_oid = if ad.access_method.as_bytes() == OID_AD_CA_REPOSITORY_RAW {
OID_AD_CA_REPOSITORY.to_string()
} else if ad.access_method.as_bytes() == OID_AD_RPKI_MANIFEST_RAW {
OID_AD_RPKI_MANIFEST.to_string()
} else if ad.access_method.as_bytes() == OID_AD_RPKI_NOTIFY_RAW {
OID_AD_RPKI_NOTIFY.to_string()
} else if ad.access_method.as_bytes() == OID_AD_SIGNED_OBJECT_RAW {
OID_AD_SIGNED_OBJECT.to_string()
} else {
ad.access_method.to_id_string()
};
let is_signed_object = access_method_oid == OID_AD_SIGNED_OBJECT;
let uri = match &ad.access_location {
x509_parser::extensions::GeneralName::URI(u) => u,
_ => {
if is_signed_object {
signed_object_access_location_not_uri = true;
}
continue;
}
};
if is_signed_object {
signed_object_uris.push(uri.to_string());
}
all.push(AccessDescription {
access_method_oid,
access_location: uri.to_string(),
});
}
Ok(SubjectInfoAccessParsed {
access_descriptions: all,
signed_object_uris,
signed_object_access_location_not_uri,
})
}
fn parse_ip_addr_blocks(ext_value: &[u8]) -> Result<IpResourceSet, ()> {
let (rem, obj) = parse_der(ext_value).map_err(|_| ())?;
if !rem.is_empty() {
return Err(());
}
let seq = obj.as_sequence().map_err(|_| ())?;
let mut families = Vec::with_capacity(seq.len());
for fam in seq {
let fam_seq = fam.as_sequence().map_err(|_| ())?;
if fam_seq.len() != 2 {
return Err(());
}
let af_bytes = fam_seq[0].as_slice().map_err(|_| ())?;
if af_bytes.len() != 2 {
return Err(());
}
let afi = match af_bytes {
[0x00, 0x01] => Afi::Ipv4,
[0x00, 0x02] => Afi::Ipv6,
_ => return Err(()),
};
let choice = match &fam_seq[1].content {
BerObjectContent::Null => IpAddressChoice::Inherit,
BerObjectContent::Sequence(_) => {
let items_seq = fam_seq[1].as_sequence().map_err(|_| ())?;
let mut items = Vec::with_capacity(items_seq.len());
for item in items_seq {
items.push(parse_ip_address_or_range(afi, item)?);
}
IpAddressChoice::AddressesOrRanges(items)
}
_ => return Err(()),
};
families.push(IpAddressFamily { afi, choice });
}
Ok(IpResourceSet { families })
}
fn parse_ip_address_or_range(afi: Afi, obj: &DerObject<'_>) -> Result<IpAddressOrRange, ()> {
match &obj.content {
BerObjectContent::BitString(_, _) => {
Ok(IpAddressOrRange::Prefix(parse_ip_prefix(afi, obj)?))
}
BerObjectContent::Sequence(_) => {
let seq = obj.as_sequence().map_err(|_| ())?;
if seq.len() != 2 {
return Err(());
}
let min = parse_ip_address_bound(afi, &seq[0], false)?;
let max = parse_ip_address_bound(afi, &seq[1], true)?;
Ok(IpAddressOrRange::Range(IpAddressRange { min, max }))
}
_ => Err(()),
}
}
fn parse_ip_prefix(afi: Afi, obj: &DerObject<'_>) -> Result<IpPrefix, ()> {
let (unused_bits, bytes) = match &obj.content {
BerObjectContent::BitString(unused, bso) => (*unused, bso.data.to_vec()),
_ => return Err(()),
};
if unused_bits > 7 {
return Err(());
}
if !bytes.is_empty() && unused_bits != 0 {
let mask = (1u8 << unused_bits) - 1;
if (bytes[bytes.len() - 1] & mask) != 0 {
return Err(());
}
} else if bytes.is_empty() && unused_bits != 0 {
return Err(());
}
let prefix_len = (bytes.len() * 8)
.checked_sub(unused_bits as usize)
.ok_or(())? as u16;
if prefix_len > afi.ub() {
return Err(());
}
let addr = canonicalize_prefix_addr(afi, prefix_len, &bytes);
Ok(IpPrefix {
afi,
prefix_len,
addr,
})
}
/// Parse an RFC 3779 `IPAddress` BIT STRING into an address-like byte array.
///
/// When used as an `IPAddressRange` endpoint, RFC 3779 allows endpoints to be encoded with
/// fewer than `ub` bits. In that case, the missing bits are interpreted as 0s for the lower
/// bound and 1s for the upper bound. This is essential to correctly interpret ranges that
/// are expressed on non-octet boundaries.
fn parse_ip_address_bound(
afi: Afi,
obj: &DerObject<'_>,
fill_remaining_ones: bool,
) -> Result<Vec<u8>, ()> {
let (unused_bits, bytes) = match &obj.content {
BerObjectContent::BitString(unused, bso) => (*unused, bso.data.to_vec()),
_ => return Err(()),
};
if unused_bits > 7 {
return Err(());
}
if !bytes.is_empty() && unused_bits != 0 {
let mask = (1u8 << unused_bits) - 1;
if (bytes[bytes.len() - 1] & mask) != 0 {
return Err(());
}
} else if bytes.is_empty() && unused_bits != 0 {
return Err(());
}
let bit_len: u16 = (bytes.len() * 8)
.checked_sub(unused_bits as usize)
.ok_or(())?
.try_into()
.map_err(|_| ())?;
if bit_len > afi.ub() {
return Err(());
}
let mut out = vec![0u8; afi.octets_len()];
let copy_len = bytes.len().min(out.len());
out[..copy_len].copy_from_slice(&bytes[..copy_len]);
if fill_remaining_ones {
if bit_len == 0 {
for b in &mut out {
*b = 0xFF;
}
return Ok(out);
}
let last_bit = (bit_len - 1) as usize;
let last_byte = last_bit / 8;
let rem = (bit_len % 8) as u8;
if rem != 0 && last_byte < out.len() {
// Set the (8-rem) trailing bits in the last byte to 1.
let mask: u8 = (1u8 << (8 - rem)) - 1;
out[last_byte] |= mask;
}
for b in out.iter_mut().skip(last_byte + 1) {
*b = 0xFF;
}
}
Ok(out)
}
fn parse_as_identifiers(ext_value: &[u8]) -> Result<AsResourceSet, ()> {
let (rem, obj) = parse_der(ext_value).map_err(|_| ())?;
if !rem.is_empty() {
return Err(());
}
let seq = obj.as_sequence().map_err(|_| ())?;
let mut asnum: Option<AsIdentifierChoice> = None;
let mut rdi: Option<AsIdentifierChoice> = None;
for item in seq {
if item.class() != Class::ContextSpecific {
return Err(());
}
match item.tag() {
Tag(0) => {
if asnum.is_some() {
return Err(());
}
let inner = parse_explicit_inner(item)?;
asnum = Some(parse_as_identifier_choice(&inner)?);
}
Tag(1) => {
if rdi.is_some() {
return Err(());
}
let inner = parse_explicit_inner(item)?;
rdi = Some(parse_as_identifier_choice(&inner)?);
}
_ => return Err(()),
}
}
Ok(AsResourceSet { asnum, rdi })
}
fn parse_explicit_inner<'a>(obj: &'a DerObject<'a>) -> Result<DerObject<'a>, ()> {
let inner_der = obj.as_slice().map_err(|_| ())?;
let (rem, inner) = parse_der(inner_der).map_err(|_| ())?;
if !rem.is_empty() {
return Err(());
}
Ok(inner)
}
fn parse_as_identifier_choice(obj: &DerObject<'_>) -> Result<AsIdentifierChoice, ()> {
match &obj.content {
BerObjectContent::Null => Ok(AsIdentifierChoice::Inherit),
BerObjectContent::Sequence(_) => {
let seq = obj.as_sequence().map_err(|_| ())?;
let mut items = Vec::with_capacity(seq.len());
for item in seq {
items.push(parse_as_id_or_range(item)?);
}
Ok(AsIdentifierChoice::AsIdsOrRanges(items))
}
_ => Err(()),
}
}
fn parse_as_id_or_range(obj: &DerObject<'_>) -> Result<AsIdOrRange, ()> {
match &obj.content {
BerObjectContent::Integer(_) => {
let v = obj.as_u64().map_err(|_| ())?;
if v > u32::MAX as u64 {
return Err(());
}
Ok(AsIdOrRange::Id(v as u32))
}
BerObjectContent::Sequence(_) => {
let seq = obj.as_sequence().map_err(|_| ())?;
if seq.len() != 2 {
return Err(());
}
let min = seq[0].as_u64().map_err(|_| ())?;
let max = seq[1].as_u64().map_err(|_| ())?;
if min > u32::MAX as u64 || max > u32::MAX as u64 || min > max {
return Err(());
}
Ok(AsIdOrRange::Range {
min: min as u32,
max: max as u32,
})
}
_ => Err(()),
}
}
fn canonicalize_prefix_addr(afi: Afi, prefix_len: u16, bytes: &[u8]) -> Vec<u8> {
let full_len = afi.octets_len();
let mut addr = vec![0u8; full_len];
let copy_len = bytes.len().min(full_len);
addr[..copy_len].copy_from_slice(&bytes[..copy_len]);
if prefix_len == 0 {
return addr;
}
let last_prefix_bit = (prefix_len - 1) as usize;
let last_prefix_byte = last_prefix_bit / 8;
let rem = (prefix_len % 8) as u8;
if rem != 0 && last_prefix_byte < addr.len() {
let mask: u8 = 0xFF << (8 - rem);
addr[last_prefix_byte] &= mask;
}
addr
}
fn prefix_covers(resource: &IpPrefix, subject: &IpPrefix) -> bool {
if resource.afi != subject.afi {
return false;
}
if resource.prefix_len > subject.prefix_len {
return false;
}
let n = resource.prefix_len as usize;
let whole = n / 8;
let rem = (n % 8) as u8;
if resource.addr.len() != subject.addr.len() {
return false;
}
if resource.addr[..whole] != subject.addr[..whole] {
return false;
}
if rem == 0 {
return true;
}
let mask = 0xFFu8 << (8 - rem);
(resource.addr[whole] & mask) == (subject.addr[whole] & mask)
}
fn prefix_range(afi: Afi, p: &IpPrefix) -> (u128, u128) {
let mut base_bytes = [0u8; 16];
match afi {
Afi::Ipv4 => {
base_bytes[12..].copy_from_slice(&p.addr[..4]);
}
Afi::Ipv6 => {
base_bytes.copy_from_slice(&p.addr[..16]);
}
}
let base = u128::from_be_bytes(base_bytes);
let host_bits = (afi.ub() - p.prefix_len) as u32;
if host_bits == 0 {
return (base, base);
}
let mask = (1u128 << host_bits) - 1;
(base, base | mask)
}
fn range_covers_prefix(afi: Afi, r: &IpAddressRange, p: &IpPrefix) -> bool {
let (p_min, p_max) = prefix_range(afi, p);
let r_min = bytes_to_u128(afi, &r.min);
let r_max = bytes_to_u128(afi, &r.max);
r_min <= p_min && p_max <= r_max
}
fn bytes_to_u128(afi: Afi, bytes: &[u8]) -> u128 {
let mut out = [0u8; 16];
match afi {
Afi::Ipv4 => {
let copy_len = bytes.len().min(4);
out[12..12 + copy_len].copy_from_slice(&bytes[..copy_len]);
}
Afi::Ipv6 => {
let copy_len = bytes.len().min(16);
out[..copy_len].copy_from_slice(&bytes[..copy_len]);
}
}
u128::from_be_bytes(out)
}

View File

@@ -0,0 +1,102 @@
// Strict RPKI name profile tests.
#[cfg(test)]
mod strict_name_tests {
use super::*;
fn name_with_attrs(attrs: &[(&[u8], u8, &[u8])]) -> X509NameDer {
let mut rdns = Vec::new();
for (oid, tag, value) in attrs {
let mut attr = Vec::new();
attr.extend(der_tlv(0x06, oid));
attr.extend(der_tlv(*tag, value));
let attr = der_tlv(0x30, &attr);
let rdn = der_tlv(0x31, &attr);
rdns.extend(rdn);
}
X509NameDer(der_tlv(0x30, &rdns))
}
fn der_tlv(tag: u8, value: &[u8]) -> Vec<u8> {
let mut out = vec![tag];
encode_len(value.len(), &mut out);
out.extend_from_slice(value);
out
}
fn encode_len(len: usize, out: &mut Vec<u8>) {
if len < 0x80 {
out.push(len as u8);
return;
}
let mut bytes = Vec::new();
let mut value = len;
while value > 0 {
bytes.push((value & 0xFF) as u8);
value >>= 8;
}
bytes.reverse();
out.push(0x80 | bytes.len() as u8);
out.extend(bytes);
}
#[test]
fn strict_name_accepts_printable_common_name_and_serial_number() {
let name = name_with_attrs(&[
(&[0x55, 0x04, 0x03], 0x13, b"CN1"),
(&[0x55, 0x04, 0x05], 0x13, b"SN1"),
]);
validate_strict_rpki_name(&name, "subject").expect("strict name");
}
#[test]
fn strict_name_rejects_utf8_common_name() {
let name = name_with_attrs(&[(&[0x55, 0x04, 0x03], 0x0C, b"CN1")]);
let err = validate_strict_rpki_name(&name, "subject").expect_err("strict name fails");
assert!(err.to_string().contains("PrintableString"), "{err}");
}
#[test]
fn strict_name_rejects_duplicate_common_name() {
let name = name_with_attrs(&[
(&[0x55, 0x04, 0x03], 0x13, b"CN1"),
(&[0x55, 0x04, 0x03], 0x13, b"CN2"),
]);
let err = validate_strict_rpki_name(&name, "subject").expect_err("strict name fails");
assert!(err.to_string().contains("exactly once"), "{err}");
}
#[test]
fn profile_rejects_rfc8360_v2_policy_oid() {
let extensions = RcExtensionsParsed {
basic_constraints: vec![BasicConstraintsProfile {
ca: true,
critical: true,
path_len_constraint: None,
}],
subject_key_identifier: Vec::new(),
authority_key_identifier: Vec::new(),
crl_distribution_points: Vec::new(),
authority_info_access: Vec::new(),
subject_info_access: Vec::new(),
certificate_policies: vec![(
vec![CertificatePoliciesProfile {
policy_oid: "1.3.6.1.5.5.7.14.3".to_string(),
qualifier_oids: Vec::new(),
}],
true,
)],
extension_oids: Vec::new(),
ip_resources: Vec::new(),
as_resources: Vec::new(),
};
let err = extensions
.validate_profile(true)
.expect_err("v2 policy OID must remain invalid");
assert!(
matches!(&err, ResourceCertificateProfileError::InvalidCertificatePolicy(oid) if oid == "1.3.6.1.5.5.7.14.3"),
"{err}"
);
}
}

View File

@@ -0,0 +1,564 @@
// Resource certificate and resource-set model types.
/// Resource Certificate kind (semantic classification).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ResourceCertKind {
Ca,
Ee,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ResourceCertificateRole {
TrustAnchor,
Ca,
SignedObjectEe,
RouterEe,
}
/// A parsed RPKI Resource Certificate (RFC 6487) data model.
///
/// This module intentionally focuses on the semantics needed by Signed Object validation and
/// object-specific EE certificate checks (MFT/ROA/ASPA), as described in
/// `rpki/specs/03_resource_certificate_rc.md`.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ResourceCertificate {
pub raw_der: Vec<u8>,
pub tbs: RpkixTbsCertificate,
pub kind: ResourceCertKind,
}
pub type ResourceCaCertificate = ResourceCertificate;
pub type ResourceEeCertificate = ResourceCertificate;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RpkixTbsCertificate {
pub version: u32,
pub serial_number: BigUint,
pub signature_algorithm: String,
pub issuer_name: X509NameDer,
pub subject_name: X509NameDer,
pub validity_not_before: UtcTime,
pub validity_not_after: UtcTime,
/// DER encoding of SubjectPublicKeyInfo.
pub subject_public_key_info: Vec<u8>,
pub extensions: RcExtensions,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RcExtensions {
pub basic_constraints_ca: bool,
pub basic_constraints: Option<BasicConstraintsProfile>,
pub subject_key_identifier: Option<Vec<u8>>,
/// Authority Key Identifier (AKI) keyIdentifier value.
pub authority_key_identifier: Option<Vec<u8>>,
/// CRL Distribution Points URIs (fullName).
pub crl_distribution_points_uris: Option<Vec<String>>,
/// Authority Information Access (AIA) caIssuers URIs.
pub ca_issuers_uris: Option<Vec<String>>,
pub subject_info_access: Option<SubjectInfoAccess>,
pub certificate_policies_oid: Option<String>,
pub certificate_policies: Option<CertificatePoliciesProfile>,
pub extension_oids: Vec<String>,
pub ip_resources: Option<IpResourceSet>,
pub as_resources: Option<AsResourceSet>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct BasicConstraintsProfile {
pub ca: bool,
pub critical: bool,
pub path_len_constraint: Option<u32>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CertificatePoliciesProfile {
pub policy_oid: String,
pub qualifier_oids: Vec<String>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ResourceCertificateParsed {
pub raw_der: Vec<u8>,
pub version: X509Version,
pub serial_number: BigUint,
pub signature_algorithm: AlgorithmIdentifierValue,
pub tbs_signature_algorithm: AlgorithmIdentifierValue,
pub issuer_name: X509NameDer,
pub subject_name: X509NameDer,
pub validity_not_before: Asn1TimeUtc,
pub validity_not_after: Asn1TimeUtc,
/// DER encoding of SubjectPublicKeyInfo.
pub subject_public_key_info: Vec<u8>,
pub extensions: RcExtensionsParsed,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AlgorithmIdentifierValue {
pub oid: String,
pub parameters: Option<AlgorithmParametersValue>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AlgorithmParametersValue {
pub class: Asn1Class,
pub tag: Asn1Tag,
pub data: Vec<u8>,
}
impl AlgorithmIdentifierValue {
pub fn params_absent_or_null(&self) -> bool {
match &self.parameters {
None => true,
Some(p) if p.class == Asn1Class::Universal && p.tag == Asn1Tag::Null => true,
Some(_p) => false,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RcExtensionsParsed {
pub basic_constraints: Vec<BasicConstraintsProfile>,
pub subject_key_identifier: Vec<(Vec<u8>, bool)>,
pub authority_key_identifier: Vec<(AuthorityKeyIdentifierParsed, bool)>,
pub crl_distribution_points: Vec<(CrlDistributionPointsParsed, bool)>,
pub authority_info_access: Vec<(AuthorityInfoAccessParsed, bool)>,
pub subject_info_access: Vec<(SubjectInfoAccessParsed, bool)>,
pub certificate_policies: Vec<(Vec<CertificatePoliciesProfile>, bool)>,
pub extension_oids: Vec<String>,
pub ip_resources: Vec<(IpResourceSet, bool)>,
pub as_resources: Vec<(AsResourceSet, bool)>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AuthorityKeyIdentifierParsed {
pub key_identifier: Option<Vec<u8>>,
pub has_authority_cert_issuer: bool,
pub has_authority_cert_serial: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AuthorityInfoAccessParsed {
pub ca_issuers_uris: Vec<String>,
pub ca_issuers_access_location_not_uri: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CrlDistributionPointsParsed {
pub distribution_points: Vec<CrlDistributionPointParsed>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CrlDistributionPointParsed {
pub distribution_point_present: bool,
pub reasons_present: bool,
pub crl_issuer_present: bool,
pub name_relative_to_crl_issuer_present: bool,
pub full_name_uris: Vec<String>,
pub full_name_not_uri: bool,
pub full_name_present: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SubjectInfoAccessParsed {
pub access_descriptions: Vec<AccessDescription>,
pub signed_object_uris: Vec<String>,
pub signed_object_access_location_not_uri: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum SubjectInfoAccess {
Ca(SubjectInfoAccessCa),
Ee(SubjectInfoAccessEe),
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SubjectInfoAccessCa {
pub access_descriptions: Vec<AccessDescription>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SubjectInfoAccessEe {
pub signed_object_uris: Vec<String>,
/// The full list of access descriptions as carried in the SIA extension.
pub access_descriptions: Vec<AccessDescription>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AccessDescription {
pub access_method_oid: String,
pub access_location: String,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
pub enum Afi {
Ipv4,
Ipv6,
}
impl Afi {
pub fn ub(self) -> u16 {
match self {
Afi::Ipv4 => 32,
Afi::Ipv6 => 128,
}
}
pub fn octets_len(self) -> usize {
match self {
Afi::Ipv4 => 4,
Afi::Ipv6 => 16,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct IpResourceSet {
pub families: Vec<IpAddressFamily>,
}
impl IpResourceSet {
/// Decode the DER bytes carried inside the X.509 `extnValue` OCTET STRING for
/// `id-pe-ipAddrBlocks` (RFC 3779 / RFC 6487).
pub fn decode_extn_value(extn_value: &[u8]) -> Result<Self, IpResourceSetDecodeError> {
parse_ip_addr_blocks(extn_value).map_err(|_| IpResourceSetDecodeError::InvalidEncoding)
}
pub fn is_all_inherit(&self) -> bool {
self.families
.iter()
.all(|f| matches!(f.choice, IpAddressChoice::Inherit))
}
pub fn has_any_inherit(&self) -> bool {
self.families
.iter()
.any(|f| matches!(f.choice, IpAddressChoice::Inherit))
}
pub fn contains_prefix(&self, prefix: &IpPrefix) -> bool {
self.families.iter().any(|fam| fam.contains_prefix(prefix))
}
}
#[derive(Debug, thiserror::Error)]
pub enum IpResourceSetDecodeError {
#[error("invalid ipAddrBlocks encoding (RFC 3779 §2.2.3; RFC 6487 §4.8.10)")]
InvalidEncoding,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct IpAddressFamily {
pub afi: Afi,
pub choice: IpAddressChoice,
}
impl IpAddressFamily {
pub fn contains_prefix(&self, prefix: &IpPrefix) -> bool {
if self.afi != prefix.afi {
return false;
}
match &self.choice {
IpAddressChoice::Inherit => true,
IpAddressChoice::AddressesOrRanges(items) => items.iter().any(|item| match item {
IpAddressOrRange::Prefix(p) => prefix_covers(p, prefix),
IpAddressOrRange::Range(r) => range_covers_prefix(self.afi, r, prefix),
}),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum IpAddressChoice {
Inherit,
AddressesOrRanges(Vec<IpAddressOrRange>),
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum IpAddressOrRange {
Prefix(IpPrefix),
Range(IpAddressRange),
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct IpAddressRange {
pub min: Vec<u8>,
pub max: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
pub struct IpPrefix {
pub afi: Afi,
pub prefix_len: u16,
pub addr: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct AsResourceSet {
pub asnum: Option<AsIdentifierChoice>,
pub rdi: Option<AsIdentifierChoice>,
}
impl AsResourceSet {
/// Decode the DER bytes carried inside the X.509 `extnValue` OCTET STRING for
/// `id-pe-autonomousSysIds` (RFC 3779 / RFC 6487).
pub fn decode_extn_value(extn_value: &[u8]) -> Result<Self, AsResourceSetDecodeError> {
parse_as_identifiers(extn_value).map_err(|_| AsResourceSetDecodeError::InvalidEncoding)
}
pub fn is_asnum_inherit(&self) -> bool {
matches!(self.asnum, Some(AsIdentifierChoice::Inherit))
}
pub fn has_any_range(&self) -> bool {
self.asnum.as_ref().map(|c| c.has_range()).unwrap_or(false)
|| self.rdi.as_ref().map(|c| c.has_range()).unwrap_or(false)
}
pub fn asnum_single_id(&self) -> Option<u32> {
match self.asnum.as_ref()? {
AsIdentifierChoice::Inherit => None,
AsIdentifierChoice::AsIdsOrRanges(items) => {
if items.len() != 1 {
return None;
}
match &items[0] {
AsIdOrRange::Id(v) => Some(*v),
AsIdOrRange::Range { .. } => None,
}
}
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum AsResourceSetDecodeError {
#[error("invalid autonomousSysIds encoding (RFC 3779 §3.2.3; RFC 6487 §4.8.11)")]
InvalidEncoding,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum AsIdentifierChoice {
Inherit,
AsIdsOrRanges(Vec<AsIdOrRange>),
}
impl AsIdentifierChoice {
pub fn has_range(&self) -> bool {
match self {
AsIdentifierChoice::Inherit => false,
AsIdentifierChoice::AsIdsOrRanges(items) => {
items.iter().any(|i| matches!(i, AsIdOrRange::Range { .. }))
}
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum AsIdOrRange {
Id(u32),
Range { min: u32, max: u32 },
}
#[derive(Debug, thiserror::Error)]
pub enum ResourceCertificateParseError {
#[error("X.509 parse error: {0} (RFC 5280 §4.1; RFC 6487 §4)")]
Parse(String),
#[error("trailing bytes after certificate DER: {0} bytes (DER; RFC 5280 §4.1)")]
TrailingBytes(usize),
#[error("invalid RFC 3779 IP resources extension encoding (RFC 6487 §4.8.10; RFC 3779 §2.2)")]
InvalidIpResourcesEncoding,
#[error("invalid RFC 3779 AS resources extension encoding (RFC 6487 §4.8.11; RFC 3779 §3.2)")]
InvalidAsResourcesEncoding,
}
#[derive(Debug, thiserror::Error)]
pub enum ResourceCertificateProfileError {
#[error("{0}")]
InvalidTimeEncoding(#[from] InvalidTimeEncodingError),
#[error("certificate version must be v3 (RFC 5280 §4.1; RFC 6487 §4)")]
InvalidVersion,
#[error("signatureAlgorithm does not match tbsCertificate.signature (RFC 5280 §4.1)")]
SignatureAlgorithmMismatch,
#[error(
"unsupported signature algorithm (expected sha256WithRSAEncryption {OID_SHA256_WITH_RSA_ENCRYPTION}) (RFC 7935 §2; RFC 6487 §4)"
)]
UnsupportedSignatureAlgorithm,
#[error("invalid signature algorithm parameters (RFC 5280 §4.1.1.2)")]
InvalidSignatureAlgorithmParameters,
#[error(
"{role} Name strict validation failed: {detail} (RFC 6487 §4.4; RFC 5280 §4.1.2.4/§4.1.2.6)"
)]
StrictName { role: &'static str, detail: String },
#[error("duplicate extension: {0} (RFC 5280 §4.2; RFC 6487 §4.8)")]
DuplicateExtension(&'static str),
#[error("SubjectKeyIdentifier criticality must be non-critical (RFC 6487 §4.8.2)")]
SkiCriticality,
#[error("SubjectInfoAccess criticality must be non-critical (RFC 6487 §4.8.8)")]
SiaCriticality,
#[error("certificatePolicies criticality must be critical (RFC 6487 §4.8.9)")]
CertificatePoliciesCriticality,
#[error("certificatePolicies must be present (RFC 6487 §4.8.9)")]
CertificatePoliciesMissing,
#[error(
"certificatePolicies must contain RPKI policy OID {OID_CP_IPADDR_ASNUMBER}, got {0} (RFC 6487 §4.8.9)"
)]
InvalidCertificatePolicy(String),
#[error("certificatePolicies may contain at most one CPS qualifier (RFC 6487 §4.8.9)")]
CertificatePoliciesTooManyQualifiers,
#[error(
"certificatePolicies qualifier must be id-qt-cps ({OID_QT_CPS}), got {0} (RFC 6487 §4.8.9)"
)]
CertificatePoliciesInvalidQualifier(String),
#[error("basicConstraints must be present in CA certificates (RFC 6487 §4.8.1)")]
BasicConstraintsMissing,
#[error("basicConstraints criticality must be critical in CA certificates (RFC 6487 §4.8.1)")]
BasicConstraintsCriticality,
#[error("basicConstraints cA must be TRUE in CA certificates (RFC 6487 §4.8.1)")]
BasicConstraintsCaFalse,
#[error(
"basicConstraints pathLenConstraint must be absent in CA certificates (RFC 6487 §4.8.1)"
)]
BasicConstraintsPathLenPresent,
#[error("basicConstraints must be absent in EE certificates (RFC 6487 §4.8.1)")]
BasicConstraintsEeMustOmit,
#[error("extension {oid} is not permitted for {role} resource certificates (RFC 6487 §4.8)")]
DisallowedExtension { role: &'static str, oid: String },
#[error("autonomousSysIds RDI field must be absent (RFC 6487 §4.8.11; RFC 3779 §3.2.3)")]
AsResourcesRdiPresent,
#[error(
"SIA id-ad-signedObject accessLocation must be URI (RFC 6487 §4.8.8.2; RFC 5280 §4.2.2.2)"
)]
SignedObjectSiaNotUri,
#[error("SIA id-ad-signedObject must include at least one rsync:// URI (RFC 6487 §4.8.8.2)")]
SignedObjectSiaNoRsync,
#[error("ipAddrBlocks criticality must be critical when present (RFC 6487 §4.8.10)")]
IpResourcesCriticality,
#[error("autonomousSysIds criticality must be critical when present (RFC 6487 §4.8.11)")]
AsResourcesCriticality,
#[error(
"authorityKeyIdentifier must be present in non-self-signed certificates (RFC 6487 §4.8.3; RFC 5280 §4.2.1.1)"
)]
AkiMissing,
#[error(
"authorityKeyIdentifier criticality must be non-critical (RFC 6487 §4.8.3; RFC 5280 §4.2.1.1)"
)]
AkiCriticality,
#[error(
"authorityKeyIdentifier authorityCertIssuer MUST NOT be present (RFC 6487 §4.8.3; RFC 5280 §4.2.1.1)"
)]
AkiAuthorityCertIssuerPresent,
#[error(
"authorityKeyIdentifier authorityCertSerialNumber MUST NOT be present (RFC 6487 §4.8.3; RFC 5280 §4.2.1.1)"
)]
AkiAuthorityCertSerialPresent,
#[error(
"self-signed certificate authorityKeyIdentifier must equal subjectKeyIdentifier when present (RFC 6487 §4.8.3)"
)]
AkiSelfSignedNotEqualSki,
#[error(
"CRLDistributionPoints must be present in non-self-signed certificates (RFC 6487 §4.8.6; RFC 5280 §4.2.1.13)"
)]
CrlDistributionPointsMissing,
#[error(
"CRLDistributionPoints criticality must be non-critical (RFC 6487 §4.8.6; RFC 5280 §4.2.1.13)"
)]
CrlDistributionPointsCriticality,
#[error("CRLDistributionPoints MUST be omitted in self-signed certificates (RFC 6487 §4.8.6)")]
CrlDistributionPointsSelfSignedMustOmit,
#[error("CRLDistributionPoints must contain exactly one DistributionPoint (RFC 6487 §4.8.6)")]
CrlDistributionPointsNotSingle,
#[error("CRLDistributionPoints distributionPoint field MUST be present (RFC 6487 §4.8.6)")]
CrlDistributionPointsNoDistributionPoint,
#[error("CRLDistributionPoints reasons field MUST be omitted (RFC 6487 §4.8.6)")]
CrlDistributionPointsHasReasons,
#[error("CRLDistributionPoints cRLIssuer field MUST be omitted (RFC 6487 §4.8.6)")]
CrlDistributionPointsHasCrlIssuer,
#[error(
"CRLDistributionPoints distributionPoint MUST contain fullName and MUST NOT contain nameRelativeToCRLIssuer (RFC 6487 §4.8.6)"
)]
CrlDistributionPointsInvalidName,
#[error(
"CRLDistributionPoints fullName must contain only URI GeneralNames (RFC 6487 §4.8.6; RFC 5280 §4.2.1.6)"
)]
CrlDistributionPointsFullNameNotUri,
#[error("CRLDistributionPoints must include at least one rsync:// URI (RFC 6487 §4.8.6)")]
CrlDistributionPointsNoRsync,
#[error(
"authorityInfoAccess must be present in non-self-signed certificates (RFC 6487 §4.8.7; RFC 5280 §4.2.2.1)"
)]
AuthorityInfoAccessMissing,
#[error(
"authorityInfoAccess criticality must be non-critical (RFC 6487 §4.8.7; RFC 5280 §4.2.2.1)"
)]
AuthorityInfoAccessCriticality,
#[error("authorityInfoAccess MUST be omitted in self-signed certificates (RFC 6487 §4.8.7)")]
AuthorityInfoAccessSelfSignedMustOmit,
#[error(
"authorityInfoAccess id-ad-caIssuers accessLocation must be URI (RFC 6487 §4.8.7; RFC 5280 §4.2.2.1)"
)]
AuthorityInfoAccessCaIssuersNotUri,
#[error("authorityInfoAccess must include at least one id-ad-caIssuers URI (RFC 6487 §4.8.7)")]
AuthorityInfoAccessMissingCaIssuers,
#[error("authorityInfoAccess must include at least one rsync:// URI (RFC 6487 §4.8.7)")]
AuthorityInfoAccessNoRsync,
}
#[derive(Debug, thiserror::Error)]
pub enum ResourceCertificateDecodeError {
#[error("{0}")]
Parse(#[from] ResourceCertificateParseError),
#[error("{0}")]
Validate(#[from] ResourceCertificateProfileError),
}
pub type ResourceCertificateError = ResourceCertificateDecodeError;

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,181 @@
// CMS reader and BER/DER content parsing primitives.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum CmsParseMode {
BerCompatible,
DerStrict,
}
struct CmsReader<'a> {
buf: &'a [u8],
mode: CmsParseMode,
}
impl<'a> CmsReader<'a> {
fn new(buf: &'a [u8], mode: CmsParseMode) -> Self {
Self { buf, mode }
}
fn is_empty(&self) -> bool {
self.buf.is_empty()
}
fn remaining_len(&self) -> usize {
self.buf.len()
}
fn peek_tag(&self) -> Result<u8, String> {
let (_rem, any) = parse_any(self.buf, self.mode)?;
header_to_single_byte_tag(&any.header)
}
fn take_any(&mut self) -> Result<(u8, &'a [u8]), String> {
let (rem, any) = parse_any(self.buf, self.mode)?;
let tag = header_to_single_byte_tag(&any.header)?;
self.buf = rem;
Ok((tag, any.data))
}
fn take_any_full(&mut self) -> Result<(u8, &'a [u8], &'a [u8]), String> {
let (rem, any) = parse_any(self.buf, self.mode)?;
let consumed = self.buf.len() - rem.len();
let full = &self.buf[..consumed];
let tag = header_to_single_byte_tag(&any.header)?;
self.buf = rem;
Ok((tag, full, any.data))
}
fn skip_any(&mut self) -> Result<(), String> {
let _ = self.take_any()?;
Ok(())
}
fn take_tag(&mut self, expected_tag: u8) -> Result<&'a [u8], String> {
let (tag, value) = self.take_any()?;
if tag != expected_tag {
return Err(format!(
"unexpected tag: got 0x{tag:02X}, expected 0x{expected_tag:02X}"
));
}
Ok(value)
}
fn take_sequence(&mut self) -> Result<CmsReader<'a>, String> {
let value = self.take_tag(0x30)?;
Ok(CmsReader::new(value, self.mode))
}
fn take_octet_string(&mut self) -> Result<Vec<u8>, String> {
let (rem, any) = parse_any(self.buf, self.mode)?;
let tag = header_to_single_byte_tag(&any.header)?;
if self.mode == CmsParseMode::DerStrict && tag != 0x04 {
return Err(format!(
"unexpected tag in DER strict mode: got 0x{tag:02X}, expected 0x04"
));
}
if tag != 0x04 && tag != 0x24 {
return Err(format!("unexpected tag: got 0x{tag:02X}, expected 0x04"));
}
let octets = flatten_octet_string(any, self.mode)?;
self.buf = rem;
Ok(octets)
}
fn take_uint_u64(&mut self) -> Result<u64, String> {
let value = self.take_tag(0x02)?;
der_uint_from_bytes(value)
}
fn take_explicit(&mut self, expected_outer_tag: u8) -> Result<(u8, &'a [u8]), String> {
let inner_der = self.take_tag(expected_outer_tag)?;
let (tag, value, rem) = cms_take_tlv(inner_der, self.mode)?;
if !rem.is_empty() {
return Err("trailing bytes inside EXPLICIT value".into());
}
Ok((tag, value))
}
fn take_explicit_der(&mut self, expected_outer_tag: u8) -> Result<&'a [u8], String> {
let inner_der = self.take_tag(expected_outer_tag)?;
let (_tag, _value, rem) = cms_take_tlv(inner_der, self.mode)?;
if !rem.is_empty() {
return Err("trailing bytes inside EXPLICIT value".into());
}
Ok(inner_der)
}
}
fn parse_signed_object_content_info(
raw_der: &[u8],
parse_der: &[u8],
mode: CmsParseMode,
) -> Result<RpkiSignedObjectParsed, SignedObjectParseError> {
let mut r = CmsReader::new(parse_der, mode);
let mut content_info_seq = r
.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
if !r.is_empty() {
return Err(SignedObjectParseError::TrailingBytes(r.remaining_len()));
}
let content_type = take_oid_string(&mut content_info_seq)?;
let signed_data = parse_signed_data_from_contentinfo_cursor(&mut content_info_seq)?;
if !content_info_seq.is_empty() {
return Err(SignedObjectParseError::Parse(
"ContentInfo must be a SEQUENCE of 2 elements".into(),
));
}
Ok(RpkiSignedObjectParsed {
raw_der: raw_der.to_vec(),
content_info_content_type: content_type,
signed_data,
})
}
fn parse_any<'a>(input: &'a [u8], mode: CmsParseMode) -> Result<(&'a [u8], Any<'a>), String> {
match mode {
CmsParseMode::BerCompatible => {
Any::from_ber(input).map_err(|e| format!("BER parse error: {e}"))
}
CmsParseMode::DerStrict => {
Any::from_der(input).map_err(|e| format!("DER parse error: {e}"))
}
}
}
fn header_to_single_byte_tag(header: &Header<'_>) -> Result<u8, String> {
let tag_no = header.tag().0;
if tag_no > 30 {
return Err(format!("high-tag-number form not supported: {tag_no}"));
}
Ok(((header.class() as u8) << 6)
| if header.constructed() { 0x20 } else { 0x00 }
| tag_no as u8)
}
fn cms_take_tlv(input: &[u8], mode: CmsParseMode) -> Result<(u8, &[u8], &[u8]), String> {
let (rem, any) = parse_any(input, mode)?;
let tag = header_to_single_byte_tag(&any.header)?;
Ok((tag, any.data, rem))
}
fn flatten_octet_string(any: Any<'_>, mode: CmsParseMode) -> Result<Vec<u8>, String> {
if any.class() != Class::Universal || any.tag() != Tag::OctetString {
return Err("expected OCTET STRING".into());
}
if !any.header.constructed() {
return Ok(any.data.to_vec());
}
if mode == CmsParseMode::DerStrict {
return Err("constructed OCTET STRING is not allowed in DER strict mode".into());
}
let mut out = Vec::new();
let mut input = any.data;
while !input.is_empty() {
let (rem, child) = Any::from_ber(input).map_err(|e| format!("BER parse error: {e}"))?;
out.extend(flatten_octet_string(child, mode)?);
input = rem;
}
Ok(out)
}

View File

@@ -0,0 +1,542 @@
// SignedData profile validation and EE certificate checks.
impl RpkiSignedObjectParsed {
pub fn validate_profile(self) -> Result<RpkiSignedObject, SignedObjectValidateError> {
self.validate_profile_with_strict_name(false)
}
pub fn validate_profile_with_strict_name(
self,
strict_name: bool,
) -> Result<RpkiSignedObject, SignedObjectValidateError> {
if self.content_info_content_type != OID_SIGNED_DATA {
return Err(SignedObjectValidateError::InvalidContentInfoContentType(
self.content_info_content_type,
));
}
let signed_data = validate_signed_data_profile(self.signed_data, strict_name)?;
Ok(RpkiSignedObject {
raw_der: self.raw_der,
content_info_content_type: OID_SIGNED_DATA.to_string(),
signed_data,
})
}
}
fn parse_signed_data_from_contentinfo_cursor(
seq: &mut CmsReader<'_>,
) -> Result<SignedDataParsed, SignedObjectParseError> {
let inner_der = seq.take_explicit_der(0xA0).map_err(|_e| {
SignedObjectParseError::Parse("ContentInfo.content must be [0] EXPLICIT".into())
})?;
let mut r = CmsReader::new(inner_der, seq.mode);
let signed_data_seq = r
.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
if !r.is_empty() {
return Err(SignedObjectParseError::Parse(
"trailing bytes inside ContentInfo.content".into(),
));
}
parse_signed_data_cursor(signed_data_seq)
}
fn parse_signed_data_cursor(
mut seq: CmsReader<'_>,
) -> Result<SignedDataParsed, SignedObjectParseError> {
let version = seq
.take_uint_u64()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let digest_set_bytes = seq
.take_tag(0x31)
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let mut digest_set = CmsReader::new(digest_set_bytes, seq.mode);
let mut digest_algorithms: Vec<AlgorithmIdentifierParsed> = Vec::new();
while !digest_set.is_empty() {
let alg = digest_set
.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let (oid, params_ok) = parse_algorithm_identifier_cursor(alg)?;
digest_algorithms.push(AlgorithmIdentifierParsed { oid, params_ok });
}
let encap_content_info = parse_encapsulated_content_info_cursor(
seq.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?,
)?;
let mut certificates: Option<Vec<Vec<u8>>> = None;
let mut crls_present = false;
let mut signer_infos: Option<Vec<SignerInfoParsed>> = None;
while !seq.is_empty() {
let tag = seq
.peek_tag()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
match tag {
0xA0 => {
if certificates.is_some() {
return Err(SignedObjectParseError::Parse(
"SignedData.certificates appears more than once".into(),
));
}
let content = seq
.take_tag(0xA0)
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
certificates = Some(split_der_objects(content, seq.mode)?);
}
0xA1 => {
crls_present = true;
seq.skip_any()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
}
0x31 => {
if signer_infos.is_some() {
return Err(SignedObjectParseError::Parse(
"SignedData.signerInfos appears more than once".into(),
));
}
let set_bytes = seq
.take_tag(0x31)
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
signer_infos = Some(parse_signer_infos_set_cursor(set_bytes, seq.mode)?);
}
_ => {
return Err(SignedObjectParseError::Parse(
"unexpected field in SignedData".into(),
));
}
}
}
let signer_infos = signer_infos
.ok_or_else(|| SignedObjectParseError::Parse("SignedData.signerInfos missing".into()))?;
Ok(SignedDataParsed {
version,
digest_algorithms,
encap_content_info,
certificates,
crls_present,
signer_infos,
})
}
fn parse_encapsulated_content_info_cursor(
mut seq: CmsReader<'_>,
) -> Result<EncapsulatedContentInfoParsed, SignedObjectParseError> {
if seq.is_empty() {
return Err(SignedObjectParseError::Parse(
"EncapsulatedContentInfo must be SEQUENCE of 1..2".into(),
));
}
let econtent_type = take_oid_string(&mut seq)?;
let econtent = if seq.is_empty() {
None
} else {
let inner_der = seq.take_explicit_der(0xA0).map_err(|_e| {
SignedObjectParseError::Parse(
"EncapsulatedContentInfo.eContent must be [0] EXPLICIT".into(),
)
})?;
let mut inner = CmsReader::new(inner_der, seq.mode);
let octets = inner
.take_octet_string()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
if !inner.is_empty() {
return Err(SignedObjectParseError::Parse(
"trailing bytes inside EncapsulatedContentInfo.eContent".into(),
));
}
Some(octets)
};
if !seq.is_empty() {
return Err(SignedObjectParseError::Parse(
"EncapsulatedContentInfo must be SEQUENCE of 1..2".into(),
));
}
Ok(EncapsulatedContentInfoParsed {
econtent_type,
econtent,
})
}
fn split_der_objects(
mut input: &[u8],
mode: CmsParseMode,
) -> Result<Vec<Vec<u8>>, SignedObjectParseError> {
let mut out: Vec<Vec<u8>> = Vec::new();
while !input.is_empty() {
let (_tag, _value, rem) =
cms_take_tlv(input, mode).map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let consumed = input.len() - rem.len();
out.push(input[..consumed].to_vec());
input = rem;
}
Ok(out)
}
fn parse_signer_infos_set_cursor(
set_bytes: &[u8],
mode: CmsParseMode,
) -> Result<Vec<SignerInfoParsed>, SignedObjectParseError> {
let mut set = CmsReader::new(set_bytes, mode);
let mut out: Vec<SignerInfoParsed> = Vec::new();
while !set.is_empty() {
let si = set
.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
out.push(parse_signer_info_cursor(si)?);
}
Ok(out)
}
fn validate_ee_certificate(
der: &[u8],
strict_name: bool,
) -> Result<ResourceEeCertificate, SignedObjectValidateError> {
let (rem, cert) = X509Certificate::from_der(der)
.map_err(|e| SignedObjectValidateError::EeCertificateParse(e.to_string()))?;
if !rem.is_empty() {
return Err(SignedObjectValidateError::EeCertificateParse(format!(
"trailing bytes after EE certificate DER: {}",
rem.len()
)));
}
let rc = match ResourceCertificate::from_der(der) {
Ok(v) => v,
Err(e) => {
return match e {
crate::data_model::rc::ResourceCertificateDecodeError::Validate(
crate::data_model::rc::ResourceCertificateProfileError::SignedObjectSiaNotUri,
) => Err(SignedObjectValidateError::EeCertificateSignedObjectSiaNotUri),
crate::data_model::rc::ResourceCertificateDecodeError::Validate(
crate::data_model::rc::ResourceCertificateProfileError::SignedObjectSiaNoRsync,
) => Err(SignedObjectValidateError::EeCertificateSignedObjectSiaNoRsync),
_ => Err(SignedObjectValidateError::EeCertificateParse(e.to_string())),
};
}
};
rc.validate_rfc6487_profile(ResourceCertificateRole::SignedObjectEe)
.map_err(|e| SignedObjectValidateError::EeCertificateParse(e.to_string()))?;
if strict_name {
rc.validate_strict_name_profile()
.map_err(|e| SignedObjectValidateError::EeCertificateParse(e.to_string()))?;
}
let ski = rc
.tbs
.extensions
.subject_key_identifier
.clone()
.ok_or(SignedObjectValidateError::EeCertificateMissingSki)?;
let spki_der = rc.tbs.subject_public_key_info.clone();
let (rem, spki) = SubjectPublicKeyInfo::from_der(&spki_der)
.map_err(|e| SignedObjectValidateError::EeCertificateParse(e.to_string()))?;
if !rem.is_empty() {
return Err(SignedObjectValidateError::EeCertificateParse(
"trailing bytes after EE SubjectPublicKeyInfo DER".to_string(),
));
}
let parsed_pk = spki.parsed().map_err(|_e| {
SignedObjectValidateError::EeCertificateParse(
"unsupported EE public key algorithm".to_string(),
)
})?;
let (rsa_public_modulus, rsa_public_exponent) = match parsed_pk {
PublicKey::RSA(rsa) => {
let modulus = strip_leading_zeros(rsa.modulus).to_vec();
let exponent = strip_leading_zeros(rsa.exponent).to_vec();
let _ = rsa.try_exponent().map_err(|_e| {
SignedObjectValidateError::EeCertificateParse("invalid EE RSA exponent".to_string())
})?;
(modulus, exponent)
}
_ => {
return Err(SignedObjectValidateError::EeCertificateParse(
"unsupported EE public key algorithm".to_string(),
));
}
};
let sia = rc
.tbs
.extensions
.subject_info_access
.as_ref()
.ok_or(SignedObjectValidateError::EeCertificateMissingSia)?;
let signed_object_uris: Vec<String> = match sia {
SubjectInfoAccess::Ee(ee) => ee.signed_object_uris.clone(),
SubjectInfoAccess::Ca(_ca) => Vec::new(),
};
if signed_object_uris.is_empty() {
return Err(SignedObjectValidateError::EeCertificateMissingSignedObjectSia);
}
if !signed_object_uris.iter().any(|u| u.starts_with("rsync://")) {
return Err(SignedObjectValidateError::EeCertificateSignedObjectSiaNoRsync);
}
Ok(ResourceEeCertificate {
raw_der: der.to_vec(),
subject_key_identifier: ski,
spki_der,
rsa_public_modulus,
rsa_public_exponent,
tbs_certificate_der: cert.tbs_certificate.as_ref().to_vec(),
signature_bytes: cert.signature_value.data.to_vec(),
key_usage_summary: summarize_ee_key_usage(&cert),
sia_signed_object_uris: signed_object_uris,
resource_cert: rc,
})
}
fn summarize_ee_key_usage(cert: &X509Certificate<'_>) -> EeKeyUsageSummary {
for ext in cert.extensions() {
if ext.oid.as_bytes() == OID_KEY_USAGE_RAW {
match ext.parsed_extension() {
ParsedExtension::KeyUsage(ku) => {
if !ext.critical {
return EeKeyUsageSummary::NotCritical;
}
let ok = ku.digital_signature()
&& !ku.key_cert_sign()
&& !ku.crl_sign()
&& !ku.non_repudiation()
&& !ku.key_encipherment()
&& !ku.data_encipherment()
&& !ku.key_agreement()
&& !ku.encipher_only()
&& !ku.decipher_only();
return if ok {
EeKeyUsageSummary::DigitalSignatureOnly
} else {
EeKeyUsageSummary::InvalidBits
};
}
other => {
return EeKeyUsageSummary::ParseError(format!(
"unexpected parsed keyUsage extension: {other:?}"
));
}
}
}
}
EeKeyUsageSummary::Missing
}
fn parse_signer_info_cursor(
mut seq: CmsReader<'_>,
) -> Result<SignerInfoParsed, SignedObjectParseError> {
let version = seq
.take_uint_u64()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let (sid_tag, sid_bytes) = seq
.take_any()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let sid = if (sid_tag & 0xC0) == 0x80 && (sid_tag & 0x1F) == 0 {
SignerIdentifierParsed::SubjectKeyIdentifier(sid_bytes.to_vec())
} else {
SignerIdentifierParsed::Other
};
let digest_alg_seq = seq
.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let (digest_oid, digest_params_ok) = parse_algorithm_identifier_cursor(digest_alg_seq)?;
let digest_algorithm = AlgorithmIdentifierParsed {
oid: digest_oid,
params_ok: digest_params_ok,
};
let mut signed_attrs_content: Option<Vec<u8>> = None;
let mut signed_attrs_der_for_signature: Option<Vec<u8>> = None;
if seq
.peek_tag()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?
== 0xA0
{
let (tag, full_tlv, value) = seq
.take_any_full()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
if tag != 0xA0 {
return Err(SignedObjectParseError::Parse(
"SignerInfo.signedAttrs must be [0] IMPLICIT".into(),
));
}
signed_attrs_content = Some(value.to_vec());
signed_attrs_der_for_signature = Some(make_signed_attrs_der_for_signature(full_tlv)?);
}
let sig_alg_seq = seq
.take_sequence()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
let (signature_oid, signature_params_ok) = parse_algorithm_identifier_cursor(sig_alg_seq)?;
let signature_algorithm = AlgorithmIdentifierParsed {
oid: signature_oid,
params_ok: signature_params_ok,
};
let signature = seq
.take_octet_string()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?
.to_vec();
let unsigned_attrs_present = !seq.is_empty();
Ok(SignerInfoParsed {
version,
sid,
digest_algorithm,
signature_algorithm,
signed_attrs_content,
signed_attrs_der_for_signature,
unsigned_attrs_present,
signature,
})
}
fn validate_signed_data_profile(
signed_data: SignedDataParsed,
strict_name: bool,
) -> Result<SignedDataProfiled, SignedObjectValidateError> {
if signed_data.version != 3 {
return Err(SignedObjectValidateError::InvalidSignedDataVersion(
signed_data.version,
));
}
if signed_data.digest_algorithms.len() != 1 {
return Err(SignedObjectValidateError::InvalidDigestAlgorithmsCount(
signed_data.digest_algorithms.len(),
));
}
let digest_alg = &signed_data.digest_algorithms[0];
if digest_alg.oid != OID_SHA256 {
return Err(SignedObjectValidateError::InvalidDigestAlgorithm(
digest_alg.oid.clone(),
));
}
if signed_data.crls_present {
return Err(SignedObjectValidateError::CrlsPresent);
}
let econtent = signed_data
.encap_content_info
.econtent
.clone()
.ok_or(SignedObjectValidateError::EContentMissing)?;
if econtent.is_empty() {
return Err(SignedObjectValidateError::EContentMissing);
}
let encap_content_info = EncapsulatedContentInfo {
econtent_type: signed_data.encap_content_info.econtent_type.clone(),
econtent: econtent.clone(),
};
let certs = signed_data
.certificates
.as_ref()
.ok_or(SignedObjectValidateError::CertificatesMissing)?;
if certs.len() != 1 {
return Err(SignedObjectValidateError::InvalidCertificatesCount(
certs.len(),
));
}
let ee = validate_ee_certificate(&certs[0], strict_name)?;
if signed_data.signer_infos.len() != 1 {
return Err(SignedObjectValidateError::InvalidSignerInfosCount(
signed_data.signer_infos.len(),
));
}
let signer = &signed_data.signer_infos[0];
if signer.version != 3 {
return Err(SignedObjectValidateError::InvalidSignerInfoVersion(
signer.version,
));
}
let sid_ski = match &signer.sid {
SignerIdentifierParsed::SubjectKeyIdentifier(ski) => ski.clone(),
SignerIdentifierParsed::Other => {
return Err(SignedObjectValidateError::InvalidSignerIdentifier);
}
};
if signer.digest_algorithm.oid != OID_SHA256 {
return Err(SignedObjectValidateError::InvalidSignerInfoDigestAlgorithm(
signer.digest_algorithm.oid.clone(),
));
}
let signed_attrs_content = signer
.signed_attrs_content
.as_deref()
.ok_or(SignedObjectValidateError::SignedAttrsMissing)?;
let signed_attrs_der_for_signature = signer
.signed_attrs_der_for_signature
.clone()
.ok_or(SignedObjectValidateError::SignedAttrsMissing)?;
let signed_attrs = parse_signed_attrs_implicit(signed_attrs_content)?;
if signer.unsigned_attrs_present {
return Err(SignedObjectValidateError::UnsignedAttrsPresent);
}
if !signer.signature_algorithm.params_ok {
return Err(SignedObjectValidateError::InvalidSignatureAlgorithmParameters);
}
let signature_algorithm = signer.signature_algorithm.oid.clone();
if signature_algorithm != OID_RSA_ENCRYPTION
&& signature_algorithm != OID_SHA256_WITH_RSA_ENCRYPTION
{
return Err(SignedObjectValidateError::InvalidSignatureAlgorithm(
signature_algorithm,
));
}
if sid_ski != ee.subject_key_identifier {
return Err(SignedObjectValidateError::SidSkiMismatch);
}
if signed_attrs.content_type != encap_content_info.econtent_type {
return Err(SignedObjectValidateError::ContentTypeAttrMismatch {
econtent_type: encap_content_info.econtent_type.clone(),
attr_content_type: signed_attrs.content_type.clone(),
});
}
let computed = digest::digest(&digest::SHA256, &encap_content_info.econtent);
if computed.as_ref() != signed_attrs.message_digest.as_slice() {
return Err(SignedObjectValidateError::MessageDigestMismatch);
}
Ok(SignedDataProfiled {
version: 3,
digest_algorithms: vec![OID_SHA256.to_string()],
encap_content_info,
certificates: vec![ee.clone()],
crls_present: false,
signer_infos: vec![SignerInfoProfiled {
version: 3,
sid_ski,
digest_algorithm: OID_SHA256.to_string(),
signature_algorithm: signer.signature_algorithm.oid.clone(),
signed_attrs,
unsigned_attrs_present: false,
signature: signer.signature.clone(),
signed_attrs_der_for_signature,
}],
})
}

View File

@@ -0,0 +1,344 @@
// Signed attributes, signing-time, and algorithm parsing helpers.
fn parse_signed_attrs_implicit(
input: &[u8],
) -> Result<SignedAttrsProfiled, SignedObjectValidateError> {
let mut content_type: Option<String> = None;
let mut message_digest: Option<Vec<u8>> = None;
let mut signing_time: Option<Asn1TimeUtc> = None;
fn count_elements(mut r: DerReader<'_>) -> Result<usize, String> {
let mut n = 0usize;
while !r.is_empty() {
r.skip_any()?;
n += 1;
}
Ok(n)
}
let mut remaining = DerReader::new(input);
while !remaining.is_empty() {
let mut attr = remaining
.take_sequence()
.map_err(|e| SignedObjectValidateError::SignedAttrsParse(e.to_string()))?;
let oid_bytes = attr
.take_tag(0x06)
.map_err(|e| SignedObjectValidateError::SignedAttrsParse(e.to_string()))?;
let oid = oid_value_bytes_to_string(oid_bytes);
let values_bytes = attr
.take_tag(0x31)
.map_err(|e| SignedObjectValidateError::SignedAttrsParse(e.to_string()))?;
if !attr.is_empty() {
return Err(SignedObjectValidateError::SignedAttrsParse(
"Attribute must be SEQUENCE of 2".into(),
));
}
let mut values = DerReader::new(values_bytes);
let count = if values.is_empty() {
0
} else {
values
.skip_any()
.map_err(|e| SignedObjectValidateError::SignedAttrsParse(e.to_string()))?;
if values.is_empty() {
1
} else {
1 + count_elements(values)
.map_err(|e| SignedObjectValidateError::SignedAttrsParse(e.to_string()))?
}
};
if count != 1 {
return Err(
SignedObjectValidateError::InvalidSignedAttributeValuesCount { oid, count },
);
}
// Re-parse the sole value.
let mut values = DerReader::new(values_bytes);
let (val_tag, val_bytes) = values
.take_any()
.map_err(|e| SignedObjectValidateError::SignedAttrsParse(e.to_string()))?;
match oid.as_str() {
OID_CMS_ATTR_CONTENT_TYPE => {
if content_type.is_some() {
return Err(SignedObjectValidateError::DuplicateSignedAttribute(oid));
}
if val_tag != 0x06 {
return Err(SignedObjectValidateError::SignedAttrsParse(
"content-type attr value must be OBJECT IDENTIFIER".into(),
));
}
content_type = Some(oid_value_bytes_to_string(val_bytes));
}
OID_CMS_ATTR_MESSAGE_DIGEST => {
if message_digest.is_some() {
return Err(SignedObjectValidateError::DuplicateSignedAttribute(oid));
}
if val_tag != 0x04 {
return Err(SignedObjectValidateError::SignedAttrsParse(
"message-digest attr value must be OCTET STRING".into(),
));
}
message_digest = Some(val_bytes.to_vec());
}
OID_CMS_ATTR_SIGNING_TIME => {
if signing_time.is_some() {
return Err(SignedObjectValidateError::DuplicateSignedAttribute(oid));
}
signing_time = Some(parse_signing_time_value_tlv(val_tag, val_bytes)?);
}
_ => {
return Err(SignedObjectValidateError::UnsupportedSignedAttribute(oid));
}
}
}
Ok(SignedAttrsProfiled {
content_type: content_type
.ok_or(SignedObjectValidateError::SignedAttrsContentTypeMissing)?,
message_digest: message_digest
.ok_or(SignedObjectValidateError::SignedAttrsMessageDigestMissing)?,
signing_time: signing_time
.ok_or(SignedObjectValidateError::SignedAttrsSigningTimeMissing)?,
other_attrs_present: false,
})
}
fn parse_signing_time_value_tlv(
tag: u8,
value: &[u8],
) -> Result<Asn1TimeUtc, SignedObjectValidateError> {
match tag {
0x17 => Ok(Asn1TimeUtc {
utc: parse_utctime(value)?,
encoding: Asn1TimeEncoding::UtcTime,
}),
0x18 => Ok(Asn1TimeUtc {
utc: parse_generalized_time(value)?,
encoding: Asn1TimeEncoding::GeneralizedTime,
}),
_ => Err(SignedObjectValidateError::InvalidSigningTimeValue),
}
}
fn make_signed_attrs_der_for_signature(full_tlv: &[u8]) -> Result<Vec<u8>, SignedObjectParseError> {
// We need the DER encoding of SignedAttributes (SET OF Attribute) as signature input.
// The SignedAttributes field in SignerInfo is `[0] IMPLICIT`, so the on-wire bytes start with
// a context-specific constructed tag (0xA0 for tag 0). For signature verification, this tag
// is replaced with the universal SET tag (0x31), leaving length+content unchanged.
//
let mut cs_der = full_tlv.to_vec();
if cs_der.is_empty() {
return Err(SignedObjectParseError::Parse(
"signedAttrs encoding is empty".into(),
));
}
// The first byte should be the context-specific tag (0xA0) for [0] constructed.
// Replace it with universal SET (0x31) for signature input.
cs_der[0] = 0x31;
Ok(cs_der)
}
fn take_oid_string(seq: &mut CmsReader<'_>) -> Result<String, SignedObjectParseError> {
let oid = seq
.take_tag(0x06)
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
Ok(oid_value_bytes_to_string(oid))
}
fn oid_value_bytes_to_string(oid_value: &[u8]) -> String {
if oid_value == OID_SHA256_RAW {
return OID_SHA256.to_string();
}
if oid_value == OID_SIGNED_DATA_RAW {
return OID_SIGNED_DATA.to_string();
}
if oid_value == OID_CMS_ATTR_CONTENT_TYPE_RAW {
return OID_CMS_ATTR_CONTENT_TYPE.to_string();
}
if oid_value == OID_CMS_ATTR_MESSAGE_DIGEST_RAW {
return OID_CMS_ATTR_MESSAGE_DIGEST.to_string();
}
if oid_value == OID_CMS_ATTR_SIGNING_TIME_RAW {
return OID_CMS_ATTR_SIGNING_TIME.to_string();
}
if oid_value == OID_RSA_ENCRYPTION_RAW {
return OID_RSA_ENCRYPTION.to_string();
}
if oid_value == OID_SHA256_WITH_RSA_ENCRYPTION_RAW {
return OID_SHA256_WITH_RSA_ENCRYPTION.to_string();
}
if oid_value == OID_CT_RPKI_MANIFEST_RAW {
return OID_CT_RPKI_MANIFEST.to_string();
}
if oid_value == OID_CT_ROUTE_ORIGIN_AUTHZ_RAW {
return OID_CT_ROUTE_ORIGIN_AUTHZ.to_string();
}
if oid_value == OID_CT_ASPA_RAW {
return OID_CT_ASPA.to_string();
}
decode_oid_to_dotted_string(oid_value)
}
fn decode_oid_to_dotted_string(value: &[u8]) -> String {
if value.is_empty() {
return "<empty-oid>".into();
}
let first = value[0];
let a = (first / 40) as u32;
let b = (first % 40) as u32;
let mut out = String::new();
out.push_str(&a.to_string());
out.push('.');
out.push_str(&b.to_string());
let mut idx = 1usize;
while idx < value.len() {
let mut v: u32 = 0;
loop {
if idx >= value.len() {
out.push_str(".<truncated>");
return out;
}
let byte = value[idx];
idx += 1;
v = (v << 7) | (byte as u32 & 0x7F);
if (byte & 0x80) == 0 {
break;
}
}
out.push('.');
out.push_str(&v.to_string());
}
out
}
fn parse_algorithm_identifier_cursor(
mut seq: CmsReader<'_>,
) -> Result<(String, bool), SignedObjectParseError> {
if seq.is_empty() {
return Err(SignedObjectParseError::Parse(
"AlgorithmIdentifier must be SEQUENCE of 1..2".into(),
));
}
let oid = take_oid_string(&mut seq)?;
let params_ok = if seq.is_empty() {
true
} else {
let (tag, value) = seq
.take_any()
.map_err(|e| SignedObjectParseError::Parse(e.to_string()))?;
tag == 0x05 && value.is_empty()
};
if !seq.is_empty() {
return Err(SignedObjectParseError::Parse(
"AlgorithmIdentifier must be SEQUENCE of 1..2".into(),
));
}
Ok((oid, params_ok))
}
fn parse_utctime(value: &[u8]) -> Result<time::OffsetDateTime, SignedObjectValidateError> {
let s = std::str::from_utf8(value)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
if !s.ends_with('Z') {
return Err(SignedObjectValidateError::InvalidSigningTimeValue);
}
let digits = &s[..s.len() - 1];
if digits.len() != 10 && digits.len() != 12 {
return Err(SignedObjectValidateError::InvalidSigningTimeValue);
}
if !digits.as_bytes().iter().all(|b| b.is_ascii_digit()) {
return Err(SignedObjectValidateError::InvalidSigningTimeValue);
}
let yy: i32 = digits[0..2]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let year = if yy <= 49 { 2000 + yy } else { 1900 + yy };
let mon: u8 = digits[2..4]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let day: u8 = digits[4..6]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let hour: u8 = digits[6..8]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let min: u8 = digits[8..10]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let sec: u8 = if digits.len() == 12 {
digits[10..12]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?
} else {
0
};
let month = time::Month::try_from(mon)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let date = time::Date::from_calendar_date(year, month, day)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let time = time::Time::from_hms(hour, min, sec)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
Ok(time::OffsetDateTime::new_utc(date, time))
}
fn parse_generalized_time(value: &[u8]) -> Result<time::OffsetDateTime, SignedObjectValidateError> {
let s = std::str::from_utf8(value)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
if !s.ends_with('Z') {
return Err(SignedObjectValidateError::InvalidSigningTimeValue);
}
let digits = &s[..s.len() - 1];
if digits.len() != 12 && digits.len() != 14 {
return Err(SignedObjectValidateError::InvalidSigningTimeValue);
}
if !digits.as_bytes().iter().all(|b| b.is_ascii_digit()) {
return Err(SignedObjectValidateError::InvalidSigningTimeValue);
}
let year: i32 = digits[0..4]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let mon: u8 = digits[4..6]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let day: u8 = digits[6..8]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let hour: u8 = digits[8..10]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let min: u8 = digits[10..12]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let sec: u8 = if digits.len() == 14 {
digits[12..14]
.parse()
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?
} else {
0
};
let month = time::Month::try_from(mon)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let date = time::Date::from_calendar_date(year, month, day)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
let time = time::Time::from_hms(hour, min, sec)
.map_err(|_| SignedObjectValidateError::InvalidSigningTimeValue)?;
Ok(time::OffsetDateTime::new_utc(date, time))
}
fn strip_leading_zeros(bytes: &[u8]) -> &[u8] {
let mut idx = 0;
while idx < bytes.len() && bytes[idx] == 0 {
idx += 1;
}
if idx == bytes.len() {
&bytes[bytes.len() - 1..]
} else {
&bytes[idx..]
}
}

View File

@@ -0,0 +1,133 @@
// Signed-object decoding and RSA signature verification API.
impl RpkiSignedObject {
/// Parse a DER-encoded RPKI Signed Object (CMS ContentInfo wrapping SignedData).
///
/// This performs encoding/structure parsing only. Profile constraints are enforced by
/// `RpkiSignedObjectParsed::validate_profile`.
pub fn parse_der(der: &[u8]) -> Result<RpkiSignedObjectParsed, SignedObjectParseError> {
parse_signed_object_content_info(der, der, CmsParseMode::BerCompatible)
}
pub fn parse_der_strict_cms(
der: &[u8],
) -> Result<RpkiSignedObjectParsed, SignedObjectParseError> {
parse_signed_object_content_info(der, der, CmsParseMode::DerStrict)
}
/// Return the strict-DER CMS parse error for an object that was otherwise
/// accepted through the normal BER-compatible CMS parser.
///
/// Callers must only surface this as a compatibility warning after normal
/// decoding and validation have succeeded; a strict parse failure alone
/// does not prove that an arbitrary byte string is an RPKI signed object.
pub fn strict_cms_der_error(der: &[u8]) -> Option<SignedObjectParseError> {
Self::parse_der_strict_cms(der).err()
}
/// Decode a DER-encoded RPKI Signed Object (CMS ContentInfo wrapping SignedData) and enforce
/// the profile constraints from RFC 6488 §2-§3 and RFC 9589 §4.
pub fn decode_der(der: &[u8]) -> Result<Self, SignedObjectDecodeError> {
let parsed = Self::parse_der(der)?;
Ok(parsed.validate_profile()?)
}
pub fn decode_der_with_strict_options(
der: &[u8],
strict_cms_der: bool,
strict_name: bool,
) -> Result<Self, SignedObjectDecodeError> {
let parsed = if strict_cms_der {
Self::parse_der_strict_cms(der)?
} else {
Self::parse_der(der)?
};
Ok(parsed.validate_profile_with_strict_name(strict_name)?)
}
/// Scheme-A naming for signature verification.
pub fn verify(&self) -> Result<(), SignedObjectVerifyError> {
self.verify_signature()
}
/// Verify the CMS signature using the embedded EE certificate public key.
pub fn verify_signature(&self) -> Result<(), SignedObjectVerifyError> {
let ee = &self.signed_data.certificates[0];
let signer = &self.signed_data.signer_infos[0];
crate::crypto_sig_cache::verify_with_cache(
crate::crypto_sig_cache::CryptoSigVerifyPoint::CmsSignedObject,
&signer.signed_attrs_der_for_signature,
&signer.signature,
&ee.spki_der,
|| {
self.verify_signature_with_rsa_components(
&ee.rsa_public_modulus,
&ee.rsa_public_exponent,
)
},
)
}
/// Verify the CMS signature using a DER-encoded SubjectPublicKeyInfo.
pub fn verify_signature_with_ee_spki_der(
&self,
ee_spki_der: &[u8],
) -> Result<(), SignedObjectVerifyError> {
let (rem, spki) = SubjectPublicKeyInfo::from_der(ee_spki_der)
.map_err(|e| SignedObjectVerifyError::EeSpkiParse(e.to_string()))?;
if !rem.is_empty() {
return Err(SignedObjectVerifyError::EeSpkiTrailingBytes(rem.len()));
}
self.verify_signature_with_ee_spki(&spki)
}
/// Verify the CMS signature using a parsed SubjectPublicKeyInfo.
pub fn verify_signature_with_ee_spki(
&self,
ee_spki: &SubjectPublicKeyInfo<'_>,
) -> Result<(), SignedObjectVerifyError> {
let pk = ee_spki
.parsed()
.map_err(|_e| SignedObjectVerifyError::UnsupportedEePublicKeyAlgorithm)?;
let (n, e) = match pk {
PublicKey::RSA(rsa) => {
let n = strip_leading_zeros(rsa.modulus).to_vec();
let e = strip_leading_zeros(rsa.exponent).to_vec();
let _exp = rsa
.try_exponent()
.map_err(|_e| SignedObjectVerifyError::InvalidEeRsaExponent)?;
(n, e)
}
_ => return Err(SignedObjectVerifyError::UnsupportedEePublicKeyAlgorithm),
};
let signer = &self.signed_data.signer_infos[0];
crate::crypto_sig_cache::verify_with_cache(
crate::crypto_sig_cache::CryptoSigVerifyPoint::CmsSignedObject,
&signer.signed_attrs_der_for_signature,
&signer.signature,
ee_spki.raw,
|| self.verify_signature_with_rsa_components(n.as_slice(), e.as_slice()),
)
}
fn verify_signature_with_rsa_components(
&self,
modulus: &[u8],
exponent: &[u8],
) -> Result<(), SignedObjectVerifyError> {
let signer = &self.signed_data.signer_infos[0];
let msg = &signer.signed_attrs_der_for_signature;
let pk = ring::signature::RsaPublicKeyComponents {
n: modulus,
e: exponent,
};
pk.verify(
&ring::signature::RSA_PKCS1_2048_8192_SHA256,
msg,
&signer.signature,
)
.map_err(|_e| SignedObjectVerifyError::InvalidSignature)
}
}

View File

@@ -0,0 +1,116 @@
// Signed-object CMS compatibility and profile tests.
use super::*;
#[test]
fn strict_cms_der_rejects_constructed_octet_string_fixture() {
let der =
std::fs::read(std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(
"tests/fixtures/repository/ca.rg.net/rpki/RGnet-OU/bW-_qXU9uNhGQz21NR2ansB8lr0.mft",
))
.expect("read manifest fixture");
let parsed = RpkiSignedObject::parse_der(&der).expect("parse fixture");
let econtent = parsed
.signed_data
.encap_content_info
.econtent
.clone()
.expect("fixture eContent");
assert_eq!(
parsed.signed_data.encap_content_info.econtent.as_deref(),
Some(econtent.as_slice())
);
let primitive_octets = der_tlv(0x04, &econtent);
let constructed_octets = same_size_constructed_octet_string(&primitive_octets, &econtent);
let mutated = replace_first_subslice(&der, &primitive_octets, &constructed_octets)
.expect("replace eContent OCTET STRING");
let compatible = RpkiSignedObject::parse_der(&mutated).expect("BER-compatible parse");
assert!(
econtent.starts_with(
compatible
.signed_data
.encap_content_info
.econtent
.as_deref()
.expect("compatible eContent")
)
);
let err = RpkiSignedObject::parse_der_strict_cms(&mutated)
.expect_err("DER strict rejects constructed OCTET STRING");
assert!(err.to_string().contains("DER"), "{err}");
let compatibility_error = RpkiSignedObject::strict_cms_der_error(&mutated)
.expect("BER-compatible fixture must report strict-DER incompatibility");
assert_eq!(compatibility_error.to_string(), err.to_string());
assert!(RpkiSignedObject::strict_cms_der_error(&der).is_none());
}
fn replace_first_subslice(input: &[u8], from: &[u8], to: &[u8]) -> Option<Vec<u8>> {
let pos = input
.windows(from.len())
.position(|candidate| candidate == from)?;
let mut out = Vec::with_capacity(input.len() - from.len() + to.len());
out.extend_from_slice(&input[..pos]);
out.extend_from_slice(to);
out.extend_from_slice(&input[pos + from.len()..]);
Some(out)
}
fn same_size_constructed_octet_string(primitive: &[u8], content: &[u8]) -> Vec<u8> {
assert_eq!(primitive[0], 0x04);
let header_len = tlv_header_len(primitive);
let outer_value_len = primitive.len() - header_len;
let child_len = (0..=outer_value_len)
.rev()
.find(|candidate| 1 + len_len(*candidate) + *candidate == outer_value_len)
.expect("find child length");
let mut out = primitive[..header_len].to_vec();
out[0] = 0x24;
out.extend(der_tlv(0x04, &content[..child_len]));
assert_eq!(out.len(), primitive.len());
out
}
fn tlv_header_len(tlv: &[u8]) -> usize {
if tlv[1] & 0x80 == 0 {
2
} else {
2 + (tlv[1] & 0x7F) as usize
}
}
fn len_len(len: usize) -> usize {
if len < 0x80 {
return 1;
}
let mut value = len;
let mut n = 0usize;
while value > 0 {
n += 1;
value >>= 8;
}
1 + n
}
fn der_tlv(tag: u8, value: &[u8]) -> Vec<u8> {
let mut out = vec![tag];
encode_len(value.len(), &mut out);
out.extend_from_slice(value);
out
}
fn encode_len(len: usize, out: &mut Vec<u8>) {
if len < 0x80 {
out.push(len as u8);
return;
}
let mut bytes = Vec::new();
let mut value = len;
while value > 0 {
bytes.push((value & 0xFF) as u8);
value >>= 8;
}
bytes.reverse();
out.push(0x80 | bytes.len() as u8);
out.extend(bytes);
}

View File

@@ -0,0 +1,300 @@
// CMS signed-object model types and parse/validation errors.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum EeKeyUsageSummary {
DigitalSignatureOnly,
Missing,
NotCritical,
InvalidBits,
ParseError(String),
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ResourceEeCertificate {
pub raw_der: Vec<u8>,
pub subject_key_identifier: Vec<u8>,
pub spki_der: Vec<u8>,
pub rsa_public_modulus: Vec<u8>,
pub rsa_public_exponent: Vec<u8>,
pub tbs_certificate_der: Vec<u8>,
pub signature_bytes: Vec<u8>,
pub key_usage_summary: EeKeyUsageSummary,
pub sia_signed_object_uris: Vec<String>,
pub resource_cert: ResourceCertificate,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RpkiSignedObject {
pub raw_der: Vec<u8>,
pub content_info_content_type: String,
pub signed_data: SignedDataProfiled,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SignedDataProfiled {
pub version: u32,
pub digest_algorithms: Vec<String>,
pub encap_content_info: EncapsulatedContentInfo,
pub certificates: Vec<ResourceEeCertificate>,
pub crls_present: bool,
pub signer_infos: Vec<SignerInfoProfiled>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct EncapsulatedContentInfo {
pub econtent_type: String,
pub econtent: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SignerInfoProfiled {
pub version: u32,
pub sid_ski: Vec<u8>,
pub digest_algorithm: String,
pub signature_algorithm: String,
pub signed_attrs: SignedAttrsProfiled,
pub unsigned_attrs_present: bool,
pub signature: Vec<u8>,
pub signed_attrs_der_for_signature: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SignedAttrsProfiled {
pub content_type: String,
pub message_digest: Vec<u8>,
pub signing_time: Asn1TimeUtc,
pub other_attrs_present: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RpkiSignedObjectParsed {
pub raw_der: Vec<u8>,
pub content_info_content_type: String,
pub signed_data: SignedDataParsed,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SignedDataParsed {
pub version: u64,
pub digest_algorithms: Vec<AlgorithmIdentifierParsed>,
pub encap_content_info: EncapsulatedContentInfoParsed,
pub certificates: Option<Vec<Vec<u8>>>,
pub crls_present: bool,
pub signer_infos: Vec<SignerInfoParsed>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AlgorithmIdentifierParsed {
pub oid: String,
pub params_ok: bool,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct EncapsulatedContentInfoParsed {
pub econtent_type: String,
pub econtent: Option<Vec<u8>>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SignerInfoParsed {
pub version: u64,
pub sid: SignerIdentifierParsed,
pub digest_algorithm: AlgorithmIdentifierParsed,
pub signature_algorithm: AlgorithmIdentifierParsed,
pub signed_attrs_content: Option<Vec<u8>>,
pub signed_attrs_der_for_signature: Option<Vec<u8>>,
pub unsigned_attrs_present: bool,
pub signature: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum SignerIdentifierParsed {
SubjectKeyIdentifier(Vec<u8>),
Other,
}
#[derive(Debug, thiserror::Error)]
pub enum SignedObjectParseError {
#[error("DER parse error: {0} (RFC 6488 §2; RFC 6488 §3(1l); RFC 5652 §3/§5)")]
Parse(String),
#[error("trailing bytes after DER object: {0} bytes (DER; RFC 6488 §3(1l))")]
TrailingBytes(usize),
}
#[derive(Debug, thiserror::Error)]
pub enum SignedObjectValidateError {
#[error(
"ContentInfo.contentType must be SignedData ({OID_SIGNED_DATA}), got {0} (RFC 6488 §3(1a); RFC 5652 §3)"
)]
InvalidContentInfoContentType(String),
#[error(
"SignedData.version must be 3, got {0} (RFC 6488 §2.1.1; RFC 6488 §3(1b); RFC 5652 §5.1)"
)]
InvalidSignedDataVersion(u64),
#[error(
"SignedData.digestAlgorithms must contain exactly one AlgorithmIdentifier, got {0} (RFC 6488 §2.1.2; RFC 6488 §3(1b); RFC 5652 §5.1)"
)]
InvalidDigestAlgorithmsCount(usize),
#[error(
"digest algorithm must be id-sha256 ({OID_SHA256}), got {0} (RFC 6488 §2.1.2; RFC 6488 §3(1b); RFC 7935 §2)"
)]
InvalidDigestAlgorithm(String),
#[error("SignedData.certificates MUST be present (RFC 6488 §3(1c); RFC 5652 §5.1)")]
CertificatesMissing,
#[error(
"SignedData.certificates must contain exactly one EE certificate, got {0} (RFC 6488 §3(1c))"
)]
InvalidCertificatesCount(usize),
#[error("SignedData.crls MUST be omitted (RFC 6488 §3(1d))")]
CrlsPresent,
#[error(
"SignedData.signerInfos must contain exactly one SignerInfo, got {0} (RFC 6488 §2.1; RFC 6488 §3(1e); RFC 5652 §5.1)"
)]
InvalidSignerInfosCount(usize),
#[error("SignerInfo.version must be 3, got {0} (RFC 6488 §3(1e); RFC 5652 §5.3)")]
InvalidSignerInfoVersion(u64),
#[error("SignerInfo.sid must be subjectKeyIdentifier [0] (RFC 6488 §3(1c); RFC 5652 §5.3)")]
InvalidSignerIdentifier,
#[error(
"SignerInfo.digestAlgorithm must be id-sha256 ({OID_SHA256}), got {0} (RFC 6488 §3(1j); RFC 7935 §2)"
)]
InvalidSignerInfoDigestAlgorithm(String),
#[error("SignerInfo.signedAttrs MUST be present (RFC 9589 §4; RFC 6488 §3(1f))")]
SignedAttrsMissing,
#[error("SignerInfo.unsignedAttrs MUST be omitted (RFC 6488 §3(1i))")]
UnsignedAttrsPresent,
#[error(
"SignerInfo.signatureAlgorithm must be rsaEncryption ({OID_RSA_ENCRYPTION}) or \
sha256WithRSAEncryption ({OID_SHA256_WITH_RSA_ENCRYPTION}), got {0} (RFC 6488 §3(1k); RFC 7935 §2)"
)]
InvalidSignatureAlgorithm(String),
#[error(
"SignerInfo.signatureAlgorithm parameters must be absent or NULL (RFC 5280 §4.1.1.2; RFC 7935 §2)"
)]
InvalidSignatureAlgorithmParameters,
#[error("signedAttrs contains unsupported attribute OID {0} (RFC 9589 §4; RFC 6488 §2.1.6.4)")]
UnsupportedSignedAttribute(String),
#[error("signedAttrs contains duplicate attribute OID {0} (RFC 6488 §2.1.6.4; RFC 9589 §4)")]
DuplicateSignedAttribute(String),
#[error("signedAttrs parse error: {0} (RFC 5652 §5.3; RFC 6488 §3(1f); RFC 9589 §4)")]
SignedAttrsParse(String),
#[error(
"signedAttrs attribute {oid} attrValues must contain exactly one value, got {count} (RFC 6488 §2.1.6.4; RFC 5652 §5.3)"
)]
InvalidSignedAttributeValuesCount { oid: String, count: usize },
#[error(
"signedAttrs missing content-type attribute (RFC 9589 §4; RFC 5652 §11.1; RFC 6488 §2.1.6.4)"
)]
SignedAttrsContentTypeMissing,
#[error(
"signedAttrs missing message-digest attribute (RFC 9589 §4; RFC 5652 §11.2; RFC 6488 §2.1.6.4)"
)]
SignedAttrsMessageDigestMissing,
#[error(
"signedAttrs missing signing-time attribute (RFC 9589 §4; RFC 5652 §11.3; RFC 6488 §2.1.6.4)"
)]
SignedAttrsSigningTimeMissing,
#[error(
"signedAttrs.content-type attrValues must equal eContentType ({econtent_type}), got {attr_content_type} (RFC 6488 §3(1h); RFC 9589 §4)"
)]
ContentTypeAttrMismatch {
econtent_type: String,
attr_content_type: String,
},
#[error("EncapsulatedContentInfo.eContent MUST be present (RFC 6488 §2.1.3; RFC 5652 §5.2)")]
EContentMissing,
#[error(
"signedAttrs.message-digest does not match SHA-256(eContent) (RFC 6488 §3(1f); RFC 5652 §11.2)"
)]
MessageDigestMismatch,
#[error("EE certificate parse error: {0} (RFC 6488 §3(1c); RFC 6487 §4)")]
EeCertificateParse(String),
#[error(
"EE certificate missing SubjectKeyIdentifier extension (RFC 6488 §3(1c); RFC 6487 §4.8.2)"
)]
EeCertificateMissingSki,
#[error(
"EE certificate missing SubjectInfoAccess extension ({OID_SUBJECT_INFO_ACCESS}) (RFC 6487 §4.8.8.2)"
)]
EeCertificateMissingSia,
#[error(
"EE certificate SIA missing id-ad-signedObject access method ({OID_AD_SIGNED_OBJECT}) (RFC 6487 §4.8.8.2)"
)]
EeCertificateMissingSignedObjectSia,
#[error(
"EE certificate SIA id-ad-signedObject accessLocation must be a URI (RFC 6487 §4.8.8.2; RFC 5280 §4.2.2.2)"
)]
EeCertificateSignedObjectSiaNotUri,
#[error(
"EE certificate SIA id-ad-signedObject must include at least one rsync:// URI (RFC 6487 §4.8.8.2)"
)]
EeCertificateSignedObjectSiaNoRsync,
#[error(
"SignerInfo.sid SKI does not match EE certificate SKI (RFC 6488 §3(1c); RFC 5652 §5.3)"
)]
SidSkiMismatch,
#[error(
"invalid signing-time attribute value (expected UTCTime or GeneralizedTime) (RFC 5652 §11.3; RFC 9589 §4)"
)]
InvalidSigningTimeValue,
}
#[derive(Debug, thiserror::Error)]
pub enum SignedObjectDecodeError {
#[error("SignedObject parse error: {0}")]
Parse(#[from] SignedObjectParseError),
#[error("SignedObject validate error: {0}")]
Validate(#[from] SignedObjectValidateError),
}
#[derive(Debug, thiserror::Error)]
pub enum SignedObjectVerifyError {
#[error("EE SubjectPublicKeyInfo parse error: {0} (RFC 5280 §4.1.2.7)")]
EeSpkiParse(String),
#[error("trailing bytes after EE SubjectPublicKeyInfo DER: {0} bytes (DER; RFC 5280 §4.1.2.7)")]
EeSpkiTrailingBytes(usize),
#[error("unsupported EE public key algorithm (only RSA is supported) (RFC 7935 §2)")]
UnsupportedEePublicKeyAlgorithm,
#[error("EE RSA public exponent invalid (RFC 8017 §A.1.1; RFC 7935 §2)")]
InvalidEeRsaExponent,
#[error("signature verification failed (RFC 6488 §3(2)-(3); RFC 5652 §5.3; RFC 7935 §2)")]
InvalidSignature,
}

View File

@@ -67,7 +67,7 @@ impl Default for HttpFetcherConfig {
}
}
/// Minimal blocking HTTP(S) fetcher for stage2.
/// Minimal blocking HTTP(S) fetcher for validation runs.
///
/// This is used for:
/// - downloading TAL / TA certificates (RFC 8630 §2)

View File

@@ -72,7 +72,7 @@ impl Default for SystemRsyncConfig {
/// A `RsyncFetcher` implementation backed by the system `rsync` binary.
///
/// This is intended for live stage2 runs. For unit tests and offline fixtures,
/// This is intended for live synchronization runs. For unit tests and offline fixtures,
/// prefer `LocalDirRsyncFetcher`.
#[derive(Clone, Debug)]
pub struct SystemRsyncFetcher {

View File

@@ -1,14 +1,12 @@
//! Public package boundary for the Panda RPKI synchronization validator.
//!
//! The package intentionally keeps the audited implementation in one crate
//! while exposing only the normal synchronization/validation CLI. A separate
//! offline revalidation workflow remains in the original private repository
//! until a follow-up backlog item defines its public contract.
//! The package intentionally keeps the synchronization and validation runtime
//! in one crate. The command-line interface is the supported integration
//! surface; internal modules are progressively documented and narrowed as the
//! public API is defined.
//!
//! The implementation is extracted from the private `rpki` tree. During the
//! parity window we intentionally keep the upstream implementation's lint
//! profile unchanged; functional and output-equivalence gates take precedence
//! over behavior-changing lint cleanup.
//! The crate preserves a stable run-artifact contract. Behavioural changes must
//! be validated against fixed offline inputs before they are released.
#![allow(clippy::all)]
#![allow(clippy::pedantic, clippy::nursery, clippy::restriction)]
#![allow(dead_code, deprecated, unused_mut)]

View File

@@ -5,7 +5,7 @@ pub struct ParallelPhase1Config {
pub max_repo_sync_workers_global: usize,
pub max_inflight_snapshot_bytes_global: usize,
pub max_pending_repo_results: usize,
/// Dead-repo transport blacklist (#141). `None` disables the feature
/// Dead-repository transport blacklist. None disables the feature
/// entirely (default, behavior unchanged).
pub dead_repo_blacklist: Option<DeadRepoBlacklistConfig>,
}

View File

@@ -1,4 +1,4 @@
//! Persistent dead-repo transport blacklist (#141).
//! Persistent dead-repository transport blacklist.
//!
//! Tracks per-(repo, transport) consecutive transport-layer fetch failures
//! across runs. Once an entry reaches the configured failure threshold it is
@@ -340,7 +340,7 @@ pub fn classify_rsync_fetch_error(detail: &str) -> RepoTransportErrorClass {
if crate::parallel::repo_scheduler::is_host_level_rsync_failure(detail)
// Stall failures are transport death too: the fail-fast mechanism
// gives up when wall-clock windows pass with no (additional) progress
// (#141 M9). Content-level errors fail immediately and never carry
// Content-level errors fail immediately and never carry
// this marker, so they still do not count.
|| detail.contains("rsync fail-fast gave up")
{

View File

@@ -371,7 +371,7 @@ fn rsync_fetch_error_classification() {
classify_rsync_fetch_error("rsync file digest mismatch after download"),
RepoTransportErrorClass::Unknown
);
// Stall failures (#141 M9): fail-fast give-up means the host delivered no
// Stall failures: fail-fast give-up means the host delivered no
// bytes inside the wall-clock window — transport death, must count.
assert_eq!(
classify_rsync_fetch_error(

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,58 @@
// Conversion from transport envelopes to runtime outcomes.
fn outcome_from_transport_result(
envelope: &RepoTransportResultEnvelope,
state: RepoRuntimeState,
) -> RepoSyncRuntimeOutcome {
match (&envelope.result, state) {
(RepoTransportResultKind::Success { source, warnings }, RepoRuntimeState::RrdpOk) => {
RepoSyncRuntimeOutcome {
repo_sync_ok: true,
repo_sync_err: None,
repo_sync_source: Some(source.clone()),
repo_sync_phase: Some("rrdp_ok".to_string()),
repo_sync_duration_ms: envelope.timing_ms,
warnings: warnings.clone(),
}
}
(RepoTransportResultKind::Success { source, warnings }, RepoRuntimeState::RsyncOk) => {
RepoSyncRuntimeOutcome {
repo_sync_ok: true,
repo_sync_err: None,
repo_sync_source: Some(source.clone()),
repo_sync_phase: Some(if envelope.repo_identity.notification_uri.is_some() {
"rrdp_failed_rsync_ok".to_string()
} else {
"rsync_only_ok".to_string()
}),
repo_sync_duration_ms: envelope.timing_ms,
warnings: warnings.clone(),
}
}
(
RepoTransportResultKind::Failed {
detail, warnings, ..
},
RepoRuntimeState::FailedTerminal,
) => RepoSyncRuntimeOutcome {
repo_sync_ok: false,
repo_sync_err: Some(detail.clone()),
repo_sync_source: None,
repo_sync_phase: Some(if envelope.repo_identity.notification_uri.is_some() {
"rrdp_failed_rsync_failed".to_string()
} else {
"rsync_failed".to_string()
}),
repo_sync_duration_ms: envelope.timing_ms,
warnings: warnings.clone(),
},
_ => RepoSyncRuntimeOutcome {
repo_sync_ok: false,
repo_sync_err: Some("repo runtime state unresolved".to_string()),
repo_sync_source: None,
repo_sync_phase: Some("repo_runtime_unresolved".to_string()),
repo_sync_duration_ms: envelope.timing_ms,
warnings: Vec::new(),
},
}
}

View File

@@ -0,0 +1,372 @@
// Phase-one runtime request registration and result pumping.
pub struct Phase1RepoSyncRuntime<E: RepoTransportExecutor> {
coordinator: Mutex<GlobalRunCoordinator>,
worker_pool: Mutex<RepoTransportWorkerPool<E>>,
transport_prefetch_recorder: Option<Mutex<TransportPrefetchRecorder>>,
retry_short_rsync_scopes: Mutex<HashSet<String>>,
rsync_scope_resolver: Arc<dyn Fn(&str) -> String + Send + Sync>,
rsync_failure_scope_resolver: Arc<dyn Fn(&str) -> Option<String> + Send + Sync>,
sync_preference: SyncPreference,
}
impl<E: RepoTransportExecutor> Phase1RepoSyncRuntime<E> {
pub fn new(
coordinator: GlobalRunCoordinator,
worker_pool: RepoTransportWorkerPool<E>,
rsync_scope_resolver: Arc<dyn Fn(&str) -> String + Send + Sync>,
sync_preference: SyncPreference,
) -> Self {
Self::new_with_failure_scope(
coordinator,
worker_pool,
rsync_scope_resolver,
Arc::new(|_base: &str| None),
sync_preference,
)
}
pub fn new_with_failure_scope(
coordinator: GlobalRunCoordinator,
worker_pool: RepoTransportWorkerPool<E>,
rsync_scope_resolver: Arc<dyn Fn(&str) -> String + Send + Sync>,
rsync_failure_scope_resolver: Arc<dyn Fn(&str) -> Option<String> + Send + Sync>,
sync_preference: SyncPreference,
) -> Self {
Self {
coordinator: Mutex::new(coordinator),
worker_pool: Mutex::new(worker_pool),
transport_prefetch_recorder: None,
retry_short_rsync_scopes: Mutex::new(HashSet::new()),
rsync_scope_resolver,
rsync_failure_scope_resolver,
sync_preference,
}
}
pub fn new_with_failure_scope_and_prefetch_recording(
coordinator: GlobalRunCoordinator,
worker_pool: RepoTransportWorkerPool<E>,
rsync_scope_resolver: Arc<dyn Fn(&str) -> String + Send + Sync>,
rsync_failure_scope_resolver: Arc<dyn Fn(&str) -> Option<String> + Send + Sync>,
sync_preference: SyncPreference,
record_transport_prefetch_requests: bool,
) -> Self {
Self {
coordinator: Mutex::new(coordinator),
worker_pool: Mutex::new(worker_pool),
transport_prefetch_recorder: record_transport_prefetch_requests
.then(|| Mutex::new(TransportPrefetchRecorder::default())),
retry_short_rsync_scopes: Mutex::new(HashSet::new()),
rsync_scope_resolver,
rsync_failure_scope_resolver,
sync_preference,
}
}
fn build_requester(ca: &CaInstanceHandle) -> RepoRequester {
RepoRequester {
tal_id: ca.tal_id.clone(),
rir_id: ca.tal_id.clone(),
parent_node_id: None,
ca_instance_handle_id: format!("{}:{}", ca.tal_id, ca.manifest_rsync_uri),
publication_point_rsync_uri: ca.publication_point_rsync_uri.clone(),
manifest_rsync_uri: ca.manifest_rsync_uri.clone(),
}
}
fn build_identity(ca: &CaInstanceHandle) -> RepoIdentity {
RepoIdentity::new(ca.rrdp_notification_uri.clone(), ca.rsync_base_uri.clone())
}
fn request_transport_for_ca(
&self,
ca: &CaInstanceHandle,
priority: u8,
) -> Result<RepoSyncRequestStatus, String> {
let identity = Self::build_identity(ca);
let requester = Self::build_requester(ca);
let rsync_scope_uri = (self.rsync_scope_resolver)(&identity.rsync_base_uri);
let rsync_failure_scope_uri = (self.rsync_failure_scope_resolver)(&identity.rsync_base_uri);
if let Some(recorder) = self.transport_prefetch_recorder.as_ref() {
let mut recorder = recorder
.lock()
.expect("transport prefetch recorder lock poisoned");
recorder.record_registered_request(
&identity,
&requester,
priority,
rsync_scope_uri.clone(),
rsync_failure_scope_uri.clone(),
self.sync_preference,
);
}
let action = {
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator.register_transport_request(
identity.clone(),
requester,
time::OffsetDateTime::now_utc(),
priority,
rsync_scope_uri,
rsync_failure_scope_uri,
self.sync_preference,
false,
)
};
match action {
TransportRequestAction::Enqueue(task) => {
crate::progress_log::emit(
"phase1_repo_task_enqueued",
serde_json::json!({
"manifest_rsync_uri": ca.manifest_rsync_uri,
"publication_point_rsync_uri": ca.publication_point_rsync_uri,
"repo_key_rsync_base_uri": task.repo_identity.rsync_base_uri,
"rsync_failure_scope_uri": task.rsync_failure_scope_uri,
"repo_key_notification_uri": task.repo_identity.notification_uri,
"priority": priority,
"transport_mode": match task.mode {
RepoTransportMode::Rrdp => "rrdp",
RepoTransportMode::Rsync => "rsync",
},
}),
);
self.drain_pending_transport_tasks()?;
Ok(RepoSyncRequestStatus::Pending {
identity,
state: self
.runtime_state_for_identity(&task.repo_identity)
.unwrap_or(RepoRuntimeState::WaitingRrdp),
})
}
TransportRequestAction::Waiting { state } => {
crate::progress_log::emit(
"phase1_repo_task_waiting",
serde_json::json!({
"manifest_rsync_uri": ca.manifest_rsync_uri,
"publication_point_rsync_uri": ca.publication_point_rsync_uri,
"repo_key_rsync_base_uri": identity.rsync_base_uri,
"rsync_failure_scope_uri": (self.rsync_failure_scope_resolver)(&identity.rsync_base_uri),
"repo_key_notification_uri": identity.notification_uri,
"priority": priority,
"runtime_state": format!("{state:?}"),
}),
);
Ok(RepoSyncRequestStatus::Pending { identity, state })
}
TransportRequestAction::ReusedSuccess(result)
| TransportRequestAction::ReusedTerminalFailure(result) => {
crate::progress_log::emit(
"phase1_repo_task_reused",
serde_json::json!({
"manifest_rsync_uri": ca.manifest_rsync_uri,
"publication_point_rsync_uri": ca.publication_point_rsync_uri,
"repo_key_rsync_base_uri": identity.rsync_base_uri,
"rsync_failure_scope_uri": result.rsync_failure_scope_uri,
"repo_key_notification_uri": identity.notification_uri,
"priority": priority,
"transport_mode": match result.mode {
RepoTransportMode::Rrdp => "rrdp",
RepoTransportMode::Rsync => "rsync",
},
}),
);
Ok(RepoSyncRequestStatus::Ready {
outcome: outcome_from_transport_result(
&result,
self.runtime_state_for_identity(&identity)
.unwrap_or(RepoRuntimeState::Init),
),
identity,
})
}
}
}
fn drain_pending_transport_tasks(&self) -> Result<(), String> {
loop {
let maybe_task = {
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator.pop_next_transport_task()
};
let Some(task) = maybe_task else {
break;
};
{
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator
.mark_transport_running(&task.dedup_key, time::OffsetDateTime::now_utc())?;
}
crate::progress_log::emit(
"phase1_repo_task_dispatched",
serde_json::json!({
"repo_key_rsync_base_uri": task.repo_identity.rsync_base_uri,
"rsync_failure_scope_uri": task.rsync_failure_scope_uri,
"repo_key_notification_uri": task.repo_identity.notification_uri,
"requester_count": task.requesters.len(),
"priority": task.priority,
"transport_mode": match task.mode {
RepoTransportMode::Rrdp => "rrdp",
RepoTransportMode::Rsync => "rsync",
},
}),
);
let pool = self.worker_pool.lock().expect("worker pool lock poisoned");
pool.submit(task)?;
}
Ok(())
}
fn pump_one_transport_result(
&self,
timeout: Duration,
) -> Result<Option<RepoSyncRuntimeEvent>, String> {
let envelope = {
let pool = self.worker_pool.lock().expect("worker pool lock poisoned");
pool.recv_result_timeout(timeout)?
};
let Some(envelope) = envelope else {
return Ok(None);
};
let transport_identity = envelope.repo_identity.clone();
let completed_envelope = envelope.clone();
if let Some(recorder) = self.transport_prefetch_recorder.as_ref() {
recorder
.lock()
.expect("transport prefetch recorder lock poisoned")
.record_result(&envelope);
}
crate::progress_log::emit(
"phase1_repo_task_result",
serde_json::json!({
"repo_key_rsync_base_uri": envelope.repo_identity.rsync_base_uri,
"rsync_failure_scope_uri": envelope.rsync_failure_scope_uri,
"repo_key_notification_uri": envelope.repo_identity.notification_uri,
"timing_ms": envelope.timing_ms,
"transport_mode": match envelope.mode {
RepoTransportMode::Rrdp => "rrdp",
RepoTransportMode::Rsync => "rsync",
},
"result": match &envelope.result {
RepoTransportResultKind::Success { .. } => "success",
RepoTransportResultKind::Failed { .. } => "failed",
},
}),
);
let finished_at = time::OffsetDateTime::now_utc();
let completion = {
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator.complete_transport_result(envelope, finished_at)?
};
if !completion.follow_up_tasks.is_empty() {
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
for mut task in completion.follow_up_tasks {
if let crate::parallel::types::RepoDedupKey::RsyncScope { rsync_scope_uri } =
&task.dedup_key
{
if self
.retry_short_rsync_scopes
.lock()
.expect("retry short rsync scopes lock poisoned")
.contains(rsync_scope_uri)
{
task.retry_short_timeout = true;
}
}
crate::progress_log::emit(
"phase1_repo_task_enqueued",
serde_json::json!({
"manifest_rsync_uri": serde_json::Value::Null,
"publication_point_rsync_uri": task.requesters.first().map(|r| r.publication_point_rsync_uri.clone()),
"repo_key_rsync_base_uri": task.repo_identity.rsync_base_uri,
"repo_key_notification_uri": task.repo_identity.notification_uri,
"priority": task.priority,
"transport_mode": "rsync",
}),
);
coordinator.push_transport_task(task);
}
}
self.drain_pending_transport_tasks()?;
let completions = {
let coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator
.finalized_runtime_records_for_transport_result(&completed_envelope)
.into_iter()
.filter_map(|record| {
let outcome = match record.state {
RepoRuntimeState::RrdpOk | RepoRuntimeState::RsyncOk => record
.last_success
.as_ref()
.map(|result| outcome_from_transport_result(result, record.state)),
RepoRuntimeState::FailedTerminal => record
.terminal_failure
.as_ref()
.map(|result| outcome_from_transport_result(result, record.state)),
_ => None,
}?;
Some(RepoSyncRuntimeCompletion {
identity: record.identity,
state: record.state,
outcome,
})
})
.collect::<Vec<_>>()
};
if completions.is_empty() {
return Ok(None);
}
Ok(Some(RepoSyncRuntimeEvent {
transport_identity,
completions,
}))
}
fn pump_transport_results(
&self,
timeout: Duration,
max_events: usize,
) -> Result<Vec<RepoSyncRuntimeEvent>, String> {
let max_events = max_events.max(1);
let mut events = Vec::new();
for index in 0..max_events {
let poll_timeout = if index == 0 {
timeout
} else {
Duration::from_millis(0)
};
let Some(event) = self.pump_one_transport_result(poll_timeout)? else {
break;
};
events.push(event);
}
Ok(events)
}
fn runtime_state_for_identity(&self, identity: &RepoIdentity) -> Option<RepoRuntimeState> {
let coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator
.runtime_record(identity)
.map(|record| record.state)
}
fn resolved_outcome_for_identity(
&self,
identity: &RepoIdentity,
) -> Option<RepoSyncRuntimeOutcome> {
let coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
let record = coordinator.runtime_record(identity)?;
match record.state {
RepoRuntimeState::RrdpOk | RepoRuntimeState::RsyncOk => record
.last_success
.as_ref()
.map(|result| outcome_from_transport_result(result, record.state)),
RepoRuntimeState::FailedTerminal => record
.terminal_failure
.as_ref()
.map(|result| outcome_from_transport_result(result, record.state)),
_ => None,
}
}
}

View File

@@ -0,0 +1,173 @@
// RepoSyncRuntime implementation for the phase-one runtime.
impl<E: RepoTransportExecutor> RepoSyncRuntime for Phase1RepoSyncRuntime<E> {
fn sync_publication_point_repo(
&self,
ca: &CaInstanceHandle,
) -> Result<RepoSyncRuntimeOutcome, String> {
if let RepoSyncRequestStatus::Ready { outcome, .. } =
self.request_publication_point_repo(ca, 0)?
{
return Ok(outcome);
}
let identity = Self::build_identity(ca);
loop {
if let Some(done) = self.resolved_outcome_for_identity(&identity) {
return Ok(done);
}
let _ = self.recv_repo_result_timeout(Duration::from_millis(50))?;
}
}
fn request_publication_point_repo(
&self,
ca: &CaInstanceHandle,
priority: u8,
) -> Result<RepoSyncRequestStatus, String> {
self.request_transport_for_ca(ca, priority)
}
fn recv_repo_result_timeout(
&self,
timeout: Duration,
) -> Result<Option<RepoSyncRuntimeEvent>, String> {
self.pump_one_transport_result(timeout)
}
fn drain_repo_results_timeout(
&self,
timeout: Duration,
max_events: usize,
) -> Result<Vec<RepoSyncRuntimeEvent>, String> {
self.pump_transport_results(timeout, max_events)
}
fn reset_run_state(&self) -> Result<(), String> {
{
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
if coordinator.stats.repo_tasks_running != 0 {
return Err(format!(
"cannot reset repo runtime with {} repo task(s) still running",
coordinator.stats.repo_tasks_running
));
}
coordinator.reset_run_state();
}
loop {
let maybe_result = {
let pool = self.worker_pool.lock().expect("worker pool lock poisoned");
pool.recv_result_timeout(Duration::from_millis(0))?
};
if maybe_result.is_none() {
break;
}
}
Ok(())
}
fn prefetch_discovered_children(
&self,
children: &[DiscoveredChildCaInstance],
) -> Result<(), String> {
for child in children {
let _ = self.request_publication_point_repo(&child.handle, 1)?;
}
Ok(())
}
fn prefetch_transport_requests(
&self,
snapshot: &TransportPrefetchSnapshot,
validation_time: time::OffsetDateTime,
) -> Result<TransportPrefetchDispatchStats, String> {
let mut stats = TransportPrefetchDispatchStats {
loaded_requests: snapshot.requests.len() as u64,
..TransportPrefetchDispatchStats::default()
};
for request in &snapshot.requests {
let identity = request.to_identity();
let current_rsync_scope_uri = (self.rsync_scope_resolver)(&identity.rsync_base_uri);
let current_rsync_failure_scope_uri =
(self.rsync_failure_scope_resolver)(&identity.rsync_base_uri);
if current_rsync_scope_uri != request.rsync_scope_uri
|| current_rsync_failure_scope_uri != request.rsync_failure_scope_uri
{
stats.skipped_incompatible += 1;
continue;
}
if request.retry_short_rsync_timeout() {
self.retry_short_rsync_scopes
.lock()
.expect("retry short rsync scopes lock poisoned")
.insert(current_rsync_scope_uri.clone());
}
let action = {
let mut coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
coordinator.register_transport_request(
identity,
request.to_requester(),
validation_time,
request.priority,
current_rsync_scope_uri,
current_rsync_failure_scope_uri,
self.sync_preference,
request.retry_short_timeout(),
)
};
match action {
TransportRequestAction::Enqueue(task) => {
stats.enqueued_tasks += 1;
crate::progress_log::emit(
"phase1_repo_prefetch_enqueued",
serde_json::json!({
"repo_key_rsync_base_uri": task.repo_identity.rsync_base_uri,
"rsync_failure_scope_uri": task.rsync_failure_scope_uri,
"repo_key_notification_uri": task.repo_identity.notification_uri,
"priority": task.priority,
"transport_mode": match task.mode {
RepoTransportMode::Rrdp => "rrdp",
RepoTransportMode::Rsync => "rsync",
},
}),
);
}
TransportRequestAction::Waiting { .. } => {
stats.waiting_requests += 1;
}
TransportRequestAction::ReusedSuccess(_)
| TransportRequestAction::ReusedTerminalFailure(_) => {
stats.reused_results += 1;
}
}
}
self.drain_pending_transport_tasks()?;
Ok(stats)
}
fn transport_prefetch_snapshot(&self) -> TransportPrefetchSnapshot {
self.transport_prefetch_recorder
.as_ref()
.map(|recorder| {
recorder
.lock()
.expect("transport prefetch recorder lock poisoned")
.snapshot(self.sync_preference)
})
.unwrap_or_else(|| TransportPrefetchSnapshot::new(self.sync_preference, Vec::new()))
}
fn dead_repo_blacklist_state(
&self,
) -> Option<(
crate::parallel::dead_repo_blacklist::DeadRepoBlacklistConfig,
crate::parallel::dead_repo_blacklist::DeadRepoBlacklist,
)> {
let coordinator = self.coordinator.lock().expect("coordinator lock poisoned");
let config = coordinator.config.dead_repo_blacklist.clone()?;
let blacklist = coordinator.dead_repo_blacklist().cloned()?;
Some((config, blacklist))
}
}

View File

@@ -0,0 +1,798 @@
// Repository runtime behavior tests.
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use crate::parallel::config::ParallelPhase1Config;
use crate::parallel::repo_runtime::{Phase1RepoSyncRuntime, RepoSyncRuntime};
use crate::parallel::repo_worker::{
RepoTransportExecutor, RepoTransportWorkerPool, RepoWorkerPoolConfig,
};
use crate::parallel::run_coordinator::GlobalRunCoordinator;
use crate::parallel::transport_prefetch::TransportPrefetchSnapshot;
use crate::parallel::types::{
RepoRuntimeState, RepoTransportMode, RepoTransportResultEnvelope, RepoTransportResultKind,
RepoTransportTask, TalInputSpec,
};
use crate::policy::SyncPreference;
use crate::report::Warning;
use crate::validation::tree::{CaCertificateRef, CaInstanceHandle, DiscoveredChildCaInstance};
fn sample_ca(manifest: &str) -> CaInstanceHandle {
CaInstanceHandle {
depth: 0,
tal_id: "arin".to_string(),
parent_manifest_rsync_uri: None,
ca_certificate: CaCertificateRef::inline_der(vec![1, 2, 3]),
ca_certificate_rsync_uri: None,
effective_ip_resources: None,
effective_as_resources: None,
rsync_base_uri: "rsync://example.test/repo/".to_string(),
manifest_rsync_uri: manifest.to_string(),
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
rrdp_notification_uri: Some("https://example.test/notify.xml".to_string()),
}
}
struct SuccessTransportExecutor;
impl RepoTransportExecutor for SuccessTransportExecutor {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: task.repo_identity,
mode: task.mode,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 7,
result: RepoTransportResultKind::Success {
source: match task.mode {
RepoTransportMode::Rrdp => "rrdp".to_string(),
RepoTransportMode::Rsync => "rsync".to_string(),
},
warnings: vec![Warning::new("transport ok")],
},
}
}
}
struct CountingSuccessTransportExecutor {
count: Arc<AtomicUsize>,
}
impl RepoTransportExecutor for CountingSuccessTransportExecutor {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
self.count.fetch_add(1, Ordering::SeqCst);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: task.repo_identity,
mode: task.mode,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 7,
result: RepoTransportResultKind::Success {
source: match task.mode {
RepoTransportMode::Rrdp => "rrdp".to_string(),
RepoTransportMode::Rsync => "rsync".to_string(),
},
warnings: vec![Warning::new("transport ok")],
},
}
}
}
struct FailRrdpThenSucceedRsyncExecutor {
rrdp_count: Arc<AtomicUsize>,
rsync_count: Arc<AtomicUsize>,
}
impl RepoTransportExecutor for FailRrdpThenSucceedRsyncExecutor {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
match task.mode {
RepoTransportMode::Rrdp => {
self.rrdp_count.fetch_add(1, Ordering::SeqCst);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rrdp,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 10,
result: RepoTransportResultKind::Failed {
detail: "rrdp failed".to_string(),
warnings: vec![Warning::new("rrdp failed")],
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
}
}
RepoTransportMode::Rsync => {
self.rsync_count.fetch_add(1, Ordering::SeqCst);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 12,
result: RepoTransportResultKind::Success {
source: "rsync".to_string(),
warnings: vec![Warning::new("rsync ok")],
},
}
}
}
}
}
struct FailRrdpThenFailRsyncExecutor {
rrdp_count: Arc<AtomicUsize>,
rsync_count: Arc<AtomicUsize>,
}
impl RepoTransportExecutor for FailRrdpThenFailRsyncExecutor {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
match task.mode {
RepoTransportMode::Rrdp => {
self.rrdp_count.fetch_add(1, Ordering::SeqCst);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rrdp,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 10,
result: RepoTransportResultKind::Failed {
detail: "rrdp failed".to_string(),
warnings: vec![Warning::new("rrdp failed")],
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
}
}
RepoTransportMode::Rsync => {
self.rsync_count.fetch_add(1, Ordering::SeqCst);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri.clone(),
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 12,
result: RepoTransportResultKind::Failed {
detail: "rsync failed".to_string(),
warnings: vec![Warning::new("rsync failed")],
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
}
}
}
}
}
#[test]
fn phase1_runtime_waits_for_rrdp_transport_and_returns_rrdp_outcome() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let outcome = runtime
.sync_publication_point_repo(&sample_ca("rsync://example.test/repo/root.mft"))
.expect("sync repo");
assert!(outcome.repo_sync_ok);
assert_eq!(outcome.repo_sync_source.as_deref(), Some("rrdp"));
assert_eq!(outcome.repo_sync_phase.as_deref(), Some("rrdp_ok"));
}
#[test]
fn phase1_runtime_request_repo_returns_pending_then_repo_ready_event() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let ca = sample_ca("rsync://example.test/repo/root.mft");
let status = runtime
.request_publication_point_repo(&ca, 0)
.expect("request repo");
let identity = match status {
super::RepoSyncRequestStatus::Pending { identity, state } => {
assert_eq!(state, RepoRuntimeState::WaitingRrdp);
identity
}
other => panic!("expected pending, got {other:?}"),
};
let event = runtime
.recv_repo_result_timeout(Duration::from_secs(1))
.expect("repo event")
.expect("event");
assert_eq!(event.transport_identity, identity);
assert_eq!(event.completions.len(), 1);
assert_eq!(event.completions[0].identity, identity);
assert_eq!(event.completions[0].state, RepoRuntimeState::RrdpOk);
assert!(event.completions[0].outcome.repo_sync_ok);
assert_eq!(
event.completions[0].outcome.repo_sync_source.as_deref(),
Some("rrdp")
);
}
#[test]
fn phase1_runtime_request_repo_reuses_ready_event_result() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let ca = sample_ca("rsync://example.test/repo/root.mft");
let first = runtime
.request_publication_point_repo(&ca, 0)
.expect("request repo");
assert!(matches!(
first,
super::RepoSyncRequestStatus::Pending { .. }
));
let _ = runtime
.recv_repo_result_timeout(Duration::from_secs(1))
.expect("repo event")
.expect("event");
let second = runtime
.request_publication_point_repo(&ca, 0)
.expect("request repo reused");
match second {
super::RepoSyncRequestStatus::Ready { outcome, .. } => {
assert!(outcome.repo_sync_ok);
assert_eq!(outcome.repo_sync_phase.as_deref(), Some("rrdp_ok"));
}
other => panic!("expected ready reuse, got {other:?}"),
}
}
#[test]
fn phase1_runtime_repo_event_reports_all_finalized_identities_for_shared_rrdp() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let ca1 = sample_ca("rsync://example.test/repo/root.mft");
let mut ca2 = sample_ca("rsync://example.test/other/root.mft");
ca2.rsync_base_uri = "rsync://example.test/other/".to_string();
ca2.publication_point_rsync_uri = "rsync://example.test/other/".to_string();
let id1 = match runtime
.request_publication_point_repo(&ca1, 0)
.expect("request first")
{
super::RepoSyncRequestStatus::Pending { identity, .. } => identity,
other => panic!("expected first pending, got {other:?}"),
};
let id2 = match runtime
.request_publication_point_repo(&ca2, 0)
.expect("request second")
{
super::RepoSyncRequestStatus::Pending { identity, .. } => identity,
other => panic!("expected second pending, got {other:?}"),
};
assert_ne!(id1, id2);
let event = runtime
.recv_repo_result_timeout(Duration::from_secs(1))
.expect("repo event")
.expect("event");
let mut identities = event
.completions
.iter()
.map(|completion| completion.identity.clone())
.collect::<Vec<_>>();
identities.sort_by(|a, b| a.rsync_base_uri.cmp(&b.rsync_base_uri));
let mut expected = vec![id1, id2];
expected.sort_by(|a, b| a.rsync_base_uri.cmp(&b.rsync_base_uri));
assert_eq!(identities, expected);
assert!(
event
.completions
.iter()
.all(|completion| completion.state == RepoRuntimeState::RrdpOk)
);
}
#[test]
fn phase1_runtime_drains_multiple_ready_transport_events() {
let count = Arc::new(AtomicUsize::new(0));
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 2 },
CountingSuccessTransportExecutor {
count: Arc::clone(&count),
},
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let ca1 = sample_ca("rsync://example.test/repo/root.mft");
let mut ca2 = sample_ca("rsync://example.net/repo/root.mft");
ca2.rsync_base_uri = "rsync://example.net/repo/".to_string();
ca2.publication_point_rsync_uri = "rsync://example.net/repo/".to_string();
ca2.rrdp_notification_uri = Some("https://example.net/notify.xml".to_string());
assert!(matches!(
runtime
.request_publication_point_repo(&ca1, 0)
.expect("request ca1"),
super::RepoSyncRequestStatus::Pending { .. }
));
assert!(matches!(
runtime
.request_publication_point_repo(&ca2, 0)
.expect("request ca2"),
super::RepoSyncRequestStatus::Pending { .. }
));
let started = Instant::now();
while count.load(Ordering::SeqCst) < 2 && started.elapsed() < Duration::from_secs(1) {
std::thread::sleep(Duration::from_millis(5));
}
assert_eq!(count.load(Ordering::SeqCst), 2);
let events = runtime
.drain_repo_results_timeout(Duration::from_millis(0), 8)
.expect("drain events");
assert_eq!(events.len(), 2);
assert_eq!(
events
.iter()
.map(|event| event.completions.len())
.sum::<usize>(),
2
);
}
#[test]
fn phase1_runtime_reset_run_state_clears_completed_transport_reuse() {
let count = Arc::new(AtomicUsize::new(0));
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
CountingSuccessTransportExecutor {
count: Arc::clone(&count),
},
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let ca = sample_ca("rsync://example.test/repo/root.mft");
assert!(matches!(
runtime
.request_publication_point_repo(&ca, 0)
.expect("first request"),
super::RepoSyncRequestStatus::Pending { .. }
));
let _ = runtime
.recv_repo_result_timeout(Duration::from_secs(1))
.expect("first event")
.expect("event");
assert_eq!(count.load(Ordering::SeqCst), 1);
assert!(matches!(
runtime
.request_publication_point_repo(&ca, 0)
.expect("ready reuse before reset"),
super::RepoSyncRequestStatus::Ready { .. }
));
runtime.reset_run_state().expect("reset");
assert!(matches!(
runtime
.request_publication_point_repo(&ca, 0)
.expect("second request after reset"),
super::RepoSyncRequestStatus::Pending { .. }
));
let _ = runtime
.recv_repo_result_timeout(Duration::from_secs(1))
.expect("second event")
.expect("event");
assert_eq!(count.load(Ordering::SeqCst), 2);
}
#[test]
fn phase1_runtime_records_prefetch_snapshot_only_when_enabled() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new_with_failure_scope_and_prefetch_recording(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
Arc::new(|_base: &str| Some("rsync://example.test/".to_string())),
SyncPreference::RrdpThenRsync,
true,
);
let ca = sample_ca("rsync://example.test/repo/root.mft");
let _ = runtime
.request_publication_point_repo(&ca, 0)
.expect("request repo");
let snapshot = runtime.transport_prefetch_snapshot();
assert_eq!(snapshot.requests.len(), 1);
assert_eq!(
snapshot.requests[0].repo_identity.rsync_base_uri,
"rsync://example.test/repo/"
);
assert_eq!(
snapshot.requests[0].rsync_failure_scope_uri.as_deref(),
Some("rsync://example.test/")
);
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
SyncPreference::RrdpThenRsync,
);
let _ = runtime
.request_publication_point_repo(&ca, 0)
.expect("request repo without recording");
assert!(runtime.transport_prefetch_snapshot().requests.is_empty());
}
#[test]
fn phase1_runtime_prefetch_dispatches_and_later_request_waits_on_same_task() {
let count = Arc::new(AtomicUsize::new(0));
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
CountingSuccessTransportExecutor {
count: Arc::clone(&count),
},
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new_with_failure_scope_and_prefetch_recording(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
Arc::new(|_base: &str| Some("rsync://example.test/".to_string())),
SyncPreference::RrdpThenRsync,
true,
);
let ca = sample_ca("rsync://example.test/repo/root.mft");
let mut recorder = crate::parallel::transport_prefetch::TransportPrefetchRecorder::default();
recorder.record_registered_request(
&super::Phase1RepoSyncRuntime::<CountingSuccessTransportExecutor>::build_identity(&ca),
&super::Phase1RepoSyncRuntime::<CountingSuccessTransportExecutor>::build_requester(&ca),
0,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RrdpThenRsync,
);
let snapshot = recorder.snapshot(SyncPreference::RrdpThenRsync);
let stats = runtime
.prefetch_transport_requests(&snapshot, time::OffsetDateTime::UNIX_EPOCH)
.expect("prefetch transport requests");
assert_eq!(stats.loaded_requests, 1);
assert_eq!(stats.enqueued_tasks, 1);
let status = runtime
.request_publication_point_repo(&ca, 0)
.expect("request after prefetch");
assert!(matches!(
status,
super::RepoSyncRequestStatus::Pending {
state: RepoRuntimeState::WaitingRrdp,
..
}
));
let _ = runtime
.recv_repo_result_timeout(Duration::from_secs(1))
.expect("repo event")
.expect("event");
assert_eq!(count.load(Ordering::SeqCst), 1);
}
#[test]
fn phase1_runtime_does_not_persist_prefetch_only_requests() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new_with_failure_scope_and_prefetch_recording(
coordinator,
pool,
Arc::new(|base: &str| base.to_string()),
Arc::new(|_base: &str| Some("rsync://example.test/".to_string())),
SyncPreference::RrdpThenRsync,
true,
);
let snapshot = TransportPrefetchSnapshot::new(
SyncPreference::RrdpThenRsync,
vec![
crate::parallel::transport_prefetch::TransportPrefetchRequest::from_registered_request(
&crate::parallel::types::RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/",
),
&crate::parallel::types::RepoRequester::with_tal_rir(
"arin",
"arin",
"rsync://example.test/repo/root.mft",
"rsync://example.test/repo/",
"arin:root",
),
0,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RrdpThenRsync,
),
],
);
let stats = runtime
.prefetch_transport_requests(&snapshot, time::OffsetDateTime::UNIX_EPOCH)
.expect("prefetch transport requests");
assert_eq!(stats.enqueued_tasks, 1);
assert!(
runtime.transport_prefetch_snapshot().requests.is_empty(),
"prefetch-only requests should not be carried forward forever"
);
}
#[test]
fn phase1_runtime_prefetch_skips_requests_when_scope_resolver_differs() {
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
SuccessTransportExecutor,
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new_with_failure_scope_and_prefetch_recording(
coordinator,
pool,
Arc::new(|_base: &str| "rsync://example.test/different/".to_string()),
Arc::new(|_base: &str| Some("rsync://example.test/".to_string())),
SyncPreference::RrdpThenRsync,
true,
);
let snapshot = TransportPrefetchSnapshot::new(
SyncPreference::RrdpThenRsync,
vec![
crate::parallel::transport_prefetch::TransportPrefetchRequest::from_registered_request(
&crate::parallel::types::RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/",
),
&crate::parallel::types::RepoRequester::with_tal_rir(
"arin",
"arin",
"rsync://example.test/repo/root.mft",
"rsync://example.test/repo/",
"arin:root",
),
0,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RrdpThenRsync,
),
],
);
let stats = runtime
.prefetch_transport_requests(&snapshot, time::OffsetDateTime::UNIX_EPOCH)
.expect("prefetch transport requests");
assert_eq!(stats.loaded_requests, 1);
assert_eq!(stats.enqueued_tasks, 0);
assert_eq!(stats.skipped_incompatible, 1);
}
#[test]
fn phase1_runtime_transitions_rrdp_failure_to_rsync_success() {
let rrdp_count = Arc::new(AtomicUsize::new(0));
let rsync_count = Arc::new(AtomicUsize::new(0));
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
FailRrdpThenSucceedRsyncExecutor {
rrdp_count: Arc::clone(&rrdp_count),
rsync_count: Arc::clone(&rsync_count),
},
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|_base: &str| "rsync://example.test/module/".to_string()),
SyncPreference::RrdpThenRsync,
);
let outcome = runtime
.sync_publication_point_repo(&sample_ca("rsync://example.test/repo/root.mft"))
.expect("sync repo");
assert!(outcome.repo_sync_ok);
assert_eq!(outcome.repo_sync_source.as_deref(), Some("rsync"));
assert_eq!(
outcome.repo_sync_phase.as_deref(),
Some("rrdp_failed_rsync_ok")
);
assert_eq!(rrdp_count.load(Ordering::SeqCst), 1);
assert_eq!(rsync_count.load(Ordering::SeqCst), 1);
}
#[test]
fn phase1_runtime_terminal_failure_keeps_rsync_failure_duration() {
let rrdp_count = Arc::new(AtomicUsize::new(0));
let rsync_count = Arc::new(AtomicUsize::new(0));
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
FailRrdpThenFailRsyncExecutor {
rrdp_count: Arc::clone(&rrdp_count),
rsync_count: Arc::clone(&rsync_count),
},
)
.expect("pool");
let runtime = Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|_base: &str| "rsync://example.test/module/".to_string()),
SyncPreference::RrdpThenRsync,
);
let outcome = runtime
.sync_publication_point_repo(&sample_ca("rsync://example.test/repo/root.mft"))
.expect("sync repo");
assert!(!outcome.repo_sync_ok);
assert_eq!(
outcome.repo_sync_phase.as_deref(),
Some("rrdp_failed_rsync_failed")
);
assert_eq!(outcome.repo_sync_duration_ms, 12);
assert_eq!(rrdp_count.load(Ordering::SeqCst), 1);
assert_eq!(rsync_count.load(Ordering::SeqCst), 1);
}
#[test]
fn phase1_runtime_prefetch_submits_transport_task_before_consumption() {
let rrdp_count = Arc::new(AtomicUsize::new(0));
let rsync_count = Arc::new(AtomicUsize::new(0));
let coordinator = GlobalRunCoordinator::new(
ParallelPhase1Config::default(),
vec![TalInputSpec::from_url("https://example.test/arin.tal")],
);
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 1 },
FailRrdpThenSucceedRsyncExecutor {
rrdp_count: Arc::clone(&rrdp_count),
rsync_count: Arc::clone(&rsync_count),
},
)
.expect("pool");
let runtime = Arc::new(Phase1RepoSyncRuntime::new(
coordinator,
pool,
Arc::new(|_base: &str| "rsync://example.test/module/".to_string()),
SyncPreference::RrdpThenRsync,
));
let child = DiscoveredChildCaInstance {
handle: sample_ca("rsync://example.test/repo/child.mft"),
discovered_from: crate::audit::DiscoveredFrom {
parent_manifest_rsync_uri: "rsync://example.test/repo/root.mft".to_string(),
child_ca_certificate_rsync_uri: "rsync://example.test/repo/child.cer".to_string(),
child_ca_certificate_sha256_hex: "00".repeat(32),
},
child_entry_projection: None,
};
runtime
.prefetch_discovered_children(std::slice::from_ref(&child))
.expect("prefetch");
let started = Instant::now();
while rrdp_count.load(Ordering::SeqCst) == 0 && started.elapsed() < Duration::from_secs(1) {
std::thread::sleep(Duration::from_millis(10));
}
assert_eq!(rrdp_count.load(Ordering::SeqCst), 1);
let outcome = runtime
.sync_publication_point_repo(&child.handle)
.expect("sync child repo");
assert!(outcome.repo_sync_ok);
assert_eq!(rsync_count.load(Ordering::SeqCst), 1);
}

View File

@@ -0,0 +1,101 @@
// Repository runtime outcomes, events, and runtime trait.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncRuntimeOutcome {
pub repo_sync_ok: bool,
pub repo_sync_err: Option<String>,
pub repo_sync_source: Option<String>,
pub repo_sync_phase: Option<String>,
pub repo_sync_duration_ms: u64,
pub warnings: Vec<Warning>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum RepoSyncRequestStatus {
Ready {
identity: RepoIdentity,
outcome: RepoSyncRuntimeOutcome,
},
Pending {
identity: RepoIdentity,
state: RepoRuntimeState,
},
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncRuntimeCompletion {
pub identity: RepoIdentity,
pub state: RepoRuntimeState,
pub outcome: RepoSyncRuntimeOutcome,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoSyncRuntimeEvent {
pub transport_identity: RepoIdentity,
pub completions: Vec<RepoSyncRuntimeCompletion>,
}
pub trait RepoSyncRuntime: Send + Sync {
fn sync_publication_point_repo(
&self,
ca: &CaInstanceHandle,
) -> Result<RepoSyncRuntimeOutcome, String>;
fn request_publication_point_repo(
&self,
ca: &CaInstanceHandle,
priority: u8,
) -> Result<RepoSyncRequestStatus, String>;
fn recv_repo_result_timeout(
&self,
timeout: Duration,
) -> Result<Option<RepoSyncRuntimeEvent>, String>;
fn drain_repo_results_timeout(
&self,
timeout: Duration,
max_events: usize,
) -> Result<Vec<RepoSyncRuntimeEvent>, String> {
let max_events = max_events.max(1);
let mut events = Vec::new();
for index in 0..max_events {
let poll_timeout = if index == 0 {
timeout
} else {
Duration::from_millis(0)
};
let Some(event) = self.recv_repo_result_timeout(poll_timeout)? else {
break;
};
events.push(event);
}
Ok(events)
}
fn reset_run_state(&self) -> Result<(), String>;
fn prefetch_discovered_children(
&self,
children: &[DiscoveredChildCaInstance],
) -> Result<(), String>;
fn prefetch_transport_requests(
&self,
snapshot: &TransportPrefetchSnapshot,
validation_time: time::OffsetDateTime,
) -> Result<TransportPrefetchDispatchStats, String>;
fn transport_prefetch_snapshot(&self) -> TransportPrefetchSnapshot;
/// Dead-repository blacklist working copy for run-end persistence.
/// `None` when the feature is disabled.
fn dead_repo_blacklist_state(
&self,
) -> Option<(
crate::parallel::dead_repo_blacklist::DeadRepoBlacklistConfig,
crate::parallel::dead_repo_blacklist::DeadRepoBlacklist,
)> {
None
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,199 @@
// Reusable transport failures and repository in-flight state.
fn reusable_rsync_failure_scope(result: &RepoTransportResultEnvelope) -> Option<&str> {
let scope = result.rsync_failure_scope_uri.as_deref()?;
match (&result.mode, &result.result) {
(RepoTransportMode::Rsync, RepoTransportResultKind::Failed { detail, .. })
if is_host_level_rsync_failure(detail) =>
{
Some(scope)
}
_ => None,
}
}
pub(crate) fn is_host_level_rsync_failure(detail: &str) -> bool {
let lower = detail.to_ascii_lowercase();
lower.contains("timeout waiting for daemon connection")
|| lower.contains("failed to connect")
|| lower.contains("no route to host")
|| lower.contains("network is unreachable")
|| lower.contains("connection refused")
|| lower.contains("name or service not known")
|| lower.contains("temporary failure in name resolution")
}
#[derive(Default)]
pub struct InFlightRepoTable {
entries: HashMap<RepoKey, InFlightRepoEntry>,
}
impl InFlightRepoTable {
pub fn new() -> Self {
Self::default()
}
pub fn len(&self) -> usize {
self.entries.len()
}
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
pub fn reset_run_state(&mut self) {
self.entries.clear();
}
pub fn get(&self, key: &RepoKey) -> Option<&InFlightRepoEntry> {
self.entries.get(key)
}
pub fn last_result(&self, key: &RepoKey) -> Option<&RepoSyncResultEnvelope> {
self.entries
.get(key)
.and_then(|entry| entry.last_result.as_ref())
}
pub fn register_request(
&mut self,
repo_key: RepoKey,
requester: RepoRequester,
validation_time: time::OffsetDateTime,
sync_preference: SyncPreference,
priority: u8,
) -> RepoRequestAction {
match self.entries.get_mut(&repo_key) {
Some(entry) => match entry.state {
RepoTaskState::Pending | RepoTaskState::Running => {
entry.waiting_requesters.push(requester);
RepoRequestAction::Waiting
}
RepoTaskState::Succeeded | RepoTaskState::Reused => {
RepoRequestAction::Reused(entry.result_ref.clone().expect("result_ref exists"))
}
RepoTaskState::Failed => RepoRequestAction::FailedReuse {
detail: entry
.last_error
.clone()
.unwrap_or_else(|| "repo sync failed".to_string()),
},
},
None => {
let task = RepoSyncTask {
repo_key: repo_key.clone(),
validation_time,
sync_preference,
tal_id: requester.tal_id.clone(),
rir_id: requester.rir_id.clone(),
priority,
requesters: vec![requester],
};
self.entries.insert(
repo_key,
InFlightRepoEntry {
state: RepoTaskState::Pending,
task_ref: Some(task.clone()),
waiting_requesters: Vec::new(),
result_ref: None,
last_result: None,
last_error: None,
started_at: None,
finished_at: None,
},
);
RepoRequestAction::Enqueued(task)
}
}
}
pub fn mark_running(
&mut self,
repo_key: &RepoKey,
started_at: time::OffsetDateTime,
) -> Result<(), String> {
let entry = self
.entries
.get_mut(repo_key)
.ok_or_else(|| format!("repo not found: {}", repo_key.rsync_base_uri))?;
match entry.state {
RepoTaskState::Pending => {
entry.state = RepoTaskState::Running;
entry.started_at = Some(started_at);
Ok(())
}
other => Err(format!("repo cannot transition to running from {other:?}")),
}
}
pub fn complete_success(
&mut self,
repo_key: &RepoKey,
result: RepoSyncResultEnvelope,
finished_at: time::OffsetDateTime,
) -> Result<RepoCompletion, String> {
let entry = self
.entries
.get_mut(repo_key)
.ok_or_else(|| format!("repo not found: {}", repo_key.rsync_base_uri))?;
match entry.state {
RepoTaskState::Pending | RepoTaskState::Running => {
let result_ref = match &result.result {
RepoSyncResultKind::Success(result_ref)
| RepoSyncResultKind::Reused(result_ref) => result_ref.clone(),
RepoSyncResultKind::Failed { detail } => {
return Err(format!(
"success completion called with failure result: {detail}"
));
}
};
entry.state = RepoTaskState::Succeeded;
entry.result_ref = Some(result_ref);
entry.last_result = Some(result);
entry.last_error = None;
entry.finished_at = Some(finished_at);
entry.task_ref = None;
let released_requesters = std::mem::take(&mut entry.waiting_requesters);
Ok(RepoCompletion {
repo_key: repo_key.clone(),
released_requesters,
})
}
other => Err(format!("repo cannot transition to success from {other:?}")),
}
}
pub fn complete_failure(
&mut self,
repo_key: &RepoKey,
result: RepoSyncResultEnvelope,
finished_at: time::OffsetDateTime,
) -> Result<RepoCompletion, String> {
let entry = self
.entries
.get_mut(repo_key)
.ok_or_else(|| format!("repo not found: {}", repo_key.rsync_base_uri))?;
match entry.state {
RepoTaskState::Pending | RepoTaskState::Running => {
let detail = match &result.result {
RepoSyncResultKind::Failed { detail } => detail.clone(),
RepoSyncResultKind::Success(_) | RepoSyncResultKind::Reused(_) => {
return Err("failure completion called with success result".to_string());
}
};
entry.state = RepoTaskState::Failed;
entry.result_ref = None;
entry.last_result = Some(result);
entry.last_error = Some(detail);
entry.finished_at = Some(finished_at);
entry.task_ref = None;
let released_requesters = std::mem::take(&mut entry.waiting_requesters);
Ok(RepoCompletion {
repo_key: repo_key.clone(),
released_requesters,
})
}
other => Err(format!("repo cannot transition to failure from {other:?}")),
}
}
}

View File

@@ -0,0 +1,231 @@
// Repository scheduler state transition tests.
#[cfg(test)]
mod tests {
use crate::parallel::repo_scheduler::{InFlightRepoTable, RepoRequestAction};
use crate::parallel::types::{
RepoKey, RepoRequester, RepoSyncResultEnvelope, RepoSyncResultKind, RepoSyncResultRef,
RepoTaskState,
};
use crate::policy::SyncPreference;
fn requester(tal_id: &str, rir_id: &str, manifest: &str) -> RepoRequester {
RepoRequester {
tal_id: tal_id.to_string(),
rir_id: rir_id.to_string(),
parent_node_id: None,
ca_instance_handle_id: format!("{tal_id}:{manifest}"),
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
manifest_rsync_uri: manifest.to_string(),
}
}
#[test]
fn inflight_repo_table_stores_entries_by_repo_key() {
let key = RepoKey::new("rsync://example.test/repo/", None);
let mut table = InFlightRepoTable::new();
let action = table.register_request(
key.clone(),
requester("arin", "arin", "rsync://example.test/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert!(matches!(action, RepoRequestAction::Enqueued(_)));
assert_eq!(table.len(), 1);
assert_eq!(
table.get(&key).map(|entry| entry.state),
Some(RepoTaskState::Pending)
);
}
#[test]
fn running_repo_request_is_merged_into_waiting_list() {
let key = RepoKey::new("rsync://example.test/repo/", None);
let mut table = InFlightRepoTable::new();
let _ = table.register_request(
key.clone(),
requester("arin", "arin", "rsync://example.test/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
table
.mark_running(&key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark running");
let action = table.register_request(
key.clone(),
requester("apnic", "apnic", "rsync://example.test/repo/other.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert_eq!(action, RepoRequestAction::Waiting);
assert_eq!(table.get(&key).unwrap().waiting_requesters.len(), 1);
}
#[test]
fn complete_success_releases_waiting_requesters_and_reuses_result() {
let key = RepoKey::new("rsync://example.test/repo/", None);
let mut table = InFlightRepoTable::new();
let _ = table.register_request(
key.clone(),
requester("arin", "arin", "rsync://example.test/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
table
.mark_running(&key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark running");
let waiting = requester("ripe", "ripe", "rsync://example.test/repo/child.mft");
let _ = table.register_request(
key.clone(),
waiting.clone(),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
let result_ref = RepoSyncResultRef {
repo_key: key.clone(),
source: "rrdp".to_string(),
};
let completion = table
.complete_success(
&key,
RepoSyncResultEnvelope {
repo_key: key.clone(),
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
result: RepoSyncResultKind::Success(result_ref.clone()),
phase: Some("rrdp_ok".to_string()),
timing_ms: 10,
warnings: Vec::new(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete success");
assert_eq!(completion.released_requesters, vec![waiting]);
assert_eq!(table.get(&key).unwrap().state, RepoTaskState::Succeeded);
let action = table.register_request(
key,
requester("afrinic", "afrinic", "rsync://example.test/repo/again.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert_eq!(action, RepoRequestAction::Reused(result_ref));
}
#[test]
fn complete_failure_releases_waiting_requesters_and_reuses_failure() {
let key = RepoKey::new("rsync://example.test/repo/", None);
let mut table = InFlightRepoTable::new();
let _ = table.register_request(
key.clone(),
requester("arin", "arin", "rsync://example.test/repo/root.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
table
.mark_running(&key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark running");
let waiting = requester("ripe", "ripe", "rsync://example.test/repo/child.mft");
let _ = table.register_request(
key.clone(),
waiting.clone(),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
let completion = table
.complete_failure(
&key,
RepoSyncResultEnvelope {
repo_key: key.clone(),
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
result: RepoSyncResultKind::Failed {
detail: "network timeout".to_string(),
},
phase: Some("rrdp_failed_rsync_failed".to_string()),
timing_ms: 10,
warnings: Vec::new(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete failure");
assert_eq!(completion.released_requesters, vec![waiting]);
assert_eq!(table.get(&key).unwrap().state, RepoTaskState::Failed);
let action = table.register_request(
key,
requester("afrinic", "afrinic", "rsync://example.test/repo/again.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
assert_eq!(
action,
RepoRequestAction::FailedReuse {
detail: "network timeout".to_string()
}
);
}
#[test]
fn requesters_from_different_tals_do_not_lose_identity() {
let key = RepoKey::new("rsync://shared.example/repo/", None);
let mut table = InFlightRepoTable::new();
let _ = table.register_request(
key.clone(),
requester("arin", "arin", "rsync://shared.example/repo/a.mft"),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
table
.mark_running(&key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark running");
let wait_apnic = requester("apnic", "apnic", "rsync://shared.example/repo/b.mft");
let wait_ripe = requester("ripe", "ripe", "rsync://shared.example/repo/c.mft");
let _ = table.register_request(
key.clone(),
wait_apnic.clone(),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
let _ = table.register_request(
key.clone(),
wait_ripe.clone(),
time::OffsetDateTime::UNIX_EPOCH,
SyncPreference::RrdpThenRsync,
0,
);
let completion = table
.complete_success(
&key,
RepoSyncResultEnvelope {
repo_key: key.clone(),
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
result: RepoSyncResultKind::Success(RepoSyncResultRef {
repo_key: key.clone(),
source: "rrdp".to_string(),
}),
phase: Some("rrdp_ok".to_string()),
timing_ms: 10,
warnings: Vec::new(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete success");
assert_eq!(completion.released_requesters, vec![wait_apnic, wait_ripe]);
}
}

View File

@@ -0,0 +1,864 @@
// Transport scheduler deduplication and fallback tests.
#[cfg(test)]
mod transport_tests {
use crate::parallel::repo_scheduler::{
TransportRequestAction, TransportStateTables, TransportTaskState,
};
use crate::parallel::types::{
RepoDedupKey, RepoIdentity, RepoRequester, RepoRuntimeState, RepoTransportMode,
RepoTransportResultEnvelope, RepoTransportResultKind,
};
use crate::policy::SyncPreference;
fn requester(id: &str) -> RepoRequester {
RepoRequester::with_tal_rir(
"apnic",
"apnic",
format!("rsync://example.test/repo/{id}.mft"),
"rsync://example.test/repo/".to_string(),
format!("node:{id}"),
)
}
#[test]
fn register_transport_request_enqueues_initial_rrdp_task() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/",
);
let action = tables.register_transport_request(
identity.clone(),
requester("root"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
let task = match action {
TransportRequestAction::Enqueue(task) => task,
other => panic!("expected enqueue, got {other:?}"),
};
assert_eq!(task.mode, RepoTransportMode::Rrdp);
assert_eq!(
task.dedup_key,
RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string()
}
);
let key = "https://example.test/notify.xml".to_string();
assert_eq!(
tables.rrdp_inflight.get(&key).unwrap().state,
TransportTaskState::Pending
);
assert_eq!(
tables.runtime_records.get(&identity).unwrap().state,
RepoRuntimeState::WaitingRrdp
);
}
#[test]
fn register_transport_request_waits_on_existing_rrdp_task() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/a/",
);
let _ = tables.register_transport_request(
identity.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
let other_identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/b/",
);
let action = tables.register_transport_request(
other_identity.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
assert_eq!(
action,
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRrdp
}
);
assert_eq!(
tables.runtime_records.get(&other_identity).unwrap().state,
RepoRuntimeState::WaitingRrdp
);
assert_eq!(
tables
.rrdp_inflight
.get("https://example.test/notify.xml")
.unwrap()
.waiting_requesters
.len(),
1
);
}
#[test]
fn complete_rrdp_success_reuses_for_later_identity_requests() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/a/",
);
let _ = tables.register_transport_request(
identity.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
tables
.mark_transport_running(
&RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("mark running");
let completion = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rrdp,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 10,
result: RepoTransportResultKind::Success {
source: "rrdp".to_string(),
warnings: Vec::new(),
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete success");
assert!(completion.follow_up_tasks.is_empty());
let later_identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/b/",
);
let action = tables.register_transport_request(
later_identity.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
assert!(matches!(action, TransportRequestAction::ReusedSuccess(_)));
assert_eq!(
tables.runtime_records.get(&later_identity).unwrap().state,
RepoRuntimeState::RrdpOk
);
let same_identity_action = tables.register_transport_request(
later_identity.clone(),
requester("b-again"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
assert!(matches!(
same_identity_action,
TransportRequestAction::ReusedSuccess(_)
));
let finalized = tables.finalized_runtime_records_for_transport(&RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
});
assert_eq!(finalized.len(), 2);
}
#[test]
fn complete_rrdp_failure_enqueues_rsync_follow_up() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/a/",
);
let _ = tables.register_transport_request(
identity.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
tables
.mark_transport_running(
&RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("mark running");
let completion = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rrdp,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 10,
result: RepoTransportResultKind::Failed {
detail: "rrdp timeout".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete failure");
assert_eq!(completion.follow_up_tasks.len(), 1);
assert_eq!(completion.follow_up_tasks[0].mode, RepoTransportMode::Rsync);
assert_eq!(
tables.runtime_records.get(&identity).unwrap().state,
RepoRuntimeState::WaitingRsync
);
let same_identity_action = tables.register_transport_request(
identity.clone(),
requester("a-again"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
assert_eq!(
same_identity_action,
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync
}
);
let later_identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/b/",
);
let later_action = tables.register_transport_request(
later_identity.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RrdpThenRsync,
);
assert_eq!(
later_action,
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync
}
);
assert_eq!(
tables.runtime_records.get(&later_identity).unwrap().state,
RepoRuntimeState::WaitingRsync
);
}
#[test]
fn host_failure_scope_probes_once_then_reuses_terminal_failure() {
let mut tables = TransportStateTables::new();
let identity_a = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/a/",
);
let identity_b = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/b/",
);
let _ = tables.register_transport_request(
identity_a.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/a/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RrdpThenRsync,
);
let _ = tables.register_transport_request(
identity_b.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/b/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RrdpThenRsync,
);
tables
.mark_transport_running(
&RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("mark running");
let completion = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: "https://example.test/notify.xml".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity_a.clone(),
mode: RepoTransportMode::Rrdp,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 10,
result: RepoTransportResultKind::Failed {
detail: "rrdp timeout".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete rrdp failure");
assert_eq!(completion.follow_up_tasks.len(), 1);
assert_eq!(
completion.follow_up_tasks[0]
.rsync_failure_scope_uri
.as_deref(),
Some("rsync://example.test/")
);
let probe_task = completion.follow_up_tasks[0].clone();
tables
.mark_transport_running(&probe_task.dedup_key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark rsync running");
let completion = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: probe_task.dedup_key,
rsync_failure_scope_uri: probe_task.rsync_failure_scope_uri,
repo_identity: probe_task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 15_000,
result: RepoTransportResultKind::Failed {
detail: "rsync error: timeout waiting for daemon connection".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete rsync failure");
assert!(completion.follow_up_tasks.is_empty());
assert_eq!(
tables.runtime_records.get(&identity_a).unwrap().state,
RepoRuntimeState::FailedTerminal
);
assert_eq!(
tables.runtime_records.get(&identity_b).unwrap().state,
RepoRuntimeState::FailedTerminal
);
let finalized = tables.finalized_runtime_records_for_transport_result(
tables
.runtime_records
.get(&identity_a)
.unwrap()
.terminal_failure
.as_ref()
.unwrap(),
);
assert_eq!(finalized.len(), 2);
}
#[test]
fn complete_rsync_failure_reuses_terminal_failure() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let _ = tables.register_transport_request(
identity.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
tables
.mark_transport_running(
&RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("mark running");
let _ = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 20,
result: RepoTransportResultKind::Failed {
detail: "rsync timeout".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete rsync failure");
let later_identity = RepoIdentity::new(None, "rsync://example.test/repo/b/");
let action = tables.register_transport_request(
later_identity,
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
assert!(matches!(
action,
TransportRequestAction::ReusedTerminalFailure(_)
));
let finalized = tables.finalized_runtime_records_for_transport(&RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
});
assert_eq!(finalized.len(), 2);
let same_identity_action = tables.register_transport_request(
identity,
requester("a-again"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
assert!(matches!(
same_identity_action,
TransportRequestAction::ReusedTerminalFailure(_)
));
}
#[test]
fn complete_rsync_success_reuses_for_later_identity_requests() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let action = tables.register_transport_request(
identity.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
assert!(matches!(action, TransportRequestAction::Enqueue(_)));
tables
.mark_transport_running(
&RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("mark running");
tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 20,
result: RepoTransportResultKind::Success {
source: "rsync".to_string(),
warnings: Vec::new(),
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete rsync success");
let later_identity = RepoIdentity::new(None, "rsync://example.test/repo/b/");
let action = tables.register_transport_request(
later_identity.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
assert!(matches!(action, TransportRequestAction::ReusedSuccess(_)));
assert_eq!(
tables.runtime_records.get(&later_identity).unwrap().state,
RepoRuntimeState::RsyncOk
);
}
#[test]
fn register_rsync_request_waits_on_existing_rsync_task() {
let mut tables = TransportStateTables::new();
let identity_a = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let identity_b = RepoIdentity::new(None, "rsync://example.test/repo/b/");
let _ = tables.register_transport_request(
identity_a,
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
let action = tables.register_transport_request(
identity_b.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
assert_eq!(
action,
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync
}
);
assert_eq!(
tables.runtime_records.get(&identity_b).unwrap().state,
RepoRuntimeState::WaitingRsync
);
assert_eq!(
tables
.rsync_inflight
.get("rsync://example.test/module/")
.unwrap()
.waiting_requesters
.len(),
1
);
}
#[test]
fn host_failure_scope_success_marks_host_reachable_and_schedules_waiters() {
let mut tables = TransportStateTables::new();
let identity_a = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let identity_b = RepoIdentity::new(None, "rsync://example.test/repo/b/");
let action_a = tables.register_transport_request(
identity_a.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/a/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
let probe_task = match action_a {
TransportRequestAction::Enqueue(task) => task,
other => panic!("expected first host probe enqueue, got {other:?}"),
};
let action_b = tables.register_transport_request(
identity_b.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/b/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
assert_eq!(
action_b,
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync
}
);
tables
.mark_transport_running(&probe_task.dedup_key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark rsync running");
let completion = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: probe_task.dedup_key,
rsync_failure_scope_uri: probe_task.rsync_failure_scope_uri,
repo_identity: probe_task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 5,
result: RepoTransportResultKind::Success {
source: "rsync".to_string(),
warnings: Vec::new(),
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete rsync success");
assert_eq!(
tables.runtime_records.get(&identity_a).unwrap().state,
RepoRuntimeState::RsyncOk
);
assert_eq!(completion.follow_up_tasks.len(), 1);
assert_eq!(
completion.follow_up_tasks[0].dedup_key,
RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/repo/b/".to_string()
}
);
assert_eq!(
tables.runtime_records.get(&identity_b).unwrap().state,
RepoRuntimeState::WaitingRsync
);
}
#[test]
fn non_host_level_rsync_failure_does_not_poison_host_scope() {
let mut tables = TransportStateTables::new();
let identity_a = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let identity_b = RepoIdentity::new(None, "rsync://example.test/repo/b/");
let action_a = tables.register_transport_request(
identity_a.clone(),
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/a/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
let probe_task = match action_a {
TransportRequestAction::Enqueue(task) => task,
other => panic!("expected first host probe enqueue, got {other:?}"),
};
let _ = tables.register_transport_request(
identity_b.clone(),
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/b/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
tables
.mark_transport_running(&probe_task.dedup_key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark rsync running");
let completion = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: probe_task.dedup_key,
rsync_failure_scope_uri: probe_task.rsync_failure_scope_uri,
repo_identity: probe_task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 5,
result: RepoTransportResultKind::Failed {
detail: "rsync file digest mismatch after download".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete rsync failure");
assert_eq!(
tables.runtime_records.get(&identity_a).unwrap().state,
RepoRuntimeState::FailedTerminal
);
assert_eq!(completion.follow_up_tasks.len(), 1);
assert_eq!(
completion.follow_up_tasks[0].dedup_key,
RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/repo/b/".to_string()
}
);
}
#[test]
fn cached_host_level_failure_reuses_for_later_rsync_only_requests() {
let mut tables = TransportStateTables::new();
let identity_a = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let action_a = tables.register_transport_request(
identity_a,
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/a/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
let probe_task = match action_a {
TransportRequestAction::Enqueue(task) => task,
other => panic!("expected first host probe enqueue, got {other:?}"),
};
tables
.mark_transport_running(&probe_task.dedup_key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark rsync running");
let _ = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: probe_task.dedup_key,
rsync_failure_scope_uri: probe_task.rsync_failure_scope_uri,
repo_identity: probe_task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 15_000,
result: RepoTransportResultKind::Failed {
detail: "rsync error: failed to connect to daemon".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete host failure");
let later_identity = RepoIdentity::new(None, "rsync://example.test/repo/c/");
let action = tables.register_transport_request(
later_identity.clone(),
requester("c"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/c/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
assert!(matches!(
action,
TransportRequestAction::ReusedTerminalFailure(_)
));
assert_eq!(
tables.runtime_records.get(&later_identity).unwrap().state,
RepoRuntimeState::FailedTerminal
);
}
#[test]
fn reset_run_state_clears_host_failure_scope_cache() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(None, "rsync://example.test/repo/a/");
let action = tables.register_transport_request(
identity,
requester("a"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/a/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
let probe_task = match action {
TransportRequestAction::Enqueue(task) => task,
other => panic!("expected host probe enqueue, got {other:?}"),
};
tables
.mark_transport_running(&probe_task.dedup_key, time::OffsetDateTime::UNIX_EPOCH)
.expect("mark rsync running");
let _ = tables
.complete_transport_result(
RepoTransportResultEnvelope {
dedup_key: probe_task.dedup_key,
rsync_failure_scope_uri: probe_task.rsync_failure_scope_uri,
repo_identity: probe_task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
timing_ms: 15_000,
result: RepoTransportResultKind::Failed {
detail: "temporary failure in name resolution".to_string(),
warnings: Vec::new(),
error_class: crate::parallel::types::RepoTransportErrorClass::Unknown,
},
},
time::OffsetDateTime::UNIX_EPOCH,
)
.expect("complete host failure");
tables.reset_run_state();
let later_identity = RepoIdentity::new(None, "rsync://example.test/repo/b/");
let action = tables.register_transport_request(
later_identity,
requester("b"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/repo/b/".to_string(),
Some("rsync://example.test/".to_string()),
SyncPreference::RsyncOnly,
);
assert!(matches!(action, TransportRequestAction::Enqueue(_)));
}
#[test]
fn register_transport_request_skips_rrdp_when_sync_preference_is_rsync_only() {
let mut tables = TransportStateTables::new();
let identity = RepoIdentity::new(
Some("https://example.test/notify.xml".to_string()),
"rsync://example.test/repo/",
);
let action = tables.register_transport_request(
identity.clone(),
requester("root"),
time::OffsetDateTime::UNIX_EPOCH,
0,
"rsync://example.test/module/".to_string(),
None,
SyncPreference::RsyncOnly,
);
let task = match action {
TransportRequestAction::Enqueue(task) => task,
other => panic!("expected enqueue, got {other:?}"),
};
assert_eq!(task.mode, RepoTransportMode::Rsync);
assert_eq!(
task.dedup_key,
RepoDedupKey::RsyncScope {
rsync_scope_uri: "rsync://example.test/module/".to_string()
}
);
assert!(tables.rrdp_inflight.is_empty());
assert_eq!(
tables.runtime_records.get(&identity).unwrap().state,
RepoRuntimeState::WaitingRsync
);
}
}

View File

@@ -0,0 +1,871 @@
// Transport request state tables and completion transitions.
impl TransportStateTables {
pub fn new() -> Self {
Self::default()
}
pub fn set_dead_repo_blacklist(&mut self, blacklist: DeadRepoBlacklist) {
self.dead_repo_blacklist = Some(blacklist);
}
fn dead_repo_terminal_envelope(
identity: &RepoIdentity,
requesters: &[RepoRequester],
rsync_scope_uri: &str,
rsync_failure_scope_uri: Option<String>,
) -> RepoTransportResultEnvelope {
let first_requester = requesters
.first()
.expect("blacklist terminal record must keep at least one requester");
RepoTransportResultEnvelope {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: rsync_scope_uri.to_string(),
},
rsync_failure_scope_uri,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rsync,
tal_id: first_requester.tal_id.clone(),
rir_id: first_requester.rir_id.clone(),
timing_ms: 0,
result: RepoTransportResultKind::Failed {
detail: format!(
"dead repo blacklist: rsync transport persistently unreachable: {}",
identity.rsync_base_uri
),
warnings: vec![Warning::new(format!(
"dead_repo_blacklist_skip_all: repository {} skipped after repeated transport failures",
identity.rsync_base_uri
))
.with_context(identity.rsync_base_uri.clone())],
error_class: RepoTransportErrorClass::Unknown,
},
}
}
pub fn runtime_record(&self, identity: &RepoIdentity) -> Option<&RepoRuntimeRecord> {
self.runtime_records.get(identity)
}
pub fn finalized_runtime_records_for_transport(
&self,
dedup_key: &RepoDedupKey,
) -> Vec<RepoRuntimeRecord> {
self.runtime_records
.values()
.filter(|record| match dedup_key {
RepoDedupKey::RrdpNotify { notification_uri } => {
record.rrdp_notification_key.as_deref() == Some(notification_uri.as_str())
}
RepoDedupKey::RsyncScope { rsync_scope_uri } => {
record.rsync_scope_key == *rsync_scope_uri
}
})
.filter(|record| {
matches!(
record.state,
RepoRuntimeState::RrdpOk
| RepoRuntimeState::RsyncOk
| RepoRuntimeState::FailedTerminal
)
})
.cloned()
.collect()
}
pub fn finalized_runtime_records_for_transport_result(
&self,
result: &RepoTransportResultEnvelope,
) -> Vec<RepoRuntimeRecord> {
let failure_scope = reusable_rsync_failure_scope(result);
self.runtime_records
.values()
.filter(|record| {
let exact_match = match &result.dedup_key {
RepoDedupKey::RrdpNotify { notification_uri } => {
record.rrdp_notification_key.as_deref() == Some(notification_uri.as_str())
}
RepoDedupKey::RsyncScope { rsync_scope_uri } => {
record.rsync_scope_key == *rsync_scope_uri
}
};
let failure_scope_match = failure_scope
.map(|scope| record.rsync_failure_scope_key.as_deref() == Some(scope))
.unwrap_or(false);
exact_match || failure_scope_match
})
.filter(|record| {
matches!(
record.state,
RepoRuntimeState::RrdpOk
| RepoRuntimeState::RsyncOk
| RepoRuntimeState::FailedTerminal
)
})
.cloned()
.collect()
}
pub fn reset_run_state(&mut self) {
self.rrdp_inflight.clear();
self.rsync_inflight.clear();
self.rsync_failure_by_scope.clear();
self.rsync_failure_probe_inflight.clear();
self.rsync_failure_scope_reachable.clear();
self.runtime_records.clear();
}
pub fn register_transport_request(
&mut self,
identity: RepoIdentity,
requester: RepoRequester,
validation_time: time::OffsetDateTime,
priority: u8,
rsync_scope_uri: String,
rsync_failure_scope_uri: Option<String>,
sync_preference: SyncPreference,
) -> TransportRequestAction {
if let Some(record) = self.runtime_records.get_mut(&identity) {
record.requesters.push(requester.clone());
return match record.state {
RepoRuntimeState::WaitingRrdp => {
if let Some(key) = record.rrdp_notification_key.as_ref() {
if let Some(entry) = self.rrdp_inflight.get_mut(key) {
entry.waiting_requesters.push(requester);
}
}
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRrdp,
}
}
RepoRuntimeState::RrdpOk | RepoRuntimeState::RsyncOk => {
TransportRequestAction::ReusedSuccess(
record
.last_success
.clone()
.expect("success state must keep last_success"),
)
}
RepoRuntimeState::RrdpFailedPendingRsync | RepoRuntimeState::WaitingRsync => {
if let Some(entry) = self.rsync_inflight.get_mut(&record.rsync_scope_key) {
entry.waiting_requesters.push(requester);
}
record.state = RepoRuntimeState::WaitingRsync;
TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync,
}
}
RepoRuntimeState::FailedTerminal => TransportRequestAction::ReusedTerminalFailure(
record
.terminal_failure
.clone()
.expect("terminal failure must keep last result"),
),
RepoRuntimeState::Init => TransportRequestAction::Waiting {
state: RepoRuntimeState::Init,
},
};
}
// Dead-repository blacklist fast paths, consulted only for fresh
// identities; existing records already encode their terminal state.
if let Some(blacklist) = self.dead_repo_blacklist.as_ref() {
let rrdp_wanted = sync_preference == SyncPreference::RrdpThenRsync
&& identity.notification_uri.is_some();
let rrdp_dead = rrdp_wanted
&& identity
.notification_uri
.as_deref()
.map(|uri| blacklist.is_blacklisted(RepoTransportMode::Rrdp, uri))
.unwrap_or(false);
let rsync_dead = blacklist
.is_blacklisted(RepoTransportMode::Rsync, identity.rsync_base_uri.as_str());
if rsync_dead && (!rrdp_wanted || rrdp_dead) {
crate::progress_log::emit(
"dead_repo_blacklist_skip_all",
serde_json::json!({
"repo_key_notification_uri": identity.notification_uri,
"repo_key_rsync_base_uri": identity.rsync_base_uri,
}),
);
let envelope = Self::dead_repo_terminal_envelope(
&identity,
std::slice::from_ref(&requester),
&rsync_scope_uri,
rsync_failure_scope_uri.clone(),
);
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::FailedTerminal,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri,
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: Some(envelope.clone()),
},
);
return TransportRequestAction::ReusedTerminalFailure(envelope);
}
if rrdp_dead {
crate::progress_log::emit(
"dead_repo_blacklist_skip_rrdp",
serde_json::json!({
"repo_key_notification_uri": identity.notification_uri,
"repo_key_rsync_base_uri": identity.rsync_base_uri,
}),
);
let mut action = self.register_rsync_request(
identity,
requester,
validation_time,
priority,
rsync_scope_uri,
rsync_failure_scope_uri,
);
// A blacklisted rrdp transport is a known-persistent failure,
// so the rsync fallback must keep the short retry profile
// The prefetch layer records no real rrdp failure in
// skipped runs, which would otherwise silently drop the task
// back to the default 15s window from the second skipped run
// on. The coordinator wrapper only ever sets the flag to true,
// so setting it here survives registration.
if let TransportRequestAction::Enqueue(task) = &mut action {
task.retry_short_timeout = true;
}
return action;
}
}
if sync_preference == SyncPreference::RrdpThenRsync {
if let Some(notification_uri) = identity.notification_uri.clone() {
if let Some(entry) = self.rrdp_inflight.get_mut(&notification_uri) {
if let Some(result) = entry.last_result.clone() {
return match result.result {
RepoTransportResultKind::Success { .. } => {
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::RrdpOk,
rrdp_notification_key: Some(notification_uri),
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: Some(result.clone()),
terminal_failure: None,
},
);
TransportRequestAction::ReusedSuccess(result)
}
RepoTransportResultKind::Failed { .. } => {
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity: identity.clone(),
state: RepoRuntimeState::RrdpFailedPendingRsync,
rrdp_notification_key: Some(notification_uri),
rsync_scope_key: rsync_scope_uri.clone(),
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester.clone()],
validation_time,
priority,
last_success: None,
terminal_failure: None,
},
);
self.register_rsync_request(
identity,
requester,
validation_time,
priority,
rsync_scope_uri,
rsync_failure_scope_uri,
)
}
};
}
entry.waiting_requesters.push(requester.clone());
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::WaitingRrdp,
rrdp_notification_key: Some(notification_uri),
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: None,
},
);
return TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRrdp,
};
}
let task = RepoTransportTask {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: notification_uri.clone(),
},
rsync_failure_scope_uri: None,
repo_identity: identity.clone(),
mode: RepoTransportMode::Rrdp,
retry_short_timeout: false,
tal_id: requester.tal_id.clone(),
rir_id: requester.rir_id.clone(),
validation_time,
priority,
requesters: vec![requester.clone()],
};
self.rrdp_inflight.insert(
notification_uri.clone(),
TransportInFlightEntry {
state: TransportTaskState::Pending,
task: task.clone(),
waiting_requesters: Vec::new(),
last_result: None,
started_at: None,
finished_at: None,
},
);
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::WaitingRrdp,
rrdp_notification_key: Some(notification_uri),
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: None,
},
);
return TransportRequestAction::Enqueue(task);
}
}
self.register_rsync_request(
identity,
requester,
validation_time,
priority,
rsync_scope_uri,
rsync_failure_scope_uri,
)
}
fn register_rsync_request(
&mut self,
identity: RepoIdentity,
requester: RepoRequester,
validation_time: time::OffsetDateTime,
priority: u8,
rsync_scope_uri: String,
rsync_failure_scope_uri: Option<String>,
) -> TransportRequestAction {
if let Some(entry) = self.rsync_inflight.get_mut(&rsync_scope_uri) {
if let Some(result) = entry.last_result.clone() {
return match result.result {
RepoTransportResultKind::Success { .. } => {
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::RsyncOk,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: Some(result.clone()),
terminal_failure: None,
},
);
TransportRequestAction::ReusedSuccess(result)
}
RepoTransportResultKind::Failed { .. } => {
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::FailedTerminal,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: Some(result.clone()),
},
);
TransportRequestAction::ReusedTerminalFailure(result)
}
};
}
entry.waiting_requesters.push(requester.clone());
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::WaitingRsync,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: None,
},
);
return TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync,
};
}
if let Some(failure_scope_uri) = rsync_failure_scope_uri.as_ref() {
if let Some(result) = self.rsync_failure_by_scope.get(failure_scope_uri).cloned() {
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::FailedTerminal,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: Some(result.clone()),
},
);
return TransportRequestAction::ReusedTerminalFailure(result);
}
if !self
.rsync_failure_scope_reachable
.contains(failure_scope_uri)
{
if self
.rsync_failure_probe_inflight
.get(failure_scope_uri)
.is_some()
{
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::WaitingRsync,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: None,
},
);
return TransportRequestAction::Waiting {
state: RepoRuntimeState::WaitingRsync,
};
}
}
}
let task = RepoTransportTask {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: rsync_scope_uri.clone(),
},
rsync_failure_scope_uri: rsync_failure_scope_uri.clone(),
repo_identity: identity.clone(),
mode: RepoTransportMode::Rsync,
retry_short_timeout: false,
tal_id: requester.tal_id.clone(),
rir_id: requester.rir_id.clone(),
validation_time,
priority,
requesters: vec![requester.clone()],
};
self.rsync_inflight.insert(
rsync_scope_uri.clone(),
TransportInFlightEntry {
state: TransportTaskState::Pending,
task: task.clone(),
waiting_requesters: Vec::new(),
last_result: None,
started_at: None,
finished_at: None,
},
);
if let Some(failure_scope_uri) = rsync_failure_scope_uri.as_ref() {
if !self
.rsync_failure_scope_reachable
.contains(failure_scope_uri)
{
self.rsync_failure_probe_inflight
.insert(failure_scope_uri.clone(), rsync_scope_uri.clone());
}
}
self.runtime_records.insert(
identity.clone(),
RepoRuntimeRecord {
identity,
state: RepoRuntimeState::WaitingRsync,
rrdp_notification_key: None,
rsync_scope_key: rsync_scope_uri,
rsync_failure_scope_key: rsync_failure_scope_uri.clone(),
requesters: vec![requester],
validation_time,
priority,
last_success: None,
terminal_failure: None,
},
);
TransportRequestAction::Enqueue(task)
}
fn schedule_rsync_for_record(
record: &mut RepoRuntimeRecord,
rsync_inflight: &mut HashMap<String, TransportInFlightEntry>,
rsync_failure_by_scope: &HashMap<String, RepoTransportResultEnvelope>,
rsync_failure_probe_inflight: &mut HashMap<String, String>,
rsync_failure_scope_reachable: &HashSet<String>,
dead_repo_blacklist: Option<&DeadRepoBlacklist>,
follow_up_tasks: &mut Vec<RepoTransportTask>,
) {
let rsync_scope_uri = record.rsync_scope_key.clone();
if let Some(entry) = rsync_inflight.get_mut(&rsync_scope_uri) {
if let Some(result) = entry.last_result.clone() {
match result.result {
RepoTransportResultKind::Success { .. } => {
record.state = RepoRuntimeState::RsyncOk;
record.last_success = Some(result);
}
RepoTransportResultKind::Failed { .. } => {
record.state = RepoRuntimeState::FailedTerminal;
record.terminal_failure = Some(result);
}
}
return;
}
entry.waiting_requesters.extend(record.requesters.clone());
record.state = RepoRuntimeState::WaitingRsync;
return;
}
if let Some(failure_scope_uri) = record.rsync_failure_scope_key.as_ref() {
if let Some(result) = rsync_failure_by_scope.get(failure_scope_uri).cloned() {
record.state = RepoRuntimeState::FailedTerminal;
record.terminal_failure = Some(result);
return;
}
if !rsync_failure_scope_reachable.contains(failure_scope_uri)
&& rsync_failure_probe_inflight.contains_key(failure_scope_uri)
{
record.state = RepoRuntimeState::WaitingRsync;
return;
}
}
// The rsync transport of this repository is
// persistently unreachable; terminate instantly instead of enqueueing
// another doomed fetch (rrdp was already attempted or skipped).
if let Some(blacklist) = dead_repo_blacklist {
if blacklist.is_blacklisted(
RepoTransportMode::Rsync,
record.identity.rsync_base_uri.as_str(),
) {
crate::progress_log::emit(
"dead_repo_blacklist_skip_all",
serde_json::json!({
"repo_key_notification_uri": record.identity.notification_uri,
"repo_key_rsync_base_uri": record.identity.rsync_base_uri,
"after_rrdp_failure": true,
}),
);
let envelope = Self::dead_repo_terminal_envelope(
&record.identity,
&record.requesters,
&rsync_scope_uri,
record.rsync_failure_scope_key.clone(),
);
record.state = RepoRuntimeState::FailedTerminal;
record.terminal_failure = Some(envelope);
return;
}
}
let first_requester = record
.requesters
.first()
.expect("rsync record must keep at least one requester");
let task = RepoTransportTask {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: rsync_scope_uri.clone(),
},
rsync_failure_scope_uri: record.rsync_failure_scope_key.clone(),
repo_identity: record.identity.clone(),
mode: RepoTransportMode::Rsync,
retry_short_timeout: false,
tal_id: first_requester.tal_id.clone(),
rir_id: first_requester.rir_id.clone(),
validation_time: record.validation_time,
priority: record.priority,
requesters: record.requesters.clone(),
};
rsync_inflight.insert(
rsync_scope_uri.clone(),
TransportInFlightEntry {
state: TransportTaskState::Pending,
task: task.clone(),
waiting_requesters: Vec::new(),
last_result: None,
started_at: None,
finished_at: None,
},
);
if let Some(failure_scope_uri) = record.rsync_failure_scope_key.as_ref() {
if !rsync_failure_scope_reachable.contains(failure_scope_uri) {
rsync_failure_probe_inflight.insert(failure_scope_uri.clone(), rsync_scope_uri);
}
}
record.state = RepoRuntimeState::WaitingRsync;
follow_up_tasks.push(task);
}
pub fn mark_transport_running(
&mut self,
dedup_key: &RepoDedupKey,
started_at: time::OffsetDateTime,
) -> Result<(), String> {
match dedup_key {
RepoDedupKey::RrdpNotify { notification_uri } => {
let entry = self
.rrdp_inflight
.get_mut(notification_uri)
.ok_or_else(|| format!("rrdp transport not found: {notification_uri}"))?;
entry.state = TransportTaskState::Running;
entry.started_at = Some(started_at);
}
RepoDedupKey::RsyncScope { rsync_scope_uri } => {
let entry = self
.rsync_inflight
.get_mut(rsync_scope_uri)
.ok_or_else(|| format!("rsync transport not found: {rsync_scope_uri}"))?;
entry.state = TransportTaskState::Running;
entry.started_at = Some(started_at);
}
}
Ok(())
}
pub fn complete_transport_result(
&mut self,
result: RepoTransportResultEnvelope,
finished_at: time::OffsetDateTime,
) -> Result<TransportCompletion, String> {
match (&result.dedup_key, &result.result) {
(
RepoDedupKey::RrdpNotify { notification_uri },
RepoTransportResultKind::Success { .. },
) => {
let entry = self
.rrdp_inflight
.get_mut(notification_uri)
.ok_or_else(|| format!("rrdp transport not found: {notification_uri}"))?;
entry.state = TransportTaskState::Finished;
entry.finished_at = Some(finished_at);
entry.last_result = Some(result.clone());
let released_requesters = std::mem::take(&mut entry.waiting_requesters);
for record in self.runtime_records.values_mut() {
if record.rrdp_notification_key.as_deref() == Some(notification_uri)
&& record.state == RepoRuntimeState::WaitingRrdp
{
record.state = RepoRuntimeState::RrdpOk;
record.last_success = Some(result.clone());
}
}
Ok(TransportCompletion {
released_requesters,
follow_up_tasks: Vec::new(),
})
}
(
RepoDedupKey::RrdpNotify { notification_uri },
RepoTransportResultKind::Failed { .. },
) => {
let entry = self
.rrdp_inflight
.get_mut(notification_uri)
.ok_or_else(|| format!("rrdp transport not found: {notification_uri}"))?;
entry.state = TransportTaskState::Finished;
entry.finished_at = Some(finished_at);
entry.last_result = Some(result.clone());
let mut follow_up_tasks = Vec::new();
for record in self.runtime_records.values_mut() {
if record.rrdp_notification_key.as_deref() == Some(notification_uri)
&& record.state == RepoRuntimeState::WaitingRrdp
{
record.state = RepoRuntimeState::RrdpFailedPendingRsync;
Self::schedule_rsync_for_record(
record,
&mut self.rsync_inflight,
&self.rsync_failure_by_scope,
&mut self.rsync_failure_probe_inflight,
&self.rsync_failure_scope_reachable,
self.dead_repo_blacklist.as_ref(),
&mut follow_up_tasks,
);
}
}
Ok(TransportCompletion {
released_requesters: Vec::new(),
follow_up_tasks,
})
}
(
RepoDedupKey::RsyncScope { rsync_scope_uri },
RepoTransportResultKind::Success { .. },
) => {
let mut follow_up_tasks = Vec::new();
let entry = self
.rsync_inflight
.get_mut(rsync_scope_uri)
.ok_or_else(|| format!("rsync transport not found: {rsync_scope_uri}"))?;
entry.state = TransportTaskState::Finished;
entry.finished_at = Some(finished_at);
entry.last_result = Some(result.clone());
if let Some(failure_scope_uri) = result.rsync_failure_scope_uri.as_ref() {
self.rsync_failure_probe_inflight.remove(failure_scope_uri);
self.rsync_failure_scope_reachable
.insert(failure_scope_uri.clone());
}
let released_requesters = std::mem::take(&mut entry.waiting_requesters);
for record in self.runtime_records.values_mut() {
if record.rsync_scope_key == *rsync_scope_uri
&& matches!(
record.state,
RepoRuntimeState::WaitingRsync
| RepoRuntimeState::RrdpFailedPendingRsync
)
{
record.state = RepoRuntimeState::RsyncOk;
record.last_success = Some(result.clone());
}
}
if let Some(failure_scope_uri) = result.rsync_failure_scope_uri.as_ref() {
for record in self.runtime_records.values_mut() {
if record.rsync_scope_key != *rsync_scope_uri
&& record.rsync_failure_scope_key.as_deref()
== Some(failure_scope_uri.as_str())
&& matches!(record.state, RepoRuntimeState::WaitingRsync)
{
Self::schedule_rsync_for_record(
record,
&mut self.rsync_inflight,
&self.rsync_failure_by_scope,
&mut self.rsync_failure_probe_inflight,
&self.rsync_failure_scope_reachable,
self.dead_repo_blacklist.as_ref(),
&mut follow_up_tasks,
);
}
}
}
Ok(TransportCompletion {
released_requesters,
follow_up_tasks,
})
}
(
RepoDedupKey::RsyncScope { rsync_scope_uri },
RepoTransportResultKind::Failed { .. },
) => {
let entry = self
.rsync_inflight
.get_mut(rsync_scope_uri)
.ok_or_else(|| format!("rsync transport not found: {rsync_scope_uri}"))?;
entry.state = TransportTaskState::Finished;
entry.finished_at = Some(finished_at);
entry.last_result = Some(result.clone());
let reusable_failure_scope =
reusable_rsync_failure_scope(&result).map(str::to_string);
if let Some(failure_scope_uri) = result.rsync_failure_scope_uri.as_ref() {
self.rsync_failure_probe_inflight.remove(failure_scope_uri);
if reusable_failure_scope.as_deref() == Some(failure_scope_uri.as_str()) {
self.rsync_failure_by_scope
.insert(failure_scope_uri.clone(), result.clone());
} else {
self.rsync_failure_scope_reachable
.insert(failure_scope_uri.clone());
}
}
let released_requesters = std::mem::take(&mut entry.waiting_requesters);
for record in self.runtime_records.values_mut() {
if (record.rsync_scope_key == *rsync_scope_uri
|| reusable_failure_scope
.as_deref()
.map(|scope| record.rsync_failure_scope_key.as_deref() == Some(scope))
.unwrap_or(false))
&& matches!(
record.state,
RepoRuntimeState::WaitingRsync
| RepoRuntimeState::RrdpFailedPendingRsync
)
{
record.state = RepoRuntimeState::FailedTerminal;
record.terminal_failure = Some(result.clone());
}
}
if let Some(failure_scope_uri) = result.rsync_failure_scope_uri.as_ref() {
if reusable_failure_scope.as_deref() != Some(failure_scope_uri.as_str()) {
let mut follow_up_tasks = Vec::new();
for record in self.runtime_records.values_mut() {
if record.rsync_scope_key != *rsync_scope_uri
&& record.rsync_failure_scope_key.as_deref()
== Some(failure_scope_uri.as_str())
&& matches!(record.state, RepoRuntimeState::WaitingRsync)
{
Self::schedule_rsync_for_record(
record,
&mut self.rsync_inflight,
&self.rsync_failure_by_scope,
&mut self.rsync_failure_probe_inflight,
&self.rsync_failure_scope_reachable,
self.dead_repo_blacklist.as_ref(),
&mut follow_up_tasks,
);
}
}
return Ok(TransportCompletion {
released_requesters,
follow_up_tasks,
});
}
}
Ok(TransportCompletion {
released_requesters,
follow_up_tasks: Vec::new(),
})
}
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,384 @@
// Repository sync and transport executor implementations.
const RETRY_SHORT_TIMEOUT: Duration = Duration::from_secs(1);
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RepoWorkerPoolConfig {
pub max_workers: usize,
}
impl From<&ParallelPhase1Config> for RepoWorkerPoolConfig {
fn from(value: &ParallelPhase1Config) -> Self {
Self {
max_workers: value.max_repo_sync_workers_global,
}
}
}
pub trait RepoSyncExecutor: Send + Sync + 'static {
fn execute(&self, task: RepoSyncTask) -> RepoSyncResultEnvelope;
}
pub trait RepoTransportExecutor: Send + Sync + 'static {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope;
}
pub struct LiveRrdpTransportExecutor<H: Fetcher> {
store: Arc<RocksStore>,
current_repo_index: CurrentRepoIndexHandle,
http_fetcher: Arc<H>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
}
impl<H: Fetcher> LiveRrdpTransportExecutor<H> {
pub fn new(
store: Arc<RocksStore>,
current_repo_index: CurrentRepoIndexHandle,
http_fetcher: Arc<H>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
) -> Self {
Self {
store,
current_repo_index,
http_fetcher,
timing,
download_log,
}
}
}
impl<H: Fetcher + 'static> RepoTransportExecutor for LiveRrdpTransportExecutor<H> {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
let started = std::time::Instant::now();
debug_assert_eq!(task.mode, RepoTransportMode::Rrdp);
let notification_uri = task
.repo_identity
.notification_uri
.as_deref()
.expect("rrdp transport requires notification uri");
let sync_result = if task.retry_short_timeout {
crate::fetch::http::with_scoped_http_timeout_override(RETRY_SHORT_TIMEOUT, || {
run_rrdp_transport(
self.store.as_ref(),
notification_uri,
Some(&self.current_repo_index),
self.http_fetcher.as_ref(),
self.timing.as_ref(),
self.download_log.as_ref(),
)
})
} else {
run_rrdp_transport(
self.store.as_ref(),
notification_uri,
Some(&self.current_repo_index),
self.http_fetcher.as_ref(),
self.timing.as_ref(),
self.download_log.as_ref(),
)
};
match sync_result {
Ok(_) => RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri,
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rrdp,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: started.elapsed().as_millis() as u64,
result: RepoTransportResultKind::Success {
source: "rrdp".to_string(),
warnings: Vec::new(),
},
},
Err(err) => {
let error_class =
crate::parallel::dead_repo_blacklist::classify_rrdp_sync_error(&err);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri,
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rrdp,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: started.elapsed().as_millis() as u64,
result: RepoTransportResultKind::Failed {
detail: err.to_string(),
warnings: Vec::new(),
error_class,
},
}
}
}
}
}
pub struct LiveRsyncTransportExecutor<R: RsyncFetcher> {
store: Arc<RocksStore>,
current_repo_index: CurrentRepoIndexHandle,
rsync_fetcher: Arc<R>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
}
impl<R: RsyncFetcher> LiveRsyncTransportExecutor<R> {
pub fn new(
store: Arc<RocksStore>,
current_repo_index: CurrentRepoIndexHandle,
rsync_fetcher: Arc<R>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
) -> Self {
Self {
store,
current_repo_index,
rsync_fetcher,
timing,
download_log,
}
}
}
impl<R: RsyncFetcher + 'static> RepoTransportExecutor for LiveRsyncTransportExecutor<R> {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
let started = std::time::Instant::now();
debug_assert_eq!(task.mode, RepoTransportMode::Rsync);
let sync_result = if task.retry_short_timeout {
with_scoped_rsync_timeout_override(RETRY_SHORT_TIMEOUT, || {
with_scoped_rsync_fail_fast_profile(
RsyncFailFastProfile {
initial_wall_clock_timeout: RETRY_SHORT_TIMEOUT,
max_wall_clock_timeout: RETRY_SHORT_TIMEOUT,
max_attempts: 1,
},
|| {
run_rsync_transport(
self.store.as_ref(),
&task.repo_identity.rsync_base_uri,
Some(&self.current_repo_index),
self.rsync_fetcher.as_ref(),
self.timing.as_ref(),
self.download_log.as_ref(),
)
},
)
})
} else {
run_rsync_transport(
self.store.as_ref(),
&task.repo_identity.rsync_base_uri,
Some(&self.current_repo_index),
self.rsync_fetcher.as_ref(),
self.timing.as_ref(),
self.download_log.as_ref(),
)
};
match sync_result {
Ok(_) => RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri,
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: started.elapsed().as_millis() as u64,
result: RepoTransportResultKind::Success {
source: "rsync".to_string(),
warnings: Vec::new(),
},
},
Err(err) => {
let error_class =
crate::parallel::dead_repo_blacklist::classify_rsync_repo_sync_error(&err);
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri,
repo_identity: task.repo_identity,
mode: RepoTransportMode::Rsync,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: started.elapsed().as_millis() as u64,
result: RepoTransportResultKind::Failed {
detail: err.to_string(),
warnings: Vec::new(),
error_class,
},
}
}
}
}
}
pub struct LiveRepoTransportExecutor<H: Fetcher, R: RsyncFetcher> {
rrdp: LiveRrdpTransportExecutor<H>,
rsync: LiveRsyncTransportExecutor<R>,
}
impl<H: Fetcher, R: RsyncFetcher> LiveRepoTransportExecutor<H, R> {
pub fn new(
store: Arc<RocksStore>,
current_repo_index: CurrentRepoIndexHandle,
http_fetcher: Arc<H>,
rsync_fetcher: Arc<R>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
) -> Self {
Self {
rrdp: LiveRrdpTransportExecutor::new(
Arc::clone(&store),
current_repo_index.clone(),
http_fetcher,
timing.clone(),
download_log.clone(),
),
rsync: LiveRsyncTransportExecutor::new(
store,
current_repo_index,
rsync_fetcher,
timing,
download_log,
),
}
}
}
impl<H: Fetcher + 'static, R: RsyncFetcher + 'static> RepoTransportExecutor
for LiveRepoTransportExecutor<H, R>
{
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
match task.mode {
RepoTransportMode::Rrdp => self.rrdp.execute_transport(task),
RepoTransportMode::Rsync => self.rsync.execute_transport(task),
}
}
}
pub struct LiveRepoSyncExecutor<H: Fetcher, R: RsyncFetcher> {
store: Arc<RocksStore>,
policy: Policy,
http_fetcher: Arc<H>,
rsync_fetcher: Arc<R>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
}
impl<H: Fetcher, R: RsyncFetcher> LiveRepoSyncExecutor<H, R> {
pub fn new(
store: Arc<RocksStore>,
policy: Policy,
http_fetcher: Arc<H>,
rsync_fetcher: Arc<R>,
timing: Option<TimingHandle>,
download_log: Option<DownloadLogHandle>,
) -> Self {
Self {
store,
policy,
http_fetcher,
rsync_fetcher,
timing,
download_log,
}
}
}
impl<H: Fetcher + 'static, R: RsyncFetcher + 'static> RepoSyncExecutor
for LiveRepoSyncExecutor<H, R>
{
fn execute(&self, task: RepoSyncTask) -> RepoSyncResultEnvelope {
let started = std::time::Instant::now();
crate::progress_log::emit(
"phase1_repo_worker_execute_start",
serde_json::json!({
"repo_key_rsync_base_uri": task.repo_key.rsync_base_uri,
"repo_key_notification_uri": task.repo_key.notification_uri,
"tal_id": task.tal_id,
"rir_id": task.rir_id,
}),
);
match sync_publication_point(
self.store.as_ref(),
&self.policy,
task.repo_key.notification_uri.as_deref(),
&task.repo_key.rsync_base_uri,
self.http_fetcher.as_ref(),
self.rsync_fetcher.as_ref(),
self.timing.as_ref(),
self.download_log.as_ref(),
) {
Ok(res) => {
let timing_ms = started.elapsed().as_millis() as u64;
crate::progress_log::emit(
"phase1_repo_worker_execute_finish",
serde_json::json!({
"repo_key_rsync_base_uri": task.repo_key.rsync_base_uri,
"repo_key_notification_uri": task.repo_key.notification_uri,
"result": "success",
"phase": repo_sync_phase_label(res.phase),
"timing_ms": timing_ms,
}),
);
RepoSyncResultEnvelope {
repo_key: task.repo_key.clone(),
tal_id: task.tal_id,
rir_id: task.rir_id,
result: super::types::RepoSyncResultKind::Success(
super::types::RepoSyncResultRef {
repo_key: task.repo_key,
source: repo_sync_source_label(res.source).to_string(),
},
),
phase: Some(repo_sync_phase_label(res.phase).to_string()),
timing_ms,
warnings: res.warnings,
}
}
Err(err) => {
let timing_ms = started.elapsed().as_millis() as u64;
crate::progress_log::emit(
"phase1_repo_worker_execute_finish",
serde_json::json!({
"repo_key_rsync_base_uri": task.repo_key.rsync_base_uri,
"repo_key_notification_uri": task.repo_key.notification_uri,
"result": "failed",
"phase": "repo_sync_failed",
"timing_ms": timing_ms,
"error": err.to_string(),
}),
);
RepoSyncResultEnvelope {
repo_key: task.repo_key,
tal_id: task.tal_id,
rir_id: task.rir_id,
result: super::types::RepoSyncResultKind::Failed {
detail: err.to_string(),
},
phase: Some("repo_sync_failed".to_string()),
timing_ms,
warnings: Vec::new(),
}
}
}
}
}
fn repo_sync_source_label(source: RepoSyncSource) -> &'static str {
match source {
RepoSyncSource::Rrdp => "rrdp",
RepoSyncSource::Rsync => "rsync",
}
}
fn repo_sync_phase_label(phase: crate::sync::repo::RepoSyncPhase) -> &'static str {
match phase {
crate::sync::repo::RepoSyncPhase::RrdpOk => "rrdp_ok",
crate::sync::repo::RepoSyncPhase::RrdpFailedRsyncOk => "rrdp_failed_rsync_ok",
crate::sync::repo::RepoSyncPhase::RsyncOnlyOk => "rsync_only_ok",
crate::sync::repo::RepoSyncPhase::ReplayRrdpOk => "replay_rrdp_ok",
crate::sync::repo::RepoSyncPhase::ReplayRsyncOk => "replay_rsync_ok",
crate::sync::repo::RepoSyncPhase::ReplayNoopRrdp => "replay_noop_rrdp",
crate::sync::repo::RepoSyncPhase::ReplayNoopRsync => "replay_noop_rsync",
}
}

View File

@@ -0,0 +1,255 @@
// Worker-pool channels, lifecycle, and worker loops.
enum RepoWorkerMessage {
Task(RepoSyncTask),
Shutdown,
}
enum TransportWorkerMessage {
Task(RepoTransportTask),
Shutdown,
}
pub struct RepoWorkerPool<E: RepoSyncExecutor> {
config: RepoWorkerPoolConfig,
task_tx: Sender<RepoWorkerMessage>,
result_rx: Receiver<RepoSyncResultEnvelope>,
workers: Vec<JoinHandle<()>>,
_executor: Arc<E>,
}
pub struct RepoTransportWorkerPool<E: RepoTransportExecutor> {
config: RepoWorkerPoolConfig,
task_tx: Sender<TransportWorkerMessage>,
result_rx: Receiver<RepoTransportResultEnvelope>,
workers: Vec<JoinHandle<()>>,
_executor: Arc<E>,
}
impl<E: RepoSyncExecutor> RepoWorkerPool<E> {
pub fn new(config: RepoWorkerPoolConfig, executor: E) -> Result<Self, String> {
if config.max_workers == 0 {
return Err("RepoWorkerPool requires at least one worker".to_string());
}
let executor = Arc::new(executor);
let (task_tx, task_rx) = mpsc::channel::<RepoWorkerMessage>();
let (result_tx, result_rx) = mpsc::channel::<RepoSyncResultEnvelope>();
let shared_task_rx = Arc::new(Mutex::new(task_rx));
let mut workers = Vec::with_capacity(config.max_workers);
for idx in 0..config.max_workers {
let task_rx = Arc::clone(&shared_task_rx);
let result_tx = result_tx.clone();
let executor = Arc::clone(&executor);
workers.push(
thread::Builder::new()
.name(format!("repo-sync-worker-{idx}"))
.spawn(move || worker_loop(task_rx, result_tx, executor))
.map_err(|e| format!("spawn repo worker failed: {e}"))?,
);
}
Ok(Self {
config,
task_tx,
result_rx,
workers,
_executor: executor,
})
}
pub fn worker_count(&self) -> usize {
self.config.max_workers
}
pub fn submit(&self, task: RepoSyncTask) -> Result<(), String> {
self.task_tx
.send(RepoWorkerMessage::Task(task))
.map_err(|e| format!("submit repo task failed: {e}"))
}
pub fn recv_result_timeout(
&self,
timeout: Duration,
) -> Result<Option<RepoSyncResultEnvelope>, String> {
match self.result_rx.recv_timeout(timeout) {
Ok(msg) => Ok(Some(msg)),
Err(RecvTimeoutError::Timeout) => Ok(None),
Err(RecvTimeoutError::Disconnected) => {
Err("repo worker result channel disconnected".to_string())
}
}
}
pub fn shutdown(mut self) -> Result<(), String> {
self.shutdown_inner()
}
fn shutdown_inner(&mut self) -> Result<(), String> {
if self.workers.is_empty() {
return Ok(());
}
for _ in 0..self.workers.len() {
self.task_tx
.send(RepoWorkerMessage::Shutdown)
.map_err(|e| format!("send shutdown to repo worker failed: {e}"))?;
}
let mut first_err: Option<String> = None;
for handle in self.workers.drain(..) {
if let Err(e) = handle.join() {
if first_err.is_none() {
first_err = Some(format!("join repo worker failed: {e:?}"));
}
}
}
if let Some(err) = first_err {
return Err(err);
}
Ok(())
}
}
impl<E: RepoSyncExecutor> Drop for RepoWorkerPool<E> {
fn drop(&mut self) {
let _ = self.shutdown_inner();
}
}
impl<E: RepoTransportExecutor> RepoTransportWorkerPool<E> {
pub fn new(config: RepoWorkerPoolConfig, executor: E) -> Result<Self, String> {
if config.max_workers == 0 {
return Err("RepoTransportWorkerPool requires at least one worker".to_string());
}
let executor = Arc::new(executor);
let (task_tx, task_rx) = mpsc::channel::<TransportWorkerMessage>();
let (result_tx, result_rx) = mpsc::channel::<RepoTransportResultEnvelope>();
let shared_task_rx = Arc::new(Mutex::new(task_rx));
let mut workers = Vec::with_capacity(config.max_workers);
for idx in 0..config.max_workers {
let task_rx = Arc::clone(&shared_task_rx);
let result_tx = result_tx.clone();
let executor = Arc::clone(&executor);
workers.push(
thread::Builder::new()
.name(format!("repo-transport-worker-{idx}"))
.spawn(move || transport_worker_loop(task_rx, result_tx, executor))
.map_err(|e| format!("spawn repo transport worker failed: {e}"))?,
);
}
Ok(Self {
config,
task_tx,
result_rx,
workers,
_executor: executor,
})
}
pub fn submit(&self, task: RepoTransportTask) -> Result<(), String> {
self.task_tx
.send(TransportWorkerMessage::Task(task))
.map_err(|e| format!("submit repo transport task failed: {e}"))
}
pub fn recv_result_timeout(
&self,
timeout: Duration,
) -> Result<Option<RepoTransportResultEnvelope>, String> {
match self.result_rx.recv_timeout(timeout) {
Ok(msg) => Ok(Some(msg)),
Err(RecvTimeoutError::Timeout) => Ok(None),
Err(RecvTimeoutError::Disconnected) => {
Err("repo transport worker result channel disconnected".to_string())
}
}
}
pub fn worker_count(&self) -> usize {
self.config.max_workers
}
pub fn shutdown(mut self) -> Result<(), String> {
self.shutdown_inner()
}
fn shutdown_inner(&mut self) -> Result<(), String> {
if self.workers.is_empty() {
return Ok(());
}
for _ in 0..self.workers.len() {
self.task_tx
.send(TransportWorkerMessage::Shutdown)
.map_err(|e| format!("send shutdown to repo transport worker failed: {e}"))?;
}
let mut first_err: Option<String> = None;
for handle in self.workers.drain(..) {
if let Err(e) = handle.join() {
if first_err.is_none() {
first_err = Some(format!("join repo transport worker failed: {e:?}"));
}
}
}
if let Some(err) = first_err {
return Err(err);
}
Ok(())
}
}
impl<E: RepoTransportExecutor> Drop for RepoTransportWorkerPool<E> {
fn drop(&mut self) {
let _ = self.shutdown_inner();
}
}
fn worker_loop<E: RepoSyncExecutor>(
task_rx: Arc<Mutex<Receiver<RepoWorkerMessage>>>,
result_tx: Sender<RepoSyncResultEnvelope>,
executor: Arc<E>,
) {
loop {
let message = {
let rx = task_rx.lock().expect("repo worker receiver lock poisoned");
rx.recv()
};
match message {
Ok(RepoWorkerMessage::Task(task)) => {
let result = executor.execute(task);
if result_tx.send(result).is_err() {
break;
}
}
Ok(RepoWorkerMessage::Shutdown) | Err(_) => break,
}
}
}
fn transport_worker_loop<E: RepoTransportExecutor>(
task_rx: Arc<Mutex<Receiver<TransportWorkerMessage>>>,
result_tx: Sender<RepoTransportResultEnvelope>,
executor: Arc<E>,
) {
loop {
let message = {
let rx = task_rx
.lock()
.expect("repo transport worker receiver lock poisoned");
rx.recv()
};
match message {
Ok(TransportWorkerMessage::Task(task)) => {
let result = executor.execute_transport(task);
if result_tx.send(result).is_err() {
break;
}
}
Ok(TransportWorkerMessage::Shutdown) | Err(_) => break,
}
}
}

View File

@@ -0,0 +1,599 @@
// Repository worker and transport executor tests.
use base64::Engine;
use sha2::Digest;
use std::collections::HashMap;
use std::fs;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Barrier, Mutex};
use std::time::Duration;
use super::{
LiveRepoSyncExecutor, LiveRrdpTransportExecutor, LiveRsyncTransportExecutor, RepoSyncExecutor,
RepoTransportExecutor, RepoTransportWorkerPool, RepoWorkerPool, RepoWorkerPoolConfig,
};
use crate::current_repo_index::CurrentRepoIndex;
use crate::fetch::rsync::{LocalDirRsyncFetcher, RsyncFetchError, RsyncFetchResult, RsyncFetcher};
use crate::parallel::config::ParallelPhase1Config;
use crate::parallel::types::{
RepoDedupKey, RepoIdentity, RepoKey, RepoRequester, RepoSyncResultEnvelope, RepoSyncResultKind,
RepoSyncResultRef, RepoSyncTask, RepoTransportMode, RepoTransportResultEnvelope,
RepoTransportResultKind, RepoTransportTask,
};
use crate::policy::SyncPreference;
use crate::report::Warning;
use crate::storage::RocksStore;
use crate::sync::rrdp::Fetcher;
fn sample_task(name: &str) -> RepoSyncTask {
RepoSyncTask {
repo_key: RepoKey::new(format!("rsync://example.test/{name}/"), None),
validation_time: time::OffsetDateTime::UNIX_EPOCH,
sync_preference: SyncPreference::RrdpThenRsync,
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
priority: 0,
requesters: vec![RepoRequester {
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
parent_node_id: None,
ca_instance_handle_id: format!("node:{name}"),
publication_point_rsync_uri: format!("rsync://example.test/{name}/"),
manifest_rsync_uri: format!("rsync://example.test/{name}/root.mft"),
}],
}
}
fn sample_rrdp_transport_task(notification_uri: &str, rsync_base_uri: &str) -> RepoTransportTask {
RepoTransportTask {
dedup_key: RepoDedupKey::RrdpNotify {
notification_uri: notification_uri.to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: RepoIdentity::new(Some(notification_uri.to_string()), rsync_base_uri),
mode: RepoTransportMode::Rrdp,
retry_short_timeout: false,
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
validation_time: time::OffsetDateTime::UNIX_EPOCH,
priority: 0,
requesters: vec![RepoRequester::with_tal_rir(
"arin",
"arin",
format!("{rsync_base_uri}root.mft"),
rsync_base_uri.to_string(),
"node:rrdp",
)],
}
}
fn sample_rsync_transport_task(rsync_scope_uri: &str, rsync_base_uri: &str) -> RepoTransportTask {
RepoTransportTask {
dedup_key: RepoDedupKey::RsyncScope {
rsync_scope_uri: rsync_scope_uri.to_string(),
},
rsync_failure_scope_uri: None,
repo_identity: RepoIdentity::new(None, rsync_base_uri),
mode: RepoTransportMode::Rsync,
retry_short_timeout: false,
tal_id: "arin".to_string(),
rir_id: "arin".to_string(),
validation_time: time::OffsetDateTime::UNIX_EPOCH,
priority: 0,
requesters: vec![RepoRequester::with_tal_rir(
"arin",
"arin",
format!("{rsync_base_uri}root.mft"),
rsync_base_uri.to_string(),
"node:rsync",
)],
}
}
struct SuccessExecutor;
impl RepoSyncExecutor for SuccessExecutor {
fn execute(&self, task: RepoSyncTask) -> RepoSyncResultEnvelope {
RepoSyncResultEnvelope {
repo_key: task.repo_key.clone(),
tal_id: task.tal_id,
rir_id: task.rir_id,
result: RepoSyncResultKind::Success(RepoSyncResultRef {
repo_key: task.repo_key,
source: "rrdp".to_string(),
}),
phase: Some("rrdp_ok".to_string()),
timing_ms: 12,
warnings: Vec::new(),
}
}
}
struct FailureExecutor;
impl RepoSyncExecutor for FailureExecutor {
fn execute(&self, task: RepoSyncTask) -> RepoSyncResultEnvelope {
RepoSyncResultEnvelope {
repo_key: task.repo_key,
tal_id: task.tal_id,
rir_id: task.rir_id,
result: RepoSyncResultKind::Failed {
detail: "timeout".to_string(),
},
phase: Some("repo_sync_failed".to_string()),
timing_ms: 33,
warnings: vec![Warning::new("timeout")],
}
}
}
struct BarrierExecutor {
barrier: Arc<Barrier>,
active: Arc<AtomicUsize>,
peak: Arc<AtomicUsize>,
}
#[derive(Clone)]
struct PanicHttpFetcher;
impl Fetcher for PanicHttpFetcher {
fn fetch(&self, _uri: &str) -> Result<Vec<u8>, String> {
panic!("http fetch should not be used in this test")
}
}
#[derive(Clone)]
struct ErrorHttpFetcher;
impl Fetcher for ErrorHttpFetcher {
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
Err(format!("fetch blocked for {uri}"))
}
}
#[derive(Clone)]
struct FailingRsyncFetcher;
#[derive(Clone, Default)]
struct MockHttpFetcher {
map: Arc<Mutex<HashMap<String, Vec<u8>>>>,
}
impl MockHttpFetcher {
fn new() -> Self {
Self::default()
}
fn insert(&self, uri: &str, bytes: Vec<u8>) {
self.map
.lock()
.expect("http fixture lock")
.insert(uri.to_string(), bytes);
}
}
impl RsyncFetcher for FailingRsyncFetcher {
fn fetch_objects(&self, _rsync_base_uri: &str) -> RsyncFetchResult<Vec<(String, Vec<u8>)>> {
Err(RsyncFetchError::Fetch("boom".to_string()))
}
}
impl Fetcher for MockHttpFetcher {
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
self.map
.lock()
.expect("http fixture lock")
.get(uri)
.cloned()
.ok_or_else(|| format!("missing fixture for {uri}"))
}
}
fn sha256_hex(bytes: &[u8]) -> String {
hex::encode(sha2::Sha256::digest(bytes))
}
fn rrdp_notification_xml(session_id: &str, serial: u64, snapshot_uri: &str) -> String {
let snapshot_body = rrdp_snapshot_xml(
session_id,
&[("rsync://example.test/repo/a.roa", b"a".as_ref())],
);
let snapshot_hash = sha256_hex(snapshot_body.as_bytes());
format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session_id}" serial="{serial}">
<snapshot uri="{snapshot_uri}" hash="{snapshot_hash}"/>
</notification>"#
)
}
fn rrdp_snapshot_xml(session_id: &str, objects: &[(&str, &[u8])]) -> String {
let mut body = String::from(&format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<snapshot xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session_id}" serial="1">
"#
));
for (uri, bytes) in objects {
body.push_str(&format!(
" <publish uri=\"{uri}\">{}</publish>\n",
base64::engine::general_purpose::STANDARD.encode(bytes)
));
}
body.push_str("</snapshot>");
body
}
impl RepoSyncExecutor for BarrierExecutor {
fn execute(&self, task: RepoSyncTask) -> RepoSyncResultEnvelope {
let now = self.active.fetch_add(1, Ordering::SeqCst) + 1;
loop {
let peak = self.peak.load(Ordering::SeqCst);
if now > peak {
if self
.peak
.compare_exchange(peak, now, Ordering::SeqCst, Ordering::SeqCst)
.is_ok()
{
break;
}
} else {
break;
}
}
self.barrier.wait();
self.active.fetch_sub(1, Ordering::SeqCst);
RepoSyncResultEnvelope {
repo_key: task.repo_key.clone(),
tal_id: task.tal_id,
rir_id: task.rir_id,
result: RepoSyncResultKind::Success(RepoSyncResultRef {
repo_key: task.repo_key,
source: "rsync".to_string(),
}),
phase: Some("rsync_only_ok".to_string()),
timing_ms: 1,
warnings: Vec::new(),
}
}
}
struct SuccessTransportExecutor;
impl RepoTransportExecutor for SuccessTransportExecutor {
fn execute_transport(&self, task: RepoTransportTask) -> RepoTransportResultEnvelope {
RepoTransportResultEnvelope {
dedup_key: task.dedup_key,
rsync_failure_scope_uri: task.rsync_failure_scope_uri,
repo_identity: task.repo_identity,
mode: task.mode,
tal_id: task.tal_id,
rir_id: task.rir_id,
timing_ms: 5,
result: RepoTransportResultKind::Success {
source: match task.mode {
RepoTransportMode::Rrdp => "rrdp".to_string(),
RepoTransportMode::Rsync => "rsync".to_string(),
},
warnings: Vec::new(),
},
}
}
}
#[test]
fn repo_worker_pool_config_uses_parallel_phase1_budget() {
let cfg = ParallelPhase1Config {
max_repo_sync_workers_global: 9,
..ParallelPhase1Config::default()
};
let pool_cfg = RepoWorkerPoolConfig::from(&cfg);
assert_eq!(pool_cfg.max_workers, 9);
}
#[test]
fn repo_worker_pool_rejects_zero_workers() {
let err = RepoWorkerPool::new(RepoWorkerPoolConfig { max_workers: 0 }, SuccessExecutor)
.err()
.expect("zero workers should fail");
assert!(err.contains("at least one worker"));
}
#[test]
fn repo_worker_pool_processes_tasks_and_returns_results() {
let pool = RepoWorkerPool::new(RepoWorkerPoolConfig { max_workers: 2 }, SuccessExecutor)
.expect("pool");
pool.submit(sample_task("a")).expect("submit task a");
pool.submit(sample_task("b")).expect("submit task b");
let mut results = Vec::new();
results.push(
pool.recv_result_timeout(Duration::from_secs(1))
.expect("recv result 1")
.expect("result 1"),
);
results.push(
pool.recv_result_timeout(Duration::from_secs(1))
.expect("recv result 2")
.expect("result 2"),
);
assert_eq!(results.len(), 2);
assert!(
results
.iter()
.all(|res| matches!(res.result, RepoSyncResultKind::Success(_)))
);
}
#[test]
fn repo_worker_pool_returns_failure_envelopes() {
let pool = RepoWorkerPool::new(RepoWorkerPoolConfig { max_workers: 1 }, FailureExecutor)
.expect("pool");
pool.submit(sample_task("fail")).expect("submit");
let result = pool
.recv_result_timeout(Duration::from_secs(1))
.expect("recv")
.expect("result");
assert!(matches!(
result.result,
RepoSyncResultKind::Failed { ref detail } if detail == "timeout"
));
assert_eq!(result.warnings, vec![Warning::new("timeout")]);
}
#[test]
fn repo_worker_pool_processes_work_in_parallel() {
let barrier = Arc::new(Barrier::new(2));
let active = Arc::new(AtomicUsize::new(0));
let peak = Arc::new(AtomicUsize::new(0));
let executor = BarrierExecutor {
barrier: Arc::clone(&barrier),
active: Arc::clone(&active),
peak: Arc::clone(&peak),
};
let pool =
RepoWorkerPool::new(RepoWorkerPoolConfig { max_workers: 2 }, executor).expect("pool");
pool.submit(sample_task("a")).expect("submit a");
pool.submit(sample_task("b")).expect("submit b");
let _ = pool
.recv_result_timeout(Duration::from_secs(1))
.expect("recv 1")
.expect("result 1");
let _ = pool
.recv_result_timeout(Duration::from_secs(1))
.expect("recv 2")
.expect("result 2");
assert!(peak.load(Ordering::SeqCst) >= 2);
}
#[test]
fn repo_worker_pool_exposes_configured_worker_count() {
let pool = RepoWorkerPool::new(RepoWorkerPoolConfig { max_workers: 3 }, SuccessExecutor)
.expect("pool");
assert_eq!(pool.worker_count(), 3);
}
#[test]
fn repo_transport_worker_pool_processes_transport_tasks() {
let pool = RepoTransportWorkerPool::new(
RepoWorkerPoolConfig { max_workers: 2 },
SuccessTransportExecutor,
)
.expect("pool");
pool.submit(sample_rrdp_transport_task(
"https://example.test/notify.xml",
"rsync://example.test/repo/",
))
.expect("submit rrdp");
pool.submit(sample_rsync_transport_task(
"rsync://example.test/module/",
"rsync://example.test/repo/",
))
.expect("submit rsync");
let first = pool
.recv_result_timeout(Duration::from_secs(1))
.expect("recv first")
.expect("first result");
let second = pool
.recv_result_timeout(Duration::from_secs(1))
.expect("recv second")
.expect("second result");
assert!(matches!(
first.result,
RepoTransportResultKind::Success { .. }
));
assert!(matches!(
second.result,
RepoTransportResultKind::Success { .. }
));
}
#[test]
fn live_repo_sync_executor_runs_rsync_sync_and_updates_store() {
let td = tempfile::tempdir().expect("tempdir");
fs::create_dir_all(td.path().join("nested")).expect("mkdir");
fs::write(td.path().join("a.roa"), b"a").expect("write a");
fs::write(td.path().join("nested").join("b.cer"), b"b").expect("write b");
let store_dir = tempfile::tempdir().expect("store tempdir");
let store = Arc::new(RocksStore::open(store_dir.path()).expect("open store"));
let executor = LiveRepoSyncExecutor::new(
Arc::clone(&store),
crate::policy::Policy::default(),
Arc::new(PanicHttpFetcher),
Arc::new(LocalDirRsyncFetcher::new(td.path())),
None,
None,
);
let result = executor.execute(sample_task("repo"));
assert!(matches!(result.result, RepoSyncResultKind::Success(_)));
assert_eq!(result.tal_id, "arin");
assert_eq!(result.rir_id, "arin");
assert_eq!(result.warnings.len(), 0);
let view = store
.get_repository_view_entry("rsync://example.test/repo/a.roa")
.expect("read view")
.expect("entry exists");
assert_eq!(view.rsync_uri, "rsync://example.test/repo/a.roa");
assert!(view.current_hash.is_some());
}
#[test]
fn live_repo_sync_executor_returns_failure_when_rsync_fails() {
let store_dir = tempfile::tempdir().expect("store tempdir");
let store = Arc::new(RocksStore::open(store_dir.path()).expect("open store"));
let executor = LiveRepoSyncExecutor::new(
Arc::clone(&store),
crate::policy::Policy::default(),
Arc::new(PanicHttpFetcher),
Arc::new(FailingRsyncFetcher),
None,
None,
);
let result = executor.execute(sample_task("repo"));
assert!(matches!(
result.result,
RepoSyncResultKind::Failed { ref detail } if detail.contains("rsync fallback failed")
));
assert!(
store
.get_repository_view_entry("rsync://example.test/repo/a.roa")
.expect("read view")
.is_none()
);
}
#[test]
fn live_rrdp_transport_executor_succeeds_on_valid_notification() {
let td = tempfile::tempdir().expect("tempdir");
let store = Arc::new(RocksStore::open(td.path()).expect("open store"));
let http = MockHttpFetcher::new();
let notify = "https://example.test/notification.xml";
let snapshot = "https://example.test/snapshot.xml";
let snapshot_bytes = rrdp_snapshot_xml(
"123e4567-e89b-12d3-a456-426614174000",
&[("rsync://example.test/repo/a.roa", b"a".as_ref())],
);
let notification_bytes =
rrdp_notification_xml("123e4567-e89b-12d3-a456-426614174000", 1, snapshot);
http.insert(notify, notification_bytes.into_bytes());
http.insert(snapshot, snapshot_bytes.into_bytes());
let current_repo_index = CurrentRepoIndex::shared();
let executor = LiveRrdpTransportExecutor::new(
Arc::clone(&store),
current_repo_index.clone(),
Arc::new(http),
None,
None,
);
let result = executor.execute_transport(sample_rrdp_transport_task(
notify,
"rsync://example.test/repo/",
));
assert!(
matches!(result.result, RepoTransportResultKind::Success { .. }),
"{result:?}"
);
let index = current_repo_index.read().expect("index read lock");
assert!(
index
.get_by_uri("rsync://example.test/repo/a.roa")
.is_some()
);
assert_eq!(
index.list_scope_uris(notify),
vec!["rsync://example.test/repo/a.roa".to_string()]
);
}
#[test]
fn live_rrdp_transport_executor_reports_failure_without_rsync_fallback() {
let td = tempfile::tempdir().expect("tempdir");
let store = Arc::new(RocksStore::open(td.path()).expect("open store"));
let executor = LiveRrdpTransportExecutor::new(
Arc::clone(&store),
CurrentRepoIndex::shared(),
Arc::new(ErrorHttpFetcher),
None,
None,
);
let result = executor.execute_transport(sample_rrdp_transport_task(
"https://example.test/notification.xml",
"rsync://example.test/repo/",
));
assert!(matches!(
result.result,
RepoTransportResultKind::Failed { .. }
));
}
#[test]
fn live_rsync_transport_executor_succeeds_on_local_repo() {
let td = tempfile::tempdir().expect("tempdir");
fs::create_dir_all(td.path().join("nested")).expect("mkdir");
fs::write(td.path().join("a.roa"), b"a").expect("write");
fs::write(td.path().join("nested").join("b.cer"), b"b").expect("write");
let store_dir = tempfile::tempdir().expect("store tempdir");
let store = Arc::new(RocksStore::open(store_dir.path()).expect("open store"));
let current_repo_index = CurrentRepoIndex::shared();
let executor = LiveRsyncTransportExecutor::new(
Arc::clone(&store),
current_repo_index.clone(),
Arc::new(LocalDirRsyncFetcher::new(td.path())),
None,
None,
);
let result = executor.execute_transport(sample_rsync_transport_task(
"rsync://example.test/repo/",
"rsync://example.test/repo/",
));
assert!(matches!(
result.result,
RepoTransportResultKind::Success { .. }
));
let index = current_repo_index.read().expect("index read lock");
assert!(
index
.get_by_uri("rsync://example.test/repo/a.roa")
.is_some()
);
assert!(
index
.get_by_uri("rsync://example.test/repo/nested/b.cer")
.is_some()
);
assert_eq!(
index.list_scope_uris("rsync://example.test/repo/"),
vec![
"rsync://example.test/repo/a.roa".to_string(),
"rsync://example.test/repo/nested/b.cer".to_string()
]
);
}
#[test]
fn live_rsync_transport_executor_reports_failure() {
let store_dir = tempfile::tempdir().expect("store tempdir");
let store = Arc::new(RocksStore::open(store_dir.path()).expect("open store"));
let executor = LiveRsyncTransportExecutor::new(
Arc::clone(&store),
CurrentRepoIndex::shared(),
Arc::new(FailingRsyncFetcher),
None,
None,
);
let result = executor.execute_transport(sample_rsync_transport_task(
"rsync://example.test/module/",
"rsync://example.test/repo/",
));
assert!(matches!(
result.result,
RepoTransportResultKind::Failed { .. }
));
}

View File

@@ -26,7 +26,7 @@ pub struct GlobalRunCoordinator {
pub pending_repo_tasks: VecDeque<RepoSyncTask>,
pub pending_transport_tasks: VecDeque<RepoTransportTask>,
pub stats: ParallelRunStats,
/// Mutable working copy of the dead-repo blacklist (#141): failure/success
/// Mutable working copy of the dead-repository blacklist: failure/success
/// counters update during the run and persist at run end. Skip decisions
/// inside `transport_tables` use the frozen run-start snapshot.
pub dead_repo_blacklist: Option<DeadRepoBlacklist>,
@@ -221,7 +221,7 @@ impl GlobalRunCoordinator {
Ok(completion)
}
/// Update dead-repo blacklist counters (#141) from a real worker result.
/// Update dead-repository blacklist counters from a real worker result.
/// Runs on the single pump thread; synthesized terminal envelopes never
/// pass through here, so no double counting is possible.
fn update_dead_repo_blacklist(
@@ -581,7 +581,7 @@ mod tests {
assert_eq!(coordinator.stats.repo_tasks_reused, 1);
}
// ---- Dead-repo blacklist (#141) integration tests ----
// ---- Dead-repository blacklist integration tests ----
use crate::parallel::dead_repo_blacklist::{DeadRepoBlacklist, DeadRepoBlacklistConfig};
use crate::parallel::types::{
@@ -751,7 +751,7 @@ mod tests {
assert_eq!(task.mode, RepoTransportMode::Rsync);
// register_dead passes retry_short_timeout=false (mirroring a prefetch
// snapshot that recorded no real rrdp failure for skipped runs); the
// skip path must still keep the short retry profile (#141).
// Skip decisions must retain the short retry profile.
assert!(task.retry_short_timeout);
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,270 @@
// Delta archive capture and filesystem loading helpers.
fn load_delta_rrdp_repo(
capture_root: &Path,
notify_uri: &str,
entry: &ReplayDeltaRrdpEntry,
) -> Result<ReplayDeltaRrdpRepo, ReplayDeltaArchiveError> {
let bucket_hash = sha256_hex(notify_uri.as_bytes());
let bucket_dir = capture_root.join("rrdp").join("repos").join(&bucket_hash);
if !bucket_dir.is_dir() {
return Err(ReplayDeltaArchiveError::MissingDeltaRepoBucket {
notify_uri: notify_uri.to_string(),
path: bucket_dir.display().to_string(),
});
}
let meta: ReplayRrdpRepoMeta =
read_delta_json_file(&bucket_dir.join("meta.json"), "delta RRDP repo meta")?;
ensure_delta_version("delta RRDP repo meta", meta.version)?;
if meta.rpki_notify != notify_uri {
return Err(ReplayDeltaArchiveError::RrdpMetaMismatch {
expected: notify_uri.to_string(),
actual: meta.rpki_notify.clone(),
});
}
let transition: ReplayDeltaTransition =
read_delta_json_file(&bucket_dir.join("transition.json"), "delta transition")?;
if transition.kind != entry.kind {
return Err(ReplayDeltaArchiveError::TransitionKindMismatch {
notify_uri: notify_uri.to_string(),
locks_kind: entry.kind.as_str().to_string(),
transition_kind: transition.kind.as_str().to_string(),
});
}
if transition.base != entry.base {
return Err(ReplayDeltaArchiveError::TransitionBaseMismatch {
notify_uri: notify_uri.to_string(),
});
}
if transition.target != entry.target {
return Err(ReplayDeltaArchiveError::TransitionTargetMismatch {
notify_uri: notify_uri.to_string(),
});
}
if transition.delta_count != entry.delta_count || transition.deltas != entry.deltas {
return Err(ReplayDeltaArchiveError::DeltaSerialListMismatch {
notify_uri: notify_uri.to_string(),
});
}
let (target_notification_path, delta_paths, target_archive_path) = match entry.kind {
ReplayDeltaRrdpKind::Delta => {
let session = entry.target.session.as_ref().ok_or_else(|| {
ReplayDeltaArchiveError::TransitionTargetMismatch {
notify_uri: notify_uri.to_string(),
}
})?;
let serial = entry.target.serial.ok_or_else(|| {
ReplayDeltaArchiveError::TransitionTargetMismatch {
notify_uri: notify_uri.to_string(),
}
})?;
let session_dir = bucket_dir.join(session);
if !session_dir.is_dir() {
return Err(ReplayDeltaArchiveError::MissingDeltaSessionDir {
notify_uri: notify_uri.to_string(),
path: session_dir.display().to_string(),
});
}
let notification = session_dir.join(format!("notification-target-{serial}.xml"));
if !notification.is_file() {
return Err(ReplayDeltaArchiveError::MissingTargetNotification {
notify_uri: notify_uri.to_string(),
path: notification.display().to_string(),
});
}
let mut delta_paths = Vec::new();
for delta_serial in &entry.deltas {
let pattern = format!("delta-{delta_serial}-");
let deltas_dir = session_dir.join("deltas");
let mut matches = if deltas_dir.is_dir() {
fs::read_dir(&deltas_dir)
.map_err(|e| {
ReplayDeltaArchiveError::Base(ReplayArchiveError::ReadFile {
entity: "delta deltas dir",
path: deltas_dir.display().to_string(),
detail: e.to_string(),
})
})?
.filter_map(|entry| entry.ok().map(|e| e.path()))
.filter(|path| path.is_file())
.filter(|path| {
path.file_name()
.and_then(|n| n.to_str())
.is_some_and(|n| n.starts_with(&pattern) && n.ends_with(".xml"))
})
.collect::<Vec<_>>()
} else {
Vec::new()
};
matches.sort();
let path = matches.into_iter().next().ok_or_else(|| {
ReplayDeltaArchiveError::MissingDeltaFile {
notify_uri: notify_uri.to_string(),
serial: *delta_serial,
path: deltas_dir
.join(format!("delta-{delta_serial}-<hash>.xml"))
.display()
.to_string(),
}
})?;
delta_paths.push((*delta_serial, path));
}
let target_archive = bucket_dir.join(format!("target-archive-{serial}.bin"));
let target_archive_path = if target_archive.is_file() {
Some(target_archive)
} else {
None
};
(Some(notification), delta_paths, target_archive_path)
}
ReplayDeltaRrdpKind::Unchanged | ReplayDeltaRrdpKind::FallbackRsync => {
(None, Vec::new(), None)
}
ReplayDeltaRrdpKind::SessionReset | ReplayDeltaRrdpKind::Gap => (None, Vec::new(), None),
};
Ok(ReplayDeltaRrdpRepo {
notify_uri: notify_uri.to_string(),
bucket_hash,
bucket_dir,
meta,
transition,
target_notification_path,
delta_paths,
target_archive_path,
})
}
fn load_delta_rsync_module(
capture_root: &Path,
module_uri: &str,
entry: &ReplayDeltaRsyncEntry,
) -> Result<ReplayDeltaRsyncModule, ReplayDeltaArchiveError> {
let canonical = canonical_rsync_module(module_uri).map_err(ReplayDeltaArchiveError::Base)?;
let bucket_hash = sha256_hex(canonical.as_bytes());
let bucket_dir = capture_root
.join("rsync")
.join("modules")
.join(&bucket_hash);
if !bucket_dir.is_dir() {
return Err(ReplayDeltaArchiveError::MissingRsyncModuleBucket {
module_uri: canonical.clone(),
path: bucket_dir.display().to_string(),
});
}
let meta_path = bucket_dir.join("meta.json");
let meta: ReplayRsyncModuleMeta = if meta_path.is_file() {
let meta: ReplayRsyncModuleMeta =
read_delta_json_file(&meta_path, "delta rsync module meta")?;
ensure_delta_version("delta rsync module meta", meta.version)?;
if meta.module != canonical {
return Err(ReplayDeltaArchiveError::RsyncMetaMismatch {
expected: canonical.clone(),
actual: meta.module.clone(),
});
}
meta
} else {
ReplayRsyncModuleMeta {
version: 1,
module: canonical.clone(),
created_at: String::new(),
last_seen_at: String::new(),
}
};
let files: ReplayDeltaRsyncFiles =
read_delta_json_file(&bucket_dir.join("files.json"), "delta rsync files")?;
ensure_delta_version("delta rsync files", files.version)?;
if files.module != canonical {
return Err(ReplayDeltaArchiveError::RsyncFilesModuleMismatch {
expected: canonical.clone(),
actual: files.module.clone(),
});
}
if files.file_count != entry.file_count || files.file_count != files.files.len() {
return Err(ReplayDeltaArchiveError::RsyncFileCountMismatch {
module_uri: canonical.clone(),
declared: entry.file_count,
actual: files.files.len(),
});
}
let tree_dir = bucket_dir.join("tree");
let mut overlay_files = Vec::new();
for uri in &files.files {
let rel = uri.strip_prefix(&canonical).ok_or_else(|| {
ReplayDeltaArchiveError::RsyncFilesModuleMismatch {
expected: canonical.clone(),
actual: uri.clone(),
}
})?;
let tree_root = module_tree_root(&canonical, &tree_dir)?;
let path = tree_root.join(rel);
if !path.is_file() {
return Err(ReplayDeltaArchiveError::MissingRsyncOverlayFile {
module_uri: canonical.clone(),
path: path.display().to_string(),
});
}
overlay_files.push((uri.clone(), path));
}
Ok(ReplayDeltaRsyncModule {
module_uri: canonical,
bucket_hash,
bucket_dir,
meta,
overlay_only: entry.overlay_only,
files,
tree_dir,
overlay_files,
})
}
fn module_tree_root(module_uri: &str, tree_dir: &Path) -> Result<PathBuf, ReplayDeltaArchiveError> {
let rest = module_uri.strip_prefix("rsync://").ok_or_else(|| {
ReplayDeltaArchiveError::Base(ReplayArchiveError::InvalidRsyncUri {
uri: module_uri.to_string(),
detail: "URI must start with rsync://".to_string(),
})
})?;
let mut parts = rest.trim_end_matches('/').split('/');
let authority = parts.next().unwrap_or_default();
let module = parts.next().unwrap_or_default();
Ok(tree_dir.join(authority).join(module))
}
fn ensure_delta_version(entity: &'static str, version: u32) -> Result<(), ReplayDeltaArchiveError> {
if version == 1 {
Ok(())
} else {
Err(ReplayDeltaArchiveError::Base(
ReplayArchiveError::UnsupportedVersion { entity, version },
))
}
}
fn read_delta_json_file<T: for<'de> Deserialize<'de>>(
path: &Path,
entity: &'static str,
) -> Result<T, ReplayDeltaArchiveError> {
let bytes = fs::read(path).map_err(|e| {
ReplayDeltaArchiveError::Base(ReplayArchiveError::ReadFile {
entity,
path: path.display().to_string(),
detail: e.to_string(),
})
})?;
serde_json::from_slice(&bytes).map_err(|e| {
ReplayDeltaArchiveError::Base(ReplayArchiveError::ParseJson {
entity,
path: path.display().to_string(),
detail: e.to_string(),
})
})
}

View File

@@ -0,0 +1,344 @@
// Payload-delta lock, capture, and archive index models.
#[derive(Debug, thiserror::Error)]
pub enum ReplayDeltaArchiveError {
#[error(transparent)]
Base(#[from] ReplayArchiveError),
#[error("delta capture directory not found: {0}")]
MissingDeltaCaptureDirectory(String),
#[error("delta capture.json captureId mismatch: locks={locks_capture}, capture={capture_json}")]
CaptureIdMismatch {
locks_capture: String,
capture_json: String,
},
#[error(
"delta base.json baseCapture mismatch: locks={locks_base_capture}, base_json={base_json_base_capture}"
)]
BaseCaptureMismatch {
locks_base_capture: String,
base_json_base_capture: String,
},
#[error(
"delta base.json baseLocksSha256 mismatch: locks={locks_sha256}, base_json={base_json_sha256}"
)]
BaseLocksShaMismatch {
locks_sha256: String,
base_json_sha256: String,
},
#[error("base locks sha256 mismatch: expected {expected}, actual {actual}")]
BaseLocksBytesShaMismatch { expected: String, actual: String },
#[error("delta repo bucket not found for {notify_uri}: {path}")]
MissingDeltaRepoBucket { notify_uri: String, path: String },
#[error("delta repo meta mismatch: expected {expected}, actual {actual}")]
RrdpMetaMismatch { expected: String, actual: String },
#[error(
"delta transition kind mismatch for {notify_uri}: locks={locks_kind}, transition={transition_kind}"
)]
TransitionKindMismatch {
notify_uri: String,
locks_kind: String,
transition_kind: String,
},
#[error("delta transition base mismatch for {notify_uri}")]
TransitionBaseMismatch { notify_uri: String },
#[error("delta transition target mismatch for {notify_uri}")]
TransitionTargetMismatch { notify_uri: String },
#[error("delta serial list mismatch for {notify_uri}")]
DeltaSerialListMismatch { notify_uri: String },
#[error("delta notification session directory not found for {notify_uri}: {path}")]
MissingDeltaSessionDir { notify_uri: String, path: String },
#[error("target notification file not found for {notify_uri}: {path}")]
MissingTargetNotification { notify_uri: String, path: String },
#[error("delta file not found for {notify_uri} serial={serial}: {path}")]
MissingDeltaFile {
notify_uri: String,
serial: u64,
path: String,
},
#[error("delta target archive missing for {notify_uri}: {path}")]
MissingTargetArchive { notify_uri: String, path: String },
#[error("delta rsync module bucket not found for {module_uri}: {path}")]
MissingRsyncModuleBucket { module_uri: String, path: String },
#[error("delta rsync module meta mismatch: expected {expected}, actual {actual}")]
RsyncMetaMismatch { expected: String, actual: String },
#[error("delta rsync files.json module mismatch: expected {expected}, actual {actual}")]
RsyncFilesModuleMismatch { expected: String, actual: String },
#[error(
"delta rsync file count mismatch for {module_uri}: declared={declared}, actual={actual}"
)]
RsyncFileCountMismatch {
module_uri: String,
declared: usize,
actual: usize,
},
#[error("delta rsync overlay file not found for {module_uri}: {path}")]
MissingRsyncOverlayFile { module_uri: String, path: String },
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaLocks {
pub version: u32,
pub capture: String,
#[serde(rename = "baseCapture")]
pub base_capture: String,
#[serde(rename = "baseLocksSha256")]
pub base_locks_sha256: String,
pub rrdp: BTreeMap<String, ReplayDeltaRrdpEntry>,
pub rsync: BTreeMap<String, ReplayDeltaRsyncEntry>,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaBaseMeta {
pub version: u32,
#[serde(rename = "baseCapture")]
pub base_capture: String,
#[serde(rename = "baseLocksSha256")]
pub base_locks_sha256: String,
#[serde(rename = "createdAt")]
pub created_at: String,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum ReplayDeltaRrdpKind {
Unchanged,
Delta,
FallbackRsync,
SessionReset,
Gap,
}
impl ReplayDeltaRrdpKind {
pub fn as_str(self) -> &'static str {
match self {
Self::Unchanged => "unchanged",
Self::Delta => "delta",
Self::FallbackRsync => "fallback-rsync",
Self::SessionReset => "session-reset",
Self::Gap => "gap",
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaRrdpState {
pub transport: ReplayTransport,
pub session: Option<String>,
pub serial: Option<u64>,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaRrdpEntry {
pub kind: ReplayDeltaRrdpKind,
pub base: ReplayDeltaRrdpState,
pub target: ReplayDeltaRrdpState,
#[serde(rename = "delta_count")]
pub delta_count: usize,
pub deltas: Vec<u64>,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaRsyncEntry {
#[serde(rename = "file_count")]
pub file_count: usize,
#[serde(rename = "overlay_only")]
pub overlay_only: bool,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaTransition {
pub kind: ReplayDeltaRrdpKind,
pub base: ReplayDeltaRrdpState,
pub target: ReplayDeltaRrdpState,
#[serde(rename = "delta_count")]
pub delta_count: usize,
pub deltas: Vec<u64>,
}
#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
pub struct ReplayDeltaRsyncFiles {
pub version: u32,
pub module: String,
#[serde(rename = "fileCount")]
pub file_count: usize,
pub files: Vec<String>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ReplayDeltaRrdpRepo {
pub notify_uri: String,
pub bucket_hash: String,
pub bucket_dir: PathBuf,
pub meta: ReplayRrdpRepoMeta,
pub transition: ReplayDeltaTransition,
pub target_notification_path: Option<PathBuf>,
pub delta_paths: Vec<(u64, PathBuf)>,
pub target_archive_path: Option<PathBuf>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ReplayDeltaRsyncModule {
pub module_uri: String,
pub bucket_hash: String,
pub bucket_dir: PathBuf,
pub meta: ReplayRsyncModuleMeta,
pub overlay_only: bool,
pub files: ReplayDeltaRsyncFiles,
pub tree_dir: PathBuf,
pub overlay_files: Vec<(String, PathBuf)>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ReplayDeltaArchiveIndex {
pub archive_root: PathBuf,
pub capture_root: PathBuf,
pub delta_locks_path: PathBuf,
pub delta_locks: ReplayDeltaLocks,
pub capture_meta: crate::replay::archive::ReplayCaptureMeta,
pub base_meta: ReplayDeltaBaseMeta,
pub rrdp_repos: BTreeMap<String, ReplayDeltaRrdpRepo>,
pub rsync_modules: BTreeMap<String, ReplayDeltaRsyncModule>,
}
impl ReplayDeltaArchiveIndex {
pub fn load(
delta_archive_root: impl AsRef<Path>,
delta_locks_path: impl AsRef<Path>,
) -> Result<Self, ReplayDeltaArchiveError> {
let archive_root = delta_archive_root.as_ref().to_path_buf();
let delta_locks_path = delta_locks_path.as_ref().to_path_buf();
let delta_locks: ReplayDeltaLocks =
read_delta_json_file(&delta_locks_path, "payload delta locks")?;
ensure_delta_version("payload delta locks", delta_locks.version)?;
let capture_root = archive_root
.join("v1")
.join("captures")
.join(&delta_locks.capture);
if !capture_root.is_dir() {
return Err(ReplayDeltaArchiveError::MissingDeltaCaptureDirectory(
capture_root.display().to_string(),
));
}
let capture_meta: crate::replay::archive::ReplayCaptureMeta =
read_delta_json_file(&capture_root.join("capture.json"), "delta capture meta")?;
ensure_delta_version("delta capture meta", capture_meta.version)?;
if capture_meta.capture_id != delta_locks.capture {
return Err(ReplayDeltaArchiveError::CaptureIdMismatch {
locks_capture: delta_locks.capture.clone(),
capture_json: capture_meta.capture_id.clone(),
});
}
let base_meta: ReplayDeltaBaseMeta =
read_delta_json_file(&capture_root.join("base.json"), "delta base meta")?;
ensure_delta_version("delta base meta", base_meta.version)?;
if base_meta.base_capture != delta_locks.base_capture {
return Err(ReplayDeltaArchiveError::BaseCaptureMismatch {
locks_base_capture: delta_locks.base_capture.clone(),
base_json_base_capture: base_meta.base_capture.clone(),
});
}
if base_meta.base_locks_sha256.to_ascii_lowercase()
!= delta_locks.base_locks_sha256.to_ascii_lowercase()
{
return Err(ReplayDeltaArchiveError::BaseLocksShaMismatch {
locks_sha256: delta_locks.base_locks_sha256.clone(),
base_json_sha256: base_meta.base_locks_sha256.clone(),
});
}
let mut rrdp_repos = BTreeMap::new();
for (notify_uri, entry) in &delta_locks.rrdp {
let repo = load_delta_rrdp_repo(&capture_root, notify_uri, entry)?;
rrdp_repos.insert(notify_uri.clone(), repo);
}
let mut rsync_modules = BTreeMap::new();
for (module_uri, entry) in &delta_locks.rsync {
let module = load_delta_rsync_module(&capture_root, module_uri, entry)?;
rsync_modules.insert(module.module_uri.clone(), module);
}
Ok(Self {
archive_root,
capture_root,
delta_locks_path,
delta_locks,
capture_meta,
base_meta,
rrdp_repos,
rsync_modules,
})
}
pub fn rrdp_repo(&self, notify_uri: &str) -> Option<&ReplayDeltaRrdpRepo> {
self.rrdp_repos.get(notify_uri)
}
pub fn rsync_module(&self, module_uri: &str) -> Option<&ReplayDeltaRsyncModule> {
self.rsync_modules.get(module_uri)
}
pub fn resolve_rsync_module_for_base_uri(
&self,
rsync_base_uri: &str,
) -> Result<&ReplayDeltaRsyncModule, ReplayDeltaArchiveError> {
let module_uri =
canonical_rsync_module(rsync_base_uri).map_err(ReplayDeltaArchiveError::Base)?;
self.rsync_modules.get(&module_uri).ok_or_else(|| {
ReplayDeltaArchiveError::MissingRsyncModuleBucket {
module_uri,
path: "<not indexed>".to_string(),
}
})
}
pub fn validate_base_locks_sha256_bytes(
&self,
base_locks_bytes: &[u8],
) -> Result<(), ReplayDeltaArchiveError> {
let actual = sha256_hex(base_locks_bytes);
let expected = self.delta_locks.base_locks_sha256.to_ascii_lowercase();
if actual != expected {
return Err(ReplayDeltaArchiveError::BaseLocksBytesShaMismatch { expected, actual });
}
Ok(())
}
pub fn validate_base_locks_sha256_file(
&self,
path: &Path,
) -> Result<(), ReplayDeltaArchiveError> {
let bytes = fs::read(path).map_err(|e| {
ReplayDeltaArchiveError::Base(ReplayArchiveError::ReadFile {
entity: "base locks file",
path: path.display().to_string(),
detail: e.to_string(),
})
})?;
self.validate_base_locks_sha256_bytes(&bytes)
}
}

View File

@@ -0,0 +1,537 @@
// Payload-delta archive loading and validation tests.
use super::*;
fn build_delta_fixture() -> (tempfile::TempDir, PathBuf, PathBuf, String, String) {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-delta-archive");
let capture = "delta-cap";
let base_capture = "base-cap";
let base_sha = "deadbeef";
let capture_root = archive_root.join("v1").join("captures").join(capture);
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
std::fs::write(
capture_root.join("capture.json"),
format!(
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-15T00:00:00Z","notes":""}}"#
),
)
.expect("write capture json");
std::fs::write(
capture_root.join("base.json"),
format!(
r#"{{"version":1,"baseCapture":"{base_capture}","baseLocksSha256":"{base_sha}","createdAt":"2026-03-15T00:00:00Z"}}"#
),
)
.expect("write base json");
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
let session = "11111111-1111-1111-1111-111111111111".to_string();
let _target_serial = 12u64;
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = capture_root
.join("rrdp/repos")
.join(&repo_hash)
.join(&session);
let deltas_dir = session_dir.join("deltas");
std::fs::create_dir_all(&deltas_dir).expect("mkdir deltas");
std::fs::write(
session_dir.parent().unwrap().join("meta.json"),
format!(
r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-15T00:00:00Z","lastSeenAt":"2026-03-15T00:00:01Z"}}"#
),
)
.expect("write meta");
std::fs::write(
session_dir.parent().unwrap().join("transition.json"),
format!(
r#"{{"kind":"delta","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}"#
),
)
.expect("write transition");
std::fs::write(
session_dir.join("notification-target-12.xml"),
b"<notification/>",
)
.expect("write notification");
std::fs::write(
deltas_dir.join("delta-11-aaaa.xml"),
b"<delta serial='11'/>",
)
.expect("write delta 11");
std::fs::write(
deltas_dir.join("delta-12-bbbb.xml"),
b"<delta serial='12'/>",
)
.expect("write delta 12");
std::fs::write(
session_dir.parent().unwrap().join("target-archive-12.bin"),
b"bin",
)
.expect("write target archive");
let module_uri = "rsync://rsync.example.test/repo/".to_string();
let module_hash = sha256_hex(module_uri.as_bytes());
let module_bucket = capture_root.join("rsync/modules").join(&module_hash);
let tree_root = module_bucket
.join("tree")
.join("rsync.example.test")
.join("repo");
std::fs::create_dir_all(tree_root.join("sub")).expect("mkdir tree root");
std::fs::write(
module_bucket.join("meta.json"),
format!(
r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-15T00:00:00Z","lastSeenAt":"2026-03-15T00:00:01Z"}}"#
),
)
.expect("write rsync meta");
std::fs::write(
module_bucket.join("files.json"),
format!(
r#"{{"version":1,"module":"{module_uri}","fileCount":2,"files":["{module_uri}a.roa","{module_uri}sub/b.cer"]}}"#
),
)
.expect("write files json");
std::fs::write(tree_root.join("a.roa"), b"roa").expect("write a.roa");
std::fs::write(tree_root.join("sub").join("b.cer"), b"cer").expect("write b.cer");
let locks_path = temp.path().join("locks-delta.json");
std::fs::write(
&locks_path,
format!(
r#"{{"version":1,"capture":"{capture}","baseCapture":"{base_capture}","baseLocksSha256":"{base_sha}","rrdp":{{"{notify_uri}":{{"kind":"delta","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}}},"rsync":{{"{module_uri}":{{"file_count":2,"overlay_only":true}}}}}}"#
),
)
.expect("write locks-delta");
(temp, archive_root, locks_path, notify_uri, module_uri)
}
#[test]
fn delta_archive_index_loads_unchanged_and_fallback_rsync_rrdp_entries() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let capture_root = archive_root.join("v1/captures/delta-cap");
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = capture_root.join("rrdp/repos").join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"unchanged","base":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"target":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"delta_count":0,"deltas":[]}"#,
)
.expect("rewrite transition unchanged");
std::fs::write(
&locks_path,
r#"{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{"https://rrdp.example.test/notification.xml":{"kind":"unchanged","base":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"target":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"delta_count":0,"deltas":[]}},"rsync":{"rsync://rsync.example.test/repo/":{"file_count":2,"overlay_only":true}}}"#,
).expect("rewrite locks unchanged");
let index =
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load unchanged index");
let repo = index.rrdp_repo(&notify_uri).expect("rrdp repo");
assert_eq!(repo.transition.kind, ReplayDeltaRrdpKind::Unchanged);
assert!(repo.target_notification_path.is_none());
assert!(repo.delta_paths.is_empty());
assert!(repo.target_archive_path.is_none());
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"fallback-rsync","base":{"transport":"rsync","session":null,"serial":null},"target":{"transport":"rsync","session":null,"serial":null},"delta_count":0,"deltas":[]}"#,
).expect("rewrite transition fallback-rsync");
std::fs::write(
&locks_path,
r#"{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{"https://rrdp.example.test/notification.xml":{"kind":"fallback-rsync","base":{"transport":"rsync","session":null,"serial":null},"target":{"transport":"rsync","session":null,"serial":null},"delta_count":0,"deltas":[]}},"rsync":{"rsync://rsync.example.test/repo/":{"file_count":2,"overlay_only":true}}}"#,
).expect("rewrite locks fallback-rsync");
let index = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path)
.expect("load fallback-rsync index");
let repo = index.rrdp_repo(&notify_uri).expect("rrdp repo");
assert_eq!(repo.transition.kind, ReplayDeltaRrdpKind::FallbackRsync);
assert!(repo.target_notification_path.is_none());
assert!(repo.delta_paths.is_empty());
}
#[test]
fn delta_archive_index_resolves_rsync_module_from_base_uri() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
let index =
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index");
let module = index
.resolve_rsync_module_for_base_uri("rsync://rsync.example.test/repo/sub/path")
.expect("resolve module");
assert_eq!(module.module_uri, "rsync://rsync.example.test/repo/");
}
#[test]
fn delta_archive_index_rejects_unsupported_versions_and_meta_mismatches() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
std::fs::write(
&locks_path,
r#"{"version":2,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{},"rsync":{}}"#,
).expect("rewrite locks version");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(
err,
ReplayDeltaArchiveError::Base(ReplayArchiveError::UnsupportedVersion {
entity: "payload delta locks",
version: 2
})
),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
std::fs::write(
archive_root.join("v1/captures/delta-cap/rrdp/repos").join(&repo_hash).join("meta.json"),
r#"{"version":1,"rpkiNotify":"https://other.example/notification.xml","createdAt":"2026-03-15T00:00:00Z","lastSeenAt":"2026-03-15T00:00:01Z"}"#,
).expect("rewrite rrdp meta");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::RrdpMetaMismatch { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_delta_fixture();
let module_hash = sha256_hex(module_uri.as_bytes());
std::fs::write(
archive_root.join("v1/captures/delta-cap/rsync/modules").join(&module_hash).join("meta.json"),
r#"{"version":1,"module":"rsync://other.example/repo/","createdAt":"2026-03-15T00:00:00Z","lastSeenAt":"2026-03-15T00:00:01Z"}"#,
).expect("rewrite rsync meta");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::RsyncMetaMismatch { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_rejects_transition_base_target_and_serial_mismatches() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"delta","base":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":9},"target":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":12},"delta_count":2,"deltas":[11,12]}"#,
).expect("rewrite transition base");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::TransitionBaseMismatch { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"delta","base":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"target":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":13},"delta_count":2,"deltas":[11,12]}"#,
).expect("rewrite transition target");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(
err,
ReplayDeltaArchiveError::TransitionTargetMismatch { .. }
),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"delta","base":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"target":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":12},"delta_count":1,"deltas":[12]}"#,
).expect("rewrite transition deltas");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::DeltaSerialListMismatch { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_rejects_missing_session_dir_and_overlay_files() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash)
.join("11111111-1111-1111-1111-111111111111");
std::fs::remove_dir_all(&session_dir).expect("remove session dir");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::MissingDeltaSessionDir { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_delta_fixture();
let module_hash = sha256_hex(module_uri.as_bytes());
let overlay_path = archive_root
.join("v1/captures/delta-cap/rsync/modules")
.join(&module_hash)
.join("tree/rsync.example.test/repo/sub/b.cer");
std::fs::remove_file(overlay_path).expect("remove overlay file");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::MissingRsyncOverlayFile { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_accepts_missing_rsync_module_meta_when_files_and_tree_exist() {
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_delta_fixture();
let module_hash = sha256_hex(module_uri.as_bytes());
let meta_path = archive_root
.join("v1/captures/delta-cap/rsync/modules")
.join(module_hash)
.join("meta.json");
std::fs::remove_file(&meta_path).expect("remove delta rsync module meta");
let index = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path)
.expect("load delta replay index without rsync meta");
let module = index
.rsync_modules
.get(&module_uri)
.expect("module present");
assert_eq!(module.meta.module, module_uri);
assert_eq!(module.meta.version, 1);
}
#[test]
fn delta_archive_index_accepts_correct_base_locks_sha_and_rejects_missing_module_resolution() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
let index =
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index");
let locks_bytes = std::fs::read(&locks_path).expect("read locks bytes");
assert!(
index
.validate_base_locks_sha256_bytes(&locks_bytes)
.is_err()
);
let err = index
.resolve_rsync_module_for_base_uri("rsync://missing.example/repo/path")
.unwrap_err();
assert!(
matches!(
err,
ReplayDeltaArchiveError::MissingRsyncModuleBucket { .. }
),
"{err}"
);
}
#[test]
fn delta_archive_index_loads_session_reset_and_gap_entries_without_target_files() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
for kind in ["session-reset", "gap"] {
std::fs::write(
repo_dir.join("transition.json"),
format!(
r#"{{"kind":"{kind}","base":{{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10}},"target":{{"transport":"rrdp","session":"22222222-2222-2222-2222-222222222222","serial":12}},"delta_count":0,"deltas":[]}}"#,
),
)
.expect("rewrite transition kind");
std::fs::write(
&locks_path,
format!(
r#"{{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"deadbeef","rrdp":{{"{notify_uri}":{{"kind":"{kind}","base":{{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10}},"target":{{"transport":"rrdp","session":"22222222-2222-2222-2222-222222222222","serial":12}},"delta_count":0,"deltas":[]}}}},"rsync":{{"rsync://rsync.example.test/repo/":{{"file_count":2,"overlay_only":true}}}}}}"#,
),
)
.expect("rewrite locks kind");
let index =
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index");
let repo = index.rrdp_repo(&notify_uri).expect("rrdp repo");
assert!(repo.target_notification_path.is_none());
assert!(repo.delta_paths.is_empty());
}
}
#[test]
fn delta_archive_index_loads_rrdp_and_rsync_entries() {
let (_temp, archive_root, locks_path, notify_uri, module_uri) = build_delta_fixture();
let index =
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index");
assert_eq!(index.capture_meta.capture_id, "delta-cap");
assert_eq!(index.base_meta.base_capture, "base-cap");
assert_eq!(index.rrdp_repos.len(), 1);
assert_eq!(index.rsync_modules.len(), 1);
let repo = index.rrdp_repo(&notify_uri).expect("rrdp repo");
assert_eq!(repo.transition.kind, ReplayDeltaRrdpKind::Delta);
assert_eq!(repo.transition.delta_count, 2);
assert_eq!(repo.delta_paths.len(), 2);
assert!(repo.target_notification_path.as_ref().unwrap().is_file());
assert!(repo.target_archive_path.as_ref().unwrap().is_file());
let module = index.rsync_module(&module_uri).expect("rsync module");
assert_eq!(module.files.file_count, 2);
assert_eq!(module.overlay_files.len(), 2);
assert!(module.overlay_files.iter().all(|(_, path)| path.is_file()));
}
#[test]
fn delta_archive_index_rejects_capture_and_sha_mismatches() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
let capture_root = archive_root.join("v1/captures/delta-cap");
std::fs::write(
capture_root.join("capture.json"),
r#"{"version":1,"captureId":"other-cap","createdAt":"2026-03-15T00:00:00Z","notes":""}"#,
)
.expect("rewrite capture json");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::CaptureIdMismatch { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
let capture_root = archive_root.join("v1/captures/delta-cap");
std::fs::write(
capture_root.join("base.json"),
r#"{"version":1,"baseCapture":"base-cap","baseLocksSha256":"beefdead","createdAt":"2026-03-15T00:00:00Z"}"#,
)
.expect("rewrite base json sha");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::BaseLocksShaMismatch { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_rejects_missing_target_notification_and_repo_bucket() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash)
.join("11111111-1111-1111-1111-111111111111");
std::fs::remove_file(session_dir.join("notification-target-12.xml"))
.expect("remove target notification");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(
err,
ReplayDeltaArchiveError::MissingTargetNotification { .. }
),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::remove_dir_all(repo_dir).expect("remove repo dir");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::MissingDeltaRepoBucket { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_rejects_base_meta_mismatch() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
let capture_root = archive_root.join("v1/captures/delta-cap");
std::fs::write(
capture_root.join("base.json"),
r#"{"version":1,"baseCapture":"other","baseLocksSha256":"deadbeef","createdAt":"2026-03-15T00:00:00Z"}"#,
)
.expect("rewrite base json");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::BaseCaptureMismatch { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_rejects_transition_mismatch_and_missing_delta_file() {
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"unchanged","base":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":10},"target":{"transport":"rrdp","session":"11111111-1111-1111-1111-111111111111","serial":12},"delta_count":2,"deltas":[11,12]}"#,
)
.expect("rewrite transition");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::TransitionKindMismatch { .. }),
"{err}"
);
let (_temp, archive_root, locks_path, notify_uri, _module_uri) = build_delta_fixture();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let delta_path = archive_root
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash)
.join("11111111-1111-1111-1111-111111111111/deltas/delta-12-bbbb.xml");
std::fs::remove_file(delta_path).expect("remove delta");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::MissingDeltaFile { .. }),
"{err}"
);
}
#[test]
fn delta_archive_index_validates_base_locks_sha256_bytes_and_file() {
let (_temp, archive_root, locks_path, _notify_uri, _module_uri) = build_delta_fixture();
let index =
ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).expect("load delta index");
let err = index
.validate_base_locks_sha256_bytes(b"not-the-right-base-locks")
.unwrap_err();
assert!(
matches!(
err,
ReplayDeltaArchiveError::BaseLocksBytesShaMismatch { .. }
),
"{err}"
);
let temp_file = tempfile::NamedTempFile::new().expect("tempfile");
std::fs::write(temp_file.path(), b"still-wrong").expect("write base locks file");
let err = index
.validate_base_locks_sha256_file(temp_file.path())
.unwrap_err();
assert!(
matches!(
err,
ReplayDeltaArchiveError::BaseLocksBytesShaMismatch { .. }
),
"{err}"
);
}
#[test]
fn delta_archive_index_rejects_rsync_files_mismatch() {
let (_temp, archive_root, locks_path, _notify_uri, module_uri) = build_delta_fixture();
let module_hash = sha256_hex(module_uri.as_bytes());
let module_dir = archive_root
.join("v1/captures/delta-cap/rsync/modules")
.join(&module_hash);
std::fs::write(
module_dir.join("files.json"),
format!(
r#"{{"version":1,"module":"{module_uri}","fileCount":3,"files":["{module_uri}a.roa"]}}"#
),
)
.expect("rewrite files json");
let err = ReplayDeltaArchiveIndex::load(&archive_root, &locks_path).unwrap_err();
assert!(
matches!(err, ReplayDeltaArchiveError::RsyncFileCountMismatch { .. }),
"{err}"
);
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,69 @@
// Batch encoders shared by the RocksDB write paths.
fn write_roa_cache_projection_to_batch(
projection_cf: &ColumnFamily,
batch: &mut WriteBatch,
vcir: &ValidatedCaInstanceResult,
context: Option<&RoaCacheProjectionContext>,
timing: Option<&mut VcirReplaceTimingBreakdown>,
) -> StorageResult<()> {
let projection_key = roa_cache_projection_key(&vcir.manifest_rsync_uri);
let projection = RoaCacheProjection::from_vcir_with_context(vcir, context)?;
match projection {
Some(projection) => {
let projection_value = encode_cbor(&projection, "roa_cache_projection")?;
if let Some(timing) = timing {
timing.roa_cache_projection_value_bytes = projection_value.len() as u64;
}
batch.put_cf(projection_cf, projection_key.as_bytes(), projection_value);
}
None => {
batch.delete_cf(projection_cf, projection_key.as_bytes());
}
}
Ok(())
}
fn write_vcir_failed_fetch_reuse_identity_to_batch(
failed_fetch_reuse_identity_cf: &ColumnFamily,
batch: &mut WriteBatch,
manifest_rsync_uri: &str,
failed_fetch_reuse_identity: Option<&VcirFailedFetchReuseIdentity>,
) -> StorageResult<()> {
let key = vcir_failed_fetch_reuse_identity_key(manifest_rsync_uri);
match failed_fetch_reuse_identity {
Some(identity) => {
identity.validate_internal()?;
let value = encode_cbor(identity, "vcir_failed_fetch_reuse_identity")?;
batch.put_cf(failed_fetch_reuse_identity_cf, key.as_bytes(), value);
}
None => batch.delete_cf(failed_fetch_reuse_identity_cf, key.as_bytes()),
}
Ok(())
}
fn write_publication_point_cache_projection_to_batch(
projection_cf: &ColumnFamily,
batch: &mut WriteBatch,
action: PublicationPointCacheProjectionWriteAction<'_>,
timing: Option<&mut VcirReplaceTimingBreakdown>,
) -> StorageResult<()> {
match action {
PublicationPointCacheProjectionWriteAction::Keep => Ok(()),
PublicationPointCacheProjectionWriteAction::Write(projection) => {
projection.validate_internal()?;
let key = publication_point_cache_projection_key(&projection.manifest_rsync_uri);
let value = encode_cbor(projection, "publication_point_cache_projection")?;
if let Some(timing) = timing {
timing.publication_point_cache_projection_value_bytes = value.len() as u64;
}
batch.put_cf(projection_cf, key.as_bytes(), value);
Ok(())
}
PublicationPointCacheProjectionWriteAction::Delete { manifest_rsync_uri } => {
let key = publication_point_cache_projection_key(manifest_rsync_uri);
batch.delete_cf(projection_cf, key.as_bytes());
Ok(())
}
}
}

View File

@@ -0,0 +1,199 @@
//! RocksDB and process-memory observation.
//!
//! This module owns the serialization-friendly memory snapshot model and the
//! mapping from RocksDB property names to that model. Storage mutation stays
//! in the parent module.
use rocksdb::DB;
use serde::Serialize;
pub(super) fn process_vm_rss_kb() -> Option<u64> {
let status = std::fs::read_to_string("/proc/self/status").ok()?;
status.lines().find_map(|line| {
let rest = line.strip_prefix("VmRSS:")?;
rest.split_whitespace().next()?.parse::<u64>().ok()
})
}
const ROCKSDB_MEMORY_PROPERTY_NAMES: &[(&str, &str)] = &[
("cur_size_all_mem_tables", "rocksdb.cur-size-all-mem-tables"),
("size_all_mem_tables", "rocksdb.size-all-mem-tables"),
(
"estimate_table_readers_mem",
"rocksdb.estimate-table-readers-mem",
),
("block_cache_capacity", "rocksdb.block-cache-capacity"),
("block_cache_usage", "rocksdb.block-cache-usage"),
(
"block_cache_pinned_usage",
"rocksdb.block-cache-pinned-usage",
),
("num_snapshots", "rocksdb.num-snapshots"),
("background_errors", "rocksdb.background-errors"),
];
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct RocksDbMemoryProperties {
pub cur_size_all_mem_tables: Option<u64>,
pub size_all_mem_tables: Option<u64>,
pub estimate_table_readers_mem: Option<u64>,
pub block_cache_capacity: Option<u64>,
pub block_cache_usage: Option<u64>,
pub block_cache_pinned_usage: Option<u64>,
pub num_snapshots: Option<u64>,
pub background_errors: Option<u64>,
pub errors: Vec<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct RocksDbColumnFamilyMemoryProperties {
pub name: String,
pub properties: RocksDbMemoryProperties,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct RocksDbMemoryDbSnapshot {
pub label: String,
pub properties: RocksDbMemoryProperties,
pub column_families: Vec<RocksDbColumnFamilyMemoryProperties>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct RocksDbMemoryTotals {
pub cur_size_all_mem_tables: u64,
pub size_all_mem_tables: u64,
pub estimate_table_readers_mem: u64,
pub block_cache_capacity: u64,
pub block_cache_usage: u64,
pub block_cache_pinned_usage: u64,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct RocksDbMemorySnapshot {
pub databases: Vec<RocksDbMemoryDbSnapshot>,
pub totals: RocksDbMemoryTotals,
}
impl RocksDbMemoryTotals {
pub(super) fn add_properties(&mut self, properties: &RocksDbMemoryProperties) {
self.cur_size_all_mem_tables += properties.cur_size_all_mem_tables.unwrap_or(0);
self.size_all_mem_tables += properties.size_all_mem_tables.unwrap_or(0);
self.estimate_table_readers_mem += properties.estimate_table_readers_mem.unwrap_or(0);
self.block_cache_capacity += properties.block_cache_capacity.unwrap_or(0);
self.block_cache_usage += properties.block_cache_usage.unwrap_or(0);
self.block_cache_pinned_usage += properties.block_cache_pinned_usage.unwrap_or(0);
}
}
fn set_memory_property(properties: &mut RocksDbMemoryProperties, name: &str, value: u64) {
match name {
"cur_size_all_mem_tables" => properties.cur_size_all_mem_tables = Some(value),
"size_all_mem_tables" => properties.size_all_mem_tables = Some(value),
"estimate_table_readers_mem" => properties.estimate_table_readers_mem = Some(value),
"block_cache_capacity" => properties.block_cache_capacity = Some(value),
"block_cache_usage" => properties.block_cache_usage = Some(value),
"block_cache_pinned_usage" => properties.block_cache_pinned_usage = Some(value),
"num_snapshots" => properties.num_snapshots = Some(value),
"background_errors" => properties.background_errors = Some(value),
_ => {}
}
}
fn parse_rocksdb_property_int(raw: Option<String>) -> Option<u64> {
raw.and_then(|value| value.trim().parse::<u64>().ok())
}
fn memory_properties_for_db(db: &DB) -> RocksDbMemoryProperties {
let mut properties = RocksDbMemoryProperties::default();
for (field_name, property_name) in ROCKSDB_MEMORY_PROPERTY_NAMES {
match db.property_value(*property_name) {
Ok(value) => {
if let Some(parsed) = parse_rocksdb_property_int(value) {
set_memory_property(&mut properties, field_name, parsed);
}
}
Err(err) => properties.errors.push(format!("{property_name}: {}", err)),
}
}
properties
}
fn memory_properties_for_cf(db: &DB, cf_name: &'static str) -> RocksDbColumnFamilyMemoryProperties {
let mut properties = RocksDbMemoryProperties::default();
let Some(cf) = db.cf_handle(cf_name) else {
properties
.errors
.push(format!("missing column family: {cf_name}"));
return RocksDbColumnFamilyMemoryProperties {
name: cf_name.to_string(),
properties,
};
};
for (field_name, property_name) in ROCKSDB_MEMORY_PROPERTY_NAMES {
match db.property_value_cf(cf, *property_name) {
Ok(value) => {
if let Some(parsed) = parse_rocksdb_property_int(value) {
set_memory_property(&mut properties, field_name, parsed);
}
}
Err(err) => properties.errors.push(format!("{property_name}: {}", err)),
}
}
RocksDbColumnFamilyMemoryProperties {
name: cf_name.to_string(),
properties,
}
}
pub(crate) fn memory_db_snapshot_for_column_families(
label: impl Into<String>,
db: &DB,
column_families: Option<&[&'static str]>,
) -> RocksDbMemoryDbSnapshot {
RocksDbMemoryDbSnapshot {
label: label.into(),
properties: memory_properties_for_db(db),
column_families: column_families
.map(|names| {
names
.iter()
.map(|name| memory_properties_for_cf(db, name))
.collect()
})
.unwrap_or_default(),
}
}
#[cfg(test)]
mod tests {
use super::{
RocksDbMemoryProperties, RocksDbMemoryTotals, parse_rocksdb_property_int,
set_memory_property,
};
#[test]
fn parses_rocksdb_integer_properties_without_panicking() {
assert_eq!(
parse_rocksdb_property_int(Some(" 42 ".to_string())),
Some(42)
);
assert_eq!(
parse_rocksdb_property_int(Some("not-a-number".to_string())),
None
);
assert_eq!(parse_rocksdb_property_int(None), None);
}
#[test]
fn aggregates_only_known_memory_properties() {
let mut properties = RocksDbMemoryProperties::default();
set_memory_property(&mut properties, "block_cache_usage", 11);
set_memory_property(&mut properties, "unknown", 99);
let mut totals = RocksDbMemoryTotals::default();
totals.add_properties(&properties);
assert_eq!(totals.block_cache_usage, 11);
assert_eq!(totals.block_cache_capacity, 0);
}
}

View File

@@ -0,0 +1,758 @@
// Core storage errors, repository records, and serde helpers.
#[derive(Debug, thiserror::Error)]
pub enum StorageError {
#[error("rocksdb error: {0}")]
RocksDb(String),
#[error("missing column family: {0}")]
MissingColumnFamily(&'static str),
#[error("cbor codec error for {entity}: {detail}")]
Codec {
entity: &'static str,
detail: String,
},
#[error("invalid {entity}: {detail}")]
InvalidData {
entity: &'static str,
detail: String,
},
}
pub type StorageResult<T> = Result<T, StorageError>;
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct RepositoryBlobVerificationSummary {
pub current_objects: u64,
pub bytes_verified: u64,
pub batches: u64,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct ChildCertificateCacheMmapLookup {
pub projections: Vec<Option<ChildCertificateCacheProjection>>,
pub hits: usize,
pub misses: usize,
pub file_bytes: u64,
}
pub struct RocksStore {
db: DB,
external_raw_store: Option<ExternalRawStoreDb>,
external_repo_bytes: Option<ExternalRepoBytesDb>,
publication_point_cache_index_dir: PathBuf,
child_certificate_cache_index_dir: PathBuf,
publication_point_cache_projection_index: Mutex<PublicationPointCacheProjectionIndexState>,
}
enum PublicationPointCacheProjectionIndexState {
Uninitialized,
Disabled,
BuildingFromEmpty {
index: HashMap<String, Arc<[u8]>>,
bytes: usize,
limit: usize,
},
Loaded {
index: HashMap<String, Arc<[u8]>>,
bytes: usize,
},
LoadedMmap {
mmap: PpCacheMmapIndexSet,
dirty: HashMap<String, Arc<[u8]>>,
dirty_bytes: usize,
load_stats: PpCacheIndexLoadStats,
},
}
#[derive(Clone, Copy)]
pub(crate) enum PublicationPointCacheProjectionWriteAction<'a> {
Keep,
Write(&'a PublicationPointCacheProjection),
Delete { manifest_rsync_uri: &'a str },
}
fn default_child_certificate_cache_index_dir(db_path: &Path) -> PathBuf {
let file_name = db_path
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("work-db");
db_path.with_file_name(format!("{file_name}.child-cert-cache-index"))
}
fn child_certificate_cache_segment_file_name(manifest_rsync_uri: &str) -> String {
format!(
"{}.idx",
hex::encode(compute_sha256_32(manifest_rsync_uri.as_bytes()))
)
}
const PP_CACHE_RAW_INDEX_ENV: &str = "RPKI_PP_CACHE_RAW_INDEX";
const PP_CACHE_RAW_INDEX_EMPTY_BUILD_LIMIT_ENV: &str =
"RPKI_PP_CACHE_RAW_INDEX_EMPTY_BUILD_LIMIT_BYTES";
const DEFAULT_PP_CACHE_RAW_INDEX_EMPTY_BUILD_LIMIT_BYTES: usize = 32 * 1024 * 1024;
const PP_CACHE_INDEX_COMPACTION_SEGMENT_THRESHOLD: usize = 16;
const PP_CACHE_INDEX_COMPACTION_BYTES_THRESHOLD: u64 = 1_610_612_736;
fn pp_cache_raw_index_enabled() -> bool {
match std::env::var(PP_CACHE_RAW_INDEX_ENV) {
Ok(value) => !matches!(
value.trim().to_ascii_lowercase().as_str(),
"0" | "false" | "off" | "no"
),
Err(_) => true,
}
}
fn pp_cache_raw_index_empty_build_limit_bytes() -> usize {
std::env::var(PP_CACHE_RAW_INDEX_EMPTY_BUILD_LIMIT_ENV)
.ok()
.and_then(|value| value.trim().parse::<usize>().ok())
.unwrap_or(DEFAULT_PP_CACHE_RAW_INDEX_EMPTY_BUILD_LIMIT_BYTES)
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum RrdpDeltaOp {
Upsert { rsync_uri: String, bytes: Vec<u8> },
Delete { rsync_uri: String },
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RepositoryViewState {
Present,
Withdrawn,
Replaced,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RepositoryViewEntry {
pub rsync_uri: String,
pub current_hash: Option<String>,
pub repository_source: Option<String>,
pub object_type: Option<String>,
pub state: RepositoryViewState,
}
impl RepositoryViewEntry {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty("repository_view.rsync_uri", &self.rsync_uri)?;
if let Some(source) = &self.repository_source {
validate_non_empty("repository_view.repository_source", source)?;
}
match self.state {
RepositoryViewState::Present | RepositoryViewState::Replaced => {
let hash = self
.current_hash
.as_deref()
.ok_or(StorageError::InvalidData {
entity: "repository_view",
detail: "current_hash is required when state is present or replaced"
.to_string(),
})?;
validate_sha256_hex("repository_view.current_hash", hash)?;
}
RepositoryViewState::Withdrawn => {
if let Some(hash) = &self.current_hash {
validate_sha256_hex("repository_view.current_hash", hash)?;
}
}
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RawByHashEntry {
pub sha256_hex: String,
pub bytes: Vec<u8>,
pub origin_uris: Vec<String>,
pub object_type: Option<String>,
pub encoding: Option<String>,
}
impl RawByHashEntry {
pub fn from_bytes(sha256_hex: impl Into<String>, bytes: Vec<u8>) -> Self {
Self {
sha256_hex: sha256_hex.into(),
bytes,
origin_uris: Vec::new(),
object_type: None,
encoding: None,
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
validate_sha256_hex("raw_by_hash.sha256_hex", &self.sha256_hex)?;
if self.bytes.is_empty() {
return Err(StorageError::InvalidData {
entity: "raw_by_hash",
detail: "bytes must not be empty".to_string(),
});
}
let computed = hex::encode(compute_sha256_32(&self.bytes));
if computed != self.sha256_hex.to_ascii_lowercase() {
return Err(StorageError::InvalidData {
entity: "raw_by_hash",
detail: "sha256_hex does not match bytes".to_string(),
});
}
let mut seen = HashSet::with_capacity(self.origin_uris.len());
for uri in &self.origin_uris {
validate_non_empty("raw_by_hash.origin_uris[]", uri)?;
if !seen.insert(uri.as_str()) {
return Err(StorageError::InvalidData {
entity: "raw_by_hash",
detail: format!("duplicate origin URI: {uri}"),
});
}
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CurrentObjectWithHash {
pub current_hash_hex: String,
pub current_hash: [u8; 32],
pub bytes: Vec<u8>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct ValidatedManifestMeta {
pub validated_manifest_number: Vec<u8>,
pub validated_manifest_this_update: PackTime,
pub validated_manifest_next_update: PackTime,
}
impl ValidatedManifestMeta {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_manifest_number_be(
"validated_manifest_meta.validated_manifest_number",
&self.validated_manifest_number,
)?;
let this_update = parse_time(
"validated_manifest_meta.validated_manifest_this_update",
&self.validated_manifest_this_update,
)?;
let next_update = parse_time(
"validated_manifest_meta.validated_manifest_next_update",
&self.validated_manifest_next_update,
)?;
if next_update < this_update {
return Err(StorageError::InvalidData {
entity: "validated_manifest_meta",
detail: "validated_manifest_next_update must be >= validated_manifest_this_update"
.to_string(),
});
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct ManifestReplayMeta {
pub manifest_rsync_uri: String,
pub manifest_number_be: Vec<u8>,
pub manifest_this_update: PackTime,
pub manifest_sha256: Vec<u8>,
pub updated_at_validation_time: PackTime,
}
impl ManifestReplayMeta {
pub fn from_vcir(vcir: &ValidatedCaInstanceResult) -> Self {
Self {
manifest_rsync_uri: vcir.manifest_rsync_uri.clone(),
manifest_number_be: vcir
.validated_manifest_meta
.validated_manifest_number
.clone(),
manifest_this_update: vcir
.validated_manifest_meta
.validated_manifest_this_update
.clone(),
manifest_sha256: vcir.ccr_manifest_projection.manifest_sha256.clone(),
updated_at_validation_time: vcir.last_successful_validation_time.clone(),
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty(
"manifest_replay_meta.manifest_rsync_uri",
&self.manifest_rsync_uri,
)?;
validate_manifest_number_be(
"manifest_replay_meta.manifest_number_be",
&self.manifest_number_be,
)?;
parse_time(
"manifest_replay_meta.manifest_this_update",
&self.manifest_this_update,
)?;
validate_sha256_digest_bytes(
"manifest_replay_meta.manifest_sha256",
&self.manifest_sha256,
)?;
parse_time(
"manifest_replay_meta.updated_at_validation_time",
&self.updated_at_validation_time,
)?;
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirCcrManifestProjection {
pub manifest_rsync_uri: String,
pub manifest_sha256: Vec<u8>,
pub manifest_size: u64,
pub manifest_ee_aki: Vec<u8>,
pub manifest_number_be: Vec<u8>,
pub manifest_this_update: PackTime,
pub manifest_sia_locations_der: Vec<Vec<u8>>,
pub subordinate_skis: Vec<Vec<u8>>,
}
impl VcirCcrManifestProjection {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty(
"vcir.ccr_manifest_projection.manifest_rsync_uri",
&self.manifest_rsync_uri,
)?;
validate_sha256_digest_bytes(
"vcir.ccr_manifest_projection.manifest_sha256",
&self.manifest_sha256,
)?;
if self.manifest_size < 1000 {
return Err(StorageError::InvalidData {
entity: "vcir.ccr_manifest_projection.manifest_size",
detail: format!("must be >= 1000, got {}", self.manifest_size),
});
}
validate_fixed_len_bytes(
"vcir.ccr_manifest_projection.manifest_ee_aki",
&self.manifest_ee_aki,
20,
)?;
validate_manifest_number_be(
"vcir.ccr_manifest_projection.manifest_number_be",
&self.manifest_number_be,
)?;
parse_time(
"vcir.ccr_manifest_projection.manifest_this_update",
&self.manifest_this_update,
)?;
if self.manifest_sia_locations_der.is_empty() {
return Err(StorageError::InvalidData {
entity: "vcir.ccr_manifest_projection.manifest_sia_locations_der",
detail: "must contain at least one AccessDescription".to_string(),
});
}
for location in &self.manifest_sia_locations_der {
validate_full_der_with_tag(
"vcir.ccr_manifest_projection.manifest_sia_locations_der[]",
location,
Some(0x30),
)?;
}
validate_sorted_unique_fixed_len_bytes(
"vcir.ccr_manifest_projection.subordinate_skis",
&self.subordinate_skis,
20,
)?;
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirInstanceGate {
pub manifest_next_update: PackTime,
pub current_crl_next_update: PackTime,
pub self_ca_not_after: PackTime,
pub instance_effective_until: PackTime,
}
impl VcirInstanceGate {
pub fn validate_internal(&self) -> StorageResult<()> {
let manifest_next_update = parse_time(
"vcir.instance_gate.manifest_next_update",
&self.manifest_next_update,
)?;
let current_crl_next_update = parse_time(
"vcir.instance_gate.current_crl_next_update",
&self.current_crl_next_update,
)?;
let self_ca_not_after = parse_time(
"vcir.instance_gate.self_ca_not_after",
&self.self_ca_not_after,
)?;
let instance_effective_until = parse_time(
"vcir.instance_gate.instance_effective_until",
&self.instance_effective_until,
)?;
let expected = manifest_next_update
.min(current_crl_next_update)
.min(self_ca_not_after);
if instance_effective_until != expected {
return Err(StorageError::InvalidData {
entity: "vcir.instance_gate",
detail: "instance_effective_until must equal min(manifest_next_update, current_crl_next_update, self_ca_not_after)".to_string(),
});
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirFailedFetchReuseIdentity {
#[serde(rename = "c")]
#[serde(with = "serde_bytes_32")]
pub current_ca_sha256: [u8; 32],
#[serde(rename = "t")]
#[serde(with = "serde_bytes_32")]
pub ta_context_digest: [u8; 32],
#[serde(rename = "p")]
#[serde(with = "serde_bytes_32")]
pub ca_validation_context_digest: [u8; 32],
#[serde(rename = "f")]
#[serde(with = "serde_bytes_32")]
pub policy_fingerprint: [u8; 32],
#[serde(rename = "nb")]
pub effective_not_before: PackTime,
#[serde(rename = "nu")]
pub effective_until: PackTime,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirReuseRecordsCleared {
pub vcir_records: u64,
pub failed_fetch_identity_records: u64,
}
impl VcirFailedFetchReuseIdentity {
pub fn validate_internal(&self) -> StorageResult<()> {
let effective_not_before = parse_time(
"vcir_failed_fetch_reuse_identity.effective_not_before",
&self.effective_not_before,
)?;
let effective_until = parse_time(
"vcir_failed_fetch_reuse_identity.effective_until",
&self.effective_until,
)?;
if effective_not_before >= effective_until {
return Err(StorageError::InvalidData {
entity: "vcir_failed_fetch_reuse_identity.effective_window",
detail: "effective_not_before must be before effective_until".to_string(),
});
}
Ok(())
}
pub fn contains_validation_time(&self, validation_time: time::OffsetDateTime) -> bool {
let Ok(effective_not_before) = self.effective_not_before.parse() else {
return false;
};
let Ok(effective_until) = self.effective_until.parse() else {
return false;
};
validation_time >= effective_not_before && validation_time < effective_until
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirChildEntry {
pub child_manifest_rsync_uri: String,
pub child_cert_rsync_uri: String,
pub child_cert_hash: String,
pub child_ski: String,
pub child_rsync_base_uri: String,
pub child_publication_point_rsync_uri: String,
pub child_rrdp_notification_uri: Option<String>,
pub child_effective_ip_resources: Option<IpResourceSet>,
pub child_effective_as_resources: Option<AsResourceSet>,
pub accepted_at_validation_time: PackTime,
}
impl VcirChildEntry {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty(
"vcir.child_entries[].child_manifest_rsync_uri",
&self.child_manifest_rsync_uri,
)?;
validate_non_empty(
"vcir.child_entries[].child_cert_rsync_uri",
&self.child_cert_rsync_uri,
)?;
validate_sha256_hex(
"vcir.child_entries[].child_cert_hash",
&self.child_cert_hash,
)?;
validate_non_empty("vcir.child_entries[].child_ski", &self.child_ski)?;
validate_non_empty(
"vcir.child_entries[].child_rsync_base_uri",
&self.child_rsync_base_uri,
)?;
validate_non_empty(
"vcir.child_entries[].child_publication_point_rsync_uri",
&self.child_publication_point_rsync_uri,
)?;
if let Some(uri) = &self.child_rrdp_notification_uri {
validate_non_empty("vcir.child_entries[].child_rrdp_notification_uri", uri)?;
}
parse_time(
"vcir.child_entries[].accepted_at_validation_time",
&self.accepted_at_validation_time,
)?;
Ok(())
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VcirOutputType {
Vrp,
Aspa,
RouterKey,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VcirSourceObjectType {
Roa,
Aspa,
RouterKey,
Other,
}
impl VcirSourceObjectType {
pub fn as_str(self) -> &'static str {
match self {
Self::Roa => "roa",
Self::Aspa => "aspa",
Self::RouterKey => "router_key",
Self::Other => "other",
}
}
}
struct FixedBytesVisitor<const N: usize>;
impl<'de, const N: usize> serde::de::Visitor<'de> for FixedBytesVisitor<N> {
type Value = [u8; N];
fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(formatter, "{N} bytes")
}
fn visit_bytes<E>(self, value: &[u8]) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
if value.len() != N {
return Err(E::invalid_length(value.len(), &self));
}
let mut out = [0u8; N];
out.copy_from_slice(value);
Ok(out)
}
fn visit_byte_buf<E>(self, value: Vec<u8>) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
self.visit_bytes(&value)
}
fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
where
A: serde::de::SeqAccess<'de>,
{
let mut out = [0u8; N];
for (idx, slot) in out.iter_mut().enumerate() {
*slot = seq
.next_element()?
.ok_or_else(|| serde::de::Error::invalid_length(idx, &self))?;
}
Ok(out)
}
}
fn deserialize_fixed_bytes<'de, D, const N: usize>(deserializer: D) -> Result<[u8; N], D::Error>
where
D: serde::Deserializer<'de>,
{
deserializer.deserialize_bytes(FixedBytesVisitor::<N>)
}
mod serde_bytes_16 {
pub(super) fn serialize<S>(value: &[u8; 16], serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serializer.serialize_bytes(value)
}
pub(super) fn deserialize<'de, D>(deserializer: D) -> Result<[u8; 16], D::Error>
where
D: serde::Deserializer<'de>,
{
super::deserialize_fixed_bytes::<D, 16>(deserializer)
}
}
mod serde_bytes_32 {
pub(super) fn serialize<S>(value: &[u8; 32], serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serializer.serialize_bytes(value)
}
pub(super) fn deserialize<'de, D>(deserializer: D) -> Result<[u8; 32], D::Error>
where
D: serde::Deserializer<'de>,
{
super::deserialize_fixed_bytes::<D, 32>(deserializer)
}
}
struct ByteVecVisitor;
impl<'de> serde::de::Visitor<'de> for ByteVecVisitor {
type Value = Vec<u8>;
fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str("byte vector")
}
fn visit_bytes<E>(self, value: &[u8]) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
Ok(value.to_vec())
}
fn visit_byte_buf<E>(self, value: Vec<u8>) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
Ok(value)
}
fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
where
A: serde::de::SeqAccess<'de>,
{
let mut out = Vec::with_capacity(seq.size_hint().unwrap_or(0));
while let Some(byte) = seq.next_element()? {
out.push(byte);
}
Ok(out)
}
}
mod serde_byte_vec {
pub(super) fn serialize<S>(value: &[u8], serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serializer.serialize_bytes(value)
}
pub(super) fn deserialize<'de, D>(deserializer: D) -> Result<Vec<u8>, D::Error>
where
D: serde::Deserializer<'de>,
{
deserializer.deserialize_bytes(super::ByteVecVisitor)
}
}
mod serde_optional_byte_vec {
pub(super) fn serialize<S>(value: &Option<Vec<u8>>, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match value {
Some(bytes) => serializer.serialize_some(bytes),
None => serializer.serialize_none(),
}
}
pub(super) fn deserialize<'de, D>(deserializer: D) -> Result<Option<Vec<u8>>, D::Error>
where
D: serde::Deserializer<'de>,
{
struct OptionalByteVecVisitor;
impl<'de> serde::de::Visitor<'de> for OptionalByteVecVisitor {
type Value = Option<Vec<u8>>;
fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str("optional byte vector")
}
fn visit_none<E>(self) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
Ok(None)
}
fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
where
D: serde::Deserializer<'de>,
{
deserializer
.deserialize_bytes(super::ByteVecVisitor)
.map(Some)
}
}
deserializer.deserialize_option(OptionalByteVecVisitor)
}
}
mod serde_optional_bytes_32 {
pub(super) fn serialize<S>(value: &Option<[u8; 32]>, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match value {
Some(bytes) => serializer.serialize_some(bytes.as_slice()),
None => serializer.serialize_none(),
}
}
pub(super) fn deserialize<'de, D>(deserializer: D) -> Result<Option<[u8; 32]>, D::Error>
where
D: serde::Deserializer<'de>,
{
struct OptionalBytes32Visitor;
impl<'de> serde::de::Visitor<'de> for OptionalBytes32Visitor {
type Value = Option<[u8; 32]>;
fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str("optional 32-byte array")
}
fn visit_none<E>(self) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
Ok(None)
}
fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
where
D: serde::Deserializer<'de>,
{
super::deserialize_fixed_bytes::<D, 32>(deserializer).map(Some)
}
}
deserializer.deserialize_option(OptionalBytes32Visitor)
}
}

View File

@@ -0,0 +1,628 @@
// Publication-point and ROA cache projections.
pub const PUBLICATION_POINT_CACHE_SCHEMA_VERSION: u32 = 1;
pub const PUBLICATION_POINT_CACHE_ALGORITHM_VERSION: u32 = 1;
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct PublicationPointCacheOutput {
#[serde(rename = "t")]
pub output_type: VcirOutputType,
#[serde(rename = "nb")]
pub item_effective_not_before: PackTime,
#[serde(rename = "nu")]
pub item_effective_until: PackTime,
#[serde(rename = "u")]
pub source_object_uri: String,
#[serde(rename = "k")]
pub source_object_type: VcirSourceObjectType,
#[serde(rename = "h")]
#[serde(with = "serde_bytes_32")]
pub source_object_hash: [u8; 32],
#[serde(rename = "c")]
#[serde(with = "serde_bytes_32")]
pub source_ee_cert_hash: [u8; 32],
#[serde(rename = "p")]
pub payload: VcirLocalOutputPayload,
#[serde(rename = "r")]
#[serde(with = "serde_bytes_32")]
pub rule_hash: [u8; 32],
}
impl PublicationPointCacheOutput {
fn from_local_output(output: &VcirLocalOutput, default_not_before: &PackTime) -> Self {
Self {
output_type: output.output_type,
item_effective_not_before: default_not_before.clone(),
item_effective_until: output.item_effective_until.clone(),
source_object_uri: output.source_object_uri.clone(),
source_object_type: output.source_object_type,
source_object_hash: output.source_object_hash,
source_ee_cert_hash: output.source_ee_cert_hash,
payload: output.payload.clone(),
rule_hash: output.rule_hash,
}
}
pub fn to_local_output(&self) -> VcirLocalOutput {
VcirLocalOutput {
output_type: self.output_type,
item_effective_until: self.item_effective_until.clone(),
source_object_uri: self.source_object_uri.clone(),
source_object_type: self.source_object_type,
source_object_hash: self.source_object_hash,
source_ee_cert_hash: self.source_ee_cert_hash,
payload: self.payload.clone(),
rule_hash: self.rule_hash,
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
parse_time(
"publication_point_cache_projection.outputs[].item_effective_not_before",
&self.item_effective_not_before,
)?;
parse_time(
"publication_point_cache_projection.outputs[].item_effective_until",
&self.item_effective_until,
)?;
validate_non_empty(
"publication_point_cache_projection.outputs[].source_object_uri",
&self.source_object_uri,
)?;
VcirLocalOutput {
output_type: self.output_type,
item_effective_until: self.item_effective_until.clone(),
source_object_uri: self.source_object_uri.clone(),
source_object_type: self.source_object_type,
source_object_hash: self.source_object_hash,
source_ee_cert_hash: self.source_ee_cert_hash,
payload: self.payload.clone(),
rule_hash: self.rule_hash,
}
.validate_internal()
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct PublicationPointCacheChild {
#[serde(rename = "cm")]
pub child_manifest_rsync_uri: String,
#[serde(rename = "cu")]
pub child_cert_rsync_uri: String,
#[serde(rename = "ch")]
pub child_cert_hash: String,
#[serde(rename = "ski")]
pub child_ski: String,
#[serde(rename = "rb")]
pub child_rsync_base_uri: String,
#[serde(rename = "pp")]
pub child_publication_point_rsync_uri: String,
#[serde(rename = "rn")]
pub child_rrdp_notification_uri: Option<String>,
#[serde(rename = "ip")]
pub child_effective_ip_resources: Option<IpResourceSet>,
#[serde(rename = "as")]
pub child_effective_as_resources: Option<AsResourceSet>,
#[serde(rename = "nb")]
pub child_effective_not_before: PackTime,
#[serde(rename = "nu")]
pub child_effective_until: PackTime,
}
impl PublicationPointCacheChild {
fn from_child_entry(
entry: &VcirChildEntry,
default_not_before: &PackTime,
default_until: &PackTime,
) -> Self {
Self {
child_manifest_rsync_uri: entry.child_manifest_rsync_uri.clone(),
child_cert_rsync_uri: entry.child_cert_rsync_uri.clone(),
child_cert_hash: entry.child_cert_hash.clone(),
child_ski: entry.child_ski.clone(),
child_rsync_base_uri: entry.child_rsync_base_uri.clone(),
child_publication_point_rsync_uri: entry.child_publication_point_rsync_uri.clone(),
child_rrdp_notification_uri: entry.child_rrdp_notification_uri.clone(),
child_effective_ip_resources: entry.child_effective_ip_resources.clone(),
child_effective_as_resources: entry.child_effective_as_resources.clone(),
child_effective_not_before: default_not_before.clone(),
child_effective_until: default_until.clone(),
}
}
pub fn to_child_entry(&self, accepted_at_validation_time: PackTime) -> VcirChildEntry {
VcirChildEntry {
child_manifest_rsync_uri: self.child_manifest_rsync_uri.clone(),
child_cert_rsync_uri: self.child_cert_rsync_uri.clone(),
child_cert_hash: self.child_cert_hash.clone(),
child_ski: self.child_ski.clone(),
child_rsync_base_uri: self.child_rsync_base_uri.clone(),
child_publication_point_rsync_uri: self.child_publication_point_rsync_uri.clone(),
child_rrdp_notification_uri: self.child_rrdp_notification_uri.clone(),
child_effective_ip_resources: self.child_effective_ip_resources.clone(),
child_effective_as_resources: self.child_effective_as_resources.clone(),
accepted_at_validation_time,
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
self.to_child_entry(self.child_effective_not_before.clone())
.validate_internal()?;
parse_time(
"publication_point_cache_projection.children[].child_effective_not_before",
&self.child_effective_not_before,
)?;
parse_time(
"publication_point_cache_projection.children[].child_effective_until",
&self.child_effective_until,
)?;
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct PublicationPointCacheObject {
#[serde(rename = "r")]
pub artifact_role: VcirArtifactRole,
#[serde(rename = "k")]
pub artifact_kind: VcirArtifactKind,
#[serde(rename = "u")]
pub uri: Option<String>,
#[serde(rename = "h")]
pub sha256: String,
#[serde(rename = "t")]
pub object_type: Option<String>,
#[serde(rename = "s")]
pub validation_status: VcirArtifactValidationStatus,
/// See `VcirRelatedArtifact::reject_reason`.
#[serde(rename = "e", default, skip_serializing_if = "Option::is_none")]
pub reject_reason: Option<String>,
}
impl PublicationPointCacheObject {
fn from_related_artifact(artifact: &VcirRelatedArtifact) -> Self {
Self {
artifact_role: artifact.artifact_role,
artifact_kind: artifact.artifact_kind,
uri: artifact.uri.clone(),
sha256: artifact.sha256.clone(),
object_type: artifact.object_type.clone(),
validation_status: artifact.validation_status,
reject_reason: artifact.reject_reason.clone(),
}
}
pub fn to_related_artifact(&self) -> VcirRelatedArtifact {
VcirRelatedArtifact {
artifact_role: self.artifact_role,
artifact_kind: self.artifact_kind,
uri: self.uri.clone(),
sha256: self.sha256.clone(),
object_type: self.object_type.clone(),
validation_status: self.validation_status,
reject_reason: self.reject_reason.clone(),
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
self.to_related_artifact().validate_internal()
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct PublicationPointCacheProjection {
#[serde(rename = "sv")]
pub schema_version: u32,
#[serde(rename = "av")]
pub algorithm_version: u32,
#[serde(rename = "m")]
pub manifest_rsync_uri: String,
#[serde(rename = "pp")]
pub publication_point_rsync_uri: String,
#[serde(rename = "cu")]
pub ca_cert_uri: Option<String>,
#[serde(rename = "ch")]
#[serde(with = "serde_bytes_32")]
pub ca_cert_sha256: [u8; 32],
#[serde(rename = "mh")]
#[serde(with = "serde_bytes_32")]
pub manifest_sha256: [u8; 32],
#[serde(rename = "tal")]
pub tal_id: String,
#[serde(rename = "ta")]
#[serde(with = "serde_bytes_32")]
pub ta_context_digest: [u8; 32],
#[serde(rename = "pc")]
#[serde(with = "serde_bytes_32")]
pub ca_validation_context_digest: [u8; 32],
#[serde(rename = "pf")]
#[serde(with = "serde_bytes_32")]
pub validation_policy_fingerprint: [u8; 32],
#[serde(rename = "nb")]
pub instance_effective_not_before: PackTime,
#[serde(rename = "nu")]
pub instance_effective_until: PackTime,
#[serde(rename = "mt")]
pub manifest_this_update: PackTime,
#[serde(rename = "mn")]
pub manifest_next_update: PackTime,
#[serde(rename = "cn")]
pub current_crl_next_update: PackTime,
#[serde(rename = "ca")]
pub self_ca_not_after: PackTime,
#[serde(rename = "ccr")]
pub ccr_manifest_projection: VcirCcrManifestProjection,
#[serde(rename = "o")]
pub outputs: Vec<PublicationPointCacheOutput>,
#[serde(rename = "c")]
pub children: Vec<PublicationPointCacheChild>,
#[serde(rename = "ra")]
pub related_objects: Vec<PublicationPointCacheObject>,
#[serde(rename = "s")]
pub summary: VcirSummary,
}
impl PublicationPointCacheProjection {
pub fn from_vcir_with_context(
vcir: &ValidatedCaInstanceResult,
publication_point_rsync_uri: String,
ca_cert_uri: Option<String>,
ca_cert_sha256: [u8; 32],
manifest_sha256: [u8; 32],
ta_context_digest: [u8; 32],
ca_validation_context_digest: [u8; 32],
validation_policy_fingerprint: [u8; 32],
) -> StorageResult<Self> {
let projection = Self {
schema_version: PUBLICATION_POINT_CACHE_SCHEMA_VERSION,
algorithm_version: PUBLICATION_POINT_CACHE_ALGORITHM_VERSION,
manifest_rsync_uri: vcir.manifest_rsync_uri.clone(),
publication_point_rsync_uri,
ca_cert_uri,
ca_cert_sha256,
manifest_sha256,
tal_id: vcir.tal_id.clone(),
ta_context_digest,
ca_validation_context_digest,
validation_policy_fingerprint,
instance_effective_not_before: vcir.last_successful_validation_time.clone(),
instance_effective_until: vcir.instance_gate.instance_effective_until.clone(),
manifest_this_update: vcir
.validated_manifest_meta
.validated_manifest_this_update
.clone(),
manifest_next_update: vcir.instance_gate.manifest_next_update.clone(),
current_crl_next_update: vcir.instance_gate.current_crl_next_update.clone(),
self_ca_not_after: vcir.instance_gate.self_ca_not_after.clone(),
ccr_manifest_projection: vcir.ccr_manifest_projection.clone(),
outputs: vcir
.local_outputs
.iter()
.map(|output| {
PublicationPointCacheOutput::from_local_output(
output,
&vcir.last_successful_validation_time,
)
})
.collect(),
children: vcir
.child_entries
.iter()
.map(|child| {
PublicationPointCacheChild::from_child_entry(
child,
&vcir.last_successful_validation_time,
&vcir.instance_gate.instance_effective_until,
)
})
.collect(),
related_objects: vcir
.related_artifacts
.iter()
.map(PublicationPointCacheObject::from_related_artifact)
.collect(),
summary: vcir.summary.clone(),
};
projection.validate_internal()?;
Ok(projection)
}
pub fn to_vcir_for_reuse(
&self,
validation_time: time::OffsetDateTime,
) -> ValidatedCaInstanceResult {
let validation_time = PackTime::from_utc_offset_datetime(validation_time);
ValidatedCaInstanceResult {
manifest_rsync_uri: self.manifest_rsync_uri.clone(),
parent_manifest_rsync_uri: None,
tal_id: self.tal_id.clone(),
ca_subject_name: String::from("publication-point-cache"),
ca_ski: hex::encode(self.ca_cert_sha256),
issuer_ski: hex::encode(self.ca_validation_context_digest),
last_successful_validation_time: validation_time.clone(),
current_manifest_rsync_uri: self.manifest_rsync_uri.clone(),
current_crl_rsync_uri: self
.related_objects
.iter()
.find(|object| object.artifact_role == VcirArtifactRole::CurrentCrl)
.and_then(|object| object.uri.clone())
.unwrap_or_default(),
validated_manifest_meta: ValidatedManifestMeta {
validated_manifest_number: self.ccr_manifest_projection.manifest_number_be.clone(),
validated_manifest_this_update: self.manifest_this_update.clone(),
validated_manifest_next_update: self.manifest_next_update.clone(),
},
ccr_manifest_projection: self.ccr_manifest_projection.clone(),
instance_gate: VcirInstanceGate {
manifest_next_update: self.manifest_next_update.clone(),
current_crl_next_update: self.current_crl_next_update.clone(),
self_ca_not_after: self.self_ca_not_after.clone(),
instance_effective_until: self.instance_effective_until.clone(),
},
child_entries: self
.children
.iter()
.map(|child| child.to_child_entry(validation_time.clone()))
.collect(),
local_outputs: self
.outputs
.iter()
.map(PublicationPointCacheOutput::to_local_output)
.collect(),
related_artifacts: self
.related_objects
.iter()
.map(PublicationPointCacheObject::to_related_artifact)
.collect(),
summary: self.summary.clone(),
audit_summary: VcirAuditSummary {
failed_fetch_eligible: false,
last_failed_fetch_reason: None,
warning_count: 0,
audit_flags: vec!["publication_point_cache_projection".to_string()],
},
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
if self.schema_version != PUBLICATION_POINT_CACHE_SCHEMA_VERSION {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_projection.schema_version",
detail: format!("unsupported schema_version {}", self.schema_version),
});
}
if self.algorithm_version != PUBLICATION_POINT_CACHE_ALGORITHM_VERSION {
return Err(StorageError::InvalidData {
entity: "publication_point_cache_projection.algorithm_version",
detail: format!("unsupported algorithm_version {}", self.algorithm_version),
});
}
validate_non_empty(
"publication_point_cache_projection.manifest_rsync_uri",
&self.manifest_rsync_uri,
)?;
validate_non_empty(
"publication_point_cache_projection.publication_point_rsync_uri",
&self.publication_point_rsync_uri,
)?;
if let Some(uri) = &self.ca_cert_uri {
validate_non_empty("publication_point_cache_projection.ca_cert_uri", uri)?;
}
validate_non_empty("publication_point_cache_projection.tal_id", &self.tal_id)?;
parse_time(
"publication_point_cache_projection.instance_effective_not_before",
&self.instance_effective_not_before,
)?;
parse_time(
"publication_point_cache_projection.instance_effective_until",
&self.instance_effective_until,
)?;
parse_time(
"publication_point_cache_projection.manifest_this_update",
&self.manifest_this_update,
)?;
parse_time(
"publication_point_cache_projection.manifest_next_update",
&self.manifest_next_update,
)?;
parse_time(
"publication_point_cache_projection.current_crl_next_update",
&self.current_crl_next_update,
)?;
parse_time(
"publication_point_cache_projection.self_ca_not_after",
&self.self_ca_not_after,
)?;
self.ccr_manifest_projection.validate_internal()?;
for output in &self.outputs {
output.validate_internal()?;
}
for child in &self.children {
child.validate_internal()?;
}
for object in &self.related_objects {
object.validate_internal()?;
}
Ok(())
}
}
impl RoaCacheProjection {
pub fn from_vcir(vcir: &ValidatedCaInstanceResult) -> StorageResult<Option<Self>> {
Self::from_vcir_with_context(vcir, None)
}
pub fn from_vcir_with_context(
vcir: &ValidatedCaInstanceResult,
context: Option<&RoaCacheProjectionContext>,
) -> StorageResult<Option<Self>> {
let Some(context) = context else {
return Ok(None);
};
let mut issuer_ca_sha256_hex = None;
let mut crl_sha256_by_uri = Vec::new();
for artifact in &vcir.related_artifacts {
if artifact.validation_status != VcirArtifactValidationStatus::Accepted {
continue;
}
match (artifact.artifact_role, artifact.artifact_kind) {
(
VcirArtifactRole::IssuerCert | VcirArtifactRole::TrustAnchorCert,
VcirArtifactKind::Cer,
) => {
issuer_ca_sha256_hex = Some(artifact.sha256.clone());
}
(_, VcirArtifactKind::Crl) => {
if let Some(uri) = artifact.uri.as_ref() {
crl_sha256_by_uri.push(RoaCacheCrlProjection {
uri: uri.clone(),
sha256: artifact.sha256.clone(),
});
}
}
_ => {}
}
}
crl_sha256_by_uri.sort_by(|left, right| left.uri.cmp(&right.uri));
let meta_by_uri = context
.object_meta
.iter()
.map(|meta| (meta.source_object_uri.as_str(), meta))
.collect::<HashMap<_, _>>();
let vcir_validation_time =
vcir.last_successful_validation_time
.parse()
.map_err(|detail| StorageError::InvalidData {
entity: "roa_cache_projection.entries[].earliest_safe_reuse_time_unix",
detail,
})?;
let mut entries: Vec<RoaCacheObjectProjection> = Vec::new();
let mut entry_index_by_uri: HashMap<String, usize> = HashMap::new();
for output in &vcir.local_outputs {
let Some(projected_output) = RoaCacheLocalOutputProjection::from_local_output(output)
else {
continue;
};
let projected_output_effective_until = projected_output
.item_effective_until
.parse()
.map(|time| time.unix_timestamp())
.map_err(|detail| StorageError::InvalidData {
entity: "roa_cache_projection.entries[].outputs_effective_until_unix",
detail,
})?;
let Some(meta) = meta_by_uri.get(output.source_object_uri.as_str()).copied() else {
continue;
};
if meta.source_object_hash != output.source_object_hash {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[]",
detail: format!(
"metadata source object hash mismatch for {}",
output.source_object_uri
),
});
}
let earliest_safe_reuse_time_unix = meta
.earliest_safe_reuse_time
.parse()
.map(|time| time.max(vcir_validation_time).unix_timestamp())
.map_err(|detail| StorageError::InvalidData {
entity: "roa_cache_projection.entries[].earliest_safe_reuse_time_unix",
detail,
})?;
if let Some(entry_index) = entry_index_by_uri.get(output.source_object_uri.as_str()) {
let entry = &mut entries[*entry_index];
if entry.source_object_hash != output.source_object_hash {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[]",
detail: format!(
"source object hash mismatch for {}",
output.source_object_uri
),
});
}
entry.outputs_effective_until_unix = entry
.outputs_effective_until_unix
.min(projected_output_effective_until);
entry.outputs.push(projected_output);
} else {
entry_index_by_uri.insert(output.source_object_uri.clone(), entries.len());
entries.push(RoaCacheObjectProjection {
source_object_uri: output.source_object_uri.clone(),
source_object_hash: output.source_object_hash,
ee_serial: Some(meta.ee_serial.clone()),
crl_uri: Some(meta.crl_uri.clone()),
earliest_safe_reuse_time_unix: Some(earliest_safe_reuse_time_unix),
outputs_effective_until_unix: projected_output_effective_until,
outputs: vec![projected_output],
});
}
}
if entries.is_empty() {
return Ok(None);
}
entries.sort_by(|left, right| left.source_object_uri.cmp(&right.source_object_uri));
let projection = Self {
manifest_rsync_uri: vcir.manifest_rsync_uri.clone(),
instance_effective_until: vcir.instance_gate.instance_effective_until.clone(),
issuer_ca_sha256_hex,
ca_validation_context_digest: Some(context.ca_validation_context_digest),
policy_fingerprint: Some(context.policy_fingerprint),
crl_sha256_by_uri,
entries,
};
projection.validate_internal()?;
Ok(Some(projection))
}
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty(
"roa_cache_projection.manifest_rsync_uri",
&self.manifest_rsync_uri,
)?;
parse_time(
"roa_cache_projection.instance_effective_until",
&self.instance_effective_until,
)?;
if let Some(hash) = &self.issuer_ca_sha256_hex {
validate_sha256_hex("roa_cache_projection.issuer_ca_sha256_hex", hash)?;
}
if self.ca_validation_context_digest.is_some() != self.policy_fingerprint.is_some() {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.context",
detail: "ca_validation_context_digest and policy_fingerprint must be both present or both absent"
.to_string(),
});
}
let mut seen_crls = HashSet::with_capacity(self.crl_sha256_by_uri.len());
for crl in &self.crl_sha256_by_uri {
crl.validate_internal()?;
if !seen_crls.insert(crl.uri.as_str()) {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.crls[]",
detail: format!("duplicate CRL URI: {}", crl.uri),
});
}
}
if self.entries.is_empty() {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries",
detail: "must not be empty".to_string(),
});
}
let mut seen_entries = HashSet::with_capacity(self.entries.len());
for entry in &self.entries {
entry.validate_internal()?;
if !seen_entries.insert(entry.source_object_uri.as_str()) {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[]",
detail: format!("duplicate ROA URI: {}", entry.source_object_uri),
});
}
}
Ok(())
}
}

View File

@@ -0,0 +1,730 @@
// VCIR summaries, RRDP records, and validation metadata.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VcirArtifactRole {
Manifest,
CurrentCrl,
ChildCaCert,
SignedObject,
EeCert,
IssuerCert,
Tal,
TrustAnchorCert,
Other,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VcirArtifactKind {
Cer,
Crl,
Mft,
Roa,
Aspa,
Gbr,
Tal,
Other,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VcirArtifactValidationStatus {
Accepted,
Rejected,
WarningOnly,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirRelatedArtifact {
#[serde(rename = "r")]
pub artifact_role: VcirArtifactRole,
#[serde(rename = "k")]
pub artifact_kind: VcirArtifactKind,
#[serde(rename = "u")]
pub uri: Option<String>,
#[serde(rename = "h")]
pub sha256: String,
#[serde(rename = "t")]
pub object_type: Option<String>,
#[serde(rename = "s")]
pub validation_status: VcirArtifactValidationStatus,
/// Reject reason captured at fresh validation time for `Rejected` artifacts.
/// `None` for accepted/warning artifacts and for cache entries written
/// before this field existed.
#[serde(rename = "e", default, skip_serializing_if = "Option::is_none")]
pub reject_reason: Option<String>,
}
impl VcirRelatedArtifact {
pub fn validate_internal(&self) -> StorageResult<()> {
if let Some(uri) = &self.uri {
validate_non_empty("vcir.related_artifacts[].uri", uri)?;
}
validate_sha256_hex("vcir.related_artifacts[].sha256", &self.sha256)?;
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirSummary {
#[serde(rename = "v")]
pub local_vrp_count: u32,
#[serde(rename = "a")]
pub local_aspa_count: u32,
#[serde(rename = "r")]
pub local_router_key_count: u32,
#[serde(rename = "c")]
pub child_count: u32,
#[serde(rename = "o")]
pub accepted_object_count: u32,
#[serde(rename = "x")]
pub rejected_object_count: u32,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirAuditSummary {
#[serde(rename = "f")]
pub failed_fetch_eligible: bool,
#[serde(rename = "r")]
pub last_failed_fetch_reason: Option<String>,
#[serde(rename = "w")]
pub warning_count: u32,
#[serde(rename = "a")]
pub audit_flags: Vec<String>,
}
impl VcirAuditSummary {
pub fn validate_internal(&self) -> StorageResult<()> {
if let Some(reason) = &self.last_failed_fetch_reason {
validate_non_empty("vcir.audit_summary.last_failed_fetch_reason", reason)?;
}
for flag in &self.audit_flags {
validate_non_empty("vcir.audit_summary.audit_flags[]", flag)?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct ValidatedCaInstanceResult {
#[serde(rename = "m")]
pub manifest_rsync_uri: String,
#[serde(rename = "pm")]
pub parent_manifest_rsync_uri: Option<String>,
#[serde(rename = "tal")]
pub tal_id: String,
#[serde(rename = "subj")]
pub ca_subject_name: String,
#[serde(rename = "ski")]
pub ca_ski: String,
#[serde(rename = "aki")]
pub issuer_ski: String,
#[serde(rename = "vt")]
pub last_successful_validation_time: PackTime,
#[serde(rename = "cm")]
pub current_manifest_rsync_uri: String,
#[serde(rename = "crl")]
pub current_crl_rsync_uri: String,
#[serde(rename = "mm")]
pub validated_manifest_meta: ValidatedManifestMeta,
#[serde(rename = "ccr")]
pub ccr_manifest_projection: VcirCcrManifestProjection,
#[serde(rename = "g")]
pub instance_gate: VcirInstanceGate,
#[serde(rename = "ch")]
pub child_entries: Vec<VcirChildEntry>,
#[serde(rename = "lo")]
pub local_outputs: Vec<VcirLocalOutput>,
#[serde(rename = "ra")]
pub related_artifacts: Vec<VcirRelatedArtifact>,
#[serde(rename = "s")]
pub summary: VcirSummary,
#[serde(rename = "as")]
pub audit_summary: VcirAuditSummary,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirFieldSizeBreakdown {
pub local_output_count: u64,
pub local_output_source_uri_bytes: u64,
pub local_output_source_type_bytes: u64,
pub local_output_source_hash_hex_bytes: u64,
pub local_output_source_ee_hash_hex_bytes: u64,
pub local_output_payload_json_bytes: u64,
pub local_output_rule_hash_hex_bytes: u64,
pub local_output_source_hash_binary_bytes: u64,
pub local_output_source_ee_hash_binary_bytes: u64,
pub local_output_payload_typed_body_bytes: u64,
pub local_output_rule_hash_binary_bytes: u64,
pub related_artifact_count: u64,
pub related_artifact_uri_bytes: u64,
pub related_artifact_hash_hex_bytes: u64,
pub related_artifact_type_bytes: u64,
pub child_entry_count: u64,
pub child_entry_uri_bytes: u64,
pub child_entry_hash_hex_bytes: u64,
}
fn serialized_cbor_len<T: Serialize>(value: &T) -> u64 {
serde_cbor::to_vec(value)
.map(|bytes| bytes.len() as u64)
.unwrap_or(0)
}
impl VcirFieldSizeBreakdown {
pub fn from_vcir(vcir: &ValidatedCaInstanceResult) -> Self {
let mut out = Self::default();
out.local_output_count = vcir.local_outputs.len() as u64;
for local in &vcir.local_outputs {
out.local_output_source_uri_bytes += local.source_object_uri.len() as u64;
out.local_output_source_type_bytes += local.source_object_type_name().len() as u64;
out.local_output_source_hash_hex_bytes += 64;
out.local_output_source_ee_hash_hex_bytes += 64;
out.local_output_payload_json_bytes += local.payload_json().len() as u64;
out.local_output_rule_hash_hex_bytes += 64;
out.local_output_source_hash_binary_bytes += 32;
out.local_output_source_ee_hash_binary_bytes += 32;
out.local_output_payload_typed_body_bytes += local.payload.typed_body_bytes();
out.local_output_rule_hash_binary_bytes += 32;
}
out.related_artifact_count = vcir.related_artifacts.len() as u64;
for artifact in &vcir.related_artifacts {
if let Some(uri) = &artifact.uri {
out.related_artifact_uri_bytes += uri.len() as u64;
}
out.related_artifact_hash_hex_bytes += artifact.sha256.len() as u64;
if let Some(object_type) = &artifact.object_type {
out.related_artifact_type_bytes += object_type.len() as u64;
}
}
out.child_entry_count = vcir.child_entries.len() as u64;
for child in &vcir.child_entries {
out.child_entry_uri_bytes += child.child_manifest_rsync_uri.len() as u64
+ child.child_cert_rsync_uri.len() as u64
+ child.child_rsync_base_uri.len() as u64
+ child.child_publication_point_rsync_uri.len() as u64
+ child
.child_rrdp_notification_uri
.as_ref()
.map(|uri| uri.len() as u64)
.unwrap_or(0);
out.child_entry_hash_hex_bytes +=
child.child_cert_hash.len() as u64 + child.child_ski.len() as u64;
}
out
}
pub fn add_assign(&mut self, other: &Self) {
self.local_output_count += other.local_output_count;
self.local_output_source_uri_bytes += other.local_output_source_uri_bytes;
self.local_output_source_type_bytes += other.local_output_source_type_bytes;
self.local_output_source_hash_hex_bytes += other.local_output_source_hash_hex_bytes;
self.local_output_source_ee_hash_hex_bytes += other.local_output_source_ee_hash_hex_bytes;
self.local_output_payload_json_bytes += other.local_output_payload_json_bytes;
self.local_output_rule_hash_hex_bytes += other.local_output_rule_hash_hex_bytes;
self.local_output_source_hash_binary_bytes += other.local_output_source_hash_binary_bytes;
self.local_output_source_ee_hash_binary_bytes +=
other.local_output_source_ee_hash_binary_bytes;
self.local_output_payload_typed_body_bytes += other.local_output_payload_typed_body_bytes;
self.local_output_rule_hash_binary_bytes += other.local_output_rule_hash_binary_bytes;
self.related_artifact_count += other.related_artifact_count;
self.related_artifact_uri_bytes += other.related_artifact_uri_bytes;
self.related_artifact_hash_hex_bytes += other.related_artifact_hash_hex_bytes;
self.related_artifact_type_bytes += other.related_artifact_type_bytes;
self.child_entry_count += other.child_entry_count;
self.child_entry_uri_bytes += other.child_entry_uri_bytes;
self.child_entry_hash_hex_bytes += other.child_entry_hash_hex_bytes;
}
pub fn local_output_old_projection_bytes(&self) -> u64 {
self.local_output_source_type_bytes
+ self.local_output_source_hash_hex_bytes
+ self.local_output_source_ee_hash_hex_bytes
+ self.local_output_payload_json_bytes
+ self.local_output_rule_hash_hex_bytes
}
pub fn local_output_typed_projection_bytes(&self) -> u64 {
self.local_output_count
+ self.local_output_source_hash_binary_bytes
+ self.local_output_source_ee_hash_binary_bytes
+ self.local_output_payload_typed_body_bytes
+ self.local_output_rule_hash_binary_bytes
}
pub fn local_output_projection_saved_bytes(&self) -> u64 {
self.local_output_old_projection_bytes()
.saturating_sub(self.local_output_typed_projection_bytes())
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirReplaceTimingBreakdown {
pub validate_ms: u64,
pub vcir_encode_ms: u64,
pub vcir_value_bytes: u64,
pub replay_meta_encode_ms: u64,
pub replay_meta_value_bytes: u64,
pub roa_cache_projection_encode_ms: u64,
pub roa_cache_projection_value_bytes: u64,
pub publication_point_cache_projection_encode_ms: u64,
pub publication_point_cache_projection_value_bytes: u64,
pub batch_build_ms: u64,
pub write_batch_ms: u64,
pub total_encoded_bytes: u64,
pub field_sizes: VcirFieldSizeBreakdown,
pub rss_before_kb: Option<u64>,
pub rss_after_validate_kb: Option<u64>,
pub rss_after_vcir_encode_kb: Option<u64>,
pub rss_after_replay_meta_encode_kb: Option<u64>,
pub rss_after_roa_cache_projection_encode_kb: Option<u64>,
pub rss_after_publication_point_cache_projection_encode_kb: Option<u64>,
pub rss_after_write_batch_kb: Option<u64>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirStorageSummary {
pub entry_count: u64,
pub vcir_value_bytes: u64,
pub vcir_value_bytes_max: u64,
pub vcir_value_bytes_max_manifest_rsync_uri: Option<String>,
pub core_fields: VcirCoreFieldSizeBreakdown,
pub ccr_projection: VcirCcrProjectionSizeBreakdown,
pub child_resources: VcirChildResourceSizeBreakdown,
pub field_sizes: VcirFieldSizeBreakdown,
pub local_output_old_projection_bytes: u64,
pub local_output_typed_projection_bytes: u64,
pub local_output_projection_saved_bytes: u64,
pub top_entries_by_vcir_value_bytes: Vec<VcirStorageEntrySummary>,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirStorageEntrySummary {
pub manifest_rsync_uri: String,
pub vcir_value_bytes: u64,
pub local_vrp_count: u32,
pub local_aspa_count: u32,
pub local_router_key_count: u32,
pub accepted_object_count: u32,
pub rejected_object_count: u32,
pub child_count: u32,
pub core_fields: VcirCoreFieldSizeBreakdown,
pub ccr_projection: VcirCcrProjectionSizeBreakdown,
pub child_resources: VcirChildResourceSizeBreakdown,
pub field_sizes: VcirFieldSizeBreakdown,
pub local_output_old_projection_bytes: u64,
pub local_output_typed_projection_bytes: u64,
pub local_output_projection_saved_bytes: u64,
}
impl VcirStorageEntrySummary {
fn from_vcir(vcir: &ValidatedCaInstanceResult, vcir_value_bytes: u64) -> Self {
let field_sizes = VcirFieldSizeBreakdown::from_vcir(vcir);
Self {
manifest_rsync_uri: vcir.manifest_rsync_uri.clone(),
vcir_value_bytes,
local_vrp_count: vcir.summary.local_vrp_count,
local_aspa_count: vcir.summary.local_aspa_count,
local_router_key_count: vcir.summary.local_router_key_count,
accepted_object_count: vcir.summary.accepted_object_count,
rejected_object_count: vcir.summary.rejected_object_count,
child_count: vcir.summary.child_count,
core_fields: VcirCoreFieldSizeBreakdown::from_vcir(vcir),
ccr_projection: VcirCcrProjectionSizeBreakdown::from_projection(
&vcir.ccr_manifest_projection,
),
child_resources: VcirChildResourceSizeBreakdown::from_vcir(vcir),
local_output_old_projection_bytes: field_sizes.local_output_old_projection_bytes(),
local_output_typed_projection_bytes: field_sizes.local_output_typed_projection_bytes(),
local_output_projection_saved_bytes: field_sizes.local_output_projection_saved_bytes(),
field_sizes,
}
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirCoreFieldSizeBreakdown {
pub manifest_rsync_uri_bytes: u64,
pub parent_manifest_rsync_uri_bytes: u64,
pub tal_id_bytes: u64,
pub ca_subject_name_bytes: u64,
pub ca_ski_bytes: u64,
pub issuer_ski_bytes: u64,
pub current_manifest_rsync_uri_bytes: u64,
pub current_crl_rsync_uri_bytes: u64,
pub validated_manifest_number_bytes: u64,
}
impl VcirCoreFieldSizeBreakdown {
fn from_vcir(vcir: &ValidatedCaInstanceResult) -> Self {
Self {
manifest_rsync_uri_bytes: vcir.manifest_rsync_uri.len() as u64,
parent_manifest_rsync_uri_bytes: vcir
.parent_manifest_rsync_uri
.as_ref()
.map(|uri| uri.len() as u64)
.unwrap_or(0),
tal_id_bytes: vcir.tal_id.len() as u64,
ca_subject_name_bytes: vcir.ca_subject_name.len() as u64,
ca_ski_bytes: vcir.ca_ski.len() as u64,
issuer_ski_bytes: vcir.issuer_ski.len() as u64,
current_manifest_rsync_uri_bytes: vcir.current_manifest_rsync_uri.len() as u64,
current_crl_rsync_uri_bytes: vcir.current_crl_rsync_uri.len() as u64,
validated_manifest_number_bytes: vcir
.validated_manifest_meta
.validated_manifest_number
.len() as u64,
}
}
fn add_assign(&mut self, other: &Self) {
self.manifest_rsync_uri_bytes += other.manifest_rsync_uri_bytes;
self.parent_manifest_rsync_uri_bytes += other.parent_manifest_rsync_uri_bytes;
self.tal_id_bytes += other.tal_id_bytes;
self.ca_subject_name_bytes += other.ca_subject_name_bytes;
self.ca_ski_bytes += other.ca_ski_bytes;
self.issuer_ski_bytes += other.issuer_ski_bytes;
self.current_manifest_rsync_uri_bytes += other.current_manifest_rsync_uri_bytes;
self.current_crl_rsync_uri_bytes += other.current_crl_rsync_uri_bytes;
self.validated_manifest_number_bytes += other.validated_manifest_number_bytes;
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirChildResourceSizeBreakdown {
pub effective_ip_resource_cbor_bytes: u64,
pub effective_as_resource_cbor_bytes: u64,
}
impl VcirChildResourceSizeBreakdown {
fn from_vcir(vcir: &ValidatedCaInstanceResult) -> Self {
let mut out = Self::default();
for child in &vcir.child_entries {
out.effective_ip_resource_cbor_bytes +=
serialized_cbor_len(&child.child_effective_ip_resources);
out.effective_as_resource_cbor_bytes +=
serialized_cbor_len(&child.child_effective_as_resources);
}
out
}
fn add_assign(&mut self, other: &Self) {
self.effective_ip_resource_cbor_bytes += other.effective_ip_resource_cbor_bytes;
self.effective_as_resource_cbor_bytes += other.effective_as_resource_cbor_bytes;
}
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct VcirCcrProjectionSizeBreakdown {
pub manifest_rsync_uri_bytes: u64,
pub manifest_sha256_bytes: u64,
pub manifest_ee_aki_bytes: u64,
pub manifest_number_bytes: u64,
pub manifest_sia_locations_count: u64,
pub manifest_sia_locations_der_bytes: u64,
pub subordinate_ski_count: u64,
pub subordinate_ski_bytes: u64,
}
impl VcirCcrProjectionSizeBreakdown {
fn from_projection(projection: &VcirCcrManifestProjection) -> Self {
Self {
manifest_rsync_uri_bytes: projection.manifest_rsync_uri.len() as u64,
manifest_sha256_bytes: projection.manifest_sha256.len() as u64,
manifest_ee_aki_bytes: projection.manifest_ee_aki.len() as u64,
manifest_number_bytes: projection.manifest_number_be.len() as u64,
manifest_sia_locations_count: projection.manifest_sia_locations_der.len() as u64,
manifest_sia_locations_der_bytes: projection
.manifest_sia_locations_der
.iter()
.map(|location| location.len() as u64)
.sum(),
subordinate_ski_count: projection.subordinate_skis.len() as u64,
subordinate_ski_bytes: projection
.subordinate_skis
.iter()
.map(|ski| ski.len() as u64)
.sum(),
}
}
fn add_assign(&mut self, other: &Self) {
self.manifest_rsync_uri_bytes += other.manifest_rsync_uri_bytes;
self.manifest_sha256_bytes += other.manifest_sha256_bytes;
self.manifest_ee_aki_bytes += other.manifest_ee_aki_bytes;
self.manifest_number_bytes += other.manifest_number_bytes;
self.manifest_sia_locations_count += other.manifest_sia_locations_count;
self.manifest_sia_locations_der_bytes += other.manifest_sia_locations_der_bytes;
self.subordinate_ski_count += other.subordinate_ski_count;
self.subordinate_ski_bytes += other.subordinate_ski_bytes;
}
}
fn push_top_vcir_storage_entry(
entries: &mut Vec<VcirStorageEntrySummary>,
entry: VcirStorageEntrySummary,
) {
const TOP_N: usize = 20;
entries.push(entry);
entries.sort_by(|left, right| {
right
.vcir_value_bytes
.cmp(&left.vcir_value_bytes)
.then_with(|| left.manifest_rsync_uri.cmp(&right.manifest_rsync_uri))
});
entries.truncate(TOP_N);
}
impl ValidatedCaInstanceResult {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty("vcir.manifest_rsync_uri", &self.manifest_rsync_uri)?;
if let Some(parent_manifest_rsync_uri) = &self.parent_manifest_rsync_uri {
validate_non_empty("vcir.parent_manifest_rsync_uri", parent_manifest_rsync_uri)?;
}
validate_non_empty("vcir.tal_id", &self.tal_id)?;
validate_non_empty("vcir.ca_subject_name", &self.ca_subject_name)?;
validate_non_empty("vcir.ca_ski", &self.ca_ski)?;
validate_non_empty("vcir.issuer_ski", &self.issuer_ski)?;
parse_time(
"vcir.last_successful_validation_time",
&self.last_successful_validation_time,
)?;
validate_non_empty(
"vcir.current_manifest_rsync_uri",
&self.current_manifest_rsync_uri,
)?;
validate_non_empty("vcir.current_crl_rsync_uri", &self.current_crl_rsync_uri)?;
self.validated_manifest_meta.validate_internal()?;
self.ccr_manifest_projection.validate_internal()?;
self.instance_gate.validate_internal()?;
let expected_manifest_next = self
.validated_manifest_meta
.validated_manifest_next_update
.parse()
.map_err(|detail| StorageError::InvalidData {
entity: "vcir",
detail: format!(
"validated_manifest_meta.validated_manifest_next_update invalid: {detail}"
),
})?;
let instance_manifest_next =
self.instance_gate
.manifest_next_update
.parse()
.map_err(|detail| StorageError::InvalidData {
entity: "vcir",
detail: format!("instance_gate.manifest_next_update invalid: {detail}"),
})?;
if expected_manifest_next != instance_manifest_next {
return Err(StorageError::InvalidData {
entity: "vcir",
detail: "instance_gate.manifest_next_update must equal validated_manifest_meta.validated_manifest_next_update".to_string(),
});
}
let mut child_manifests = HashSet::with_capacity(self.child_entries.len());
for child in &self.child_entries {
child.validate_internal()?;
if !child_manifests.insert(child.child_manifest_rsync_uri.as_str()) {
return Err(StorageError::InvalidData {
entity: "vcir",
detail: format!(
"duplicate child_manifest_rsync_uri: {}",
child.child_manifest_rsync_uri
),
});
}
}
let mut vrp_count = 0u32;
let mut aspa_count = 0u32;
let mut router_key_count = 0u32;
for output in &self.local_outputs {
output.validate_internal()?;
match output.output_type {
VcirOutputType::Vrp => vrp_count += 1,
VcirOutputType::Aspa => aspa_count += 1,
VcirOutputType::RouterKey => router_key_count += 1,
}
}
let mut output_ids = self
.local_outputs
.iter()
.map(VcirLocalOutput::output_id)
.collect::<Vec<_>>();
output_ids.sort_unstable();
if let Some(duplicate) = output_ids
.windows(2)
.find_map(|pair| (pair[0] == pair[1]).then(|| pair[0].clone()))
{
return Err(StorageError::InvalidData {
entity: "vcir",
detail: format!("duplicate output_id: {duplicate}"),
});
}
if self.summary.local_vrp_count != vrp_count {
return Err(StorageError::InvalidData {
entity: "vcir.summary",
detail: format!(
"local_vrp_count={} does not match local_outputs count {}",
self.summary.local_vrp_count, vrp_count
),
});
}
if self.summary.local_aspa_count != aspa_count {
return Err(StorageError::InvalidData {
entity: "vcir.summary",
detail: format!(
"local_aspa_count={} does not match local_outputs count {}",
self.summary.local_aspa_count, aspa_count
),
});
}
if self.summary.local_router_key_count != router_key_count {
return Err(StorageError::InvalidData {
entity: "vcir.summary",
detail: format!(
"local_router_key_count={} does not match local_outputs count {}",
self.summary.local_router_key_count, router_key_count
),
});
}
if self.summary.child_count != self.child_entries.len() as u32 {
return Err(StorageError::InvalidData {
entity: "vcir.summary",
detail: format!(
"child_count={} does not match child_entries length {}",
self.summary.child_count,
self.child_entries.len()
),
});
}
for artifact in &self.related_artifacts {
artifact.validate_internal()?;
}
self.audit_summary.validate_internal()?;
Ok(())
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RrdpSourceSyncState {
Empty,
SnapshotOnly,
DeltaReady,
Error,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RrdpSourceRecord {
pub notify_uri: String,
pub last_session_id: Option<String>,
pub last_serial: Option<u64>,
pub first_seen_at: PackTime,
pub last_seen_at: PackTime,
pub last_sync_at: Option<PackTime>,
pub sync_state: RrdpSourceSyncState,
pub last_snapshot_uri: Option<String>,
pub last_snapshot_hash: Option<String>,
pub last_error: Option<String>,
}
impl RrdpSourceRecord {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty("rrdp_source.notify_uri", &self.notify_uri)?;
if let Some(session_id) = &self.last_session_id {
validate_non_empty("rrdp_source.last_session_id", session_id)?;
}
parse_time("rrdp_source.first_seen_at", &self.first_seen_at)?;
parse_time("rrdp_source.last_seen_at", &self.last_seen_at)?;
if let Some(last_sync_at) = &self.last_sync_at {
parse_time("rrdp_source.last_sync_at", last_sync_at)?;
}
if let Some(last_snapshot_uri) = &self.last_snapshot_uri {
validate_non_empty("rrdp_source.last_snapshot_uri", last_snapshot_uri)?;
}
if let Some(last_snapshot_hash) = &self.last_snapshot_hash {
validate_sha256_hex("rrdp_source.last_snapshot_hash", last_snapshot_hash)?;
}
if let Some(last_error) = &self.last_error {
validate_non_empty("rrdp_source.last_error", last_error)?;
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RrdpSourceMemberRecord {
pub notify_uri: String,
pub rsync_uri: String,
pub current_hash: Option<String>,
pub object_type: Option<String>,
pub present: bool,
pub last_confirmed_session_id: String,
pub last_confirmed_serial: u64,
pub last_changed_at: PackTime,
}
impl RrdpSourceMemberRecord {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty("rrdp_source_member.notify_uri", &self.notify_uri)?;
validate_non_empty("rrdp_source_member.rsync_uri", &self.rsync_uri)?;
validate_non_empty(
"rrdp_source_member.last_confirmed_session_id",
&self.last_confirmed_session_id,
)?;
if self.present {
let hash = self
.current_hash
.as_deref()
.ok_or(StorageError::InvalidData {
entity: "rrdp_source_member",
detail: "current_hash is required when present=true".to_string(),
})?;
validate_sha256_hex("rrdp_source_member.current_hash", hash)?;
} else if let Some(hash) = &self.current_hash {
validate_sha256_hex("rrdp_source_member.current_hash", hash)?;
}
parse_time("rrdp_source_member.last_changed_at", &self.last_changed_at)?;
Ok(())
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RrdpUriOwnerState {
Active,
Conflict,
Withdrawn,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RrdpUriOwnerRecord {
pub rsync_uri: String,
pub notify_uri: String,
pub current_hash: Option<String>,
pub last_confirmed_session_id: String,
pub last_confirmed_serial: u64,
pub last_changed_at: PackTime,
pub owner_state: RrdpUriOwnerState,
}
impl RrdpUriOwnerRecord {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty("rrdp_uri_owner.rsync_uri", &self.rsync_uri)?;
validate_non_empty("rrdp_uri_owner.notify_uri", &self.notify_uri)?;
validate_non_empty(
"rrdp_uri_owner.last_confirmed_session_id",
&self.last_confirmed_session_id,
)?;
if let Some(hash) = &self.current_hash {
validate_sha256_hex("rrdp_uri_owner.current_hash", hash)?;
}
parse_time("rrdp_uri_owner.last_changed_at", &self.last_changed_at)?;
Ok(())
}
}

View File

@@ -0,0 +1,548 @@
// VCIR local-output and child-certificate cache projections.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VcirLocalOutputPayload {
Vrp {
asn: u32,
afi: crate::data_model::roa::RoaAfi,
prefix_len: u16,
#[serde(with = "serde_bytes_16")]
addr: [u8; 16],
max_length: u16,
},
Aspa {
customer_as_id: u32,
provider_as_ids: Vec<u32>,
},
RouterKey {
as_id: u32,
#[serde(with = "serde_byte_vec")]
ski: Vec<u8>,
#[serde(with = "serde_byte_vec")]
spki_der: Vec<u8>,
},
}
impl VcirLocalOutputPayload {
pub fn typed_body_bytes(&self) -> u64 {
match self {
Self::Vrp { .. } => 4 + 1 + 2 + 16 + 2,
Self::Aspa {
provider_as_ids, ..
} => 4 + (provider_as_ids.len() as u64 * 4),
Self::RouterKey { ski, spki_der, .. } => 4 + ski.len() as u64 + spki_der.len() as u64,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct VcirLocalOutput {
#[serde(rename = "t")]
pub output_type: VcirOutputType,
#[serde(rename = "e")]
pub item_effective_until: PackTime,
#[serde(rename = "u")]
pub source_object_uri: String,
#[serde(rename = "k")]
pub source_object_type: VcirSourceObjectType,
#[serde(rename = "h")]
#[serde(with = "serde_bytes_32")]
pub source_object_hash: [u8; 32],
#[serde(rename = "c")]
#[serde(with = "serde_bytes_32")]
pub source_ee_cert_hash: [u8; 32],
#[serde(rename = "p")]
pub payload: VcirLocalOutputPayload,
#[serde(rename = "r")]
#[serde(with = "serde_bytes_32")]
pub rule_hash: [u8; 32],
}
impl VcirLocalOutput {
pub fn output_id(&self) -> String {
self.rule_hash_hex()
}
pub fn source_object_hash_hex(&self) -> String {
hex::encode(self.source_object_hash)
}
pub fn source_ee_cert_hash_hex(&self) -> String {
hex::encode(self.source_ee_cert_hash)
}
pub fn rule_hash_hex(&self) -> String {
hex::encode(self.rule_hash)
}
pub fn source_object_type_name(&self) -> &'static str {
self.source_object_type.as_str()
}
pub fn payload_json(&self) -> String {
match &self.payload {
VcirLocalOutputPayload::Vrp {
asn,
afi,
prefix_len,
addr,
max_length,
} => {
let prefix = match afi {
crate::data_model::roa::RoaAfi::Ipv4 => {
let ip = std::net::Ipv4Addr::new(addr[0], addr[1], addr[2], addr[3]);
format!("{ip}/{prefix_len}")
}
crate::data_model::roa::RoaAfi::Ipv6 => {
let ip = std::net::Ipv6Addr::from(*addr);
format!("{ip}/{prefix_len}")
}
};
format!(r#"{{"asn":{asn},"max_length":{max_length},"prefix":"{prefix}"}}"#)
}
VcirLocalOutputPayload::Aspa {
customer_as_id,
provider_as_ids,
} => {
let providers = provider_as_ids
.iter()
.map(u32::to_string)
.collect::<Vec<_>>()
.join(",");
format!(r#"{{"customer_as_id":{customer_as_id},"provider_as_ids":[{providers}]}}"#)
}
VcirLocalOutputPayload::RouterKey {
as_id,
ski,
spki_der,
} => {
let ski_hex = hex::encode(ski);
let spki_der_base64 = base64::engine::general_purpose::STANDARD.encode(spki_der);
format!(
r#"{{"as_id":{as_id},"ski_hex":"{ski_hex}","spki_der_base64":"{spki_der_base64}"}}"#
)
}
}
}
pub fn validate_internal(&self) -> StorageResult<()> {
parse_time(
"vcir.local_outputs[].item_effective_until",
&self.item_effective_until,
)?;
validate_non_empty(
"vcir.local_outputs[].source_object_uri",
&self.source_object_uri,
)?;
validate_local_output_type_matches_payload(self)?;
Ok(())
}
}
fn validate_local_output_type_matches_payload(output: &VcirLocalOutput) -> StorageResult<()> {
let matches_payload = matches!(
(&output.output_type, &output.payload),
(VcirOutputType::Vrp, VcirLocalOutputPayload::Vrp { .. })
| (VcirOutputType::Aspa, VcirLocalOutputPayload::Aspa { .. })
| (
VcirOutputType::RouterKey,
VcirLocalOutputPayload::RouterKey { .. }
)
);
if !matches_payload {
return Err(StorageError::InvalidData {
entity: "vcir.local_outputs[]",
detail: "output_type must match payload variant".to_string(),
});
}
Ok(())
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RoaCacheCrlProjection {
#[serde(rename = "u")]
pub uri: String,
#[serde(rename = "h")]
pub sha256: String,
}
impl RoaCacheCrlProjection {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty("roa_cache_projection.crls[].uri", &self.uri)?;
validate_sha256_hex("roa_cache_projection.crls[].sha256", &self.sha256)?;
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaCacheObjectMeta {
pub source_object_uri: String,
pub source_object_hash: [u8; 32],
pub ee_serial: Vec<u8>,
pub crl_uri: String,
pub earliest_safe_reuse_time: PackTime,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RoaCacheProjectionContext {
pub ca_validation_context_digest: [u8; 32],
pub policy_fingerprint: [u8; 32],
pub object_meta: Vec<RoaCacheObjectMeta>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RoaCacheLocalOutputProjection {
#[serde(rename = "e")]
pub item_effective_until: PackTime,
#[serde(rename = "c")]
#[serde(with = "serde_bytes_32")]
pub source_ee_cert_hash: [u8; 32],
#[serde(rename = "p")]
pub payload: VcirLocalOutputPayload,
#[serde(rename = "r")]
#[serde(with = "serde_bytes_32")]
pub rule_hash: [u8; 32],
}
impl RoaCacheLocalOutputProjection {
fn from_local_output(output: &VcirLocalOutput) -> Option<Self> {
if output.output_type != VcirOutputType::Vrp
|| output.source_object_type != VcirSourceObjectType::Roa
{
return None;
}
Some(Self {
item_effective_until: output.item_effective_until.clone(),
source_ee_cert_hash: output.source_ee_cert_hash,
payload: output.payload.clone(),
rule_hash: output.rule_hash,
})
}
pub fn validate_internal(&self) -> StorageResult<()> {
parse_time(
"roa_cache_projection.entries[].outputs[].item_effective_until",
&self.item_effective_until,
)?;
if !matches!(self.payload, VcirLocalOutputPayload::Vrp { .. }) {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[].outputs[]",
detail: "payload must be VRP".to_string(),
});
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RoaCacheObjectProjection {
#[serde(rename = "u")]
pub source_object_uri: String,
#[serde(rename = "h")]
#[serde(with = "serde_bytes_32")]
pub source_object_hash: [u8; 32],
#[serde(rename = "s", default, skip_serializing_if = "Option::is_none")]
#[serde(with = "serde_optional_byte_vec")]
pub ee_serial: Option<Vec<u8>>,
#[serde(rename = "c", default, skip_serializing_if = "Option::is_none")]
pub crl_uri: Option<String>,
#[serde(rename = "n", default, skip_serializing_if = "Option::is_none")]
pub earliest_safe_reuse_time_unix: Option<i64>,
#[serde(rename = "x")]
pub outputs_effective_until_unix: i64,
#[serde(rename = "o")]
pub outputs: Vec<RoaCacheLocalOutputProjection>,
}
impl RoaCacheObjectProjection {
pub fn validate_internal(&self) -> StorageResult<()> {
validate_non_empty(
"roa_cache_projection.entries[].source_object_uri",
&self.source_object_uri,
)?;
if let Some(serial) = &self.ee_serial {
if serial.is_empty() {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[].ee_serial",
detail: "must not be empty when present".to_string(),
});
}
}
if let Some(crl_uri) = &self.crl_uri {
validate_non_empty("roa_cache_projection.entries[].crl_uri", crl_uri)?;
}
if let Some(earliest_safe_reuse_time_unix) = self.earliest_safe_reuse_time_unix
&& earliest_safe_reuse_time_unix >= self.outputs_effective_until_unix
{
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[].effective_window",
detail: "earliest_safe_reuse_time_unix must be before outputs_effective_until_unix"
.to_string(),
});
}
if self.outputs.is_empty() {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[]",
detail: "outputs must not be empty".to_string(),
});
}
for output in &self.outputs {
output.validate_internal()?;
}
let expected_effective_until = self
.outputs
.iter()
.map(|output| {
output
.item_effective_until
.parse()
.map(|time| time.unix_timestamp())
.map_err(|detail| StorageError::InvalidData {
entity: "roa_cache_projection.entries[].outputs_effective_until_unix",
detail,
})
})
.collect::<StorageResult<Vec<_>>>()?
.into_iter()
.min()
.expect("outputs must not be empty");
if self.outputs_effective_until_unix != expected_effective_until {
return Err(StorageError::InvalidData {
entity: "roa_cache_projection.entries[].outputs_effective_until_unix",
detail: "must equal the earliest output item_effective_until".to_string(),
});
}
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RoaCacheProjection {
#[serde(rename = "m")]
pub manifest_rsync_uri: String,
#[serde(rename = "e")]
pub instance_effective_until: PackTime,
#[serde(rename = "i")]
pub issuer_ca_sha256_hex: Option<String>,
#[serde(rename = "p", default, skip_serializing_if = "Option::is_none")]
#[serde(with = "serde_optional_bytes_32")]
pub ca_validation_context_digest: Option<[u8; 32]>,
#[serde(rename = "f", default, skip_serializing_if = "Option::is_none")]
#[serde(with = "serde_optional_bytes_32")]
pub policy_fingerprint: Option<[u8; 32]>,
#[serde(rename = "c")]
pub crl_sha256_by_uri: Vec<RoaCacheCrlProjection>,
#[serde(rename = "r")]
pub entries: Vec<RoaCacheObjectProjection>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct ChildCertificateCacheRouterKeyProjection {
#[serde(rename = "a")]
pub as_id: u32,
#[serde(rename = "s")]
#[serde(with = "serde_byte_vec")]
pub ski: Vec<u8>,
#[serde(rename = "p")]
#[serde(with = "serde_byte_vec")]
pub spki_der: Vec<u8>,
#[serde(rename = "e")]
pub item_effective_until: PackTime,
}
impl ChildCertificateCacheRouterKeyProjection {
pub fn validate_internal(&self) -> StorageResult<()> {
if self.ski.is_empty() {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.router_keys[].ski",
detail: "must not be empty".to_string(),
});
}
if self.spki_der.is_empty() {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.router_keys[].spki_der",
detail: "must not be empty".to_string(),
});
}
parse_time(
"child_certificate_cache_projection.router_keys[].item_effective_until",
&self.item_effective_until,
)?;
Ok(())
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ChildCertificateCachePayload {
ChildCa {
#[serde(rename = "cm")]
child_manifest_rsync_uri: String,
#[serde(rename = "ski")]
child_ski: String,
#[serde(rename = "rb")]
child_rsync_base_uri: String,
#[serde(rename = "pp")]
child_publication_point_rsync_uri: String,
#[serde(rename = "rn")]
child_rrdp_notification_uri: Option<String>,
#[serde(rename = "ip")]
child_effective_ip_resources: Option<IpResourceSet>,
#[serde(rename = "as")]
child_effective_as_resources: Option<AsResourceSet>,
},
Router {
#[serde(rename = "r")]
router_keys: Vec<ChildCertificateCacheRouterKeyProjection>,
},
}
impl ChildCertificateCachePayload {
pub fn validate_internal(&self) -> StorageResult<()> {
match self {
Self::ChildCa {
child_manifest_rsync_uri,
child_ski,
child_rsync_base_uri,
child_publication_point_rsync_uri,
child_rrdp_notification_uri,
..
} => {
validate_non_empty(
"child_certificate_cache_projection.child_manifest_rsync_uri",
child_manifest_rsync_uri,
)?;
validate_non_empty("child_certificate_cache_projection.child_ski", child_ski)?;
validate_non_empty(
"child_certificate_cache_projection.child_rsync_base_uri",
child_rsync_base_uri,
)?;
validate_non_empty(
"child_certificate_cache_projection.child_publication_point_rsync_uri",
child_publication_point_rsync_uri,
)?;
if let Some(uri) = child_rrdp_notification_uri {
validate_non_empty(
"child_certificate_cache_projection.child_rrdp_notification_uri",
uri,
)?;
}
Ok(())
}
Self::Router { router_keys } => {
if router_keys.is_empty() {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.router_keys",
detail: "must not be empty".to_string(),
});
}
for router_key in router_keys {
router_key.validate_internal()?;
}
Ok(())
}
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct ChildCertificateCacheProjection {
#[serde(rename = "sv")]
pub schema_version: u32,
#[serde(rename = "av")]
pub algorithm_version: u32,
#[serde(rename = "k")]
pub cache_key_sha256_hex: String,
#[serde(rename = "cu")]
pub child_cert_uri: String,
#[serde(rename = "ch")]
pub child_cert_sha256_hex: String,
#[serde(rename = "cs")]
#[serde(with = "serde_byte_vec")]
pub child_cert_serial: Vec<u8>,
#[serde(rename = "ih")]
pub issuer_ca_sha256_hex: String,
#[serde(rename = "cru")]
pub issuer_crl_uri: String,
#[serde(rename = "crh")]
pub issuer_crl_sha256_hex: String,
#[serde(rename = "pc")]
#[serde(with = "serde_bytes_32")]
pub ca_validation_context_digest: [u8; 32],
#[serde(rename = "pf")]
#[serde(with = "serde_bytes_32")]
pub validation_policy_fingerprint: [u8; 32],
#[serde(rename = "nb")]
pub effective_not_before: PackTime,
#[serde(rename = "nu")]
pub effective_until: PackTime,
#[serde(rename = "p")]
pub payload: ChildCertificateCachePayload,
}
pub const CHILD_CERTIFICATE_CACHE_SCHEMA_VERSION: u32 = 2;
pub const CHILD_CERTIFICATE_CACHE_ALGORITHM_VERSION: u32 = 3;
impl ChildCertificateCacheProjection {
pub fn validate_internal(&self) -> StorageResult<()> {
if self.schema_version != CHILD_CERTIFICATE_CACHE_SCHEMA_VERSION {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.schema_version",
detail: format!("unsupported schema_version {}", self.schema_version),
});
}
if self.algorithm_version != CHILD_CERTIFICATE_CACHE_ALGORITHM_VERSION {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.algorithm_version",
detail: format!("unsupported algorithm_version {}", self.algorithm_version),
});
}
validate_sha256_hex(
"child_certificate_cache_projection.cache_key_sha256_hex",
&self.cache_key_sha256_hex,
)?;
validate_non_empty(
"child_certificate_cache_projection.child_cert_uri",
&self.child_cert_uri,
)?;
validate_sha256_hex(
"child_certificate_cache_projection.child_cert_sha256_hex",
&self.child_cert_sha256_hex,
)?;
if self.child_cert_serial.is_empty() {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.child_cert_serial",
detail: "must not be empty".to_string(),
});
}
validate_sha256_hex(
"child_certificate_cache_projection.issuer_ca_sha256_hex",
&self.issuer_ca_sha256_hex,
)?;
validate_non_empty(
"child_certificate_cache_projection.issuer_crl_uri",
&self.issuer_crl_uri,
)?;
validate_sha256_hex(
"child_certificate_cache_projection.issuer_crl_sha256_hex",
&self.issuer_crl_sha256_hex,
)?;
let effective_not_before = parse_time(
"child_certificate_cache_projection.effective_not_before",
&self.effective_not_before,
)?;
let effective_until = parse_time(
"child_certificate_cache_projection.effective_until",
&self.effective_until,
)?;
if effective_not_before >= effective_until {
return Err(StorageError::InvalidData {
entity: "child_certificate_cache_projection.effective_window",
detail: "effective_not_before must be before effective_until".to_string(),
});
}
self.payload.validate_internal()?;
Ok(())
}
}

View File

@@ -0,0 +1,300 @@
// VCIR reads and child-certificate cache operations.
impl RocksStore {
pub fn get_vcir(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<ValidatedCaInstanceResult>> {
let cf = self.cf(CF_VCIR)?;
let key = vcir_key(manifest_rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let vcir = decode_cbor::<ValidatedCaInstanceResult>(&bytes, "vcir")?;
vcir.validate_internal()?;
Ok(Some(vcir))
}
pub fn get_vcir_failed_fetch_reuse_identity(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<VcirFailedFetchReuseIdentity>> {
let cf = self.cf(CF_VCIR_FAILED_FETCH_REUSE_IDENTITY)?;
let key = vcir_failed_fetch_reuse_identity_key(manifest_rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let identity = decode_cbor::<VcirFailedFetchReuseIdentity>(
&bytes,
"vcir_failed_fetch_reuse_identity",
)?;
identity.validate_internal()?;
Ok(Some(identity))
}
pub fn get_manifest_replay_meta(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<ManifestReplayMeta>> {
let cf = self.cf(CF_MANIFEST_REPLAY_META)?;
let key = manifest_replay_meta_key(manifest_rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let meta = decode_cbor::<ManifestReplayMeta>(&bytes, "manifest_replay_meta")?;
meta.validate_internal()?;
Ok(Some(meta))
}
pub fn get_roa_cache_projection(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<RoaCacheProjection>> {
let cf = self.cf(CF_ROA_CACHE_PROJECTION)?;
let key = roa_cache_projection_key(manifest_rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let projection = decode_cbor::<RoaCacheProjection>(&bytes, "roa_cache_projection")?;
projection.validate_internal()?;
Ok(Some(projection))
}
pub fn put_child_certificate_cache_projection(
&self,
projection: &ChildCertificateCacheProjection,
) -> StorageResult<()> {
projection.validate_internal()?;
let cf = self.cf(CF_CHILD_CERTIFICATE_CACHE_PROJECTION)?;
let key = child_certificate_cache_projection_key(&projection.cache_key_sha256_hex);
let value = encode_cbor(projection, "child_certificate_cache_projection")?;
self.db
.put_cf(cf, key.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn get_child_certificate_cache_projection(
&self,
cache_key_sha256_hex: &str,
) -> StorageResult<Option<ChildCertificateCacheProjection>> {
validate_sha256_hex(
"child_certificate_cache_projection.cache_key_sha256_hex",
cache_key_sha256_hex,
)?;
let cf = self.cf(CF_CHILD_CERTIFICATE_CACHE_PROJECTION)?;
let key = child_certificate_cache_projection_key(cache_key_sha256_hex);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let projection = decode_cbor::<ChildCertificateCacheProjection>(
&bytes,
"child_certificate_cache_projection",
)?;
projection.validate_internal()?;
Ok(Some(projection))
}
pub fn get_child_certificate_cache_projections_batch(
&self,
cache_key_sha256_hexes: &[String],
) -> StorageResult<Vec<Option<ChildCertificateCacheProjection>>> {
if cache_key_sha256_hexes.is_empty() {
return Ok(Vec::new());
}
for cache_key_sha256_hex in cache_key_sha256_hexes {
validate_sha256_hex(
"child_certificate_cache_projection.cache_key_sha256_hex",
cache_key_sha256_hex,
)?;
}
let cf = self.cf(CF_CHILD_CERTIFICATE_CACHE_PROJECTION)?;
let keys: Vec<String> = cache_key_sha256_hexes
.iter()
.map(|key| child_certificate_cache_projection_key(key))
.collect();
self.db
.multi_get_cf(keys.iter().map(|key| (cf, key.as_bytes())))
.into_iter()
.map(|res| {
let Some(bytes) = res.map_err(|e| StorageError::RocksDb(e.to_string()))? else {
return Ok(None);
};
let projection = decode_cbor::<ChildCertificateCacheProjection>(
&bytes,
"child_certificate_cache_projection",
)?;
projection.validate_internal()?;
Ok(Some(projection))
})
.collect()
}
fn child_certificate_cache_segment_path(&self, manifest_rsync_uri: &str) -> PathBuf {
self.child_certificate_cache_index_dir
.join(child_certificate_cache_segment_file_name(
manifest_rsync_uri,
))
}
pub fn get_child_certificate_cache_projections_mmap_segment(
&self,
manifest_rsync_uri: &str,
cache_key_sha256_hexes: &[String],
) -> StorageResult<Option<ChildCertificateCacheMmapLookup>> {
if cache_key_sha256_hexes.is_empty() {
return Ok(Some(ChildCertificateCacheMmapLookup::default()));
}
validate_non_empty(
"child_certificate_cache_mmap_segment.manifest_rsync_uri",
manifest_rsync_uri,
)?;
for cache_key_sha256_hex in cache_key_sha256_hexes {
validate_sha256_hex(
"child_certificate_cache_projection.cache_key_sha256_hex",
cache_key_sha256_hex,
)?;
}
let path = self.child_certificate_cache_segment_path(manifest_rsync_uri);
if !path.exists() {
return Ok(None);
}
let (index, _) = load_pp_cache_mmap_index(&path)?;
let file_bytes = index.file_bytes();
let mut hits = 0usize;
let mut misses = 0usize;
let mut projections = Vec::with_capacity(cache_key_sha256_hexes.len());
for cache_key_sha256_hex in cache_key_sha256_hexes {
match index.lookup(cache_key_sha256_hex) {
Some(PpCacheIndexLookup::Hit(bytes)) => {
let projection = decode_cbor::<ChildCertificateCacheProjection>(
bytes,
"child_certificate_cache_projection_mmap_segment",
)?;
projection.validate_internal()?;
hits = hits.saturating_add(1);
projections.push(Some(projection));
}
Some(PpCacheIndexLookup::Deleted) | None => {
misses = misses.saturating_add(1);
projections.push(None);
}
}
}
Ok(Some(ChildCertificateCacheMmapLookup {
projections,
hits,
misses,
file_bytes,
}))
}
pub fn write_child_certificate_cache_mmap_segment(
&self,
manifest_rsync_uri: &str,
projections: &[ChildCertificateCacheProjection],
) -> StorageResult<PpCacheIndexRefreshStats> {
validate_non_empty(
"child_certificate_cache_mmap_segment.manifest_rsync_uri",
manifest_rsync_uri,
)?;
let mut entries = Vec::with_capacity(projections.len());
for projection in projections {
projection.validate_internal()?;
entries.push((
projection.cache_key_sha256_hex.clone(),
encode_cbor(
projection,
"child_certificate_cache_projection_mmap_segment",
)?,
));
}
let path = self.child_certificate_cache_segment_path(manifest_rsync_uri);
write_pp_cache_index_atomic(&path, entries)
}
pub fn write_child_certificate_cache_mmap_segment_overlay(
&self,
manifest_rsync_uri: &str,
cache_key_sha256_hexes: &[String],
projections: &[ChildCertificateCacheProjection],
) -> StorageResult<PpCacheIndexRefreshStats> {
validate_non_empty(
"child_certificate_cache_mmap_segment.manifest_rsync_uri",
manifest_rsync_uri,
)?;
let mut dirty = HashMap::<String, Vec<u8>>::with_capacity(projections.len());
for projection in projections {
projection.validate_internal()?;
dirty.insert(
projection.cache_key_sha256_hex.clone(),
encode_cbor(
projection,
"child_certificate_cache_projection_mmap_segment",
)?,
);
}
let path = self.child_certificate_cache_segment_path(manifest_rsync_uri);
let existing = if path.exists() {
Some(load_pp_cache_mmap_index(&path)?.0)
} else {
None
};
let mut entries = Vec::with_capacity(cache_key_sha256_hexes.len());
let mut emitted = HashSet::<String>::new();
for cache_key_sha256_hex in cache_key_sha256_hexes {
validate_sha256_hex(
"child_certificate_cache_projection.cache_key_sha256_hex",
cache_key_sha256_hex,
)?;
if !emitted.insert(cache_key_sha256_hex.clone()) {
continue;
}
if let Some(value) = dirty.remove(cache_key_sha256_hex) {
entries.push((cache_key_sha256_hex.clone(), value));
continue;
}
if let Some(existing) = existing.as_ref() {
if let Some(PpCacheIndexLookup::Hit(bytes)) = existing.lookup(cache_key_sha256_hex)
{
entries.push((cache_key_sha256_hex.clone(), bytes.to_vec()));
}
}
}
for (key, value) in dirty {
if emitted.insert(key.clone()) {
entries.push((key, value));
}
}
write_pp_cache_index_atomic(&path, entries)
}
}

View File

@@ -0,0 +1,199 @@
// RocksDB lifecycle, external stores, and cache-index setup.
impl RocksStore {
pub fn create_read_only_checkpoint(source: &Path, destination: &Path) -> StorageResult<()> {
if destination.exists() {
return Err(StorageError::InvalidData {
entity: "work_db_checkpoint",
detail: format!(
"checkpoint destination already exists: {}",
destination.display()
),
});
}
if let Some(parent) = destination.parent() {
std::fs::create_dir_all(parent)
.map_err(|error| StorageError::RocksDb(error.to_string()))?;
}
let mut options = Options::default();
let blob_mode = work_db_blob_mode_from_env();
let memory_profile = work_db_memory_profile_from_env();
configure_work_db_options(&mut options, blob_mode, memory_profile);
let db = DB::open_cf_descriptors_read_only(
&options,
source,
column_family_descriptors_for_blob_mode(blob_mode),
false,
)
.map_err(|error| StorageError::RocksDb(error.to_string()))?;
let checkpoint =
Checkpoint::new(&db).map_err(|error| StorageError::RocksDb(error.to_string()))?;
checkpoint
.create_checkpoint(destination)
.map_err(|error| StorageError::RocksDb(error.to_string()))
}
pub fn open(path: &Path) -> StorageResult<Self> {
let mut base_opts = Options::default();
base_opts.create_if_missing(true);
base_opts.create_missing_column_families(true);
let blob_mode = work_db_blob_mode_from_env();
let memory_profile = work_db_memory_profile_from_env();
configure_work_db_options(&mut base_opts, blob_mode, memory_profile);
let db = DB::open_cf_descriptors(
&base_opts,
path,
column_family_descriptors_for_blob_mode(blob_mode),
)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(Self {
db,
external_raw_store: None,
external_repo_bytes: None,
publication_point_cache_index_dir: default_pp_cache_index_dir(path),
child_certificate_cache_index_dir: default_child_certificate_cache_index_dir(path),
publication_point_cache_projection_index: Mutex::new(
PublicationPointCacheProjectionIndexState::Uninitialized,
),
})
}
pub fn open_with_external_raw_store(path: &Path, raw_store_path: &Path) -> StorageResult<Self> {
Self::open_with_external_stores(path, Some(raw_store_path), None)
}
pub fn open_with_external_repo_bytes(
path: &Path,
repo_bytes_path: &Path,
) -> StorageResult<Self> {
Self::open_with_external_stores(path, None, Some(repo_bytes_path))
}
pub fn open_with_external_repo_bytes_read_only(
path: &Path,
repo_bytes_path: &Path,
) -> StorageResult<Self> {
let mut store = Self::open(path)?;
store.external_repo_bytes = Some(ExternalRepoBytesDb::open_read_only(repo_bytes_path)?);
Ok(store)
}
pub fn open_with_external_stores(
path: &Path,
raw_store_path: Option<&Path>,
repo_bytes_path: Option<&Path>,
) -> StorageResult<Self> {
let mut store = Self::open(path)?;
if let Some(raw_store_path) = raw_store_path {
store.external_raw_store = Some(ExternalRawStoreDb::open(raw_store_path)?);
}
if let Some(repo_bytes_path) = repo_bytes_path {
store.external_repo_bytes = Some(ExternalRepoBytesDb::open(repo_bytes_path)?);
}
Ok(store)
}
pub(crate) fn external_raw_store_ref(&self) -> Option<&ExternalRawStoreDb> {
self.external_raw_store.as_ref()
}
pub(crate) fn external_repo_bytes_ref(&self) -> Option<&ExternalRepoBytesDb> {
self.external_repo_bytes.as_ref()
}
pub fn memory_snapshot(&self) -> RocksDbMemorySnapshot {
let mut databases = Vec::new();
databases.push(memory_db_snapshot_for_column_families(
"work-db",
&self.db,
Some(ALL_COLUMN_FAMILY_NAMES),
));
if let Some(raw_store) = self.external_raw_store.as_ref() {
databases.push(raw_store.memory_snapshot("raw-store.db"));
}
if let Some(repo_bytes) = self.external_repo_bytes.as_ref() {
databases.push(repo_bytes.memory_snapshot("repo-bytes.db"));
}
let mut totals = RocksDbMemoryTotals::default();
for db in &databases {
totals.add_properties(&db.properties);
}
RocksDbMemorySnapshot { databases, totals }
}
pub fn publication_point_cache_mmap_index_load_stats(&self) -> Option<PpCacheIndexLoadStats> {
let guard = self.publication_point_cache_projection_index.lock().ok()?;
match &*guard {
PublicationPointCacheProjectionIndexState::LoadedMmap { load_stats, .. } => {
Some(load_stats.clone())
}
PublicationPointCacheProjectionIndexState::Uninitialized
| PublicationPointCacheProjectionIndexState::Disabled
| PublicationPointCacheProjectionIndexState::BuildingFromEmpty { .. }
| PublicationPointCacheProjectionIndexState::Loaded { .. } => None,
}
}
fn try_load_publication_point_cache_mmap_index_for_update(
&self,
reason: &'static str,
) -> StorageResult<()> {
if !pp_cache_raw_index_enabled() {
return Ok(());
}
let mut guard = self
.publication_point_cache_projection_index
.lock()
.map_err(|e| {
StorageError::RocksDb(format!("publication point cache index lock poisoned: {e}"))
})?;
if !matches!(
*guard,
PublicationPointCacheProjectionIndexState::Uninitialized
) {
return Ok(());
}
match load_pp_cache_mmap_index_set(&self.publication_point_cache_index_dir) {
Ok((mmap, stats)) => {
crate::progress_log::emit(
"publication_point_cache_mmap_index_load",
serde_json::json!({
"state": "loaded",
"reason": reason,
"entries": stats.entries,
"bytes": stats.bytes,
"file_bytes": stats.file_bytes,
"load_ms": stats.load_ms,
}),
);
*guard = PublicationPointCacheProjectionIndexState::LoadedMmap {
mmap,
dirty: HashMap::new(),
dirty_bytes: 0,
load_stats: stats,
};
}
Err(e) => {
crate::progress_log::emit(
"publication_point_cache_mmap_index_load",
serde_json::json!({
"state": "deferred_fallback_scan",
"reason": reason,
"error": e.to_string(),
}),
);
}
}
Ok(())
}
fn cf(&self, name: &'static str) -> StorageResult<&ColumnFamily> {
self.db
.cf_handle(name)
.ok_or(StorageError::MissingColumnFamily(name))
}
}

View File

@@ -0,0 +1,509 @@
// Publication-point cache projection and mmap index operations.
impl RocksStore {
pub fn get_publication_point_cache_projection(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<PublicationPointCacheProjection>> {
self.get_publication_point_cache_projection_from_db(manifest_rsync_uri)
}
pub fn get_publication_point_cache_projection_cached(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<PublicationPointCacheProjection>> {
let mut owned_bytes: Option<Arc<[u8]>> = None;
{
let mut guard = self
.publication_point_cache_projection_index
.lock()
.map_err(|e| {
StorageError::RocksDb(format!(
"publication point cache index lock poisoned: {e}"
))
})?;
if matches!(
*guard,
PublicationPointCacheProjectionIndexState::Uninitialized
) {
let load_started = std::time::Instant::now();
let raw_index_enabled = pp_cache_raw_index_enabled();
*guard = if !raw_index_enabled {
crate::progress_log::emit(
"publication_point_cache_raw_index",
serde_json::json!({
"state": "disabled_by_env",
"entries": 0,
"bytes": 0,
"load_ms": load_started.elapsed().as_millis() as u64,
}),
);
PublicationPointCacheProjectionIndexState::Disabled
} else {
match load_pp_cache_mmap_index_set(&self.publication_point_cache_index_dir) {
Ok((mmap, stats)) => {
crate::progress_log::emit(
"publication_point_cache_mmap_index_load",
serde_json::json!({
"state": "loaded",
"entries": stats.entries,
"bytes": stats.bytes,
"file_bytes": stats.file_bytes,
"load_ms": stats.load_ms,
}),
);
PublicationPointCacheProjectionIndexState::LoadedMmap {
mmap,
dirty: HashMap::new(),
dirty_bytes: 0,
load_stats: stats,
}
}
Err(e) => {
crate::progress_log::emit(
"publication_point_cache_mmap_index_load",
serde_json::json!({
"state": "fallback_scan",
"error": e.to_string(),
"load_ms": load_started.elapsed().as_millis() as u64,
}),
);
let scan_started = std::time::Instant::now();
let (index, bytes) =
self.load_publication_point_cache_projection_index()?;
if index.is_empty() {
let limit = pp_cache_raw_index_empty_build_limit_bytes();
crate::progress_log::emit(
"publication_point_cache_raw_index",
serde_json::json!({
"state": "empty_building_bounded",
"entries": 0,
"bytes": 0,
"empty_build_limit_bytes": limit,
"load_ms": scan_started.elapsed().as_millis() as u64,
}),
);
PublicationPointCacheProjectionIndexState::BuildingFromEmpty {
index,
bytes: 0,
limit,
}
} else {
crate::progress_log::emit(
"publication_point_cache_raw_index",
serde_json::json!({
"state": "loaded",
"entries": index.len(),
"bytes": bytes,
"load_ms": scan_started.elapsed().as_millis() as u64,
}),
);
PublicationPointCacheProjectionIndexState::Loaded { index, bytes }
}
}
}
};
}
match &*guard {
PublicationPointCacheProjectionIndexState::Loaded { index, .. } => {
owned_bytes = index.get(manifest_rsync_uri).cloned();
}
PublicationPointCacheProjectionIndexState::BuildingFromEmpty { index, .. } => {
owned_bytes = index.get(manifest_rsync_uri).cloned();
}
PublicationPointCacheProjectionIndexState::LoadedMmap { mmap, dirty, .. } => {
if let Some(bytes) = dirty.get(manifest_rsync_uri).cloned() {
if bytes.is_empty() {
return Ok(None);
}
owned_bytes = Some(bytes);
} else if let Some(lookup) = mmap.lookup(manifest_rsync_uri) {
match lookup {
PpCacheIndexLookup::Hit(bytes) => {
let projection = decode_cbor::<PublicationPointCacheProjection>(
bytes,
"publication_point_cache_projection",
)?;
projection.validate_internal()?;
return Ok(Some(projection));
}
PpCacheIndexLookup::Deleted => return Ok(None),
}
}
}
PublicationPointCacheProjectionIndexState::Disabled
| PublicationPointCacheProjectionIndexState::Uninitialized => {}
}
}
let bytes = owned_bytes;
let Some(bytes) = bytes else {
return Ok(None);
};
let projection = decode_cbor::<PublicationPointCacheProjection>(
bytes.as_ref(),
"publication_point_cache_projection",
)?;
projection.validate_internal()?;
Ok(Some(projection))
}
fn get_publication_point_cache_projection_from_db(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<Option<PublicationPointCacheProjection>> {
let cf = self.cf(CF_PUBLICATION_POINT_CACHE_PROJECTION)?;
let key = publication_point_cache_projection_key(manifest_rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let projection = decode_cbor::<PublicationPointCacheProjection>(
&bytes,
"publication_point_cache_projection",
)?;
projection.validate_internal()?;
Ok(Some(projection))
}
fn load_publication_point_cache_projection_index(
&self,
) -> StorageResult<(HashMap<String, Arc<[u8]>>, usize)> {
let cf = self.cf(CF_PUBLICATION_POINT_CACHE_PROJECTION)?;
let mode = IteratorMode::Start;
let mut index = HashMap::new();
let mut bytes_total = 0usize;
for res in self.db.iterator_cf(cf, mode) {
let (key, value) = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
let Some(manifest_rsync_uri) =
publication_point_cache_projection_key_manifest_uri(&key)
else {
continue;
};
bytes_total = bytes_total.saturating_add(value.len());
index.insert(manifest_rsync_uri, Arc::<[u8]>::from(value.to_vec()));
}
Ok((index, bytes_total))
}
fn load_publication_point_cache_projection_entries(
&self,
) -> StorageResult<Vec<(String, Vec<u8>)>> {
let cf = self.cf(CF_PUBLICATION_POINT_CACHE_PROJECTION)?;
let mode = IteratorMode::Start;
let mut entries = Vec::new();
for res in self.db.iterator_cf(cf, mode) {
let (key, value) = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
let Some(manifest_rsync_uri) =
publication_point_cache_projection_key_manifest_uri(&key)
else {
continue;
};
entries.push((manifest_rsync_uri, value.to_vec()));
}
Ok(entries)
}
fn apply_publication_point_cache_projection_index_action(
&self,
action: PublicationPointCacheProjectionWriteAction<'_>,
) -> StorageResult<()> {
match action {
PublicationPointCacheProjectionWriteAction::Keep => Ok(()),
PublicationPointCacheProjectionWriteAction::Write(projection) => {
self.update_publication_point_cache_projection_index(projection)
}
PublicationPointCacheProjectionWriteAction::Delete { manifest_rsync_uri } => {
self.delete_publication_point_cache_projection_index_entry(manifest_rsync_uri)
}
}
}
fn update_publication_point_cache_projection_index(
&self,
projection: &PublicationPointCacheProjection,
) -> StorageResult<()> {
self.try_load_publication_point_cache_mmap_index_for_update("write")?;
let mut guard = self
.publication_point_cache_projection_index
.lock()
.map_err(|e| {
StorageError::RocksDb(format!("publication point cache index lock poisoned: {e}"))
})?;
let bytes = encode_cbor(projection, "publication_point_cache_projection")?;
match &mut *guard {
PublicationPointCacheProjectionIndexState::Loaded {
index,
bytes: total_bytes,
} => {
if let Some(previous) = index.insert(
projection.manifest_rsync_uri.clone(),
Arc::<[u8]>::from(bytes.clone()),
) {
*total_bytes = total_bytes.saturating_sub(previous.len());
}
*total_bytes = total_bytes.saturating_add(bytes.len());
}
PublicationPointCacheProjectionIndexState::BuildingFromEmpty {
index,
bytes: total_bytes,
limit,
} => {
if let Some(previous) = index.remove(&projection.manifest_rsync_uri) {
*total_bytes = total_bytes.saturating_sub(previous.len());
}
if total_bytes.saturating_add(bytes.len()) <= *limit {
*total_bytes += bytes.len();
index.insert(
projection.manifest_rsync_uri.clone(),
Arc::<[u8]>::from(bytes),
);
} else {
*guard = PublicationPointCacheProjectionIndexState::Disabled;
}
}
PublicationPointCacheProjectionIndexState::LoadedMmap {
dirty, dirty_bytes, ..
} => {
if let Some(previous) = dirty.insert(
projection.manifest_rsync_uri.clone(),
Arc::<[u8]>::from(bytes.clone()),
) {
*dirty_bytes = dirty_bytes.saturating_sub(previous.len());
}
*dirty_bytes = dirty_bytes.saturating_add(bytes.len());
}
PublicationPointCacheProjectionIndexState::Uninitialized
| PublicationPointCacheProjectionIndexState::Disabled => {}
}
Ok(())
}
fn delete_publication_point_cache_projection_index_entry(
&self,
manifest_rsync_uri: &str,
) -> StorageResult<()> {
self.try_load_publication_point_cache_mmap_index_for_update("delete")?;
let mut guard = self
.publication_point_cache_projection_index
.lock()
.map_err(|e| {
StorageError::RocksDb(format!("publication point cache index lock poisoned: {e}"))
})?;
match &mut *guard {
PublicationPointCacheProjectionIndexState::Loaded {
index,
bytes: total_bytes,
} => {
if let Some(previous) = index.remove(manifest_rsync_uri) {
*total_bytes = total_bytes.saturating_sub(previous.len());
}
}
PublicationPointCacheProjectionIndexState::BuildingFromEmpty {
index,
bytes: total_bytes,
..
} => {
if let Some(previous) = index.remove(manifest_rsync_uri) {
*total_bytes = total_bytes.saturating_sub(previous.len());
}
}
PublicationPointCacheProjectionIndexState::LoadedMmap {
dirty, dirty_bytes, ..
} => {
if let Some(previous) =
dirty.insert(manifest_rsync_uri.to_string(), Arc::<[u8]>::from([]))
{
*dirty_bytes = dirty_bytes.saturating_sub(previous.len());
}
}
PublicationPointCacheProjectionIndexState::Uninitialized
| PublicationPointCacheProjectionIndexState::Disabled => {}
}
Ok(())
}
pub fn refresh_publication_point_cache_mmap_index(
&self,
) -> StorageResult<Option<PpCacheIndexRefreshStats>> {
if !pp_cache_raw_index_enabled() {
return Ok(None);
}
self.try_load_publication_point_cache_mmap_index_for_update("refresh")?;
enum RefreshAction {
Entries {
entries: Vec<(String, Vec<u8>)>,
write_segment: bool,
old_entries: usize,
dirty_entries: usize,
},
ScanDb,
}
let action = {
let guard = self
.publication_point_cache_projection_index
.lock()
.map_err(|e| {
StorageError::RocksDb(format!(
"publication point cache index lock poisoned: {e}"
))
})?;
match &*guard {
PublicationPointCacheProjectionIndexState::LoadedMmap { mmap, dirty, .. } => {
RefreshAction::Entries {
entries: dirty
.iter()
.map(|(key, value)| (key.clone(), value.as_ref().to_vec()))
.collect::<Vec<_>>(),
write_segment: true,
old_entries: mmap.entries(),
dirty_entries: dirty.len(),
}
}
PublicationPointCacheProjectionIndexState::Loaded { index, .. }
| PublicationPointCacheProjectionIndexState::BuildingFromEmpty { index, .. } => {
RefreshAction::Entries {
entries: index
.iter()
.map(|(key, value)| (key.clone(), value.as_ref().to_vec()))
.collect::<Vec<_>>(),
write_segment: false,
old_entries: 0,
dirty_entries: 0,
}
}
PublicationPointCacheProjectionIndexState::Disabled
| PublicationPointCacheProjectionIndexState::Uninitialized => RefreshAction::ScanDb,
}
};
let (entries, write_segment, old_entries, dirty_entries) = match action {
RefreshAction::Entries {
entries,
write_segment,
old_entries,
dirty_entries,
} => (entries, write_segment, old_entries, dirty_entries),
RefreshAction::ScanDb => (
self.load_publication_point_cache_projection_entries()?,
false,
0,
0,
),
};
if entries.is_empty() {
return Ok(None);
}
let current_path = self.publication_point_cache_index_dir.join("current.idx");
let mut stats = if write_segment && current_path.exists() {
write_pp_cache_index_segment(&self.publication_point_cache_index_dir, entries)?
} else {
write_pp_cache_index_atomic(&current_path, entries)?
};
stats.old_entries = old_entries;
stats.dirty_entries = dirty_entries;
crate::progress_log::emit(
"publication_point_cache_mmap_index_refresh",
serde_json::json!({
"state": stats.state,
"old_entries": stats.old_entries,
"dirty_entries": stats.dirty_entries,
"new_entries": stats.new_entries,
"file_bytes": stats.file_bytes,
"write_ms": stats.write_ms,
}),
);
let directory_stats =
pp_cache_index_directory_stats(&self.publication_point_cache_index_dir)?;
let compaction_reason = if directory_stats.segment_count
>= PP_CACHE_INDEX_COMPACTION_SEGMENT_THRESHOLD
{
Some(format!(
"segment_count>={PP_CACHE_INDEX_COMPACTION_SEGMENT_THRESHOLD}"
))
} else if directory_stats.total_file_bytes >= PP_CACHE_INDEX_COMPACTION_BYTES_THRESHOLD {
Some(format!(
"total_file_bytes>={PP_CACHE_INDEX_COMPACTION_BYTES_THRESHOLD}"
))
} else {
None
};
if let Some(reason) = compaction_reason {
crate::progress_log::emit(
"publication_point_cache_mmap_index_compaction",
serde_json::json!({
"state": "started",
"reason": reason,
"segment_count": directory_stats.segment_count,
"total_file_bytes": directory_stats.total_file_bytes,
}),
);
match compact_pp_cache_index(&self.publication_point_cache_index_dir) {
Ok(mut compact_stats) => {
compact_stats.compaction_reason = Some(reason);
compact_stats.old_entries = stats.old_entries;
compact_stats.dirty_entries = stats.dirty_entries;
crate::progress_log::emit(
"publication_point_cache_mmap_index_compaction",
serde_json::json!({
"state": "completed",
"reason": compact_stats.compaction_reason,
"segments_before": compact_stats.compaction_segments_before,
"total_file_bytes_before": compact_stats.compaction_total_file_bytes_before,
"live_entries": compact_stats.compaction_live_entries,
"file_bytes": compact_stats.compaction_file_bytes,
"reclaimed_bytes": compact_stats.compaction_reclaimed_bytes,
"deleted_segments": compact_stats.compaction_deleted_segments,
"compaction_ms": compact_stats.compaction_ms,
}),
);
stats.compaction_triggered = compact_stats.compaction_triggered;
stats.compaction_reason = compact_stats.compaction_reason;
stats.compaction_segments_before = compact_stats.compaction_segments_before;
stats.compaction_total_file_bytes_before =
compact_stats.compaction_total_file_bytes_before;
stats.compaction_live_entries = compact_stats.compaction_live_entries;
stats.compaction_file_bytes = compact_stats.compaction_file_bytes;
stats.compaction_reclaimed_bytes = compact_stats.compaction_reclaimed_bytes;
stats.compaction_ms = compact_stats.compaction_ms;
stats.compaction_deleted_segments = compact_stats.compaction_deleted_segments;
}
Err(e) => {
let error = e.to_string();
crate::progress_log::emit(
"publication_point_cache_mmap_index_compaction",
serde_json::json!({
"state": "failed",
"reason": reason,
"segment_count": directory_stats.segment_count,
"total_file_bytes": directory_stats.total_file_bytes,
"error": error,
}),
);
stats.compaction_triggered = true;
stats.compaction_reason = Some(reason);
stats.compaction_segments_before = directory_stats.segment_count;
stats.compaction_total_file_bytes_before = directory_stats.total_file_bytes;
stats.compaction_error = Some(error);
}
}
}
let mut guard = self
.publication_point_cache_projection_index
.lock()
.map_err(|e| {
StorageError::RocksDb(format!("publication point cache index lock poisoned: {e}"))
})?;
if let PublicationPointCacheProjectionIndexState::LoadedMmap {
dirty, dirty_bytes, ..
} = &mut *guard
{
dirty.clear();
*dirty_bytes = 0;
}
Ok(Some(stats))
}
}

View File

@@ -0,0 +1,363 @@
// Repository-view and raw-object/blob storage operations.
impl RocksStore {
pub fn put_repository_view_entry(&self, entry: &RepositoryViewEntry) -> StorageResult<()> {
entry.validate_internal()?;
let cf = self.cf(CF_REPOSITORY_VIEW)?;
let key = repository_view_key(&entry.rsync_uri);
let value = encode_cbor(entry, "repository_view")?;
self.db
.put_cf(cf, key.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn get_repository_view_entry(
&self,
rsync_uri: &str,
) -> StorageResult<Option<RepositoryViewEntry>> {
let cf = self.cf(CF_REPOSITORY_VIEW)?;
let key = repository_view_key(rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let entry = decode_cbor::<RepositoryViewEntry>(&bytes, "repository_view")?;
entry.validate_internal()?;
Ok(Some(entry))
}
pub fn delete_repository_view_entry(&self, rsync_uri: &str) -> StorageResult<()> {
let cf = self.cf(CF_REPOSITORY_VIEW)?;
let key = repository_view_key(rsync_uri);
self.db
.delete_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn put_projection_batch(
&self,
repository_view_entries: &[RepositoryViewEntry],
member_records: &[RrdpSourceMemberRecord],
owner_records: &[RrdpUriOwnerRecord],
) -> StorageResult<()> {
if repository_view_entries.is_empty()
&& member_records.is_empty()
&& owner_records.is_empty()
{
return Ok(());
}
let repo_cf = self.cf(CF_REPOSITORY_VIEW)?;
let member_cf = self.cf(CF_RRDP_SOURCE_MEMBER)?;
let owner_cf = self.cf(CF_RRDP_URI_OWNER)?;
let mut batch = WriteBatch::default();
for entry in repository_view_entries {
entry.validate_internal()?;
let key = repository_view_key(&entry.rsync_uri);
let value = encode_cbor(entry, "repository_view")?;
batch.put_cf(repo_cf, key.as_bytes(), value);
}
for record in member_records {
record.validate_internal()?;
let key = rrdp_source_member_key(&record.notify_uri, &record.rsync_uri);
let value = encode_cbor(record, "rrdp_source_member")?;
batch.put_cf(member_cf, key.as_bytes(), value);
}
for record in owner_records {
record.validate_internal()?;
let key = rrdp_uri_owner_key(&record.rsync_uri);
let value = encode_cbor(record, "rrdp_uri_owner")?;
batch.put_cf(owner_cf, key.as_bytes(), value);
}
self.write_batch(batch)
}
pub fn list_repository_view_entries_with_prefix(
&self,
rsync_uri_prefix: &str,
) -> StorageResult<Vec<RepositoryViewEntry>> {
let cf = self.cf(CF_REPOSITORY_VIEW)?;
let prefix = repository_view_prefix(rsync_uri_prefix);
let mode = IteratorMode::From(prefix.as_bytes(), Direction::Forward);
self.db
.iterator_cf(cf, mode)
.take_while(|res| match res {
Ok((key, _)) => key.starts_with(prefix.as_bytes()),
Err(_) => false,
})
.map(|res| {
let (_key, value) = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
let entry = decode_cbor::<RepositoryViewEntry>(&value, "repository_view")?;
entry.validate_internal()?;
Ok(entry)
})
.collect()
}
pub fn verify_current_repository_blobs(
&self,
batch_size: usize,
) -> StorageResult<RepositoryBlobVerificationSummary> {
if batch_size == 0 {
return Err(StorageError::InvalidData {
entity: "repository_blob_verification",
detail: "batch_size must be greater than zero".to_string(),
});
}
let cf = self.cf(CF_REPOSITORY_VIEW)?;
let prefix = repository_view_prefix("");
let mode = IteratorMode::From(prefix.as_bytes(), Direction::Forward);
let mut summary = RepositoryBlobVerificationSummary::default();
let mut batch = Vec::with_capacity(batch_size);
for item in self.db.iterator_cf(cf, mode) {
let (key, value) = item.map_err(|error| StorageError::RocksDb(error.to_string()))?;
if !key.starts_with(prefix.as_bytes()) {
break;
}
let entry = decode_cbor::<RepositoryViewEntry>(&value, "repository_view")?;
entry.validate_internal()?;
if !matches!(
entry.state,
RepositoryViewState::Present | RepositoryViewState::Replaced
) {
continue;
}
let hash = entry
.current_hash
.clone()
.ok_or(StorageError::InvalidData {
entity: "repository_blob_verification",
detail: format!("current hash missing for {}", entry.rsync_uri),
})?;
batch.push((entry.rsync_uri, hash));
if batch.len() >= batch_size {
verify_repository_blob_batch(self, &batch, &mut summary)?;
batch.clear();
}
}
if !batch.is_empty() {
verify_repository_blob_batch(self, &batch, &mut summary)?;
}
Ok(summary)
}
pub fn put_raw_by_hash_entry(&self, entry: &RawByHashEntry) -> StorageResult<()> {
entry.validate_internal()?;
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.put_raw_entry(entry);
}
let cf = self.cf(CF_RAW_BY_HASH)?;
let key = raw_by_hash_key(&entry.sha256_hex);
let value = encode_cbor(entry, "raw_by_hash")?;
self.db
.put_cf(cf, key.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn put_raw_by_hash_entries_batch(&self, entries: &[RawByHashEntry]) -> StorageResult<()> {
if entries.is_empty() {
return Ok(());
}
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.put_raw_entries_batch(entries);
}
let cf = self.cf(CF_RAW_BY_HASH)?;
let mut batch = WriteBatch::default();
for entry in entries {
entry.validate_internal()?;
let key = raw_by_hash_key(&entry.sha256_hex);
let value = encode_cbor(entry, "raw_by_hash")?;
batch.put_cf(cf, key.as_bytes(), value);
}
self.write_batch(batch)
}
pub fn put_raw_by_hash_entries_batch_unchecked(
&self,
entries: &[RawByHashEntry],
) -> StorageResult<()> {
if entries.is_empty() {
return Ok(());
}
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.put_raw_entries_batch(entries);
}
let cf = self.cf(CF_RAW_BY_HASH)?;
let mut batch = WriteBatch::default();
for entry in entries {
let key = raw_by_hash_key(&entry.sha256_hex);
let value = encode_cbor(entry, "raw_by_hash")?;
batch.put_cf(cf, key.as_bytes(), value);
}
self.write_batch(batch)
}
pub fn put_blob_bytes_batch(&self, blobs: &[(String, Vec<u8>)]) -> StorageResult<()> {
if blobs.is_empty() {
return Ok(());
}
if let Some(repo_bytes) = self.external_repo_bytes.as_ref() {
if repo_bytes.is_read_only() {
return repo_bytes.require_existing_blob_bytes_batch(blobs);
}
return repo_bytes.put_blob_bytes_batch(blobs);
}
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.put_blob_bytes_batch(blobs);
}
let cf = self.cf(CF_RAW_BLOB)?;
let mut batch = WriteBatch::default();
for (sha256_hex, bytes) in blobs {
validate_sha256_hex("raw_blob.sha256_hex", sha256_hex)?;
if bytes.is_empty() {
return Err(StorageError::InvalidData {
entity: "raw_blob",
detail: "bytes must not be empty".to_string(),
});
}
let key = raw_blob_key(sha256_hex);
batch.put_cf(cf, key.as_bytes(), bytes.as_slice());
}
self.write_batch(batch)
}
pub fn delete_raw_by_hash_entry(&self, sha256_hex: &str) -> StorageResult<()> {
validate_sha256_hex("raw_by_hash.sha256_hex", sha256_hex)?;
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.delete_raw_entry(sha256_hex);
}
let cf = self.cf(CF_RAW_BY_HASH)?;
let key = raw_by_hash_key(sha256_hex);
self.db
.delete_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn get_raw_by_hash_entry(&self, sha256_hex: &str) -> StorageResult<Option<RawByHashEntry>> {
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.get_raw_entry(sha256_hex);
}
let cf = self.cf(CF_RAW_BY_HASH)?;
let key = raw_by_hash_key(sha256_hex);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let entry = decode_cbor::<RawByHashEntry>(&bytes, "raw_by_hash")?;
entry.validate_internal()?;
Ok(Some(entry))
}
pub fn get_raw_by_hash_entries_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<RawByHashEntry>>> {
if sha256_hexes.is_empty() {
return Ok(Vec::new());
}
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.get_raw_entries_batch(sha256_hexes);
}
let cf = self.cf(CF_RAW_BY_HASH)?;
let keys: Vec<String> = sha256_hexes
.iter()
.map(|hash| raw_by_hash_key(hash))
.collect();
self.db
.multi_get_cf(keys.iter().map(|key| (cf, key.as_bytes())))
.into_iter()
.map(|res| {
let maybe = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
match maybe {
Some(bytes) => {
let entry = decode_cbor::<RawByHashEntry>(&bytes, "raw_by_hash")?;
entry.validate_internal()?;
Ok(Some(entry))
}
None => Ok(None),
}
})
.collect()
}
pub fn get_blob_bytes(&self, sha256_hex: &str) -> StorageResult<Option<Vec<u8>>> {
if let Some(repo_bytes) = self.external_repo_bytes.as_ref() {
return repo_bytes.get_blob_bytes(sha256_hex);
}
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.get_blob_bytes(sha256_hex);
}
validate_sha256_hex("raw_blob.sha256_hex", sha256_hex)?;
let cf = self.cf(CF_RAW_BLOB)?;
let key = raw_blob_key(sha256_hex);
if let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
{
return Ok(Some(bytes));
}
self.get_raw_by_hash_entry(sha256_hex)
.map(|entry| entry.map(|entry| entry.bytes))
}
pub fn get_blob_bytes_batch(
&self,
sha256_hexes: &[String],
) -> StorageResult<Vec<Option<Vec<u8>>>> {
if sha256_hexes.is_empty() {
return Ok(Vec::new());
}
if let Some(repo_bytes) = self.external_repo_bytes.as_ref() {
return repo_bytes.get_blob_bytes_batch(sha256_hexes);
}
if let Some(raw_store) = self.external_raw_store.as_ref() {
return raw_store.get_blob_bytes_batch(sha256_hexes);
}
let cf = self.cf(CF_RAW_BLOB)?;
let keys: Vec<String> = sha256_hexes
.iter()
.map(|hash| {
validate_sha256_hex("raw_blob.sha256_hex", hash)?;
Ok::<String, StorageError>(raw_blob_key(hash))
})
.collect::<Result<_, _>>()?;
let blob_results: Vec<Option<Vec<u8>>> = self
.db
.multi_get_cf(keys.iter().map(|key| (cf, key.as_bytes())))
.into_iter()
.map(|res| res.map_err(|e| StorageError::RocksDb(e.to_string())))
.collect::<Result<_, _>>()?;
let mut out = Vec::with_capacity(sha256_hexes.len());
for (sha256_hex, maybe_blob) in sha256_hexes.iter().zip(blob_results.into_iter()) {
if maybe_blob.is_some() {
out.push(maybe_blob);
} else {
out.push(
self.get_raw_by_hash_entry(sha256_hex)?
.map(|entry| entry.bytes),
);
}
}
Ok(out)
}
}

View File

@@ -0,0 +1,332 @@
// Transport prefetch, VCIR listing, and RRDP metadata operations.
impl RocksStore {
pub fn put_transport_prefetch_snapshot(
&self,
snapshot: &crate::parallel::transport_prefetch::TransportPrefetchSnapshot,
) -> StorageResult<()> {
let cf = self.cf(CF_TRANSPORT_PREFETCH)?;
let value = encode_cbor(snapshot, "transport_prefetch_snapshot")?;
self.db
.put_cf(cf, TRANSPORT_PREFETCH_LAST_SNAPSHOT_KEY.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))
}
pub fn get_transport_prefetch_snapshot(
&self,
) -> StorageResult<Option<crate::parallel::transport_prefetch::TransportPrefetchSnapshot>> {
let cf = self.cf(CF_TRANSPORT_PREFETCH)?;
let Some(bytes) = self
.db
.get_cf(cf, TRANSPORT_PREFETCH_LAST_SNAPSHOT_KEY.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
decode_cbor::<crate::parallel::transport_prefetch::TransportPrefetchSnapshot>(
&bytes,
"transport_prefetch_snapshot",
)
.map(Some)
}
pub fn list_vcirs(&self) -> StorageResult<Vec<ValidatedCaInstanceResult>> {
let cf = self.cf(CF_VCIR)?;
let mode = IteratorMode::Start;
let mut out = Vec::new();
for res in self.db.iterator_cf(cf, mode) {
let (_key, bytes) = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
let vcir = decode_cbor::<ValidatedCaInstanceResult>(&bytes, "vcir")?;
vcir.validate_internal()?;
out.push(vcir);
}
Ok(out)
}
/// Remove every reusable VCIR and its failed-fetch identity while retaining
/// manifest replay metadata. The replay metadata is a fresh-validation
/// guard, not a reusable validation result; removing it changes the normal
/// traversal semantics for manifests that are not selected for replay.
///
/// This is intentionally narrower than [`Self::delete_vcir`]. It preserves
/// manifest replay metadata while clearing reusable validation records.
pub fn clear_vcir_reuse_records(&self) -> StorageResult<VcirReuseRecordsCleared> {
let vcir_cf = self.cf(CF_VCIR)?;
let identity_cf = self.cf(CF_VCIR_FAILED_FETCH_REUSE_IDENTITY)?;
let mut batch = WriteBatch::default();
let mut summary = VcirReuseRecordsCleared::default();
for entry in self.db.iterator_cf(vcir_cf, IteratorMode::Start) {
let (key, _value) = entry.map_err(|error| StorageError::RocksDb(error.to_string()))?;
batch.delete_cf(vcir_cf, key);
summary.vcir_records += 1;
}
for entry in self.db.iterator_cf(identity_cf, IteratorMode::Start) {
let (key, _value) = entry.map_err(|error| StorageError::RocksDb(error.to_string()))?;
batch.delete_cf(identity_cf, key);
summary.failed_fetch_identity_records += 1;
}
if summary.vcir_records > 0 || summary.failed_fetch_identity_records > 0 {
self.write_batch(batch)?;
}
Ok(summary)
}
pub fn summarize_vcir_storage(&self) -> StorageResult<VcirStorageSummary> {
let cf = self.cf(CF_VCIR)?;
let mode = IteratorMode::Start;
let mut summary = VcirStorageSummary::default();
for res in self.db.iterator_cf(cf, mode) {
let (_key, bytes) = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
let vcir = decode_cbor::<ValidatedCaInstanceResult>(&bytes, "vcir")?;
vcir.validate_internal()?;
summary.entry_count += 1;
let value_bytes = bytes.len() as u64;
summary.vcir_value_bytes += value_bytes;
if value_bytes > summary.vcir_value_bytes_max {
summary.vcir_value_bytes_max = value_bytes;
summary.vcir_value_bytes_max_manifest_rsync_uri =
Some(vcir.manifest_rsync_uri.clone());
}
let entry_summary = VcirStorageEntrySummary::from_vcir(&vcir, value_bytes);
summary.core_fields.add_assign(&entry_summary.core_fields);
summary
.ccr_projection
.add_assign(&entry_summary.ccr_projection);
summary
.child_resources
.add_assign(&entry_summary.child_resources);
summary.field_sizes.add_assign(&entry_summary.field_sizes);
push_top_vcir_storage_entry(
&mut summary.top_entries_by_vcir_value_bytes,
entry_summary,
);
}
summary.local_output_old_projection_bytes =
summary.field_sizes.local_output_old_projection_bytes();
summary.local_output_typed_projection_bytes =
summary.field_sizes.local_output_typed_projection_bytes();
summary.local_output_projection_saved_bytes =
summary.field_sizes.local_output_projection_saved_bytes();
Ok(summary)
}
pub fn delete_vcir(&self, manifest_rsync_uri: &str) -> StorageResult<()> {
let vcir_cf = self.cf(CF_VCIR)?;
let failed_fetch_reuse_identity_cf = self.cf(CF_VCIR_FAILED_FETCH_REUSE_IDENTITY)?;
let replay_cf = self.cf(CF_MANIFEST_REPLAY_META)?;
let projection_cf = self.cf(CF_ROA_CACHE_PROJECTION)?;
let mut batch = WriteBatch::default();
let key = vcir_key(manifest_rsync_uri);
batch.delete_cf(vcir_cf, key.as_bytes());
let failed_fetch_reuse_identity_key =
vcir_failed_fetch_reuse_identity_key(manifest_rsync_uri);
batch.delete_cf(
failed_fetch_reuse_identity_cf,
failed_fetch_reuse_identity_key.as_bytes(),
);
let replay_key = manifest_replay_meta_key(manifest_rsync_uri);
batch.delete_cf(replay_cf, replay_key.as_bytes());
let projection_key = roa_cache_projection_key(manifest_rsync_uri);
batch.delete_cf(projection_cf, projection_key.as_bytes());
self.write_batch(batch)
}
pub fn put_rrdp_source_record(&self, record: &RrdpSourceRecord) -> StorageResult<()> {
record.validate_internal()?;
let cf = self.cf(CF_RRDP_SOURCE)?;
let key = rrdp_source_key(&record.notify_uri);
let value = encode_cbor(record, "rrdp_source")?;
self.db
.put_cf(cf, key.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn get_rrdp_source_record(
&self,
notify_uri: &str,
) -> StorageResult<Option<RrdpSourceRecord>> {
let cf = self.cf(CF_RRDP_SOURCE)?;
let key = rrdp_source_key(notify_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let record = decode_cbor::<RrdpSourceRecord>(&bytes, "rrdp_source")?;
record.validate_internal()?;
Ok(Some(record))
}
pub fn put_rrdp_source_member_record(
&self,
record: &RrdpSourceMemberRecord,
) -> StorageResult<()> {
record.validate_internal()?;
let cf = self.cf(CF_RRDP_SOURCE_MEMBER)?;
let key = rrdp_source_member_key(&record.notify_uri, &record.rsync_uri);
let value = encode_cbor(record, "rrdp_source_member")?;
self.db
.put_cf(cf, key.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn get_rrdp_source_member_record(
&self,
notify_uri: &str,
rsync_uri: &str,
) -> StorageResult<Option<RrdpSourceMemberRecord>> {
let cf = self.cf(CF_RRDP_SOURCE_MEMBER)?;
let key = rrdp_source_member_key(notify_uri, rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let record = decode_cbor::<RrdpSourceMemberRecord>(&bytes, "rrdp_source_member")?;
record.validate_internal()?;
Ok(Some(record))
}
pub fn list_rrdp_source_member_records(
&self,
notify_uri: &str,
) -> StorageResult<Vec<RrdpSourceMemberRecord>> {
let cf = self.cf(CF_RRDP_SOURCE_MEMBER)?;
let prefix = rrdp_source_member_prefix(notify_uri);
let mode = IteratorMode::From(prefix.as_bytes(), Direction::Forward);
self.db
.iterator_cf(cf, mode)
.take_while(|res| match res {
Ok((key, _)) => key.starts_with(prefix.as_bytes()),
Err(_) => false,
})
.map(|res| {
let (_key, value) = res.map_err(|e| StorageError::RocksDb(e.to_string()))?;
let record = decode_cbor::<RrdpSourceMemberRecord>(&value, "rrdp_source_member")?;
record.validate_internal()?;
Ok(record)
})
.collect()
}
pub fn list_current_rrdp_source_members(
&self,
notify_uri: &str,
) -> StorageResult<Vec<RrdpSourceMemberRecord>> {
let mut records = self.list_rrdp_source_member_records(notify_uri)?;
records.retain(|record| record.present);
records.sort_by(|a, b| a.rsync_uri.cmp(&b.rsync_uri));
Ok(records)
}
pub fn is_current_rrdp_source_member(
&self,
notify_uri: &str,
rsync_uri: &str,
) -> StorageResult<bool> {
Ok(matches!(
self.get_rrdp_source_member_record(notify_uri, rsync_uri)?,
Some(record) if record.present
))
}
pub fn load_current_object_bytes_by_uri(
&self,
rsync_uri: &str,
) -> StorageResult<Option<Vec<u8>>> {
Ok(self
.load_current_object_with_hash_by_uri(rsync_uri)?
.map(|obj| obj.bytes))
}
pub fn load_current_object_with_hash_by_uri(
&self,
rsync_uri: &str,
) -> StorageResult<Option<CurrentObjectWithHash>> {
let Some(view) = self.get_repository_view_entry(rsync_uri)? else {
return Ok(None);
};
match view.state {
RepositoryViewState::Withdrawn => Ok(None),
RepositoryViewState::Present | RepositoryViewState::Replaced => {
let hash = view
.current_hash
.as_deref()
.ok_or(StorageError::InvalidData {
entity: "repository_view",
detail: format!("current_hash missing for current object URI: {rsync_uri}"),
})?;
let bytes = self
.get_blob_bytes(hash)?
.ok_or(StorageError::InvalidData {
entity: "repository_view",
detail: format!(
"blob bytes missing for current object URI: {rsync_uri} (hash={hash})"
),
})?;
let current_hash = decode_sha256_hex_32("repository_view.current_hash", hash)?;
Ok(Some(CurrentObjectWithHash {
current_hash_hex: hash.to_ascii_lowercase(),
current_hash,
bytes,
}))
}
}
}
pub fn put_rrdp_uri_owner_record(&self, record: &RrdpUriOwnerRecord) -> StorageResult<()> {
record.validate_internal()?;
let cf = self.cf(CF_RRDP_URI_OWNER)?;
let key = rrdp_uri_owner_key(&record.rsync_uri);
let value = encode_cbor(record, "rrdp_uri_owner")?;
self.db
.put_cf(cf, key.as_bytes(), value)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
pub fn get_rrdp_uri_owner_record(
&self,
rsync_uri: &str,
) -> StorageResult<Option<RrdpUriOwnerRecord>> {
let cf = self.cf(CF_RRDP_URI_OWNER)?;
let key = rrdp_uri_owner_key(rsync_uri);
let Some(bytes) = self
.db
.get_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?
else {
return Ok(None);
};
let record = decode_cbor::<RrdpUriOwnerRecord>(&bytes, "rrdp_uri_owner")?;
record.validate_internal()?;
Ok(Some(record))
}
pub fn delete_rrdp_uri_owner_record(&self, rsync_uri: &str) -> StorageResult<()> {
let cf = self.cf(CF_RRDP_URI_OWNER)?;
let key = rrdp_uri_owner_key(rsync_uri);
self.db
.delete_cf(cf, key.as_bytes())
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
#[allow(dead_code)]
pub fn write_batch(&self, batch: WriteBatch) -> StorageResult<()> {
self.db
.write(batch)
.map_err(|e| StorageError::RocksDb(e.to_string()))?;
Ok(())
}
}

View File

@@ -0,0 +1,233 @@
// VCIR replacement and failed-fetch reuse operations.
impl RocksStore {
pub fn put_vcir(&self, vcir: &ValidatedCaInstanceResult) -> StorageResult<()> {
self.put_vcir_with_publication_point_cache_projection(vcir, None)
}
pub fn put_vcir_with_failed_fetch_reuse_identity(
&self,
vcir: &ValidatedCaInstanceResult,
failed_fetch_reuse_identity: &VcirFailedFetchReuseIdentity,
) -> StorageResult<()> {
self.put_vcir_with_projection_action(
vcir,
None,
PublicationPointCacheProjectionWriteAction::Keep,
Some(failed_fetch_reuse_identity),
)
}
pub fn put_vcir_with_publication_point_cache_projection(
&self,
vcir: &ValidatedCaInstanceResult,
publication_point_projection: Option<&PublicationPointCacheProjection>,
) -> StorageResult<()> {
self.put_vcir_with_projections(vcir, None, publication_point_projection)
}
pub fn put_vcir_with_projections(
&self,
vcir: &ValidatedCaInstanceResult,
roa_cache_context: Option<&RoaCacheProjectionContext>,
publication_point_projection: Option<&PublicationPointCacheProjection>,
) -> StorageResult<()> {
let publication_point_projection_action = publication_point_projection
.map(PublicationPointCacheProjectionWriteAction::Write)
.unwrap_or(PublicationPointCacheProjectionWriteAction::Keep);
self.put_vcir_with_projection_action(
vcir,
roa_cache_context,
publication_point_projection_action,
None,
)
}
fn put_vcir_with_projection_action(
&self,
vcir: &ValidatedCaInstanceResult,
roa_cache_context: Option<&RoaCacheProjectionContext>,
publication_point_projection_action: PublicationPointCacheProjectionWriteAction<'_>,
failed_fetch_reuse_identity: Option<&VcirFailedFetchReuseIdentity>,
) -> StorageResult<()> {
vcir.validate_internal()?;
let vcir_cf = self.cf(CF_VCIR)?;
let failed_fetch_reuse_identity_cf = self.cf(CF_VCIR_FAILED_FETCH_REUSE_IDENTITY)?;
let replay_cf = self.cf(CF_MANIFEST_REPLAY_META)?;
let projection_cf = self.cf(CF_ROA_CACHE_PROJECTION)?;
let pp_projection_cf = self.cf(CF_PUBLICATION_POINT_CACHE_PROJECTION)?;
let replay_meta = ManifestReplayMeta::from_vcir(vcir);
replay_meta.validate_internal()?;
let mut batch = WriteBatch::default();
let key = vcir_key(&vcir.manifest_rsync_uri);
let value = encode_cbor(vcir, "vcir")?;
batch.put_cf(vcir_cf, key.as_bytes(), value);
write_vcir_failed_fetch_reuse_identity_to_batch(
failed_fetch_reuse_identity_cf,
&mut batch,
&vcir.manifest_rsync_uri,
failed_fetch_reuse_identity,
)?;
let replay_key = manifest_replay_meta_key(&replay_meta.manifest_rsync_uri);
let replay_value = encode_cbor(&replay_meta, "manifest_replay_meta")?;
batch.put_cf(replay_cf, replay_key.as_bytes(), replay_value);
write_roa_cache_projection_to_batch(
projection_cf,
&mut batch,
vcir,
roa_cache_context,
None,
)?;
write_publication_point_cache_projection_to_batch(
pp_projection_cf,
&mut batch,
publication_point_projection_action,
None,
)?;
self.write_batch(batch)?;
self.apply_publication_point_cache_projection_index_action(
publication_point_projection_action,
)?;
Ok(())
}
pub fn replace_vcir_and_manifest_replay_meta(
&self,
vcir: &ValidatedCaInstanceResult,
) -> StorageResult<VcirReplaceTimingBreakdown> {
self.replace_vcir_manifest_replay_meta_and_publication_point_cache_projection(vcir, None)
}
pub fn replace_vcir_manifest_replay_meta_and_publication_point_cache_projection(
&self,
vcir: &ValidatedCaInstanceResult,
publication_point_projection: Option<&PublicationPointCacheProjection>,
) -> StorageResult<VcirReplaceTimingBreakdown> {
self.replace_vcir_manifest_replay_meta_and_projections(
vcir,
None,
publication_point_projection,
)
}
pub fn replace_vcir_manifest_replay_meta_and_projections(
&self,
vcir: &ValidatedCaInstanceResult,
roa_cache_context: Option<&RoaCacheProjectionContext>,
publication_point_projection: Option<&PublicationPointCacheProjection>,
) -> StorageResult<VcirReplaceTimingBreakdown> {
let publication_point_projection_action = publication_point_projection
.map(PublicationPointCacheProjectionWriteAction::Write)
.unwrap_or(PublicationPointCacheProjectionWriteAction::Keep);
self.replace_vcir_manifest_replay_meta_and_projection_action(
vcir,
roa_cache_context,
publication_point_projection_action,
)
}
pub(crate) fn replace_vcir_manifest_replay_meta_and_projection_action(
&self,
vcir: &ValidatedCaInstanceResult,
roa_cache_context: Option<&RoaCacheProjectionContext>,
publication_point_projection_action: PublicationPointCacheProjectionWriteAction<'_>,
) -> StorageResult<VcirReplaceTimingBreakdown> {
self.replace_vcir_manifest_replay_meta_and_projection_action_with_failed_fetch_reuse_identity(
vcir,
roa_cache_context,
publication_point_projection_action,
None,
)
}
pub(crate) fn replace_vcir_manifest_replay_meta_and_projection_action_with_failed_fetch_reuse_identity(
&self,
vcir: &ValidatedCaInstanceResult,
roa_cache_context: Option<&RoaCacheProjectionContext>,
publication_point_projection_action: PublicationPointCacheProjectionWriteAction<'_>,
failed_fetch_reuse_identity: Option<&VcirFailedFetchReuseIdentity>,
) -> StorageResult<VcirReplaceTimingBreakdown> {
let mut timing = VcirReplaceTimingBreakdown {
rss_before_kb: process_vm_rss_kb(),
..VcirReplaceTimingBreakdown::default()
};
let validate_started = std::time::Instant::now();
vcir.validate_internal()?;
timing.validate_ms = validate_started.elapsed().as_millis() as u64;
timing.rss_after_validate_kb = process_vm_rss_kb();
timing.field_sizes = VcirFieldSizeBreakdown::from_vcir(vcir);
let batch_build_started = std::time::Instant::now();
let vcir_cf = self.cf(CF_VCIR)?;
let failed_fetch_reuse_identity_cf = self.cf(CF_VCIR_FAILED_FETCH_REUSE_IDENTITY)?;
let replay_cf = self.cf(CF_MANIFEST_REPLAY_META)?;
let projection_cf = self.cf(CF_ROA_CACHE_PROJECTION)?;
let pp_projection_cf = self.cf(CF_PUBLICATION_POINT_CACHE_PROJECTION)?;
let mut batch = WriteBatch::default();
let vcir_key = vcir_key(&vcir.manifest_rsync_uri);
let vcir_encode_started = std::time::Instant::now();
let vcir_value = encode_cbor(vcir, "vcir")?;
timing.vcir_encode_ms = vcir_encode_started.elapsed().as_millis() as u64;
timing.vcir_value_bytes = vcir_value.len() as u64;
batch.put_cf(vcir_cf, vcir_key.as_bytes(), vcir_value);
timing.rss_after_vcir_encode_kb = process_vm_rss_kb();
write_vcir_failed_fetch_reuse_identity_to_batch(
failed_fetch_reuse_identity_cf,
&mut batch,
&vcir.manifest_rsync_uri,
failed_fetch_reuse_identity,
)?;
let replay_meta_encode_started = std::time::Instant::now();
let replay_meta = ManifestReplayMeta::from_vcir(vcir);
replay_meta.validate_internal()?;
let replay_key = manifest_replay_meta_key(&replay_meta.manifest_rsync_uri);
let replay_value = encode_cbor(&replay_meta, "manifest_replay_meta")?;
timing.replay_meta_encode_ms = replay_meta_encode_started.elapsed().as_millis() as u64;
timing.replay_meta_value_bytes = replay_value.len() as u64;
batch.put_cf(replay_cf, replay_key.as_bytes(), replay_value);
timing.rss_after_replay_meta_encode_kb = process_vm_rss_kb();
let projection_encode_started = std::time::Instant::now();
write_roa_cache_projection_to_batch(
projection_cf,
&mut batch,
vcir,
roa_cache_context,
Some(&mut timing),
)?;
timing.roa_cache_projection_encode_ms =
projection_encode_started.elapsed().as_millis() as u64;
timing.rss_after_roa_cache_projection_encode_kb = process_vm_rss_kb();
let pp_projection_encode_started = std::time::Instant::now();
write_publication_point_cache_projection_to_batch(
pp_projection_cf,
&mut batch,
publication_point_projection_action,
Some(&mut timing),
)?;
timing.publication_point_cache_projection_encode_ms =
pp_projection_encode_started.elapsed().as_millis() as u64;
timing.rss_after_publication_point_cache_projection_encode_kb = process_vm_rss_kb();
timing.total_encoded_bytes = timing.vcir_value_bytes
+ timing.replay_meta_value_bytes
+ timing.roa_cache_projection_value_bytes
+ timing.publication_point_cache_projection_value_bytes;
timing.batch_build_ms = batch_build_started.elapsed().as_millis() as u64;
let write_batch_started = std::time::Instant::now();
self.write_batch(batch)?;
self.apply_publication_point_cache_projection_index_action(
publication_point_projection_action,
)?;
timing.write_batch_ms = write_batch_started.elapsed().as_millis() as u64;
timing.rss_after_write_batch_kb = process_vm_rss_kb();
Ok(timing)
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,357 @@
// Storage test group: child cache.
#[test]
fn get_child_certificate_cache_projections_batch_preserves_order_and_misses() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let first_key = sha256_hex(b"child-cache-key-first");
let missing_key = sha256_hex(b"child-cache-key-missing");
let second_key = sha256_hex(b"child-cache-key-second");
let first_hash = sha256_hex(b"first-child-cert");
let second_hash = sha256_hex(b"second-child-cert");
let first = sample_child_certificate_cache_projection(
first_key.clone(),
"rsync://example.test/repo/first.cer",
&first_hash,
);
let second = sample_child_certificate_cache_projection(
second_key.clone(),
"rsync://example.test/repo/second.cer",
&second_hash,
);
store
.put_child_certificate_cache_projection(&first)
.expect("put first projection");
store
.put_child_certificate_cache_projection(&second)
.expect("put second projection");
let got = store
.get_child_certificate_cache_projections_batch(&[
second_key.clone(),
missing_key,
first_key.clone(),
])
.expect("batch get child projections");
assert_eq!(got.len(), 3);
assert_eq!(
got[0]
.as_ref()
.expect("second projection")
.child_cert_sha256_hex,
second_hash
);
assert!(got[1].is_none());
assert_eq!(
got[2]
.as_ref()
.expect("first projection")
.child_cert_sha256_hex,
first_hash
);
}
#[test]
fn child_certificate_cache_mmap_segment_preserves_order_and_misses() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let manifest_uri = "rsync://example.test/repo/parent.mft";
let first_key = sha256_hex(b"child-cache-mmap-key-first");
let missing_key = sha256_hex(b"child-cache-mmap-key-missing");
let second_key = sha256_hex(b"child-cache-mmap-key-second");
let first_hash = sha256_hex(b"first-child-cert-mmap");
let second_hash = sha256_hex(b"second-child-cert-mmap");
let first = sample_child_certificate_cache_projection(
first_key.clone(),
"rsync://example.test/repo/first.cer",
&first_hash,
);
let second = sample_child_certificate_cache_projection(
second_key.clone(),
"rsync://example.test/repo/second.cer",
&second_hash,
);
assert!(
store
.get_child_certificate_cache_projections_mmap_segment(
manifest_uri,
&[first_key.clone()]
)
.expect("missing segment lookup")
.is_none()
);
let write_stats = store
.write_child_certificate_cache_mmap_segment(manifest_uri, &[first.clone(), second.clone()])
.expect("write child projection segment");
assert_eq!(write_stats.new_entries, 2);
assert!(write_stats.file_bytes > 0);
let got = store
.get_child_certificate_cache_projections_mmap_segment(
manifest_uri,
&[second_key.clone(), missing_key, first_key.clone()],
)
.expect("lookup child projection segment")
.expect("segment exists");
assert_eq!(got.hits, 2);
assert_eq!(got.misses, 1);
assert!(got.file_bytes > 0);
assert_eq!(got.projections.len(), 3);
assert_eq!(
got.projections[0]
.as_ref()
.expect("second projection")
.child_cert_sha256_hex,
second_hash
);
assert!(got.projections[1].is_none());
assert_eq!(
got.projections[2]
.as_ref()
.expect("first projection")
.child_cert_sha256_hex,
first_hash
);
}
#[test]
fn child_certificate_cache_mmap_segment_overlay_preserves_existing_values() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let manifest_uri = "rsync://example.test/repo/parent-overlay.mft";
let first_key = sha256_hex(b"child-cache-overlay-key-first");
let second_key = sha256_hex(b"child-cache-overlay-key-second");
let missing_key = sha256_hex(b"child-cache-overlay-key-missing");
let first_hash = sha256_hex(b"first-child-cert-overlay");
let old_second_hash = sha256_hex(b"old-second-child-cert-overlay");
let new_second_hash = sha256_hex(b"new-second-child-cert-overlay");
let first = sample_child_certificate_cache_projection(
first_key.clone(),
"rsync://example.test/repo/first-overlay.cer",
&first_hash,
);
let old_second = sample_child_certificate_cache_projection(
second_key.clone(),
"rsync://example.test/repo/second-overlay.cer",
&old_second_hash,
);
let new_second = sample_child_certificate_cache_projection(
second_key.clone(),
"rsync://example.test/repo/second-overlay.cer",
&new_second_hash,
);
store
.write_child_certificate_cache_mmap_segment(
manifest_uri,
&[first.clone(), old_second.clone()],
)
.expect("write initial segment");
let stats = store
.write_child_certificate_cache_mmap_segment_overlay(
manifest_uri,
&[first_key.clone(), second_key.clone(), missing_key.clone()],
std::slice::from_ref(&new_second),
)
.expect("write segment overlay");
assert_eq!(stats.new_entries, 2);
let got = store
.get_child_certificate_cache_projections_mmap_segment(
manifest_uri,
&[first_key.clone(), second_key.clone(), missing_key],
)
.expect("lookup segment")
.expect("segment exists");
assert_eq!(got.hits, 2);
assert_eq!(got.misses, 1);
assert_eq!(
got.projections[0]
.as_ref()
.expect("first projection")
.child_cert_sha256_hex,
first_hash
);
assert_eq!(
got.projections[1]
.as_ref()
.expect("updated second projection")
.child_cert_sha256_hex,
new_second_hash
);
assert!(got.projections[2].is_none());
}
#[test]
fn storage_helpers_cover_optional_validation_paths() {
let withdrawn = RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/withdrawn.cer".to_string(),
current_hash: Some(sha256_hex(b"withdrawn")),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("cer".to_string()),
state: RepositoryViewState::Withdrawn,
};
withdrawn
.validate_internal()
.expect("withdrawn repository view validates");
let raw = RawByHashEntry::from_bytes(sha256_hex(b"helper-bytes"), b"helper-bytes".to_vec());
raw.validate_internal()
.expect("raw_by_hash helper validates");
let empty_raw = RawByHashEntry {
sha256_hex: sha256_hex(b"x"),
bytes: Vec::new(),
origin_uris: Vec::new(),
object_type: None,
encoding: None,
};
let err = empty_raw
.validate_internal()
.expect_err("empty raw bytes must fail");
assert!(err.to_string().contains("bytes must not be empty"));
let duplicate_origin_raw = RawByHashEntry {
sha256_hex: sha256_hex(b"dup-origin"),
bytes: b"dup-origin".to_vec(),
origin_uris: vec![
"rsync://example.test/repo/object.cer".to_string(),
"rsync://example.test/repo/object.cer".to_string(),
],
object_type: Some("cer".to_string()),
encoding: Some("der".to_string()),
};
let err = duplicate_origin_raw
.validate_internal()
.expect_err("duplicate origin URI must fail");
assert!(err.to_string().contains("duplicate origin URI"));
}
#[test]
fn rrdp_source_optional_fields_and_owner_without_hash_validate() {
let source = RrdpSourceRecord {
notify_uri: "https://rrdp.example.test/notification.xml".to_string(),
last_session_id: None,
last_serial: None,
first_seen_at: pack_time(0),
last_seen_at: pack_time(1),
last_sync_at: None,
sync_state: RrdpSourceSyncState::Empty,
last_snapshot_uri: None,
last_snapshot_hash: None,
last_error: Some("network timeout".to_string()),
};
source
.validate_internal()
.expect("source with optional fields validates");
let owner = RrdpUriOwnerRecord {
rsync_uri: "rsync://example.test/repo/object.cer".to_string(),
notify_uri: "https://rrdp.example.test/notification.xml".to_string(),
current_hash: None,
last_confirmed_session_id: "session-1".to_string(),
last_confirmed_serial: 5,
last_changed_at: pack_time(5),
owner_state: RrdpUriOwnerState::Withdrawn,
};
owner
.validate_internal()
.expect("owner without hash validates when withdrawn");
}
#[test]
fn rrdp_source_binding_records_roundtrip_and_prefix_iteration() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let notify_uri = "https://rrdp.example.test/notification.xml";
let source = sample_rrdp_source_record(notify_uri);
store
.put_rrdp_source_record(&source)
.expect("put rrdp source record");
let got_source = store
.get_rrdp_source_record(notify_uri)
.expect("get rrdp source record")
.expect("rrdp source exists");
assert_eq!(got_source, source);
let member1 =
sample_rrdp_source_member_record(notify_uri, "rsync://example.test/repo/a.cer", 1);
let member2 =
sample_rrdp_source_member_record(notify_uri, "rsync://example.test/repo/b.roa", 2);
let other_member = sample_rrdp_source_member_record(
"https://other.example.test/notification.xml",
"rsync://other.example.test/repo/c.cer",
3,
);
store
.put_rrdp_source_member_record(&member1)
.expect("put member1");
store
.put_rrdp_source_member_record(&member2)
.expect("put member2");
store
.put_rrdp_source_member_record(&other_member)
.expect("put other member");
let mut members = store
.list_rrdp_source_member_records(notify_uri)
.expect("list rrdp source members");
members.sort_by(|a, b| a.rsync_uri.cmp(&b.rsync_uri));
assert_eq!(members, vec![member1.clone(), member2.clone()]);
let got_member = store
.get_rrdp_source_member_record(notify_uri, &member1.rsync_uri)
.expect("get member1")
.expect("member1 exists");
assert_eq!(got_member, member1);
let owner = sample_rrdp_uri_owner_record(notify_uri, &member1.rsync_uri);
store
.put_rrdp_uri_owner_record(&owner)
.expect("put uri owner record");
let got_owner = store
.get_rrdp_uri_owner_record(&member1.rsync_uri)
.expect("get uri owner record")
.expect("uri owner exists");
assert_eq!(got_owner, owner);
store
.delete_rrdp_uri_owner_record(&member1.rsync_uri)
.expect("delete uri owner record");
assert!(
store
.get_rrdp_uri_owner_record(&member1.rsync_uri)
.expect("get deleted uri owner")
.is_none()
);
let mut invalid_source = sample_rrdp_source_record("https://invalid.example/notification.xml");
invalid_source.last_snapshot_hash = Some("bad".to_string());
let err = store
.put_rrdp_source_record(&invalid_source)
.expect_err("invalid source hash must fail");
assert!(err.to_string().contains("last_snapshot_hash"));
let invalid_member = RrdpSourceMemberRecord {
notify_uri: notify_uri.to_string(),
rsync_uri: "rsync://example.test/repo/deleted.cer".to_string(),
current_hash: None,
object_type: None,
present: true,
last_confirmed_session_id: "session-1".to_string(),
last_confirmed_serial: 10,
last_changed_at: pack_time(10),
};
let err = store
.put_rrdp_source_member_record(&invalid_member)
.expect_err("present member without hash must fail");
assert!(err.to_string().contains("current_hash is required"));
}

View File

@@ -0,0 +1,310 @@
// Storage test group: helpers and models.
fn pack_time(hour: i64) -> PackTime {
PackTime::from_utc_offset_datetime(
time::OffsetDateTime::UNIX_EPOCH + time::Duration::hours(hour),
)
}
fn sha256_hex(input: &[u8]) -> String {
hex::encode(compute_sha256_32(input))
}
fn sha256_32(input: &[u8]) -> [u8; 32] {
compute_sha256_32(input)
}
fn sample_child_certificate_cache_projection(
cache_key_sha256_hex: String,
child_cert_uri: &str,
child_cert_sha256_hex: &str,
) -> ChildCertificateCacheProjection {
ChildCertificateCacheProjection {
schema_version: CHILD_CERTIFICATE_CACHE_SCHEMA_VERSION,
algorithm_version: CHILD_CERTIFICATE_CACHE_ALGORITHM_VERSION,
cache_key_sha256_hex,
child_cert_uri: child_cert_uri.to_string(),
child_cert_sha256_hex: child_cert_sha256_hex.to_string(),
child_cert_serial: vec![1],
issuer_ca_sha256_hex: sha256_hex(b"issuer-ca"),
issuer_crl_uri: "rsync://example.test/repo/issuer.crl".to_string(),
issuer_crl_sha256_hex: sha256_hex(b"issuer-crl"),
ca_validation_context_digest: sha256_32(b"parent-context"),
validation_policy_fingerprint: sha256_32(b"policy"),
effective_not_before: pack_time(0),
effective_until: pack_time(24),
payload: ChildCertificateCachePayload::ChildCa {
child_manifest_rsync_uri: format!("{child_cert_uri}.mft"),
child_ski: "11".repeat(20),
child_rsync_base_uri: "rsync://example.test/repo/child/".to_string(),
child_publication_point_rsync_uri: "rsync://example.test/repo/child/".to_string(),
child_rrdp_notification_uri: Some("https://example.test/notify.xml".to_string()),
child_effective_ip_resources: None,
child_effective_as_resources: None,
},
}
}
#[test]
fn parse_work_db_blob_mode_accepts_supported_values() {
assert_eq!(default_work_db_blob_mode(), WorkDbBlobMode::Disabled);
assert_eq!(
parse_work_db_blob_mode("default"),
Some(WorkDbBlobMode::Disabled)
);
assert_eq!(
parse_work_db_blob_mode("current"),
Some(WorkDbBlobMode::Current)
);
assert_eq!(
parse_work_db_blob_mode("legacy"),
Some(WorkDbBlobMode::Current)
);
assert_eq!(
parse_work_db_blob_mode("disabled"),
Some(WorkDbBlobMode::Disabled)
);
assert_eq!(
parse_work_db_blob_mode("no-blob"),
Some(WorkDbBlobMode::Disabled)
);
assert_eq!(parse_work_db_blob_mode("lz4"), Some(WorkDbBlobMode::Lz4));
assert_eq!(
parse_work_db_blob_mode("blob-lz4"),
Some(WorkDbBlobMode::Lz4)
);
assert_eq!(parse_work_db_blob_mode("unexpected"), None);
}
#[test]
fn parse_work_db_memory_profile_accepts_supported_values() {
assert_eq!(
parse_work_db_memory_profile("default"),
Some(WorkDbMemoryProfile::Default)
);
assert_eq!(
parse_work_db_memory_profile("none"),
Some(WorkDbMemoryProfile::Default)
);
assert_eq!(
parse_work_db_memory_profile("compact"),
Some(WorkDbMemoryProfile::Compact)
);
assert_eq!(
parse_work_db_memory_profile("low-memory"),
Some(WorkDbMemoryProfile::Compact)
);
assert_eq!(parse_work_db_memory_profile("unexpected"), None);
}
#[test]
fn vcir_field_size_breakdown_counts_local_outputs_and_artifacts() {
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let breakdown = VcirFieldSizeBreakdown::from_vcir(&vcir);
assert_eq!(breakdown.local_output_count, 2);
assert_eq!(
breakdown.local_output_payload_json_bytes,
vcir.local_outputs
.iter()
.map(|output| output.payload_json().len() as u64)
.sum::<u64>()
);
assert_eq!(
breakdown.local_output_rule_hash_hex_bytes,
vcir.local_outputs.len() as u64 * 64
);
assert_eq!(breakdown.related_artifact_count, 2);
assert!(breakdown.related_artifact_uri_bytes > 0);
assert_eq!(breakdown.child_entry_count, 1);
assert!(breakdown.child_entry_uri_bytes > 0);
assert_eq!(
breakdown.local_output_old_projection_bytes(),
breakdown.local_output_source_type_bytes
+ breakdown.local_output_source_hash_hex_bytes
+ breakdown.local_output_source_ee_hash_hex_bytes
+ breakdown.local_output_payload_json_bytes
+ breakdown.local_output_rule_hash_hex_bytes
);
assert!(
breakdown.local_output_old_projection_bytes()
> breakdown.local_output_typed_projection_bytes()
);
}
fn sample_repository_view_entry(rsync_uri: &str, bytes: &[u8]) -> RepositoryViewEntry {
RepositoryViewEntry {
rsync_uri: rsync_uri.to_string(),
current_hash: Some(sha256_hex(bytes)),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("cer".to_string()),
state: RepositoryViewState::Present,
}
}
fn sample_raw_by_hash_entry(bytes: Vec<u8>) -> RawByHashEntry {
RawByHashEntry {
sha256_hex: sha256_hex(&bytes),
bytes,
origin_uris: vec!["rsync://example.test/repo/object.cer".to_string()],
object_type: Some("cer".to_string()),
encoding: Some("der".to_string()),
}
}
fn sample_ccr_manifest_projection(
manifest_rsync_uri: &str,
manifest_this_update: PackTime,
subordinate_skis: Vec<Vec<u8>>,
) -> VcirCcrManifestProjection {
VcirCcrManifestProjection {
manifest_rsync_uri: manifest_rsync_uri.to_string(),
manifest_sha256: vec![0x11; 32],
manifest_size: 4096,
manifest_ee_aki: vec![0x22; 20],
manifest_number_be: vec![3],
manifest_this_update,
manifest_sia_locations_der: vec![vec![
0x30, 0x11, 0x06, 0x08, 0x2b, 0x06, 0x01, 0x05, 0x05, 0x07, 0x30, 0x05, 0x86, 0x05,
b'r', b's', b'y', b'n', b'c',
]],
subordinate_skis,
}
}
fn sample_vcir(manifest_rsync_uri: &str) -> ValidatedCaInstanceResult {
let roa_bytes = b"roa-object".to_vec();
let ee_bytes = b"ee-cert".to_vec();
let child_bytes = b"child-cert".to_vec();
let child_ski = "1234567890abcdef1234567890abcdef12345678".to_string();
ValidatedCaInstanceResult {
manifest_rsync_uri: manifest_rsync_uri.to_string(),
parent_manifest_rsync_uri: Some("rsync://example.test/repo/parent/parent.mft".to_string()),
tal_id: "apnic".to_string(),
ca_subject_name: "CN=Example CA".to_string(),
ca_ski: "00112233445566778899aabbccddeeff00112233".to_string(),
issuer_ski: "ffeeddccbbaa99887766554433221100ffeeddcc".to_string(),
last_successful_validation_time: pack_time(0),
current_manifest_rsync_uri: manifest_rsync_uri.to_string(),
current_crl_rsync_uri: "rsync://example.test/repo/current.crl".to_string(),
validated_manifest_meta: ValidatedManifestMeta {
validated_manifest_number: vec![3],
validated_manifest_this_update: pack_time(0),
validated_manifest_next_update: pack_time(24),
},
ccr_manifest_projection: sample_ccr_manifest_projection(
manifest_rsync_uri,
pack_time(0),
vec![hex::decode(&child_ski).expect("decode child ski")],
),
instance_gate: VcirInstanceGate {
manifest_next_update: pack_time(24),
current_crl_next_update: pack_time(12),
self_ca_not_after: pack_time(48),
instance_effective_until: pack_time(12),
},
child_entries: vec![VcirChildEntry {
child_manifest_rsync_uri: "rsync://example.test/repo/child/child.mft".to_string(),
child_cert_rsync_uri: "rsync://example.test/repo/child/child.cer".to_string(),
child_cert_hash: sha256_hex(&child_bytes),
child_ski,
child_rsync_base_uri: "rsync://example.test/repo/child/".to_string(),
child_publication_point_rsync_uri: "rsync://example.test/repo/child/".to_string(),
child_rrdp_notification_uri: Some("https://example.test/child-notify.xml".to_string()),
child_effective_ip_resources: None,
child_effective_as_resources: None,
accepted_at_validation_time: pack_time(0),
}],
local_outputs: vec![
VcirLocalOutput {
output_type: VcirOutputType::Vrp,
item_effective_until: pack_time(12),
source_object_uri: "rsync://example.test/repo/object.roa".to_string(),
source_object_type: VcirSourceObjectType::Roa,
source_object_hash: sha256_32(&roa_bytes),
source_ee_cert_hash: sha256_32(&ee_bytes),
payload: VcirLocalOutputPayload::Vrp {
asn: 64496,
afi: crate::data_model::roa::RoaAfi::Ipv4,
prefix_len: 24,
addr: {
let mut addr = [0u8; 16];
addr[..4].copy_from_slice(&[203, 0, 113, 0]);
addr
},
max_length: 24,
},
rule_hash: sha256_32(b"vrp-rule-1"),
},
VcirLocalOutput {
output_type: VcirOutputType::Aspa,
item_effective_until: pack_time(10),
source_object_uri: "rsync://example.test/repo/object.asa".to_string(),
source_object_type: VcirSourceObjectType::Aspa,
source_object_hash: sha256_32(b"aspa-object"),
source_ee_cert_hash: sha256_32(b"aspa-ee-cert"),
payload: VcirLocalOutputPayload::Aspa {
customer_as_id: 64496,
provider_as_ids: vec![64497],
},
rule_hash: sha256_32(b"aspa-rule-1"),
},
],
related_artifacts: vec![
VcirRelatedArtifact {
artifact_role: VcirArtifactRole::Manifest,
artifact_kind: VcirArtifactKind::Mft,
uri: Some(manifest_rsync_uri.to_string()),
sha256: sha256_hex(b"manifest-object"),
object_type: Some("mft".to_string()),
validation_status: VcirArtifactValidationStatus::Accepted,
reject_reason: None,
},
VcirRelatedArtifact {
artifact_role: VcirArtifactRole::CurrentCrl,
artifact_kind: VcirArtifactKind::Crl,
uri: Some("rsync://example.test/repo/current.crl".to_string()),
sha256: sha256_hex(b"current-crl"),
object_type: Some("crl".to_string()),
validation_status: VcirArtifactValidationStatus::Accepted,
reject_reason: None,
},
],
summary: VcirSummary {
local_vrp_count: 1,
local_aspa_count: 1,
local_router_key_count: 0,
child_count: 1,
accepted_object_count: 4,
rejected_object_count: 0,
},
audit_summary: VcirAuditSummary {
failed_fetch_eligible: true,
last_failed_fetch_reason: None,
warning_count: 0,
audit_flags: vec!["validated-fresh".to_string()],
},
}
}
fn roa_cache_projection_context(vcir: &ValidatedCaInstanceResult) -> RoaCacheProjectionContext {
let roa_output = vcir
.local_outputs
.iter()
.find(|output| {
output.output_type == VcirOutputType::Vrp
&& output.source_object_type == VcirSourceObjectType::Roa
})
.expect("sample VCIR has a ROA output");
RoaCacheProjectionContext {
ca_validation_context_digest: [0x31; 32],
policy_fingerprint: [0x32; 32],
object_meta: vec![RoaCacheObjectMeta {
source_object_uri: roa_output.source_object_uri.clone(),
source_object_hash: roa_output.source_object_hash,
ee_serial: vec![0x01],
crl_uri: vcir.current_crl_rsync_uri.clone(),
earliest_safe_reuse_time: vcir.last_successful_validation_time.clone(),
}],
}
}

View File

@@ -0,0 +1,266 @@
// Storage test group: object loading.
#[test]
fn load_current_object_with_hash_by_uri_uses_internal_blob_cf_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let rsync_uri = "rsync://example.test/repo/blob-only.roa";
let bytes = b"blob-only-current-object".to_vec();
let hash = sha256_hex(&bytes);
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("put blob bytes");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: rsync_uri.to_string(),
current_hash: Some(hash.clone()),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put view");
let got = store
.load_current_object_with_hash_by_uri(rsync_uri)
.expect("load current object")
.expect("current object exists");
assert_eq!(got.current_hash_hex, hash);
assert_eq!(got.current_hash, compute_sha256_32(&bytes));
assert_eq!(got.bytes, bytes);
assert!(
store
.get_raw_by_hash_entry(&got.current_hash_hex)
.expect("get raw entry")
.is_none()
);
}
#[test]
fn load_current_object_bytes_by_uri_uses_internal_blob_cf_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let rsync_uri = "rsync://example.test/repo/blob-only-bytes.roa";
let bytes = b"blob-only-current-object-bytes".to_vec();
let hash = sha256_hex(&bytes);
store
.put_blob_bytes_batch(&[(hash, bytes.clone())])
.expect("put blob bytes");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: rsync_uri.to_string(),
current_hash: Some(sha256_hex(&bytes)),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put view");
assert_eq!(
store
.load_current_object_bytes_by_uri(rsync_uri)
.expect("load current object bytes"),
Some(bytes)
);
}
#[test]
fn pack_file_can_lazy_load_bytes_from_external_raw_store() {
let td = tempfile::tempdir().expect("tempdir");
let raw_store = std::sync::Arc::new(
ExternalRawStoreDb::open(td.path().join("raw-store.db")).expect("open raw store"),
);
let bytes = b"lazy-pack-file".to_vec();
let sha256_hex = sha256_hex(&bytes);
raw_store
.put_raw_entry(&RawByHashEntry::from_bytes(
sha256_hex.clone(),
bytes.clone(),
))
.expect("put raw entry");
let file = PackFile::from_lazy_external_raw_store(
"rsync://example.test/repo/a.roa",
sha256_hex,
compute_sha256_32(&bytes),
raw_store,
);
assert_eq!(file.bytes().expect("lazy bytes"), bytes.as_slice());
assert_eq!(file.bytes_cloned().expect("cloned bytes"), bytes);
}
#[test]
fn pack_file_can_lazy_load_bytes_from_external_repo_bytes_store() {
let td = tempfile::tempdir().expect("tempdir");
let repo_bytes_store = std::sync::Arc::new(
ExternalRepoBytesDb::open(td.path().join("repo-bytes.db")).expect("open repo bytes"),
);
let bytes = b"repo-object-pack-file".to_vec();
let sha256_hex = sha256_hex(&bytes);
repo_bytes_store
.put_blob_bytes_batch(&[(sha256_hex.clone(), bytes.clone())])
.expect("put repo bytes");
let file = PackFile::from_lazy_repo_bytes(
"rsync://example.test/repo/a.roa",
sha256_hex,
compute_sha256_32(&bytes),
repo_bytes_store,
);
assert_eq!(file.bytes().expect("lazy repo bytes"), bytes.as_slice());
assert_eq!(file.bytes_cloned().expect("cloned repo bytes"), bytes);
assert_eq!(file.compute_sha256().expect("compute sha256"), file.sha256);
}
#[test]
fn read_only_checkpoint_isolated_from_source_work_db() {
let td = tempfile::tempdir().expect("tempdir");
let source_path = td.path().join("source-work-db");
let checkpoint_path = td.path().join("checkpoint-work-db");
let uri = "rsync://example.test/repo/a.roa";
let bytes = b"checkpoint-object";
let hash = sha256_hex(bytes);
{
let source = RocksStore::open(&source_path).expect("open source");
source
.put_blob_bytes_batch(&[(hash.clone(), bytes.to_vec())])
.expect("put blob");
source
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: uri.to_string(),
current_hash: Some(hash),
repository_source: Some("fixture".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put view");
}
RocksStore::create_read_only_checkpoint(&source_path, &checkpoint_path)
.expect("create checkpoint");
let checkpoint = RocksStore::open(&checkpoint_path).expect("open checkpoint");
checkpoint
.delete_repository_view_entry(uri)
.expect("delete checkpoint entry");
drop(checkpoint);
let source = RocksStore::open(&source_path).expect("reopen source");
assert!(
source
.get_repository_view_entry(uri)
.expect("get source entry")
.is_some()
);
}
#[test]
fn read_only_external_repo_bytes_rejects_writes() {
let td = tempfile::tempdir().expect("tempdir");
let path = td.path().join("repo-bytes.db");
let hash = sha256_hex(b"repo-object");
{
let writable = ExternalRepoBytesDb::open(&path).expect("open writable repo bytes");
writable
.put_blob_bytes_batch(&[(hash.clone(), b"repo-object".to_vec())])
.expect("seed repo bytes");
}
let read_only = ExternalRepoBytesDb::open_read_only(&path).expect("open read-only repo bytes");
assert_eq!(
read_only
.get_blob_bytes(&hash)
.expect("read repo bytes")
.expect("blob"),
b"repo-object"
);
assert!(
read_only
.put_blob_bytes_batch(&[(hash, b"repo-object".to_vec())])
.is_err()
);
}
#[test]
fn read_only_store_accepts_only_idempotent_existing_blob_apply() {
let td = tempfile::tempdir().expect("tempdir");
let work_db_path = td.path().join("work-db");
let repo_bytes_path = td.path().join("repo-bytes.db");
let bytes = b"repo-object".to_vec();
let hash = sha256_hex(&bytes);
{
let repo_bytes = ExternalRepoBytesDb::open(&repo_bytes_path).expect("open repo bytes");
repo_bytes
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("seed repo bytes");
}
let store =
RocksStore::open_with_external_repo_bytes_read_only(&work_db_path, &repo_bytes_path)
.expect("open read-only store");
store
.put_blob_bytes_batch(&[(hash.clone(), bytes)])
.expect("idempotent apply");
assert!(
store
.put_blob_bytes_batch(&[(hash, b"different".to_vec())])
.is_err()
);
}
#[test]
fn repository_blob_verification_detects_missing_external_blob() {
let td = tempfile::tempdir().expect("tempdir");
let repo_bytes_path = td.path().join("repo-bytes.db");
ExternalRepoBytesDb::open(&repo_bytes_path).expect("create repo bytes");
let store = RocksStore::open_with_external_repo_bytes_read_only(
&td.path().join("work-db"),
&repo_bytes_path,
)
.expect("open work db");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/missing.roa".to_string(),
current_hash: Some(sha256_hex(b"missing")),
repository_source: Some("fixture".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put view");
let error = store
.verify_current_repository_blobs(16)
.expect_err("missing blob must fail");
assert!(error.to_string().contains("missing.roa"), "{error}");
}
#[test]
fn vcir_related_artifact_reject_reason_serde_backward_compatible() {
// Legacy cache JSON written before the reject_reason ("e") field existed.
let legacy = r#"{"r":"manifest","k":"mft","u":"rsync://example.test/a.mft","h":"0000000000000000000000000000000000000000000000000000000000000000","t":"mft","s":"accepted"}"#;
let artifact: VcirRelatedArtifact = serde_json::from_str(legacy).expect("legacy artifact");
assert_eq!(artifact.reject_reason, None);
// None reason is skipped on serialization, keeping cache entries compact.
let json_none = serde_json::to_string(&artifact).expect("serialize none");
assert!(!json_none.contains("\"e\""));
// A recorded reason round-trips through the "e" field.
let mut with_reason = artifact.clone();
with_reason.validation_status = VcirArtifactValidationStatus::Rejected;
with_reason.reject_reason = Some("bad object".to_string());
let json = serde_json::to_string(&with_reason).expect("serialize");
assert!(json.contains("\"e\":\"bad object\""));
let decoded: VcirRelatedArtifact = serde_json::from_str(&json).expect("deserialize");
assert_eq!(decoded.reject_reason.as_deref(), Some("bad object"));
// PublicationPointCacheObject conversion carries the reason both ways.
let cache_object = PublicationPointCacheObject::from_related_artifact(&with_reason);
assert_eq!(cache_object.reject_reason.as_deref(), Some("bad object"));
let back = cache_object.to_related_artifact();
assert_eq!(back.reject_reason.as_deref(), Some("bad object"));
// Legacy PublicationPointCacheObject JSON without "e" still deserializes.
let legacy_object: PublicationPointCacheObject =
serde_json::from_str(legacy).expect("legacy cache object");
assert_eq!(legacy_object.reject_reason, None);
}

View File

@@ -0,0 +1,473 @@
// Storage test group: projection.
#[test]
fn vcir_ccr_manifest_projection_validate_accepts_valid_projection() {
let projection = sample_ccr_manifest_projection(
"rsync://example.test/repo/current.mft",
pack_time(0),
vec![vec![0x33; 20], vec![0x44; 20]],
);
projection.validate_internal().expect("valid projection");
}
#[test]
fn vcir_ccr_manifest_projection_validate_rejects_invalid_fields() {
let mut bad_hash = sample_ccr_manifest_projection(
"rsync://example.test/repo/current.mft",
pack_time(0),
vec![vec![0x33; 20]],
);
bad_hash.manifest_sha256 = vec![0x11; 31];
assert!(matches!(
bad_hash.validate_internal(),
Err(StorageError::InvalidData { .. })
));
let mut bad_locations = sample_ccr_manifest_projection(
"rsync://example.test/repo/current.mft",
pack_time(0),
vec![vec![0x33; 20]],
);
bad_locations.manifest_sia_locations_der = vec![vec![0x04, 0x00]];
assert!(matches!(
bad_locations.validate_internal(),
Err(StorageError::InvalidData { .. })
));
let bad_subordinates = sample_ccr_manifest_projection(
"rsync://example.test/repo/current.mft",
pack_time(0),
vec![vec![0x44; 20], vec![0x33; 20]],
);
assert!(matches!(
bad_subordinates.validate_internal(),
Err(StorageError::InvalidData { .. })
));
}
#[test]
fn roa_cache_projection_from_vcir_keeps_only_roa_vrp_outputs() {
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let context = roa_cache_projection_context(&vcir);
let projection = RoaCacheProjection::from_vcir_with_context(&vcir, Some(&context))
.expect("projection build")
.expect("projection exists");
assert_eq!(projection.manifest_rsync_uri, vcir.manifest_rsync_uri);
assert_eq!(projection.instance_effective_until, pack_time(12));
assert_eq!(projection.crl_sha256_by_uri.len(), 1);
assert_eq!(projection.entries.len(), 1);
assert_eq!(
projection.entries[0].source_object_uri,
"rsync://example.test/repo/object.roa"
);
assert_eq!(
projection.entries[0].outputs_effective_until_unix,
12 * 3600
);
assert_eq!(projection.entries[0].outputs.len(), 1);
assert!(matches!(
projection.entries[0].outputs[0].payload,
VcirLocalOutputPayload::Vrp { .. }
));
}
#[test]
fn roa_cache_projection_groups_multiple_outputs_by_roa_uri() {
let mut vcir = sample_vcir("rsync://example.test/repo/current.mft");
let mut second = vcir.local_outputs[0].clone();
second.rule_hash = sha256_32(b"vrp-rule-2");
if let VcirLocalOutputPayload::Vrp { max_length, .. } = &mut second.payload {
*max_length = 25;
}
vcir.local_outputs.push(second);
vcir.summary.local_vrp_count = 2;
let context = roa_cache_projection_context(&vcir);
let projection = RoaCacheProjection::from_vcir_with_context(&vcir, Some(&context))
.expect("projection build")
.expect("projection exists");
assert_eq!(projection.entries.len(), 1);
assert_eq!(projection.entries[0].outputs.len(), 2);
assert_eq!(
projection.entries[0].outputs_effective_until_unix,
12 * 3600
);
}
#[test]
fn publication_point_cache_projection_roundtrips_with_vcir() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put vcir with publication point projection");
let got_vcir = store
.get_vcir(&vcir.manifest_rsync_uri)
.expect("get vcir")
.expect("vcir exists");
assert_eq!(got_vcir, vcir);
let got_projection = store
.get_publication_point_cache_projection(&vcir.manifest_rsync_uri)
.expect("get publication point projection")
.expect("projection exists");
assert_eq!(got_projection, projection);
assert_eq!(got_projection.outputs.len(), 2);
assert_eq!(got_projection.children.len(), 1);
assert_eq!(got_projection.related_objects.len(), 2);
}
#[test]
fn publication_point_cache_projection_index_updates_after_first_read() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let mut projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest-old"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build old publication point projection");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put old projection");
let got_old = store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get old projection")
.expect("old projection exists");
assert_eq!(got_old.manifest_sha256, sha256_32(b"manifest-old"));
projection.manifest_sha256 = sha256_32(b"manifest-new");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put new projection");
let got_new = store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get new projection")
.expect("new projection exists");
assert_eq!(got_new.manifest_sha256, sha256_32(b"manifest-new"));
}
#[test]
fn publication_point_cache_projection_cached_empty_db_accepts_bounded_new_entries() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
assert!(
store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("cached empty lookup")
.is_none()
);
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put projection after cached empty lookup");
assert_eq!(
store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("cached lookup sees bounded new entry")
.expect("cached projection exists"),
projection
);
assert_eq!(
store
.get_publication_point_cache_projection(&vcir.manifest_rsync_uri)
.expect("direct db lookup")
.expect("projection exists"),
projection
);
}
#[test]
fn publication_point_cache_mmap_index_refresh_roundtrips_after_reopen() {
let td = tempfile::tempdir().expect("tempdir");
let db_path = td.path().join("work-db");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
{
let store = RocksStore::open(&db_path).expect("open rocksdb");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put projection");
let stats = store
.refresh_publication_point_cache_mmap_index()
.expect("refresh mmap index")
.expect("refresh stats");
assert_eq!(stats.new_entries, 1);
assert_eq!(stats.state, "written");
}
let store = RocksStore::open(&db_path).expect("reopen rocksdb");
let got = store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get cached projection from mmap")
.expect("projection exists");
assert_eq!(got, projection);
}
#[test]
fn publication_point_cache_mmap_index_dirty_overlay_wins_and_refreshes_segment() {
let td = tempfile::tempdir().expect("tempdir");
let db_path = td.path().join("work-db");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let mut projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest-old"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
{
let store = RocksStore::open(&db_path).expect("open rocksdb");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put old projection");
store
.refresh_publication_point_cache_mmap_index()
.expect("refresh base mmap index");
}
{
let store = RocksStore::open(&db_path).expect("reopen rocksdb");
assert_eq!(
store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get old projection")
.expect("old projection exists")
.manifest_sha256,
sha256_32(b"manifest-old")
);
projection.manifest_sha256 = sha256_32(b"manifest-new");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put new projection");
assert_eq!(
store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get dirty projection")
.expect("dirty projection exists")
.manifest_sha256,
sha256_32(b"manifest-new")
);
let stats = store
.refresh_publication_point_cache_mmap_index()
.expect("refresh dirty mmap segment")
.expect("refresh stats");
assert_eq!(stats.state, "segment_written");
assert_eq!(stats.dirty_entries, 1);
}
let store = RocksStore::open(&db_path).expect("reopen rocksdb after segment");
let got = store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get refreshed projection")
.expect("projection exists");
assert_eq!(got.manifest_sha256, sha256_32(b"manifest-new"));
}
#[test]
fn publication_point_cache_mmap_index_write_before_read_refreshes_segment() {
let td = tempfile::tempdir().expect("tempdir");
let db_path = td.path().join("work-db");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let mut projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest-old"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
{
let store = RocksStore::open(&db_path).expect("open rocksdb");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put old projection");
let stats = store
.refresh_publication_point_cache_mmap_index()
.expect("refresh base mmap index")
.expect("refresh stats");
assert_eq!(stats.state, "written");
}
{
let store = RocksStore::open(&db_path).expect("reopen rocksdb");
projection.manifest_sha256 = sha256_32(b"manifest-new");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put new projection before any cached read");
let stats = store
.refresh_publication_point_cache_mmap_index()
.expect("refresh dirty mmap segment")
.expect("refresh stats");
assert_eq!(stats.state, "segment_written");
assert_eq!(stats.dirty_entries, 1);
assert_eq!(stats.new_entries, 1);
}
let store = RocksStore::open(&db_path).expect("reopen rocksdb after segment");
let got = store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("get refreshed projection")
.expect("projection exists");
assert_eq!(got.manifest_sha256, sha256_32(b"manifest-new"));
}
#[test]
fn publication_point_cache_mmap_index_delete_before_read_refreshes_tombstone_segment() {
let td = tempfile::tempdir().expect("tempdir");
let db_path = td.path().join("work-db");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
{
let store = RocksStore::open(&db_path).expect("open rocksdb");
store
.put_vcir_with_publication_point_cache_projection(&vcir, Some(&projection))
.expect("put projection");
let stats = store
.refresh_publication_point_cache_mmap_index()
.expect("refresh base mmap index")
.expect("refresh stats");
assert_eq!(stats.state, "written");
}
{
let store = RocksStore::open(&db_path).expect("reopen rocksdb");
store
.replace_vcir_manifest_replay_meta_and_projection_action(
&vcir,
None,
PublicationPointCacheProjectionWriteAction::Delete {
manifest_rsync_uri: &vcir.manifest_rsync_uri,
},
)
.expect("delete projection before any cached read");
assert!(
store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("dirty tombstone lookup")
.is_none()
);
let stats = store
.refresh_publication_point_cache_mmap_index()
.expect("refresh tombstone mmap segment")
.expect("refresh stats");
assert_eq!(stats.state, "segment_written");
assert_eq!(stats.dirty_entries, 1);
assert_eq!(stats.new_entries, 1);
}
let store = RocksStore::open(&db_path).expect("reopen rocksdb after tombstone segment");
assert!(
store
.get_publication_point_cache_projection_cached(&vcir.manifest_rsync_uri)
.expect("tombstone shadows old current index")
.is_none()
);
assert!(
store
.get_publication_point_cache_projection(&vcir.manifest_rsync_uri)
.expect("direct db lookup")
.is_none()
);
}
#[test]
fn publication_point_cache_projection_rejects_version_mismatch() {
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let mut projection = PublicationPointCacheProjection::from_vcir_with_context(
&vcir,
"rsync://example.test/repo/".to_string(),
Some("rsync://example.test/repo/ca.cer".to_string()),
sha256_32(b"ca-cert"),
sha256_32(b"manifest"),
sha256_32(b"ta-context"),
sha256_32(b"parent-context"),
sha256_32(b"policy"),
)
.expect("build publication point projection");
projection.schema_version = PUBLICATION_POINT_CACHE_SCHEMA_VERSION + 1;
assert!(matches!(
projection.validate_internal(),
Err(StorageError::InvalidData { .. })
));
}
#[test]
fn roa_cache_projection_rejects_duplicate_uri_with_different_hash() {
let mut vcir = sample_vcir("rsync://example.test/repo/current.mft");
let mut duplicate = vcir.local_outputs[0].clone();
duplicate.source_object_hash = sha256_32(b"different-roa");
duplicate.rule_hash = sha256_32(b"vrp-rule-2");
vcir.local_outputs.push(duplicate);
vcir.summary.local_vrp_count = 2;
let context = roa_cache_projection_context(&vcir);
let err = RoaCacheProjection::from_vcir_with_context(&vcir, Some(&context))
.expect_err("same ROA URI with different hash must fail");
assert!(err.to_string().contains("source object hash mismatch"));
}

View File

@@ -0,0 +1,418 @@
// Storage test group: repository.
fn sample_rrdp_source_record(notify_uri: &str) -> RrdpSourceRecord {
RrdpSourceRecord {
notify_uri: notify_uri.to_string(),
last_session_id: Some("session-1".to_string()),
last_serial: Some(42),
first_seen_at: pack_time(0),
last_seen_at: pack_time(1),
last_sync_at: Some(pack_time(1)),
sync_state: RrdpSourceSyncState::DeltaReady,
last_snapshot_uri: Some("https://rrdp.example.test/snapshot.xml".to_string()),
last_snapshot_hash: Some(sha256_hex(b"snapshot-bytes")),
last_error: None,
}
}
fn sample_rrdp_source_member_record(
notify_uri: &str,
rsync_uri: &str,
serial: u64,
) -> RrdpSourceMemberRecord {
RrdpSourceMemberRecord {
notify_uri: notify_uri.to_string(),
rsync_uri: rsync_uri.to_string(),
current_hash: Some(sha256_hex(rsync_uri.as_bytes())),
object_type: Some("cer".to_string()),
present: true,
last_confirmed_session_id: "session-1".to_string(),
last_confirmed_serial: serial,
last_changed_at: pack_time(serial as i64),
}
}
fn sample_rrdp_uri_owner_record(notify_uri: &str, rsync_uri: &str) -> RrdpUriOwnerRecord {
RrdpUriOwnerRecord {
rsync_uri: rsync_uri.to_string(),
notify_uri: notify_uri.to_string(),
current_hash: Some(sha256_hex(rsync_uri.as_bytes())),
last_confirmed_session_id: "session-1".to_string(),
last_confirmed_serial: 7,
last_changed_at: pack_time(7),
owner_state: RrdpUriOwnerState::Active,
}
}
#[test]
fn repository_view_and_raw_by_hash_roundtrip() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let entry1 = sample_repository_view_entry("rsync://example.test/repo/a.cer", b"object-a");
let entry2 = sample_repository_view_entry("rsync://example.test/repo/sub/b.roa", b"object-b");
store
.put_repository_view_entry(&entry1)
.expect("put repository view entry1");
store
.put_repository_view_entry(&entry2)
.expect("put repository view entry2");
let got1 = store
.get_repository_view_entry(&entry1.rsync_uri)
.expect("get repository view entry1")
.expect("entry1 exists");
assert_eq!(got1, entry1);
let got_prefix = store
.list_repository_view_entries_with_prefix("rsync://example.test/repo/sub/")
.expect("list repository view prefix");
assert_eq!(got_prefix, vec![entry2.clone()]);
store
.delete_repository_view_entry(&entry1.rsync_uri)
.expect("delete repository view entry1");
assert!(
store
.get_repository_view_entry(&entry1.rsync_uri)
.expect("get deleted repository view entry1")
.is_none()
);
let raw = sample_raw_by_hash_entry(b"raw-der-object".to_vec());
store
.put_raw_by_hash_entry(&raw)
.expect("put raw_by_hash entry");
let got_raw = store
.get_raw_by_hash_entry(&raw.sha256_hex)
.expect("get raw_by_hash entry")
.expect("raw entry exists");
assert_eq!(got_raw, raw);
}
#[test]
fn raw_by_hash_routes_to_external_raw_store_when_configured() {
let td = tempfile::tempdir().expect("tempdir");
let main_db = td.path().join("main-db");
let raw_db = td.path().join("raw-store.db");
let raw = sample_raw_by_hash_entry(b"external-raw".to_vec());
{
let store =
RocksStore::open_with_external_raw_store(&main_db, &raw_db).expect("open store");
store.put_raw_by_hash_entry(&raw).expect("put external raw");
let got = store
.get_raw_by_hash_entry(&raw.sha256_hex)
.expect("get external raw")
.expect("raw exists");
assert_eq!(got, raw);
}
let main_store = RocksStore::open(&main_db).expect("open main only");
assert!(
main_store
.get_raw_by_hash_entry(&raw.sha256_hex)
.expect("read main store")
.is_none(),
"main db should not contain raw entry when external raw store is configured"
);
}
#[test]
fn put_blob_bytes_batch_uses_internal_blob_cf_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let bytes = b"internal-blob-only".to_vec();
let hash = sha256_hex(&bytes);
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("put blob bytes");
assert_eq!(
store.get_blob_bytes(&hash).expect("get blob bytes"),
Some(bytes.clone())
);
assert!(
store
.get_raw_by_hash_entry(&hash)
.expect("get raw entry")
.is_none()
);
}
#[test]
fn put_blob_bytes_batch_routes_to_external_raw_store_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open_with_external_raw_store(
&td.path().join("main-db"),
&td.path().join("raw-store.db"),
)
.expect("open store");
let bytes = b"external-blob-only".to_vec();
let hash = sha256_hex(&bytes);
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("put external blob bytes");
assert_eq!(store.get_blob_bytes(&hash).unwrap(), Some(bytes));
assert!(store.get_raw_by_hash_entry(&hash).unwrap().is_none());
}
#[test]
fn repo_bytes_db_is_physically_separate_from_external_raw_store() {
let td = tempfile::tempdir().expect("tempdir");
let main_db = td.path().join("main-db");
let raw_db = td.path().join("raw-store.db");
let repo_bytes_db = td.path().join("repo-bytes.db");
let store =
RocksStore::open_with_external_stores(&main_db, Some(&raw_db), Some(&repo_bytes_db))
.expect("open store");
let repo_bytes = b"repo-object".to_vec();
let repo_hash = sha256_hex(&repo_bytes);
let raw = sample_raw_by_hash_entry(b"raw-evidence".to_vec());
store
.put_blob_bytes_batch(&[(repo_hash.clone(), repo_bytes.clone())])
.expect("put repo bytes");
store.put_raw_by_hash_entry(&raw).expect("put raw evidence");
assert_eq!(store.get_blob_bytes(&repo_hash).unwrap(), Some(repo_bytes));
assert_eq!(
store.get_raw_by_hash_entry(&raw.sha256_hex).unwrap(),
Some(raw.clone())
);
drop(store);
let raw_only = RocksStore::open_with_external_raw_store(&td.path().join("raw-reader"), &raw_db)
.expect("open raw only");
assert!(
raw_only.get_blob_bytes(&repo_hash).unwrap().is_none(),
"repo object bytes must not be written into raw-store.db"
);
let repo_only =
RocksStore::open_with_external_repo_bytes(&td.path().join("repo-reader"), &repo_bytes_db)
.expect("open repo bytes only");
assert_eq!(
repo_only.get_blob_bytes(&repo_hash).unwrap(),
Some(b"repo-object".to_vec())
);
assert!(
repo_only.get_blob_bytes(&raw.sha256_hex).unwrap().is_none(),
"raw evidence bytes must not be written into repo-bytes.db"
);
}
#[test]
fn memory_snapshot_includes_work_db_and_external_stores() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open_with_external_stores(
&td.path().join("main-db"),
Some(&td.path().join("raw-store.db")),
Some(&td.path().join("repo-bytes.db")),
)
.expect("open store");
let snapshot = store.memory_snapshot();
let labels: Vec<&str> = snapshot
.databases
.iter()
.map(|db| db.label.as_str())
.collect();
assert_eq!(labels, vec!["work-db", "raw-store.db", "repo-bytes.db"]);
assert!(
snapshot.databases[0]
.column_families
.iter()
.any(|cf| cf.name == CF_REPOSITORY_VIEW)
);
serde_json::to_value(&snapshot).expect("serialize memory snapshot");
}
#[test]
fn put_blob_bytes_batch_accepts_empty_batch_with_external_raw_store() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open_with_external_raw_store(
&td.path().join("main-db"),
&td.path().join("raw-store.db"),
)
.expect("open store");
store
.put_blob_bytes_batch(&[])
.expect("empty external blob batch should be a no-op");
}
#[test]
fn get_blob_bytes_internal_falls_back_to_raw_entry_when_blob_missing() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let raw = sample_raw_by_hash_entry(b"raw-fallback".to_vec());
store.put_raw_by_hash_entry(&raw).expect("put raw entry");
assert_eq!(
store
.get_blob_bytes(&raw.sha256_hex)
.expect("get blob bytes via raw fallback"),
Some(raw.bytes.clone())
);
}
#[test]
fn get_blob_bytes_batch_internal_prefers_blob_cf_and_falls_back_to_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let blob_bytes = b"blob-cf-object".to_vec();
let blob_hash = sha256_hex(&blob_bytes);
store
.put_blob_bytes_batch(&[(blob_hash.clone(), blob_bytes.clone())])
.expect("put blob bytes");
let raw = sample_raw_by_hash_entry(b"raw-fallback-batch".to_vec());
store.put_raw_by_hash_entry(&raw).expect("put raw fallback");
let batch = store
.get_blob_bytes_batch(&[blob_hash.clone(), raw.sha256_hex.clone(), "00".repeat(32)])
.expect("get blob bytes batch");
assert_eq!(batch, vec![Some(blob_bytes), Some(raw.bytes.clone()), None]);
}
#[test]
fn get_blob_bytes_batch_routes_to_external_raw_store_without_raw_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open_with_external_raw_store(
&td.path().join("main-db"),
&td.path().join("raw-store.db"),
)
.expect("open store");
let bytes = b"external-batch-blob".to_vec();
let hash = sha256_hex(&bytes);
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("put external blob bytes");
assert_eq!(
store
.get_blob_bytes_batch(&[hash, "00".repeat(32)])
.expect("get external blob batch"),
vec![Some(bytes), None]
);
}
#[test]
fn get_blob_bytes_rejects_invalid_hash_for_internal_store() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let err = store
.get_blob_bytes("not-a-valid-hash")
.expect_err("invalid hash must fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn get_blob_bytes_batch_rejects_invalid_hash_for_internal_store() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let err = store
.get_blob_bytes_batch(&["not-a-valid-hash".to_string()])
.expect_err("invalid hash must fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn get_blob_bytes_batch_returns_empty_for_empty_request_internal() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
assert!(
store
.get_blob_bytes_batch(&[])
.expect("empty blob batch request")
.is_empty()
);
}
#[test]
fn put_blob_bytes_batch_accepts_empty_batch() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
store
.put_blob_bytes_batch(&[])
.expect("empty blob batch should be a no-op");
}
#[test]
fn put_blob_bytes_batch_rejects_empty_bytes() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let err = store
.put_blob_bytes_batch(&[(sha256_hex(b"valid"), Vec::new())])
.expect_err("empty bytes must fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn delete_raw_by_hash_entry_internal_preserves_blob_bytes() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let bytes = b"blob-persists-after-raw-delete".to_vec();
let hash = sha256_hex(&bytes);
let raw = RawByHashEntry::from_bytes(hash.clone(), bytes.clone());
store
.put_blob_bytes_batch(&[(hash.clone(), bytes.clone())])
.expect("put blob bytes");
store.put_raw_by_hash_entry(&raw).expect("put raw entry");
store
.delete_raw_by_hash_entry(&hash)
.expect("delete raw entry only");
assert!(store.get_raw_by_hash_entry(&hash).unwrap().is_none());
assert_eq!(store.get_blob_bytes(&hash).unwrap(), Some(bytes));
}
#[test]
fn delete_raw_by_hash_entry_rejects_invalid_hash() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let err = store
.delete_raw_by_hash_entry("not-a-valid-hash")
.expect_err("invalid hash must fail");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn delete_raw_by_hash_entry_routes_to_external_raw_store() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open_with_external_raw_store(
&td.path().join("main-db"),
&td.path().join("raw-store.db"),
)
.expect("open store");
let raw = sample_raw_by_hash_entry(b"external-delete".to_vec());
store.put_raw_by_hash_entry(&raw).expect("put raw entry");
store
.delete_raw_by_hash_entry(&raw.sha256_hex)
.expect("delete external raw entry");
assert!(
store
.get_raw_by_hash_entry(&raw.sha256_hex)
.unwrap()
.is_none()
);
assert!(store.get_blob_bytes(&raw.sha256_hex).unwrap().is_none());
}

View File

@@ -0,0 +1,258 @@
// Storage test group: rrdp.
#[test]
fn projection_batch_roundtrip_writes_repository_view_member_and_owner_records() {
let dir = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(dir.path()).expect("open store");
let view = RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
current_hash: Some(hex::encode([1u8; 32])),
repository_source: Some("https://example.test/notify.xml".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
};
let member = RrdpSourceMemberRecord {
notify_uri: "https://example.test/notify.xml".to_string(),
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
current_hash: Some(hex::encode([1u8; 32])),
object_type: Some("roa".to_string()),
present: true,
last_confirmed_session_id: "session-1".to_string(),
last_confirmed_serial: 7,
last_changed_at: pack_time(1),
};
let owner = RrdpUriOwnerRecord {
rsync_uri: "rsync://example.test/repo/a.roa".to_string(),
notify_uri: "https://example.test/notify.xml".to_string(),
current_hash: Some(hex::encode([1u8; 32])),
last_confirmed_session_id: "session-1".to_string(),
last_confirmed_serial: 7,
last_changed_at: pack_time(1),
owner_state: RrdpUriOwnerState::Active,
};
store
.put_projection_batch(&[view.clone()], &[member.clone()], &[owner.clone()])
.expect("write projection batch");
assert_eq!(
store
.get_repository_view_entry(&view.rsync_uri)
.expect("get view")
.expect("present view"),
view
);
assert_eq!(
store
.get_rrdp_source_member_record(&member.notify_uri, &member.rsync_uri)
.expect("get member")
.expect("present member"),
member
);
assert_eq!(
store
.get_rrdp_uri_owner_record(&owner.rsync_uri)
.expect("get owner")
.expect("present owner"),
owner
);
}
#[test]
fn current_rrdp_source_member_helpers_filter_present_records() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let notify_uri = "https://rrdp.example.test/notification.xml";
let mut present_a =
sample_rrdp_source_member_record(notify_uri, "rsync://example.test/repo/a.cer", 1);
let mut withdrawn_b =
sample_rrdp_source_member_record(notify_uri, "rsync://example.test/repo/b.roa", 2);
withdrawn_b.present = false;
let present_c =
sample_rrdp_source_member_record(notify_uri, "rsync://example.test/repo/c.crl", 3);
let other_source = sample_rrdp_source_member_record(
"https://other.example.test/notification.xml",
"rsync://other.example.test/repo/x.cer",
4,
);
present_a.last_confirmed_serial = 10;
store
.put_rrdp_source_member_record(&present_a)
.expect("put present a");
store
.put_rrdp_source_member_record(&withdrawn_b)
.expect("put withdrawn b");
store
.put_rrdp_source_member_record(&present_c)
.expect("put present c");
store
.put_rrdp_source_member_record(&other_source)
.expect("put other source");
let members = store
.list_current_rrdp_source_members(notify_uri)
.expect("list current members");
assert_eq!(
members
.iter()
.map(|record| record.rsync_uri.as_str())
.collect::<Vec<_>>(),
vec![
"rsync://example.test/repo/a.cer",
"rsync://example.test/repo/c.crl",
]
);
assert!(
store
.is_current_rrdp_source_member(notify_uri, &present_a.rsync_uri)
.expect("current a")
);
assert!(
!store
.is_current_rrdp_source_member(notify_uri, &withdrawn_b.rsync_uri)
.expect("withdrawn b")
);
assert!(
!store
.is_current_rrdp_source_member(notify_uri, &other_source.rsync_uri)
.expect("other source")
);
}
#[test]
fn load_current_object_bytes_by_uri_uses_repository_view_and_raw_by_hash() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let present_bytes = b"present-object".to_vec();
let present_hash = sha256_hex(&present_bytes);
let mut present_raw = RawByHashEntry::from_bytes(present_hash.clone(), present_bytes.clone());
present_raw
.origin_uris
.push("rsync://example.test/repo/present.roa".to_string());
present_raw.object_type = Some("roa".to_string());
store
.put_raw_by_hash_entry(&present_raw)
.expect("put present raw");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/present.roa".to_string(),
current_hash: Some(present_hash),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put present view");
let replaced_bytes = b"replaced-object".to_vec();
let replaced_hash = sha256_hex(&replaced_bytes);
let mut replaced_raw =
RawByHashEntry::from_bytes(replaced_hash.clone(), replaced_bytes.clone());
replaced_raw
.origin_uris
.push("rsync://example.test/repo/replaced.cer".to_string());
replaced_raw.object_type = Some("cer".to_string());
store
.put_raw_by_hash_entry(&replaced_raw)
.expect("put replaced raw");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/replaced.cer".to_string(),
current_hash: Some(replaced_hash),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("cer".to_string()),
state: RepositoryViewState::Replaced,
})
.expect("put replaced view");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/withdrawn.crl".to_string(),
current_hash: Some(sha256_hex(b"withdrawn")),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("crl".to_string()),
state: RepositoryViewState::Withdrawn,
})
.expect("put withdrawn view");
assert_eq!(
store
.load_current_object_bytes_by_uri("rsync://example.test/repo/present.roa")
.expect("load present"),
Some(present_bytes)
);
assert_eq!(
store
.load_current_object_bytes_by_uri("rsync://example.test/repo/replaced.cer")
.expect("load replaced"),
Some(replaced_bytes)
);
assert_eq!(
store
.load_current_object_bytes_by_uri("rsync://example.test/repo/withdrawn.crl")
.expect("load withdrawn"),
None
);
assert_eq!(
store
.load_current_object_bytes_by_uri("rsync://example.test/repo/missing.roa")
.expect("load missing"),
None
);
}
#[test]
fn load_current_object_bytes_by_uri_errors_when_raw_by_hash_is_missing() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let rsync_uri = "rsync://example.test/repo/missing.cer";
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: rsync_uri.to_string(),
current_hash: Some(hex::encode([0x11; 32])),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("cer".to_string()),
state: RepositoryViewState::Present,
})
.expect("put view");
let err = store
.load_current_object_bytes_by_uri(rsync_uri)
.expect_err("missing raw_by_hash should error");
assert!(matches!(err, StorageError::InvalidData { .. }));
}
#[test]
fn load_current_object_with_hash_by_uri_returns_hash_and_bytes() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let rsync_uri = "rsync://example.test/repo/present.roa";
let bytes = b"present-object".to_vec();
let hash = sha256_hex(&bytes);
let mut raw = RawByHashEntry::from_bytes(hash.clone(), bytes.clone());
raw.origin_uris.push(rsync_uri.to_string());
raw.object_type = Some("roa".to_string());
store.put_raw_by_hash_entry(&raw).expect("put raw");
store
.put_repository_view_entry(&RepositoryViewEntry {
rsync_uri: rsync_uri.to_string(),
current_hash: Some(hash.clone()),
repository_source: Some("https://rrdp.example.test/notification.xml".to_string()),
object_type: Some("roa".to_string()),
state: RepositoryViewState::Present,
})
.expect("put view");
let got = store
.load_current_object_with_hash_by_uri(rsync_uri)
.expect("load current object")
.expect("current object exists");
assert_eq!(got.current_hash_hex, hash);
assert_eq!(got.current_hash, compute_sha256_32(&bytes));
assert_eq!(got.bytes, bytes);
}

View File

@@ -0,0 +1,386 @@
// Storage test group: vcir.
#[test]
fn repository_view_and_raw_by_hash_validation_errors_are_reported() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let invalid_view = RepositoryViewEntry {
rsync_uri: "rsync://example.test/repo/withdrawn.cer".to_string(),
current_hash: None,
repository_source: None,
object_type: None,
state: RepositoryViewState::Present,
};
let err = store
.put_repository_view_entry(&invalid_view)
.expect_err("missing current_hash must fail");
assert!(err.to_string().contains("current_hash is required"));
let invalid_raw = RawByHashEntry {
sha256_hex: sha256_hex(b"expected"),
bytes: b"actual".to_vec(),
origin_uris: vec!["rsync://example.test/repo/object.cer".to_string()],
object_type: None,
encoding: None,
};
let err = store
.put_raw_by_hash_entry(&invalid_raw)
.expect_err("mismatched raw_by_hash entry must fail");
assert!(err.to_string().contains("does not match bytes"));
}
#[test]
fn vcir_roundtrip_and_validation_failures_are_reported() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
let roa_context = roa_cache_projection_context(&vcir);
store
.put_vcir_with_projections(&vcir, Some(&roa_context), None)
.expect("put vcir");
let got = store
.get_vcir(&vcir.manifest_rsync_uri)
.expect("get vcir")
.expect("vcir exists");
assert_eq!(got, vcir);
let replay_meta = store
.get_manifest_replay_meta(&vcir.manifest_rsync_uri)
.expect("get manifest replay meta")
.expect("manifest replay meta exists");
assert_eq!(
replay_meta,
ManifestReplayMeta {
manifest_rsync_uri: vcir.manifest_rsync_uri.clone(),
manifest_number_be: vcir
.validated_manifest_meta
.validated_manifest_number
.clone(),
manifest_this_update: vcir
.validated_manifest_meta
.validated_manifest_this_update
.clone(),
manifest_sha256: vcir.ccr_manifest_projection.manifest_sha256.clone(),
updated_at_validation_time: vcir.last_successful_validation_time.clone(),
}
);
let projection = store
.get_roa_cache_projection(&vcir.manifest_rsync_uri)
.expect("get roa cache projection")
.expect("roa cache projection exists");
assert_eq!(projection.manifest_rsync_uri, vcir.manifest_rsync_uri);
assert_eq!(projection.entries.len(), 1);
assert_eq!(
projection.entries[0].source_object_uri,
"rsync://example.test/repo/object.roa"
);
let mut invalid = sample_vcir("rsync://example.test/repo/invalid.mft");
invalid.summary.local_vrp_count = 9;
let err = store
.put_vcir(&invalid)
.expect_err("invalid vcir must fail");
assert!(err.to_string().contains("local_vrp_count=9"));
let mut invalid = sample_vcir("rsync://example.test/repo/invalid-2.mft");
invalid.instance_gate.instance_effective_until = pack_time(11);
let err = store
.put_vcir(&invalid)
.expect_err("invalid instance gate must fail");
assert!(err.to_string().contains("instance_effective_until"));
store
.delete_vcir(&vcir.manifest_rsync_uri)
.expect("delete vcir");
assert!(
store
.get_vcir(&vcir.manifest_rsync_uri)
.expect("get deleted vcir")
.is_none()
);
assert!(
store
.get_manifest_replay_meta(&vcir.manifest_rsync_uri)
.expect("get deleted manifest replay meta")
.is_none()
);
assert!(
store
.get_roa_cache_projection(&vcir.manifest_rsync_uri)
.expect("get deleted roa cache projection")
.is_none()
);
}
#[test]
fn clear_vcir_reuse_records_preserves_manifest_replay_meta() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir = sample_vcir("rsync://example.test/repo/current.mft");
store.put_vcir(&vcir).expect("put vcir");
let replay_meta = store
.get_manifest_replay_meta(&vcir.manifest_rsync_uri)
.expect("get replay meta")
.expect("replay meta exists");
let cleared = store
.clear_vcir_reuse_records()
.expect("clear reusable VCIR records");
assert_eq!(cleared.vcir_records, 1);
assert_eq!(cleared.failed_fetch_identity_records, 0);
assert!(
store
.get_vcir(&vcir.manifest_rsync_uri)
.expect("get cleared vcir")
.is_none()
);
assert_eq!(
store
.get_manifest_replay_meta(&vcir.manifest_rsync_uri)
.expect("get preserved replay meta"),
Some(replay_meta)
);
}
#[test]
fn transport_prefetch_snapshot_roundtrips() {
use crate::parallel::transport_prefetch::{
TransportPrefetchDedupKey, TransportPrefetchMode, TransportPrefetchRepoIdentity,
TransportPrefetchRequest, TransportPrefetchRequester, TransportPrefetchSnapshot,
};
use crate::policy::SyncPreference;
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
assert!(
store
.get_transport_prefetch_snapshot()
.expect("get empty prefetch snapshot")
.is_none()
);
let snapshot = TransportPrefetchSnapshot::new(
SyncPreference::RrdpThenRsync,
vec![TransportPrefetchRequest {
dedup_key: TransportPrefetchDedupKey::RrdpNotify {
notification_uri: "https://example.test/notification.xml".to_string(),
},
rsync_scope_uri: "rsync://example.test/repo/".to_string(),
rsync_failure_scope_uri: Some("rsync://example.test/".to_string()),
repo_identity: TransportPrefetchRepoIdentity {
notification_uri: Some("https://example.test/notification.xml".to_string()),
rsync_base_uri: "rsync://example.test/repo/".to_string(),
},
mode: TransportPrefetchMode::Rrdp,
last_result: None,
last_rsync_result: None,
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
priority: 0,
requesters: vec![TransportPrefetchRequester {
tal_id: "apnic".to_string(),
rir_id: "apnic".to_string(),
parent_node_id: None,
ca_instance_handle_id: "apnic:rsync://example.test/repo/root.mft".to_string(),
publication_point_rsync_uri: "rsync://example.test/repo/".to_string(),
manifest_rsync_uri: "rsync://example.test/repo/root.mft".to_string(),
}],
}],
);
store
.put_transport_prefetch_snapshot(&snapshot)
.expect("put prefetch snapshot");
let got = store
.get_transport_prefetch_snapshot()
.expect("get prefetch snapshot")
.expect("snapshot exists");
assert_eq!(got, snapshot);
}
#[test]
fn manifest_replay_meta_validation_reports_invalid_fields() {
let mut meta = ManifestReplayMeta {
manifest_rsync_uri: "rsync://example.test/repo/current.mft".to_string(),
manifest_number_be: vec![3],
manifest_this_update: pack_time(0),
manifest_sha256: vec![0x11; 32],
updated_at_validation_time: pack_time(1),
};
meta.validate_internal().expect("valid replay meta");
meta.manifest_sha256 = vec![0x11; 31];
let err = meta
.validate_internal()
.expect_err("short manifest sha must fail");
assert!(err.to_string().contains("must be 32 bytes"));
meta.manifest_sha256 = vec![0x11; 32];
meta.manifest_number_be = vec![0, 3];
let err = meta
.validate_internal()
.expect_err("non-minimal manifest number must fail");
assert!(err.to_string().contains("minimal big-endian"));
}
#[test]
fn list_vcirs_returns_all_entries() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir1 = sample_vcir("rsync://example.test/repo/a.mft");
let vcir2 = sample_vcir("rsync://example.test/repo/b.mft");
store.put_vcir(&vcir1).expect("put vcir1");
store.put_vcir(&vcir2).expect("put vcir2");
let mut got = store.list_vcirs().expect("list vcirs");
got.sort_by(|a, b| a.manifest_rsync_uri.cmp(&b.manifest_rsync_uri));
assert_eq!(got, vec![vcir1, vcir2]);
}
#[test]
fn summarize_vcir_storage_aggregates_values_and_field_sizes() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let vcir1 = sample_vcir("rsync://example.test/repo/a.mft");
let vcir2 = sample_vcir("rsync://example.test/repo/b.mft");
store.put_vcir(&vcir1).expect("put vcir1");
store.put_vcir(&vcir2).expect("put vcir2");
let summary = store.summarize_vcir_storage().expect("summarize vcirs");
let mut expected_fields = VcirFieldSizeBreakdown::default();
expected_fields.add_assign(&VcirFieldSizeBreakdown::from_vcir(&vcir1));
expected_fields.add_assign(&VcirFieldSizeBreakdown::from_vcir(&vcir2));
assert_eq!(summary.entry_count, 2);
assert!(summary.vcir_value_bytes > 0);
assert!(summary.vcir_value_bytes_max > 0);
assert!(summary.vcir_value_bytes_max_manifest_rsync_uri.is_some());
assert_eq!(summary.top_entries_by_vcir_value_bytes.len(), 2);
assert!(
summary.top_entries_by_vcir_value_bytes[0].vcir_value_bytes
>= summary.top_entries_by_vcir_value_bytes[1].vcir_value_bytes
);
assert_eq!(summary.field_sizes, expected_fields);
assert!(summary.core_fields.manifest_rsync_uri_bytes > 0);
assert!(summary.ccr_projection.manifest_sha256_bytes > 0);
assert!(summary.child_resources.effective_ip_resource_cbor_bytes > 0);
assert_eq!(
summary.local_output_old_projection_bytes,
expected_fields.local_output_old_projection_bytes()
);
assert_eq!(
summary.local_output_typed_projection_bytes,
expected_fields.local_output_typed_projection_bytes()
);
assert_eq!(
summary.local_output_projection_saved_bytes,
expected_fields.local_output_projection_saved_bytes()
);
}
#[test]
fn replace_vcir_and_manifest_replay_meta_replaces_current_entry() {
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let mut previous = sample_vcir("rsync://example.test/repo/current.mft");
previous.local_outputs = vec![VcirLocalOutput {
output_type: VcirOutputType::Vrp,
item_effective_until: pack_time(10),
source_object_uri: "rsync://example.test/repo/old.roa".to_string(),
source_object_type: VcirSourceObjectType::Roa,
source_object_hash: sha256_32(b"old-roa"),
source_ee_cert_hash: sha256_32(b"old-ee"),
payload: VcirLocalOutputPayload::Vrp {
asn: 64496,
afi: crate::data_model::roa::RoaAfi::Ipv4,
prefix_len: 24,
addr: {
let mut addr = [0u8; 16];
addr[..4].copy_from_slice(&[203, 0, 113, 0]);
addr
},
max_length: 24,
},
rule_hash: sha256_32(b"old-rule"),
}];
previous.summary.local_vrp_count = 1;
previous.summary.local_aspa_count = 0;
previous.summary.local_router_key_count = 0;
let previous_roa_context = roa_cache_projection_context(&previous);
let previous_timing = store
.replace_vcir_manifest_replay_meta_and_projections(
&previous,
Some(&previous_roa_context),
None,
)
.expect("store previous vcir");
assert!(previous_timing.vcir_value_bytes > 0);
assert!(previous_timing.replay_meta_value_bytes > 0);
assert!(previous_timing.roa_cache_projection_value_bytes > 0);
assert_eq!(
previous_timing.total_encoded_bytes,
previous_timing.vcir_value_bytes
+ previous_timing.replay_meta_value_bytes
+ previous_timing.roa_cache_projection_value_bytes
);
assert!(
store
.get_roa_cache_projection(&previous.manifest_rsync_uri)
.expect("get previous projection")
.is_some()
);
let mut current = sample_vcir("rsync://example.test/repo/current.mft");
current.local_outputs = vec![VcirLocalOutput {
output_type: VcirOutputType::Aspa,
item_effective_until: pack_time(11),
source_object_uri: "rsync://example.test/repo/new.asa".to_string(),
source_object_type: VcirSourceObjectType::Aspa,
source_object_hash: sha256_32(b"new-aspa"),
source_ee_cert_hash: sha256_32(b"new-ee"),
payload: VcirLocalOutputPayload::Aspa {
customer_as_id: 64496,
provider_as_ids: vec![64497],
},
rule_hash: sha256_32(b"new-rule"),
}];
current.summary.local_vrp_count = 0;
current.summary.local_aspa_count = 1;
let current_timing = store
.replace_vcir_and_manifest_replay_meta(&current)
.expect("replace vcir and replay meta");
assert!(current_timing.vcir_value_bytes > 0);
assert!(current_timing.replay_meta_value_bytes > 0);
assert_eq!(current_timing.roa_cache_projection_value_bytes, 0);
assert_eq!(
current_timing.total_encoded_bytes,
current_timing.vcir_value_bytes + current_timing.replay_meta_value_bytes
);
let got = store
.get_vcir(&current.manifest_rsync_uri)
.expect("get replaced vcir")
.expect("vcir exists");
assert_eq!(got, current);
let replay_meta = store
.get_manifest_replay_meta(&current.manifest_rsync_uri)
.expect("get replaced replay meta")
.expect("replay meta exists");
assert_eq!(
replay_meta.manifest_number_be,
current.validated_manifest_meta.validated_manifest_number
);
assert_eq!(
replay_meta.manifest_sha256,
current.ccr_manifest_projection.manifest_sha256
);
assert!(
store
.get_roa_cache_projection(&current.manifest_rsync_uri)
.expect("get current projection")
.is_none()
);
}

View File

@@ -0,0 +1,32 @@
// Repository blob integrity verification helper.
fn verify_repository_blob_batch(
store: &RocksStore,
batch: &[(String, String)],
summary: &mut RepositoryBlobVerificationSummary,
) -> StorageResult<()> {
let hashes = batch
.iter()
.map(|(_, hash)| hash.clone())
.collect::<Vec<_>>();
let blobs = store.get_blob_bytes_batch(&hashes)?;
for ((uri, expected_hash), blob) in batch.iter().zip(blobs.into_iter()) {
let bytes = blob.as_ref().ok_or(StorageError::InvalidData {
entity: "repository_blob_verification",
detail: format!("blob missing for URI {uri} (hash={expected_hash})"),
})?;
let actual_hash = hex::encode(compute_sha256_32(bytes));
if !actual_hash.eq_ignore_ascii_case(expected_hash) {
return Err(StorageError::InvalidData {
entity: "repository_blob_verification",
detail: format!(
"blob hash mismatch for URI {uri}: expected={expected_hash}, actual={actual_hash}"
),
});
}
summary.current_objects += 1;
summary.bytes_verified += bytes.len() as u64;
}
summary.batches += 1;
Ok(())
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,279 @@
// Repository sync test group: delta replay.
#[test]
fn delta_replay_sync_applies_rrdp_deltas_when_base_state_matches() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_fixture,
base_archive,
base_locks,
delta_archive,
delta_locks,
notify_uri,
_fallback_notify,
module_uri,
) = build_delta_replay_fixture();
let base_index =
Arc::new(ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"));
let delta_index = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let http = PayloadDeltaReplayHttpFetcher::from_index(delta_index.clone())
.expect("build delta http fetcher");
let rsync = PayloadDeltaReplayRsyncFetcher::new(base_index, delta_index.clone());
let state = RrdpState {
session_id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string(),
serial: 10,
};
persist_rrdp_local_state(
&store,
&notify_uri,
&state,
RrdpSourceSyncState::DeltaReady,
None,
None,
)
.expect("seed base state");
let out = sync_publication_point_replay_delta(
&store,
&delta_index,
Some(&notify_uri),
&module_uri,
&http,
&rsync,
None,
None,
)
.expect("delta sync ok");
assert_eq!(out.source, RepoSyncSource::Rrdp);
assert_eq!(out.objects_written, 2);
assert_current_object(&store, "rsync://example.test/repo/a.mft", b"delta-a");
assert_current_object(&store, "rsync://example.test/repo/sub/b.roa", b"delta-b");
let new_state = load_rrdp_local_state(&store, &notify_uri)
.expect("load current state")
.expect("rrdp state present");
assert_eq!(new_state.serial, 12);
}
#[test]
fn delta_replay_sync_rejects_base_state_mismatch() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_fixture,
base_archive,
base_locks,
delta_archive,
delta_locks,
notify_uri,
_fallback_notify,
module_uri,
) = build_delta_replay_fixture();
let base_index =
Arc::new(ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"));
let delta_index = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let http = PayloadDeltaReplayHttpFetcher::from_index(delta_index.clone())
.expect("build delta http fetcher");
let rsync = PayloadDeltaReplayRsyncFetcher::new(base_index, delta_index.clone());
let err = sync_publication_point_replay_delta(
&store,
&delta_index,
Some(&notify_uri),
&module_uri,
&http,
&rsync,
None,
None,
)
.unwrap_err();
assert!(matches!(err, RepoSyncError::Replay(_)), "{err}");
}
#[test]
fn delta_replay_sync_noops_unchanged_rrdp_repo() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_fixture,
base_archive,
base_locks,
delta_archive,
delta_locks,
notify_uri,
_fallback_notify,
module_uri,
) = build_delta_replay_fixture();
let state = RrdpState {
session_id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string(),
serial: 10,
};
persist_rrdp_local_state(
&store,
&notify_uri,
&state,
RrdpSourceSyncState::DeltaReady,
None,
None,
)
.expect("seed base state");
let base_locks_body = std::fs::read_to_string(&base_locks).expect("read base locks");
let base_locks_sha = sha256_hex(base_locks_body.as_bytes());
std::fs::write(
&delta_locks,
format!(r#"{{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"{base_locks_sha}","rrdp":{{"{notify_uri}":{{"kind":"unchanged","base":{{"transport":"rrdp","session":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa","serial":10}},"target":{{"transport":"rrdp","session":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa","serial":10}},"delta_count":0,"deltas":[]}},"https://rrdp-fallback.example.test/notification.xml":{{"kind":"fallback-rsync","base":{{"transport":"rsync","session":null,"serial":null}},"target":{{"transport":"rsync","session":null,"serial":null}},"delta_count":0,"deltas":[]}}}},"rsync":{{"rsync://rsync.example.test/repo/":{{"file_count":1,"overlay_only":true}}}}}}"#),
)
.expect("rewrite delta locks");
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = delta_archive
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
r#"{"kind":"unchanged","base":{"transport":"rrdp","session":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa","serial":10},"target":{"transport":"rrdp","session":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa","serial":10},"delta_count":0,"deltas":[]}"#,
).expect("rewrite transition");
let delta_index =
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index");
let http = PayloadDeltaReplayHttpFetcher::from_index(Arc::new(delta_index.clone()))
.expect("build delta http fetcher");
let base_index =
Arc::new(ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"));
let rsync = PayloadDeltaReplayRsyncFetcher::new(base_index, Arc::new(delta_index.clone()));
let out = sync_publication_point_replay_delta(
&store,
&delta_index,
Some(&notify_uri),
&module_uri,
&http,
&rsync,
None,
None,
)
.expect("unchanged delta sync ok");
assert_eq!(out.source, RepoSyncSource::Rrdp);
assert_eq!(out.objects_written, 0);
}
#[test]
fn delta_replay_sync_uses_rsync_overlay_for_fallback_rsync_kind() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_fixture,
base_archive,
base_locks,
delta_archive,
delta_locks,
_notify_uri,
fallback_notify,
module_uri,
) = build_delta_replay_fixture();
let base_index =
Arc::new(ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"));
let delta_index = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let http = PayloadDeltaReplayHttpFetcher::from_index(delta_index.clone())
.expect("build delta http fetcher");
let rsync = PayloadDeltaReplayRsyncFetcher::new(base_index, delta_index.clone());
let out = sync_publication_point_replay_delta(
&store,
&delta_index,
Some(&fallback_notify),
&module_uri,
&http,
&rsync,
None,
None,
)
.expect("fallback-rsync delta sync ok");
assert_eq!(out.source, RepoSyncSource::Rsync);
assert_eq!(out.objects_written, 2);
assert_current_object(&store, "rsync://rsync.example.test/repo/a.mft", b"base");
assert_current_object(
&store,
"rsync://rsync.example.test/repo/sub/x.cer",
b"overlay-cer",
);
}
#[test]
fn delta_replay_sync_rejects_session_reset_and_gap() {
for kind in ["session-reset", "gap"] {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_fixture,
base_archive,
base_locks,
delta_archive,
delta_locks,
notify_uri,
_fallback_notify,
module_uri,
) = build_delta_replay_fixture();
let base_index = Arc::new(
ReplayArchiveIndex::load(&base_archive, &base_locks).expect("load base index"),
);
let state = RrdpState {
session_id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string(),
serial: 10,
};
persist_rrdp_local_state(
&store,
&notify_uri,
&state,
RrdpSourceSyncState::DeltaReady,
None,
None,
)
.expect("seed base state");
let locks_body = std::fs::read_to_string(&delta_locks).expect("read delta locks");
let rewritten = locks_body.replace("\"kind\":\"delta\"", &format!("\"kind\":\"{}\"", kind));
std::fs::write(&delta_locks, rewritten).expect("rewrite locks kind");
let repo_hash = sha256_hex(notify_uri.as_bytes());
let repo_dir = delta_archive
.join("v1/captures/delta-cap/rrdp/repos")
.join(&repo_hash);
std::fs::write(
repo_dir.join("transition.json"),
format!(
r#"{{"kind":"{kind}","base":{{"transport":"rrdp","session":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa","serial":10}},"target":{{"transport":"rrdp","session":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa","serial":12}},"delta_count":2,"deltas":[11,12]}}"#,
),
)
.expect("rewrite transition kind");
let delta_index = Arc::new(
ReplayDeltaArchiveIndex::load(&delta_archive, &delta_locks).expect("load delta index"),
);
let http = PayloadDeltaReplayHttpFetcher::from_index(delta_index.clone())
.expect("build delta http fetcher");
let rsync = PayloadDeltaReplayRsyncFetcher::new(base_index.clone(), delta_index.clone());
let err = sync_publication_point_replay_delta(
&store,
&delta_index,
Some(&notify_uri),
&module_uri,
&http,
&rsync,
None,
None,
)
.unwrap_err();
assert!(matches!(err, RepoSyncError::Replay(_)), "{err}");
}
}

View File

@@ -0,0 +1,344 @@
// Repository sync test group: fallback and replay.
#[test]
fn rrdp_protocol_error_does_not_retry_and_falls_back_to_rsync() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let timing = TimingHandle::new(TimingMeta {
recorded_at_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
validation_time_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
tal_url: None,
db_path: Some(store_dir.to_string_lossy().into_owned()),
});
let notification_uri = "https://example.test/notification.xml";
let snapshot_uri = "https://example.test/snapshot.xml";
let published_uri = "rsync://example.test/repo/a.mft";
let published_bytes = b"x";
let snapshot = snapshot_xml(
"9df4b597-af9e-4dca-bdda-719cce2c4e28",
1,
&[(published_uri, published_bytes)],
);
// Intentionally wrong hash to trigger protocol error (SnapshotHashMismatch).
let wrong_hash = "00".repeat(32);
let notif = notification_xml(
"9df4b597-af9e-4dca-bdda-719cce2c4e28",
1,
snapshot_uri,
&wrong_hash,
);
let mut map = HashMap::new();
map.insert(notification_uri.to_string(), notif);
map.insert(snapshot_uri.to_string(), snapshot);
let http = MapFetcher { map };
struct EmptyRsyncFetcher;
impl RsyncFetcher for EmptyRsyncFetcher {
fn fetch_objects(
&self,
_rsync_base_uri: &str,
) -> Result<Vec<(String, Vec<u8>)>, RsyncFetchError> {
Ok(Vec::new())
}
}
let policy = Policy {
sync_preference: SyncPreference::RrdpThenRsync,
..Policy::default()
};
let download_log = DownloadLogHandle::new();
let out = sync_publication_point(
&store,
&policy,
Some(notification_uri),
"rsync://example.test/repo/",
&http,
&EmptyRsyncFetcher,
Some(&timing),
Some(&download_log),
)
.expect("sync ok");
assert_eq!(out.source, RepoSyncSource::Rsync);
assert!(
out.warnings
.iter()
.any(|w| w.message.contains("RRDP failed; falling back to rsync")),
"expected RRDP fallback warning"
);
let events = download_log.snapshot_events();
assert_eq!(
events.len(),
3,
"expected notification + snapshot + rsync fallback"
);
assert_eq!(events[0].kind, AuditDownloadKind::RrdpNotification);
assert!(events[0].success);
assert_eq!(events[1].kind, AuditDownloadKind::RrdpSnapshot);
assert!(!events[1].success);
assert_eq!(events[2].kind, AuditDownloadKind::Rsync);
assert!(events[2].success);
let v = timing_to_json(temp.path(), &timing);
let counts = v.get("counts").expect("counts");
assert_eq!(
counts
.get("rrdp_retry_attempt_total")
.and_then(|v| v.as_u64()),
Some(1)
);
assert_eq!(
counts
.get("rrdp_failed_protocol_total")
.and_then(|v| v.as_u64()),
Some(1)
);
assert_eq!(
counts
.get("repo_sync_rrdp_failed_total")
.and_then(|v| v.as_u64()),
Some(1)
);
assert_eq!(
counts
.get("repo_sync_rsync_fallback_ok_total")
.and_then(|v| v.as_u64()),
Some(1)
);
}
#[test]
fn rrdp_delta_fetches_are_logged_even_if_snapshot_fallback_is_used() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let timing = TimingHandle::new(TimingMeta {
recorded_at_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
validation_time_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
tal_url: None,
db_path: Some(store_dir.to_string_lossy().into_owned()),
});
let notification_uri = "https://example.test/notification.xml";
let snapshot_uri = "https://example.test/snapshot.xml";
let delta_2_uri = "https://example.test/delta_2.xml";
let delta_3_uri = "https://example.test/delta_3.xml";
let published_uri = "rsync://example.test/repo/a.mft";
let published_bytes = b"x";
let sid = "9df4b597-af9e-4dca-bdda-719cce2c4e28";
// Seed old RRDP state so sync_from_notification tries deltas (RFC 8182 §3.4.1).
let state = RrdpState {
session_id: sid.to_string(),
serial: 1,
};
persist_rrdp_local_state(
&store,
notification_uri,
&state,
RrdpSourceSyncState::DeltaReady,
Some(snapshot_uri),
None,
)
.expect("seed state");
let delta_2 = format!(
r#"<delta xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{sid}" serial="2"></delta>"#
)
.into_bytes();
let delta_3 = format!(
r#"<delta xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{sid}" serial="3"></delta>"#
)
.into_bytes();
let delta_2_hash = hex::encode(sha2::Sha256::digest(&delta_2));
let delta_3_hash = hex::encode(sha2::Sha256::digest(&delta_3));
let snapshot = snapshot_xml(sid, 3, &[(published_uri, published_bytes)]);
let snapshot_hash = hex::encode(sha2::Sha256::digest(&snapshot));
let notif = format!(
r#"<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{sid}" serial="3"><snapshot uri="{snapshot_uri}" hash="{snapshot_hash}"/><delta serial="2" uri="{delta_2_uri}" hash="{delta_2_hash}"/><delta serial="3" uri="{delta_3_uri}" hash="{delta_3_hash}"/></notification>"#
)
.into_bytes();
let mut map = HashMap::new();
map.insert(notification_uri.to_string(), notif);
map.insert(snapshot_uri.to_string(), snapshot);
map.insert(delta_2_uri.to_string(), delta_2);
map.insert(delta_3_uri.to_string(), delta_3);
let http = MapFetcher { map };
let policy = Policy {
sync_preference: SyncPreference::RrdpThenRsync,
..Policy::default()
};
let download_log = DownloadLogHandle::new();
let out = sync_publication_point(
&store,
&policy,
Some(notification_uri),
"rsync://example.test/repo/",
&http,
&PanicRsyncFetcher,
Some(&timing),
Some(&download_log),
)
.expect("sync ok");
assert_eq!(out.source, RepoSyncSource::Rrdp);
assert_eq!(out.objects_written, 1);
assert_current_object(&store, published_uri, published_bytes);
let events = download_log.snapshot_events();
assert_eq!(events.len(), 4);
assert_eq!(
events
.iter()
.filter(|e| e.kind == AuditDownloadKind::RrdpNotification)
.count(),
1
);
assert_eq!(
events
.iter()
.filter(|e| e.kind == AuditDownloadKind::RrdpDelta)
.count(),
2
);
assert_eq!(
events
.iter()
.filter(|e| e.kind == AuditDownloadKind::RrdpSnapshot)
.count(),
1
);
assert!(events.iter().all(|e| e.success));
}
#[test]
fn replay_sync_uses_rrdp_when_locked_to_rrdp() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_archive_temp,
archive_root,
locks_path,
notify_uri,
_rsync_locked_notify,
_rsync_base_uri,
published_uri,
) = build_replay_archive_fixture();
let replay_index =
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
let http = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher");
let rsync = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
let out = sync_publication_point_replay(
&store,
&replay_index,
Some(&notify_uri),
"rsync://example.test/repo/",
&http,
&rsync,
None,
None,
)
.expect("replay sync ok");
assert_eq!(out.source, RepoSyncSource::Rrdp);
assert_eq!(out.objects_written, 1);
assert_current_object(&store, &published_uri, b"mft");
}
#[test]
fn replay_sync_uses_rsync_when_notification_is_locked_to_rsync() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_archive_temp,
archive_root,
locks_path,
_notify_uri,
rsync_locked_notify,
rsync_base_uri,
_published_uri,
) = build_replay_archive_fixture();
let replay_index =
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
let http = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher");
let rsync = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
let out = sync_publication_point_replay(
&store,
&replay_index,
Some(&rsync_locked_notify),
&rsync_base_uri,
&http,
&rsync,
None,
None,
)
.expect("replay rsync sync ok");
assert_eq!(out.source, RepoSyncSource::Rsync);
assert_eq!(out.objects_written, 1);
assert_eq!(out.warnings.len(), 0);
assert_current_object(
&store,
"rsync://rsync.example.test/repo/sub/fallback.cer",
b"cer",
);
}
#[test]
fn replay_sync_errors_when_lock_is_missing() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let (
_archive_temp,
archive_root,
locks_path,
_notify_uri,
_rsync_locked_notify,
_rsync_base_uri,
_published_uri,
) = build_replay_archive_fixture();
let replay_index =
ReplayArchiveIndex::load(&archive_root, &locks_path).expect("load replay index");
let http = PayloadReplayHttpFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay http fetcher");
let rsync = PayloadReplayRsyncFetcher::from_paths(&archive_root, &locks_path)
.expect("build replay rsync fetcher");
let err = sync_publication_point_replay(
&store,
&replay_index,
Some("https://missing.example/notification.xml"),
"rsync://missing.example/repo/",
&http,
&rsync,
None,
None,
)
.unwrap_err();
assert!(matches!(err, RepoSyncError::Replay(_)), "{err}");
}

View File

@@ -0,0 +1,713 @@
// Repository sync test group: setup and sync.
use super::*;
use crate::analysis::timing::{TimingHandle, TimingMeta};
use crate::fetch::rsync::LocalDirRsyncFetcher;
use crate::replay::archive::{ReplayArchiveIndex, sha256_hex};
use crate::replay::delta_archive::ReplayDeltaArchiveIndex;
use crate::replay::delta_fetch_http::PayloadDeltaReplayHttpFetcher;
use crate::replay::delta_fetch_rsync::PayloadDeltaReplayRsyncFetcher;
use crate::replay::fetch_http::PayloadReplayHttpFetcher;
use crate::replay::fetch_rsync::PayloadReplayRsyncFetcher;
use crate::storage::RepositoryViewState;
use crate::sync::rrdp::Fetcher as HttpFetcher;
use crate::sync::rrdp::RrdpState;
use crate::sync::store_projection::{build_repository_view_present_entry, compute_sha256_hex};
use base64::Engine;
use sha2::Digest;
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
struct DummyHttpFetcher;
impl HttpFetcher for DummyHttpFetcher {
fn fetch(&self, _url: &str) -> Result<Vec<u8>, String> {
panic!("http fetcher must not be used in rsync-only mode")
}
}
struct PanicRsyncFetcher;
impl RsyncFetcher for PanicRsyncFetcher {
fn fetch_objects(
&self,
_rsync_base_uri: &str,
) -> Result<Vec<(String, Vec<u8>)>, RsyncFetchError> {
panic!("rsync must not be used in this test")
}
}
struct MapFetcher {
map: HashMap<String, Vec<u8>>,
}
impl HttpFetcher for MapFetcher {
fn fetch(&self, uri: &str) -> Result<Vec<u8>, String> {
self.map
.get(uri)
.cloned()
.ok_or_else(|| format!("not found: {uri}"))
}
}
fn assert_current_object(store: &RocksStore, uri: &str, expected: &[u8]) {
assert_eq!(
store
.load_current_object_bytes_by_uri(uri)
.expect("load current object"),
Some(expected.to_vec())
);
}
#[test]
fn rsync_sync_uses_fetcher_dedup_scope_for_repository_view_projection() {
struct ScopeFetcher;
impl RsyncFetcher for ScopeFetcher {
fn fetch_objects(
&self,
_rsync_base_uri: &str,
) -> Result<Vec<(String, Vec<u8>)>, RsyncFetchError> {
Ok(vec![(
"rsync://example.net/repo/child/a.mft".to_string(),
b"manifest".to_vec(),
)])
}
fn dedup_key(&self, _rsync_base_uri: &str) -> String {
"rsync://example.net/repo/".to_string()
}
}
let td = tempfile::tempdir().expect("tempdir");
let store = RocksStore::open(td.path()).expect("open rocksdb");
let seeded = build_repository_view_present_entry(
"rsync://example.net/repo/",
"rsync://example.net/repo/sibling/old.roa",
&compute_sha256_hex(b"old"),
);
store
.put_projection_batch(&[seeded], &[], &[])
.expect("seed repository view");
let fetcher = ScopeFetcher;
let written = rsync_sync_into_current_store(
&store,
"rsync://example.net/repo/child/",
None,
&fetcher,
None,
None,
)
.expect("sync ok");
assert_eq!(written, 1);
let entries = store
.list_repository_view_entries_with_prefix("rsync://example.net/repo/")
.expect("list repository view");
let sibling = entries
.iter()
.find(|entry| entry.rsync_uri == "rsync://example.net/repo/sibling/old.roa")
.expect("sibling entry exists");
assert_eq!(sibling.state, RepositoryViewState::Withdrawn);
let child = entries
.iter()
.find(|entry| entry.rsync_uri == "rsync://example.net/repo/child/a.mft")
.expect("child entry exists");
assert_eq!(child.state, RepositoryViewState::Present);
}
fn notification_xml(
session_id: &str,
serial: u64,
snapshot_uri: &str,
snapshot_hash: &str,
) -> Vec<u8> {
format!(
r#"<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session_id}" serial="{serial}"><snapshot uri="{snapshot_uri}" hash="{snapshot_hash}"/></notification>"#
)
.into_bytes()
}
fn snapshot_xml(session_id: &str, serial: u64, published: &[(&str, &[u8])]) -> Vec<u8> {
let mut out = format!(
r#"<snapshot xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session_id}" serial="{serial}">"#
);
for (uri, bytes) in published {
let b64 = base64::engine::general_purpose::STANDARD.encode(bytes);
out.push_str(&format!(r#"<publish uri="{uri}">{b64}</publish>"#));
}
out.push_str("</snapshot>");
out.into_bytes()
}
fn build_replay_archive_fixture() -> (
tempfile::TempDir,
std::path::PathBuf,
std::path::PathBuf,
String,
String,
String,
String,
) {
let temp = tempfile::tempdir().expect("tempdir");
let archive_root = temp.path().join("payload-archive");
let capture = "repo-replay";
let capture_root = archive_root.join("v1").join("captures").join(capture);
std::fs::create_dir_all(&capture_root).expect("mkdir capture root");
std::fs::write(
capture_root.join("capture.json"),
format!(
r#"{{"version":1,"captureId":"{capture}","createdAt":"2026-03-13T00:00:00Z","notes":""}}"#
),
)
.expect("write capture json");
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
let snapshot_uri = "https://rrdp.example.test/snapshot.xml".to_string();
let session = "00000000-0000-0000-0000-000000000001".to_string();
let serial = 7u64;
let published_uri = "rsync://example.test/repo/a.mft".to_string();
let published_bytes = b"mft";
let snapshot = snapshot_xml(&session, serial, &[(&published_uri, published_bytes)]);
let snapshot_hash = hex::encode(sha2::Sha256::digest(&snapshot));
let notification = notification_xml(&session, serial, &snapshot_uri, &snapshot_hash);
let repo_hash = sha256_hex(notify_uri.as_bytes());
let session_dir = capture_root
.join("rrdp/repos")
.join(&repo_hash)
.join(&session);
std::fs::create_dir_all(&session_dir).expect("mkdir session dir");
std::fs::write(
session_dir.parent().unwrap().join("meta.json"),
format!(
r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write repo meta");
std::fs::write(session_dir.join("notification-7.xml"), notification)
.expect("write notification");
std::fs::write(
session_dir.join(format!("snapshot-7-{snapshot_hash}.xml")),
&snapshot,
)
.expect("write snapshot");
let rsync_base_uri = "rsync://rsync.example.test/repo/".to_string();
let rsync_locked_notify = "https://rrdp-fallback.example.test/notification.xml".to_string();
let mod_hash = sha256_hex(rsync_base_uri.as_bytes());
let module_bucket_dir = capture_root.join("rsync/modules").join(&mod_hash);
let module_root = module_bucket_dir
.join("tree")
.join("rsync.example.test")
.join("repo");
std::fs::create_dir_all(module_root.join("sub")).expect("mkdir module tree");
std::fs::write(
module_bucket_dir.join("meta.json"),
format!(
r#"{{"version":1,"module":"{rsync_base_uri}","createdAt":"2026-03-13T00:00:00Z","lastSeenAt":"2026-03-13T00:00:01Z"}}"#
),
)
.expect("write rsync meta");
std::fs::write(module_root.join("sub").join("fallback.cer"), b"cer")
.expect("write rsync object");
let locks_path = temp.path().join("locks.json");
std::fs::write(
&locks_path,
format!(
r#"{{
"version":1,
"capture":"{capture}",
"rrdp":{{
"{notify_uri}":{{"transport":"rrdp","session":"{session}","serial":{serial}}},
"{rsync_locked_notify}":{{"transport":"rsync","session":null,"serial":null}}
}},
"rsync":{{
"{rsync_base_uri}":{{"transport":"rsync"}}
}}
}}"#
),
)
.expect("write locks");
(
temp,
archive_root,
locks_path,
notify_uri,
rsync_locked_notify,
rsync_base_uri,
published_uri,
)
}
fn build_delta_replay_fixture() -> (
tempfile::TempDir,
std::path::PathBuf,
std::path::PathBuf,
std::path::PathBuf,
std::path::PathBuf,
String,
String,
String,
) {
let temp = tempfile::tempdir().expect("tempdir");
let base_archive = temp.path().join("payload-archive");
let base_capture_root = base_archive.join("v1/captures/base-cap");
std::fs::create_dir_all(&base_capture_root).expect("mkdir base capture");
std::fs::write(
base_capture_root.join("capture.json"),
r#"{"version":1,"captureId":"base-cap","createdAt":"2026-03-16T00:00:00Z","notes":""}"#,
)
.expect("write base capture meta");
let notify_uri = "https://rrdp.example.test/notification.xml".to_string();
let snapshot_uri = "https://rrdp.example.test/snapshot.xml".to_string();
let session = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa".to_string();
let base_serial = 10u64;
let delta1_uri = "https://rrdp.example.test/d1.xml".to_string();
let delta2_uri = "https://rrdp.example.test/d2.xml".to_string();
let repo_hash = sha256_hex(notify_uri.as_bytes());
let base_session_dir = base_capture_root
.join("rrdp/repos")
.join(&repo_hash)
.join(&session);
std::fs::create_dir_all(&base_session_dir).expect("mkdir base session dir");
std::fs::write(
base_session_dir.parent().unwrap().join("meta.json"),
format!(r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
)
.expect("write base rrdp meta");
let base_snapshot = snapshot_xml(
&session,
base_serial,
&[("rsync://example.test/repo/a.mft", b"base")],
);
let base_snapshot_hash = hex::encode(sha2::Sha256::digest(&base_snapshot));
let base_notification =
notification_xml(&session, base_serial, &snapshot_uri, &base_snapshot_hash);
std::fs::write(
base_session_dir.join("notification-10.xml"),
base_notification,
)
.expect("write base notif");
std::fs::write(
base_session_dir.join(format!("snapshot-10-{base_snapshot_hash}.xml")),
base_snapshot,
)
.expect("write base snapshot");
let module_uri = "rsync://rsync.example.test/repo/".to_string();
let module_hash = sha256_hex(module_uri.as_bytes());
let base_module_bucket = base_capture_root.join("rsync/modules").join(&module_hash);
let base_module_tree = base_module_bucket.join("tree/rsync.example.test/repo");
std::fs::create_dir_all(base_module_tree.join("sub")).expect("mkdir base rsync tree");
std::fs::write(
base_module_bucket.join("meta.json"),
format!(r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
)
.expect("write base module meta");
std::fs::write(base_module_tree.join("a.mft"), b"base").expect("write base a.mft");
std::fs::write(base_module_tree.join("sub").join("x.cer"), b"base-cer")
.expect("write base x.cer");
let base_locks = temp.path().join("base-locks.json");
let fallback_notify = "https://rrdp-fallback.example.test/notification.xml".to_string();
let base_locks_body = format!(
r#"{{"version":1,"capture":"base-cap","rrdp":{{"{notify_uri}":{{"transport":"rrdp","session":"{session}","serial":10}},"{fallback_notify}":{{"transport":"rsync","session":null,"serial":null}}}},"rsync":{{"{module_uri}":{{"transport":"rsync"}}}}}}"#
);
std::fs::write(&base_locks, &base_locks_body).expect("write base locks");
let base_locks_sha = sha256_hex(base_locks_body.as_bytes());
let delta_archive = temp.path().join("payload-delta-archive");
let delta_capture_root = delta_archive.join("v1/captures/delta-cap");
std::fs::create_dir_all(&delta_capture_root).expect("mkdir delta capture");
std::fs::write(
delta_capture_root.join("capture.json"),
r#"{"version":1,"captureId":"delta-cap","createdAt":"2026-03-16T00:00:00Z","notes":""}"#,
)
.expect("write delta capture meta");
std::fs::write(
delta_capture_root.join("base.json"),
format!(r#"{{"version":1,"baseCapture":"base-cap","baseLocksSha256":"{base_locks_sha}","createdAt":"2026-03-16T00:00:00Z"}}"#),
)
.expect("write delta base meta");
let delta_session_dir = delta_capture_root
.join("rrdp/repos")
.join(&repo_hash)
.join(&session);
let delta_deltas_dir = delta_session_dir.join("deltas");
std::fs::create_dir_all(&delta_deltas_dir).expect("mkdir delta deltas");
std::fs::write(
delta_session_dir.parent().unwrap().join("meta.json"),
format!(r#"{{"version":1,"rpkiNotify":"{notify_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
)
.expect("write delta meta");
std::fs::write(
delta_session_dir.parent().unwrap().join("transition.json"),
format!(r#"{{"kind":"delta","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}}"#),
)
.expect("write delta transition");
let delta1 = format!(
r#"<delta xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="11"><publish uri="rsync://example.test/repo/a.mft">{}</publish></delta>"#,
base64::engine::general_purpose::STANDARD.encode(b"delta-a")
);
let delta2 = format!(
r#"<delta xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="12"><publish uri="rsync://example.test/repo/sub/b.roa">{}</publish></delta>"#,
base64::engine::general_purpose::STANDARD.encode(b"delta-b")
);
let delta1_hash = hex::encode(sha2::Sha256::digest(delta1.as_bytes()));
let delta2_hash = hex::encode(sha2::Sha256::digest(delta2.as_bytes()));
let target_notification = format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="http://www.ripe.net/rpki/rrdp" version="1" session_id="{session}" serial="12">
<snapshot uri="{snapshot_uri}" hash="00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" />
<delta serial="11" uri="{delta1_uri}" hash="{delta1_hash}" />
<delta serial="12" uri="{delta2_uri}" hash="{delta2_hash}" />
</notification>"#
);
std::fs::write(
delta_session_dir.join("notification-target-12.xml"),
target_notification,
)
.expect("write target notification");
std::fs::write(delta_deltas_dir.join("delta-11-aaaa.xml"), delta1).expect("write delta11");
std::fs::write(delta_deltas_dir.join("delta-12-bbbb.xml"), delta2).expect("write delta12");
let delta_module_bucket = delta_capture_root.join("rsync/modules").join(&module_hash);
let delta_module_tree = delta_module_bucket.join("tree/rsync.example.test/repo");
std::fs::create_dir_all(delta_module_tree.join("sub")).expect("mkdir delta rsync tree");
std::fs::write(
delta_module_bucket.join("meta.json"),
format!(r#"{{"version":1,"module":"{module_uri}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
)
.expect("write delta rsync meta");
std::fs::write(
delta_module_bucket.join("files.json"),
format!(r#"{{"version":1,"module":"{module_uri}","fileCount":1,"files":["{module_uri}sub/x.cer"]}}"#),
)
.expect("write delta files");
std::fs::write(delta_module_tree.join("sub").join("x.cer"), b"overlay-cer")
.expect("write overlay file");
let fallback_hash = sha256_hex(fallback_notify.as_bytes());
let fallback_repo_dir = delta_capture_root.join("rrdp/repos").join(&fallback_hash);
std::fs::create_dir_all(&fallback_repo_dir).expect("mkdir fallback repo dir");
std::fs::write(
fallback_repo_dir.join("meta.json"),
format!(r#"{{"version":1,"rpkiNotify":"{fallback_notify}","createdAt":"2026-03-16T00:00:00Z","lastSeenAt":"2026-03-16T00:00:01Z"}}"#),
)
.expect("write fallback meta");
std::fs::write(
fallback_repo_dir.join("transition.json"),
r#"{"kind":"fallback-rsync","base":{"transport":"rsync","session":null,"serial":null},"target":{"transport":"rsync","session":null,"serial":null},"delta_count":0,"deltas":[]}"#,
)
.expect("write fallback transition");
let delta_locks = temp.path().join("locks-delta.json");
std::fs::write(
&delta_locks,
format!(r#"{{"version":1,"capture":"delta-cap","baseCapture":"base-cap","baseLocksSha256":"{base_locks_sha}","rrdp":{{"{notify_uri}":{{"kind":"delta","base":{{"transport":"rrdp","session":"{session}","serial":10}},"target":{{"transport":"rrdp","session":"{session}","serial":12}},"delta_count":2,"deltas":[11,12]}},"{fallback_notify}":{{"kind":"fallback-rsync","base":{{"transport":"rsync","session":null,"serial":null}},"target":{{"transport":"rsync","session":null,"serial":null}},"delta_count":0,"deltas":[]}}}},"rsync":{{"{module_uri}":{{"file_count":1,"overlay_only":false}}}}}}"#),
)
.expect("write delta locks");
(
temp,
base_archive,
base_locks,
delta_archive,
delta_locks,
notify_uri,
fallback_notify,
module_uri,
)
}
fn timing_to_json(temp_dir: &std::path::Path, timing: &TimingHandle) -> serde_json::Value {
let timing_path = temp_dir.join("timing_retry.json");
timing.write_json(&timing_path, 50).expect("write json");
serde_json::from_slice(&std::fs::read(&timing_path).expect("read json")).expect("parse json")
}
#[test]
fn rsync_sync_writes_current_store_and_records_counts() {
let temp = tempfile::tempdir().expect("tempdir");
let repo_dir = temp.path().join("repo");
std::fs::create_dir_all(repo_dir.join("sub")).expect("mkdir");
std::fs::write(repo_dir.join("a.mft"), b"mft").expect("write");
std::fs::write(repo_dir.join("sub").join("b.roa"), b"roa").expect("write");
std::fs::write(repo_dir.join("sub").join("c.cer"), b"cer").expect("write");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let timing = TimingHandle::new(TimingMeta {
recorded_at_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
validation_time_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
tal_url: None,
db_path: Some(store_dir.to_string_lossy().into_owned()),
});
let policy = Policy {
sync_preference: SyncPreference::RsyncOnly,
..Policy::default()
};
let http = DummyHttpFetcher;
let rsync = LocalDirRsyncFetcher::new(&repo_dir);
let download_log = DownloadLogHandle::new();
let out = sync_publication_point(
&store,
&policy,
None,
"rsync://example.test/repo/",
&http,
&rsync,
Some(&timing),
Some(&download_log),
)
.expect("sync ok");
assert_eq!(out.source, RepoSyncSource::Rsync);
assert_eq!(out.objects_written, 3);
let events = download_log.snapshot_events();
assert_eq!(events.len(), 1);
assert_eq!(events[0].kind, AuditDownloadKind::Rsync);
assert!(events[0].success);
assert_eq!(events[0].bytes, Some(9));
let objects = events[0].objects.as_ref().expect("objects stat");
assert_eq!(objects.objects_count, 3);
assert_eq!(objects.objects_bytes_total, 9);
assert_current_object(&store, "rsync://example.test/repo/a.mft", b"mft");
assert_current_object(&store, "rsync://example.test/repo/sub/b.roa", b"roa");
assert_current_object(&store, "rsync://example.test/repo/sub/c.cer", b"cer");
let view = store
.get_repository_view_entry("rsync://example.test/repo/a.mft")
.expect("get repository view")
.expect("repository view entry present");
assert_eq!(
view.current_hash.as_deref(),
Some(hex::encode(sha2::Sha256::digest(b"mft")).as_str())
);
assert_eq!(
view.repository_source.as_deref(),
Some("rsync://example.test/repo/")
);
let current_bytes = store
.load_current_object_bytes_by_uri("rsync://example.test/repo/sub/b.roa")
.expect("load current bytes")
.expect("current object bytes exist");
assert_eq!(current_bytes, b"roa".to_vec());
assert!(
store
.get_raw_by_hash_entry(hex::encode(sha2::Sha256::digest(b"roa")).as_str())
.expect("get raw_by_hash")
.is_none()
);
let timing_path = temp.path().join("timing.json");
timing.write_json(&timing_path, 5).expect("write json");
let v: serde_json::Value =
serde_json::from_slice(&std::fs::read(&timing_path).expect("read json"))
.expect("parse json");
let counts = v.get("counts").expect("counts");
assert_eq!(
counts
.get("rsync_objects_fetched_total")
.and_then(|v| v.as_u64()),
Some(3)
);
assert_eq!(
counts
.get("rsync_objects_bytes_total")
.and_then(|v| v.as_u64()),
Some(3 * 3)
);
}
#[test]
fn rsync_second_sync_marks_missing_repository_view_entries_withdrawn() {
let temp = tempfile::tempdir().expect("tempdir");
let repo_dir = temp.path().join("repo");
std::fs::create_dir_all(repo_dir.join("sub")).expect("mkdir");
std::fs::write(repo_dir.join("a.mft"), b"mft-v1").expect("write a");
std::fs::write(repo_dir.join("sub").join("b.roa"), b"roa-v1").expect("write b");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let policy = Policy {
sync_preference: SyncPreference::RsyncOnly,
..Policy::default()
};
let http = DummyHttpFetcher;
let rsync = LocalDirRsyncFetcher::new(&repo_dir);
sync_publication_point(
&store,
&policy,
None,
"rsync://example.test/repo/",
&http,
&rsync,
None,
None,
)
.expect("first sync ok");
std::fs::remove_file(repo_dir.join("sub").join("b.roa")).expect("remove b");
std::fs::write(repo_dir.join("c.crl"), b"crl-v2").expect("write c");
sync_publication_point(
&store,
&policy,
None,
"rsync://example.test/repo/",
&http,
&rsync,
None,
None,
)
.expect("second sync ok");
let withdrawn = store
.get_repository_view_entry("rsync://example.test/repo/sub/b.roa")
.expect("get withdrawn repo view")
.expect("withdrawn entry exists");
assert_eq!(
withdrawn.state,
crate::storage::RepositoryViewState::Withdrawn
);
assert_eq!(
withdrawn.repository_source.as_deref(),
Some("rsync://example.test/repo/")
);
let added = store
.get_repository_view_entry("rsync://example.test/repo/c.crl")
.expect("get added repo view")
.expect("added entry exists");
assert_eq!(added.state, crate::storage::RepositoryViewState::Present);
}
#[test]
fn rrdp_fetch_error_falls_back_to_rsync_without_retry() {
let temp = tempfile::tempdir().expect("tempdir");
let store_dir = temp.path().join("db");
let store = RocksStore::open(&store_dir).expect("open rocksdb");
let timing = TimingHandle::new(TimingMeta {
recorded_at_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
validation_time_utc_rfc3339: "2026-02-28T00:00:00Z".to_string(),
tal_url: None,
db_path: Some(store_dir.to_string_lossy().into_owned()),
});
let notification_uri = "https://example.test/notification.xml";
let published_uri = "rsync://example.test/repo/a.mft";
let published_bytes = b"x";
struct AlwaysFailHttp {
notification_calls: AtomicUsize,
}
impl HttpFetcher for AlwaysFailHttp {
fn fetch(&self, _uri: &str) -> Result<Vec<u8>, String> {
self.notification_calls.fetch_add(1, Ordering::SeqCst);
Err("http request failed: simulated transient".to_string())
}
}
struct SingleObjectRsync {
uri: String,
bytes: Vec<u8>,
}
impl RsyncFetcher for SingleObjectRsync {
fn fetch_objects(
&self,
_rsync_base_uri: &str,
) -> Result<Vec<(String, Vec<u8>)>, RsyncFetchError> {
Ok(vec![(self.uri.clone(), self.bytes.clone())])
}
}
let http = AlwaysFailHttp {
notification_calls: AtomicUsize::new(0),
};
let policy = Policy {
sync_preference: SyncPreference::RrdpThenRsync,
..Policy::default()
};
let download_log = DownloadLogHandle::new();
let out = sync_publication_point(
&store,
&policy,
Some(notification_uri),
"rsync://example.test/repo/",
&http,
&SingleObjectRsync {
uri: published_uri.to_string(),
bytes: published_bytes.to_vec(),
},
Some(&timing),
Some(&download_log),
)
.expect("sync ok");
assert_eq!(out.source, RepoSyncSource::Rsync);
assert_current_object(&store, published_uri, published_bytes);
assert_eq!(http.notification_calls.load(Ordering::SeqCst), 1);
let events = download_log.snapshot_events();
assert_eq!(events.len(), 2, "expected 1x notification + 1x rsync");
assert_eq!(
events
.iter()
.filter(|e| e.kind == AuditDownloadKind::RrdpNotification)
.count(),
1
);
assert_eq!(
events
.iter()
.filter(|e| e.kind == AuditDownloadKind::RrdpNotification && !e.success)
.count(),
1
);
assert_eq!(
events
.iter()
.filter(|e| e.kind == AuditDownloadKind::Rsync)
.count(),
1
);
let v = timing_to_json(temp.path(), &timing);
let counts = v.get("counts").expect("counts");
assert_eq!(
counts
.get("rrdp_retry_attempt_total")
.and_then(|v| v.as_u64()),
Some(1)
);
assert_eq!(
counts
.get("repo_sync_rrdp_failed_total")
.and_then(|v| v.as_u64()),
Some(1)
);
assert_eq!(
counts
.get("repo_sync_rsync_fallback_ok_total")
.and_then(|v| v.as_u64()),
Some(1)
);
}

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More