NASP/rpki
59
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
rpki/scripts/manual_sync
History

README.md

Manual RRDP sync (APNIC-focused)

This directory contains manual, command-line scripts to reproduce the workflow described in:

  • specs/develop/20260226/apnic_rrdp_delta_analysis_after_manifest_revalidation_fix_20260227T022606Z.md

They are meant for hands-on validation / acceptance runs, not for CI.

Prerequisites

  • Rust toolchain (cargo)
  • rsync available on PATH (for rsync fallback/objects)
  • Network access (RRDP over HTTPS)

What the scripts do

full_sync.sh

  • Creates a fresh RocksDB directory
  • Runs a full serial validation from a TAL URL (default: APNIC RFC7730 TAL)
  • Writes:
    • run log
    • audit report JSON
    • run meta JSON (includes durations + download_stats)
    • short summary Markdown (includes durations + download_stats)
    • RocksDB key statistics (db_stats --exact)
    • RRDP legacy session/serial dump (rrdp_state_dump --view legacy-state)

delta_sync.sh

  • Copies an existing “baseline snapshot DB” to a new DB directory (so the baseline is not modified)
  • Runs another validation against the copied DB (RRDP will prefer delta when available)
  • Produces the same artifacts as full_sync.sh
  • Additionally generates a Markdown delta analysis report by comparing:
    • base vs delta report JSON
    • base vs delta rrdp_state_dump --view legacy-state TSV
    • and includes a duration comparison (base vs delta) if the base meta JSON is available
    • delta meta JSON includes download_stats copied from delta report JSON

Audit report fields (report.json)

The rpki binary writes an audit report JSON with:

  • format_version: 2
  • downloads: per-download RRDP/rsync events (URI, timestamps, duration, ok/fail, error, bytes, objects stats)
  • download_stats: aggregate counters (by kind)

These are useful for diagnosing why a run is slow (e.g. RRDP snapshot vs delta vs rsync fallback).

The standalone rrdp_state_dump tool also supports source, members, owners, and all views. The manual sync scripts intentionally call --view legacy-state so delta analysis keeps using a stable session/serial TSV format.

Meta fields (meta.json)

The scripts generate *_meta.json next to *_report.json and include:

  • durations_secs: wall-clock duration breakdown for the script steps
  • download_stats: copied from report_json.download_stats

Usage

Run from rpki/:

./scripts/manual_sync/full_sync.sh

After you have a baseline run, run delta against it:

./scripts/manual_sync/delta_sync.sh target/live/manual_sync/apnic_full_db_YYYYMMDDTHHMMSSZ \
  target/live/manual_sync/apnic_full_report_YYYYMMDDTHHMMSSZ.json

If the baseline was produced by full_sync.sh, the delta script will auto-discover the base meta JSON next to the base report (by replacing _report.json with _meta.json) and include base durations in the delta analysis report.

Configuration (env vars)

Both scripts accept overrides via env vars:

  • TAL_URL (default: APNIC TAL URL)
  • HTTP_TIMEOUT_SECS (default: 1800)
  • RSYNC_TIMEOUT_SECS (default: 1800)
  • RSYNC_MIRROR_ROOT (default: disabled; when set, passes --rsync-mirror-root to rpki)
  • VALIDATION_TIME (RFC3339; default: now UTC)
  • OUT_DIR (default: rpki/target/live/manual_sync)
  • RUN_NAME (default: auto timestamped)

Example:

TAL_URL="https://tal.apnic.net/tal-archive/apnic-rfc7730-https.tal" \
HTTP_TIMEOUT_SECS=1800 RSYNC_TIMEOUT_SECS=1800 \
./scripts/manual_sync/full_sync.sh