20260914 完善OSS日志与daemon部署

docs/command-line-reference.md

@@ -73,6 +73,7 @@ can increase memory consumption and does not guarantee proportional speedup.
 | Option | Default | Values |
 | --- | --- | --- |
 | `--log-level <level>` | `info` | `off`, `error`, `warn`, `info`, `debug`, `trace` |
+| `--log-progress-interval-secs <seconds>` | `10` | Periodic progress; `0` disables periodic events, not stage events. CLI overrides `PANDA_RPKI_LOG_PROGRESS_INTERVAL_SECS`. |
 | `--log-format <format>` | `text` | `text`, `json` |
 
 Precedence: explicit option, then `PANDA_RPKI_LOG_LEVEL` or
@@ -125,7 +126,7 @@ has stopped before restarting. Stop the entire container for Docker deployments.
 Daemon adds `daemon.lock`, `daemon-status.json`, `lifecycle.json`,
 `run-summary.jsonl`, `state/repository-db/`, and `runs/run_000001/` etc.
 Each run includes validator outputs, `run-meta.json`, `run-summary.json`,
-`stdout.log`, `stderr.log` and optional `process-time.txt`. `/usr/bin/time`
+`process-output.txt`, `validator.log` and optional `process-time.txt`. `/usr/bin/time`
 provides user/system CPU and peak RSS when installed; Docker includes it.
 Controller states: `starting`, `running`, `sleeping`, `exited`, `failed`.
 

docs/docker.md

@@ -47,6 +47,7 @@ docker compose -f docker/compose.all5.yaml run --build --rm validator
 | `PANDA_RPKI_WORKERS` | 8 | all5 and daemon object workers |
 | `PANDA_RPKI_REPO_WORKERS` | 8 | all5 and daemon transport workers |
 | `PANDA_RPKI_WORKER_QUEUE_CAPACITY` | 256 | all5 object queue |
+| `PANDA_RPKI_REPO_QUEUE_CAPACITY` | 256 | all5 daemon repository queue |
 | `PANDA_RPKI_HTTP_TIMEOUT_SECS` | 600 | all5 HTTP timeout (native CLI default is 300) |
 | `PANDA_RPKI_RRDP_SYNC_MODE` | auto | single-anchor and all5 |
 
@@ -54,7 +55,121 @@ The all5 CCR is `OUTPUT_DIR/all5.ccr`; the single-anchor CCR is
 `OUTPUT_DIR/result.ccr`. Reuse `STATE_DIR` and select new `OUTPUT_DIR` values
 to retain multiple cycles. These templates take no bundled RIR input files.
 
-## Daemon service
+## All-five daemon service
+
+The all-five daemon template combines the five regional TAL/TA pairs with the
+continuous daemon controller. Prepare the same `input/` directory used by the
+all-five one-shot example, then run:
+
+```bash
+mkdir -p data/all5-daemon
+export INPUT_DIR="$PWD/input" DAEMON_DIR="$PWD/data/all5-daemon"
+export AFRINIC_TAL_FILE=afrinic.tal AFRINIC_TA_FILE=afrinic.cer
+export APNIC_TAL_FILE=apnic.tal APNIC_TA_FILE=apnic.cer
+export ARIN_TAL_FILE=arin.tal ARIN_TA_FILE=arin.cer
+export LACNIC_TAL_FILE=lacnic.tal LACNIC_TA_FILE=lacnic.cer
+export RIPE_TAL_FILE=ripe.tal RIPE_TA_FILE=ripe.cer
+export PUID="$(id -u)" PGID="$(id -g)"
+docker compose -f docker/compose.all5-daemon.yaml up -d --build
+docker compose -f docker/compose.all5-daemon.yaml logs -f validator
+```
+
+The template uses a 60-second interval by default, keeps 10 completed runs,
+and leaves the daemon running until it is stopped. It writes each run below
+`DAEMON_DIR/runs/` and keeps shared protocol state below `DAEMON_DIR/state/`.
+The default diagnostic format is `text`, with a source file and line appended
+to each diagnostic record; select `json` explicitly for machine consumers.
+Set `PANDA_RPKI_INTERVAL_SECS` only when a different interval is intended. Stop
+the service without deleting its data with:
+
+```bash
+docker compose -f docker/compose.all5-daemon.yaml stop
+```
+
+## Upgrade a daemon deployment
+
+An upgrade replaces the image and recreates the container. It does not replace
+the host data directory. Keep the same `DAEMON_DIR`, Compose project, TAL/TA
+inputs and controller settings so the new image can resume the existing
+protocol state. Do not run two writable daemons against the same state root.
+
+Run the following from the deployment directory that owns the Compose file. Use
+a unique image tag for every candidate so the image used by a run is auditable:
+
+```bash
+export COMPOSE_FILE="${COMPOSE_FILE:-docker/compose.all5-daemon.yaml}"
+export COMPOSE_PROJECT_NAME=panda-rpki-all5
+export PANDA_RPKI_IMAGE=panda-rpki:2026-09-11-logging
+export SOURCE_DIR=/path/to/panda-rpki-oss
+
+# Keep the existing INPUT_DIR, DAEMON_DIR, TAL/TA variables and settings.
+docker compose -p "$COMPOSE_PROJECT_NAME" -f "$COMPOSE_FILE" ps
+test "$(jq -r .state "$DAEMON_DIR/daemon-status.json")" = sleeping
+
+# Build from the source tree before stopping the running service.
+docker build -f "$SOURCE_DIR/docker/Dockerfile" -t "$PANDA_RPKI_IMAGE" "$SOURCE_DIR"
+docker image inspect "$PANDA_RPKI_IMAGE" --format '{{.Id}}'
+
+# Save configuration and create a consistent stopped-state backup.
+backup="upgrade-backup-$(date -u +%Y%m%dT%H%M%SZ)"
+mkdir -p "$backup"
+cp "$COMPOSE_FILE" "$backup/compose.yaml"
+if test -f .env; then cp .env "$backup/env"; fi
+docker compose -p "$COMPOSE_PROJECT_NAME" -f "$COMPOSE_FILE" stop
+cp -a "$DAEMON_DIR/state" "$backup/state"
+cp -a "$DAEMON_DIR/daemon-status.json" "$DAEMON_DIR/lifecycle.json" "$backup/"
+
+# Recreate the same service with the new image; bind-mounted data is retained.
+docker compose -p "$COMPOSE_PROJECT_NAME" -f "$COMPOSE_FILE" up -d --no-build --force-recreate
+docker inspect "$(docker compose -p "$COMPOSE_PROJECT_NAME" -f "$COMPOSE_FILE" ps -q validator)" \
+  --format 'image={{.Config.Image}} running={{.State.Running}}'
+```
+
+After the first upgraded run, verify that the run sequence advanced, the run
+completed, and the new log/outputs exist. The exact run directory is printed by
+the daemon status and can be selected with `find`:
+
+```bash
+latest_run="$(find "$DAEMON_DIR/runs" -mindepth 1 -maxdepth 1 -type d -name 'run_*' | sort | tail -1)"
+test -s "$latest_run/run-summary.json"
+test -s "$latest_run/validator.log"
+test -s "$latest_run/result.ccr"
+grep -Eq '\[src/[^]]+\.rs:[0-9]+\]$' "$latest_run/validator.log"
+jq . "$latest_run/run-summary.json"
+```
+
+The upgrade process preserves completed run directories until normal retention
+removes them. Archive important runs separately if they must outlive
+`PANDA_RPKI_RETAIN_RUNS`.
+
+### Roll back
+
+Stop the upgraded service before changing the image. Restore the previous
+Compose file and `.env`, or set `PANDA_RPKI_IMAGE` to the previously verified
+tag, then recreate the service with the same project and data directory:
+
+```bash
+docker compose -p "$COMPOSE_PROJECT_NAME" -f "$COMPOSE_FILE" stop
+cp "$backup/compose.yaml" "$COMPOSE_FILE"
+if test -f "$backup/env"; then cp "$backup/env" .env; fi
+unset PANDA_RPKI_IMAGE
+docker compose -p "$COMPOSE_PROJECT_NAME" -f "$COMPOSE_FILE" up -d --no-build --force-recreate
+```
+
+If the new binary changed the state schema or wrote incompatible state, stop the
+service and move the current state aside before restoring the matching
+stopped-state backup:
+
+```bash
+mv "$DAEMON_DIR/state" "$DAEMON_DIR/state-incompatible-$(date -u +%Y%m%dT%H%M%SZ)"
+cp -a "$backup/state" "$DAEMON_DIR/state"
+```
+
+Never delete `DAEMON_DIR` or use `docker compose down -v` as part of a routine
+upgrade. Check [state compatibility](command-line-reference.md#state-and-network)
+before deploying a binary with storage changes.
+
+## Single-anchor daemon service
 
 ```bash
 mkdir -p data/daemon
@@ -69,7 +184,7 @@ docker compose -f docker/compose.daemon.yaml stop
 ```
 
 Defaults: interval 60 seconds, retain 10, `PANDA_RPKI_RUN_TIMEOUT_SECS=0`
-(disabled), log level `info`, log format `json`. The daemon template passes
+(disabled), log level `info`, log format `text`. The daemon template passes
 `PANDA_RPKI_LOG_LEVEL` and `PANDA_RPKI_LOG_FORMAT` into the container.
 It restarts unless stopped and allows 40 seconds for shutdown, exceeding the
 controller's default 30-second grace plus termination wait.

docs/logging.md

@@ -0,0 +1,124 @@
+# Diagnostic logging
+
+The validator writes diagnostics to stderr. Under daemon, that stream is saved
+as `runs/run_XXXXXX/validator.log`; it contains all enabled levels, not only errors.
+`process-output.txt` captures ordinary output and is normally empty. The daemon's
+own lifecycle diagnostics remain in the container log. No duplicate log forwarding
+is performed. Historical `stderr.log` files are not renamed.
+
+CLI and both daemon Compose files default to `info` and `text`.
+Text is written for people, with independently authored English sentences for
+run lifecycle, validation stages, progress, output files and failures. It does not
+enumerate event fields or translate event identifiers into prose. Progress reports
+elapsed time, publication points processed when known, and active repository sync
+attempts. Old progress snapshots are explicitly described as the last reported
+update. UUIDs, worker queues and unrelated counters are not printed.
+The synthetic initializing-stage completion is omitted in text only. Control
+characters are escaped to keep each record on one line. Diagnostic filenames do
+not select the format. Text is not a stable machine-parsing interface.
+
+```text
+2026-09-11T10:44:14Z INFO [run 230] Starting validation with delta synchronization across 5 trust anchors.
+2026-09-11T10:44:24Z INFO [run 230] Synchronizing repositories and validating objects; 10.0 seconds elapsed. Processed 32000 publication points so far. 3 repository sync attempts are still in progress.
+2026-09-11T10:45:36Z INFO [run 230] Validation completed in 82.0 seconds.
+```
+
+Use `--log-format json` or `PANDA_RPKI_LOG_FORMAT=json` for machine consumers;
+the JSON event names and fields are unchanged. Both formats redact sensitive data.
+Text error reasons are limited to 240 Unicode characters with an explicit notice
+when shortened. Unknown extension/debug events retain a diagnostic identifier and
+their message, not an arbitrary field dump. Use JSON (and debug/trace as needed)
+for full technical details, category examples and counter snapshots.
+JSON is one complete object per line: `timestamp`, `level`, `event`, `fields`, `source`.
+Timestamps are UTC RFC3339. Run context is in `fields.run_id` and `fields.run_seq`
+(null for standalone validation). Logging failures do not change validation results.
+
+Every emitted text line ends with `[source/file.rs:line]`. JSON records include
+`source: {"file": "source/file.rs", "line": 123}` at the top level. These are
+compile-time source paths and one-based line numbers from the logger call site,
+not the shared output sink; progress forwarding preserves the caller location.
+Logs emitted inside artifact/stage helpers point to the emit call inside that
+helper. Match locations against the source revision used to build the binary.
+
+```text
+2026-09-11T10:45:36Z INFO [run 230] Validation completed in 82.0 seconds. [src/cli/validate.rs:123]
+```
+
+The source line in this example is illustrative; actual logs report the compiled location.
+
+## Event contract
+
+| Events | Level | Meaning / frequency |
+| --- | --- | --- |
+| validation_started/completed/summary | info | Run boundaries and exact final business counts |
+| validation_failed, command_failed | error | Run/command failure, with error and elapsed time where available |
+| validation_stage_started/completed/failed | info/error | State store, setup, TA discovery, online validation, output, CSV conversion |
+| validation_progress | info | Default every 10s, independent of blocked network/store calls |
+| trust_anchor_started/completed/failed | info/error | One pair per TAL, fatal failure includes TAL ID |
+| validation_warnings | warn | Existing final total, not a declaration that the run failed |
+| validation_warning_category | info | Category count and up to three redacted 1KiB examples |
+| validation_warning_detail | debug | Full warning message and context |
+| artifact_write_started/completed/failed | info/error | Type, path, wall milliseconds and bytes; CCR, CSV, summary, timings |
+| rrdp_sync_attempt/success | debug | Transport attempt and successful object writes/duration |
+| rrdp_sync_failed, http_fetch_failed | warn | Recoverable attempt failures; retryable does not imply will_retry |
+| rrdp_sync_selected | debug | Actual snapshot/delta/unchanged selection, reason, serial/session |
+| rrdp_download_completed | debug | Snapshot/delta decoded payload bytes, not wire traffic |
+| rrdp_cross_origin_rejected, rrdp_delta_fallback | warn | Protocol security rejection / delta application fallback |
+| repo_terminal_failure | warn | Failed publication point, not a fatal command error |
+| http_fetch_slow, rsync_sync_slow, phase2_*_slow | debug | Detailed slow operations, aggregated into info progress |
+| phase1_repo_* | debug | Repository task scheduling and worker decisions |
+| phase2 batch/queue/object events, publication_point_* | trace | Detailed high-volume processing; not recommended for continuous all5 |
+| daemon_started/state_recovered/run_started/child_started | info | Effective configuration, recovery and per-run log path |
+| daemon_run_completed/sleeping/retention_applied/exited | info/warn | Outcome, next scheduled time and retention decisions |
+| daemon_stop_requested | info | Graceful stop request observed outside the signal handler |
+| daemon_child_terminate/child_kill | warn | One event per signal escalation, with reason |
+
+`src/logging/progress.rs` explicitly registers progress event levels, including
+the dynamic finalize-worker label; a source-inventory regression detects new
+unregistered literal call sites. Unknown library-supplied names default to debug.
+
+## Progress and volume
+
+`--log-progress-interval-secs` overrides `PANDA_RPKI_LOG_PROGRESS_INTERVAL_SECS`,
+default `10`. `0` suppresses periodic events, not stage events. `off/error/warn`
+do not create a progress thread. Queue snapshots are refreshed at most every 250ms.
+`statistics_age_ms` is the age of the queue/PP snapshot, not the repository event
+counters. `progress_stalled` means no observed completion for 30s, not proof of deadlock.
+Progress does not predict the total tree size or invent completion percentages.
+
+- Repository counters count attempts, not publication points or unique repositories.
+- `repository_objects_written` counts writes, including updates; it is not unique validated objects.
+- `roa_tasks_completed` counts returned parallel ROA tasks, not all object types.
+- `downloaded_snapshot_delta_bytes` excludes notifications and measures decoded payload.
+- The serial path reports PP completed/failed counts; absent parallel queue fields are not zeroes.
+- Slow-operation counts are cumulative per run. Detailed slow events stay debug.
+
+Warning categories are assigned at typed source sites. Opaque legacy string
+errors remain `unclassified`; they are not classified by substring guesses.
+The category total equals the existing warning total, which may contain both tree
+and audit warnings. Examples are redacted before UTF-8-safe truncation.
+
+| Category | Assignment at currently instrumented sites | Meaning / example |
+| --- | --- | --- |
+| unclassified | No typed classification retained | Opaque repository-sync error; do not guess its cause |
+| ber_compatible_cms_encoding | Existing BER-compatible CMS warning | Accepted non-DER encoding |
+| transport | Typed fetch error in RRDP fallback warning | Download failure |
+| rrdp_protocol | Typed RRDP error in fallback warning | Invalid RRDP response |
+| security_policy | Cross-origin fallback, manifest location/hash/rollback rejection | Integrity or anti-rollback check |
+| certificate | Child discovery or typed manifest EE/issuer certificate errors | Certificate/path/resource failure |
+| manifest_crl | Typed manifest errors or missing required CRL | Stale manifest or missing CRL |
+| object_validation | Parallel ROA/ASPA rejection | Invalid signed routing object |
+
+This is incremental typed coverage, not a promise that every historical warning
+site is already classified. Uninstrumented sites intentionally remain unclassified.
+
+```bash
+tail -f /path/to/data/runs/run_000001/validator.log
+jq -c 'select(.event == "validation_progress")' /path/to/validator.log
+jq -c 'select(.level == "warn" or .level == "error")' /path/to/validator.log
+```
+
+The `jq` examples require JSON logging to have been explicitly enabled.
+
+Retention removes completed run directories including their logs. Archive needed
+runs before the configured retention window expires. Summary history is append-only.

docs/usage.md

@@ -76,6 +76,15 @@ The append-only summary history is not trimmed by retention.
 
 ## Outputs and diagnostics
 
+`validator.log` is the per-run diagnostic log, not an error-only file.
+`process-output.txt` captures ordinary child output and is normally empty.
+Old runs keep their historical filenames; collectors support `stderr.log` as a fallback.
+Every ten seconds, info logs report the current stage and available progress counters.
+Use `--log-progress-interval-secs 0` to disable periodic events; stage events remain.
+`statistics_age_ms` identifies stale statistics; `progress_stalled` means no observed
+completion for thirty seconds, not a proven deadlock. Totals and percentages are not guessed.
+Events include `run_id` and, under daemon, `run_seq`. Controller logs remain separate.
+
 `vrps.csv` contains `asn,prefix,max_length`. CCR contains canonical state;
 duplicate routing payloads can collapse into a single CCR entry. Count raw
 rows and normalized unique payloads separately when comparing outputs.
@@ -83,7 +92,7 @@ ASPA canonicalization groups providers by customer ASN.
 
 Use `--log-level debug` for diagnosis, or `trace` for detailed object/worker
 events. Logs are written to stderr and can be redirected independently from
-outputs. In daemon mode, inspect per-run `stderr.log` and the controller's
+outputs. In daemon mode, inspect per-run `validator.log` and the controller's
 `daemon-status.json` and `run-summary.jsonl`.
 
 | Symptom | Action |