138 lines
9.0 KiB
Markdown
138 lines
9.0 KiB
Markdown
# Diagnostic logging
|
|
|
|
For every field, default, constraint and complete example, see the [TOML configuration reference](configuration.md)。
|
|
|
|
The validator writes diagnostics to stderr. Under daemon, that stream is saved
|
|
as `runs/run_XXXXXX/validator.log`; it contains all enabled levels, not only errors.
|
|
`process-output.txt` captures ordinary output and is normally empty. The daemon's
|
|
own lifecycle diagnostics remain in the container log. No duplicate log forwarding
|
|
is performed. Historical `stderr.log` files are not renamed.
|
|
|
|
TOML defaults and the shipped Compose TOML examples use `info` and `text`.
|
|
Compose does not independently select the logging format.
|
|
Text is written for people, with independently authored English sentences for
|
|
run lifecycle, validation stages, progress, output files and failures. It does not
|
|
enumerate event fields or translate event identifiers into prose. Progress reports
|
|
elapsed time, publication points processed when known, and active repository sync
|
|
attempts. Old progress snapshots are explicitly described as the last reported
|
|
update. UUIDs, worker queues and unrelated counters are not printed.
|
|
The synthetic initializing-stage completion is omitted in text only. Control
|
|
characters are escaped to keep each record on one line. Diagnostic filenames do
|
|
not select the format. Text is not a stable machine-parsing interface.
|
|
|
|
```text
|
|
2026-09-11T10:44:14Z INFO [run 230] Starting validation with delta synchronization across 5 trust anchors.
|
|
2026-09-11T10:44:24Z INFO [run 230] Synchronizing repositories and validating objects; 10.0 seconds elapsed. Processed 32000 publication points so far. 3 repository sync attempts are still in progress.
|
|
2026-09-11T10:45:36Z INFO [run 230] Validation completed in 82.0 seconds.
|
|
```
|
|
|
|
Set `log-format = "json"` in the TOML for machine consumers; the JSON event
|
|
names and fields are unchanged. Both formats redact sensitive data.
|
|
Text error reasons are limited to 240 Unicode characters with an explicit notice
|
|
when shortened. Unknown extension/debug events retain a diagnostic identifier and
|
|
their message, not an arbitrary field dump. Use JSON (and debug/trace as needed)
|
|
for full technical details, category examples and counter snapshots.
|
|
JSON is one complete object per line: `timestamp`, `level`, `event`, `fields`, `source`.
|
|
Timestamps are UTC RFC3339. Run context is in `fields.run_id` and `fields.run_seq`
|
|
(null for standalone validation). Logging failures do not change validation results.
|
|
|
|
Every emitted text line ends with `[source/file.rs:line]`. JSON records include
|
|
`source: {"file": "source/file.rs", "line": 123}` at the top level. These are
|
|
compile-time source paths and one-based line numbers from the logger call site,
|
|
not the shared output sink; progress forwarding preserves the caller location.
|
|
Logs emitted inside artifact/stage helpers point to the emit call inside that
|
|
helper. Match locations against the source revision used to build the binary.
|
|
|
|
```text
|
|
2026-09-11T10:45:36Z INFO [run 230] Validation completed in 82.0 seconds. [src/cli/validate.rs:123]
|
|
```
|
|
|
|
The source line in this example is illustrative; actual logs report the compiled location.
|
|
|
|
## Event contract
|
|
|
|
| Events | Level | Meaning / frequency |
|
|
| --- | --- | --- |
|
|
| validation_started/completed/summary | info | Run boundaries and exact final business counts |
|
|
| validation_failed, command_failed | error | Run/command failure, with error and elapsed time where available |
|
|
| validation_stage_started/completed/failed | info/error | State store, setup, TA discovery, online validation, output, CSV conversion |
|
|
| validation_progress | info | Default every 10s, independent of blocked network/store calls |
|
|
| trust_anchor_started/completed/failed | info/error | One pair per TAL, fatal failure includes TAL ID |
|
|
| validation_warnings | warn | Existing final total, not a declaration that the run failed |
|
|
| validation_warning_category | info | Category count and up to three redacted 1KiB examples |
|
|
| validation_warning_detail | debug | Full warning message and context |
|
|
| artifact_write_started/completed/failed | info/error | Type, path, wall milliseconds and bytes; CCR, CSV, summary, timings |
|
|
| rrdp_sync_attempt/success | debug | Transport attempt and successful object writes/duration |
|
|
| rrdp_sync_failed, http_fetch_failed | warn | Recoverable attempt failures; retryable does not imply will_retry |
|
|
| rrdp_sync_selected | debug | Actual snapshot/delta/unchanged selection, reason, serial/session |
|
|
| rrdp_download_completed | debug | Snapshot/delta decoded payload bytes, not wire traffic |
|
|
| rrdp_cross_origin_rejected, rrdp_delta_fallback | warn | Protocol security rejection / delta application fallback |
|
|
| repo_terminal_failure | warn | Failed publication point, not a fatal command error |
|
|
| http_fetch_slow, rsync_sync_slow, phase2_*_slow | debug | Detailed slow operations, aggregated into info progress |
|
|
| phase1_repo_* | debug | Repository task scheduling and worker decisions |
|
|
| phase2 batch/queue/object events, publication_point_* | trace | Detailed high-volume processing; not recommended for continuous all5 |
|
|
| daemon_started/state_recovered/run_started/child_started | info | Effective configuration, recovery and per-run log path |
|
|
| daemon_run_completed/sleeping/retention_applied/exited | info/warn | Outcome, next scheduled time and retention decisions |
|
|
| daemon_stop_requested | info | Graceful stop request observed outside the signal handler |
|
|
| daemon_child_terminate/child_kill | warn | One event per signal escalation, with reason |
|
|
|
|
`src/logging/progress.rs` explicitly registers progress event levels, including
|
|
the dynamic finalize-worker label; a source-inventory regression detects new
|
|
unregistered literal call sites. Unknown library-supplied names default to debug.
|
|
|
|
## Progress and volume
|
|
|
|
`log-progress-interval-secs` is configured in TOML and defaults to `10`. `0`
|
|
suppresses periodic events, not stage events. `off/error/warn`
|
|
do not create a progress thread. Queue snapshots are refreshed at most every 250ms.
|
|
`statistics_age_ms` is the age of the queue/PP snapshot, not the repository event
|
|
counters. `progress_stalled` means no observed completion for 30s, not proof of deadlock.
|
|
Progress does not predict the total tree size or invent completion percentages.
|
|
|
|
- Repository counters count attempts, not publication points or unique repositories.
|
|
- `repository_objects_written` counts writes, including updates; it is not unique validated objects.
|
|
- `roa_tasks_completed` counts returned parallel ROA tasks, not all object types.
|
|
- `downloaded_snapshot_delta_bytes` excludes notifications and measures decoded payload.
|
|
- The serial path reports PP completed/failed counts; absent parallel queue fields are not zeroes.
|
|
- Slow-operation counts are cumulative per run. Detailed slow events stay debug.
|
|
|
|
Warning categories are assigned at typed source sites. Opaque legacy string
|
|
errors remain `unclassified`; they are not classified by substring guesses.
|
|
The category total equals the existing warning total, which may contain both tree
|
|
and audit warnings. Examples are redacted before UTF-8-safe truncation.
|
|
|
|
| Category | Assignment at currently instrumented sites | Meaning / example |
|
|
| --- | --- | --- |
|
|
| unclassified | No typed classification retained | Opaque repository-sync error; do not guess its cause |
|
|
| ber_compatible_cms_encoding | Existing BER-compatible CMS warning | Accepted non-DER encoding |
|
|
| transport | Typed fetch error in RRDP fallback warning | Download failure |
|
|
| rrdp_protocol | Typed RRDP error in fallback warning | Invalid RRDP response |
|
|
| security_policy | Cross-origin fallback, manifest location/hash/rollback rejection | Integrity or anti-rollback check |
|
|
| certificate | Child discovery or typed manifest EE/issuer certificate errors | Certificate/path/resource failure |
|
|
| manifest_crl | Typed manifest errors or missing required CRL | Stale manifest or missing CRL |
|
|
| object_validation | Parallel ROA/ASPA rejection | Invalid signed routing object |
|
|
|
|
This is incremental typed coverage, not a promise that every historical warning
|
|
site is already classified. Uninstrumented sites intentionally remain unclassified.
|
|
|
|
```bash
|
|
tail -f /path/to/data/runs/run_000001/validator.log
|
|
jq -c 'select(.event == "validation_progress")' /path/to/validator.log
|
|
jq -c 'select(.level == "warn" or .level == "error")' /path/to/validator.log
|
|
```
|
|
|
|
The `jq` examples require JSON logging to have been explicitly enabled.
|
|
|
|
Retention removes completed run directories including their logs. Archive needed
|
|
runs before the configured retention window expires. Summary history is append-only.
|
|
|
|
## Change an existing daemon to text
|
|
|
|
Set `log-format = "text"` at the TOML top level, before `[[anchors]]`.
|
|
Keep `log-level = "info"` or the desired existing level. Do not use the removed
|
|
`--log-format` flag or `PANDA_RPKI_LOG_FORMAT` environment override.
|
|
TOML is read at process start, so stop and recreate the service as described in
|
|
[Docker configuration reload](docker.md#graceful-stop-edit-and-reload).
|
|
Each new validator child inherits the controller configuration; prior run logs
|
|
and JSON summary/metadata files are not converted.
|