NASP/rpki
84
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

20260609 增加三RP持续对比监控

Browse Source
This commit is contained in:
yuyr
2026-06-09 17:44:30 +08:00
parent d0224d53f6
commit 74012c686d
22 changed files with 3911 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

91
monitor/README.md
View File

@@ -49,6 +49,40 @@ docker compose up -d
PROMETHEUS_PORT=19090 GRAFANA_PORT=13000 docker compose up -d
```
Prometheus 默认保留 7 天数据；可通过 `PROMETHEUS_RETENTION` 覆盖：
```bash
PROMETHEUS_RETENTION=7d docker compose up -d
```
## 长期稳定性测试
portable soak package 内置 `run_24h_soak_with_metrics.sh`，用于连续运行 ours RP、启动 metrics sidecar、启动本监控栈，并每小时生成报告：
```bash
cd /path/to/portable-soak
SOAK_DURATION_SECS=0 \
HOURLY_REPORT_INTERVAL_SECS=3600 \
SOAK_RETAIN_RUNS=100 \
CLEAN_TMP_AFTER_RUN=1 \
PROMETHEUS_RETENTION=7d \
STOP_MONITOR_STACK_ON_EXIT=0 \
FEISHU_WEBHOOK_SCRIPT=/home/yuyr/.codex/skills/user/feishu-webhook/scripts/send_feishu_text.py \
./run_24h_soak_with_metrics.sh
```
`SOAK_DURATION_SECS=0` 表示持续运行不自动停止；如需 24 小时自然停止，可设置为 `86400`，脚本会等当前 run 完成后退出，不会直接 kill 半轮验证。
关键产物：
- `runs/run_xxxx/`：最近 100 个 run 原始产物；
- `hourly_reports/hour_*.md`：小时级报告；
- `hourly_reports/hourly_summary.jsonl`：小时级结构化汇总；
- `incident_runs/run_xxxx/`：异常 run 固化副本；
- `logs/metrics.*`、`logs/24h-soak.*`、`logs/hourly-reporter.*`：运行日志。
短周期联调可把 `SOAK_DURATION_SECS` 和 `HOURLY_REPORT_INTERVAL_SECS` 调小，并设置 `FEISHU_DRY_RUN=1` 避免真实飞书通知。
## 停止
```bash
@@ -122,10 +156,65 @@ Grafana dashboard：
- `ours_rp_run_completed_total`
- `ours_rp_run_duration_seconds`
- `ours_rp_run_max_rss_bytes`
- `ours_rp_vrps`
- `ours_rp_vrps{kind="total|unique"}`：`total` 为去重前 VRP 条目数，`unique` 按 `(ASN, IP Prefix, Max Length)` 去重。
- `ours_rp_vaps`
- `ours_rp_publication_points`
- `ours_rp_repo_sync_phase_count`
- `ours_rp_large_publication_points{object_count_gt="10|50|100|..."}`
- `ours_rp_cir_objects`
- `ours_rp_ccr_state_items`
## Inter-RP 持续对比监控
`rpki_inter_rp_metrics` 用于汇总三方 RP 的最新产物：
- ours RP：读取当前 portable soak 的 `runs/run_xxxx/run-summary.json`、`result.ccr`、CSV 产物；
- Routinator：读取远端200同步来的 `routinator/latest/run-meta.json`、`vrps.csv`、`vaps.csv`；
- rpki-client 9.8：读取远端200同步来的 `rpki-client/latest/run-meta.json`、`vrps.csv`、`vaps.csv`、`result.ccr`。
远端231 启动 sidecar 示例：
```bash
rpki_inter_rp_metrics \
--ours-run-root /root/rpki_20260608_2_feature062_24h_20260608T075547Z/portable-soak \
--peer-root /root/inter-rp-aggregator/synced-from-200 \
--listen 0.0.0.0:9557 \
--poll-secs 30 \
--instance remote231-inter-rp
```
Prometheus 已新增 `ours-rp-inter-rp-metrics` scrape job，默认访问 `host.docker.internal:9557`。
远端200 runner 与远端231同步脚本位于：
```text
scripts/inter_rp/run_remote200_rp_loops.sh
scripts/inter_rp/run_single_rp_with_rss.sh
scripts/inter_rp/sync_remote200_to_231.sh
scripts/inter_rp/run_inter_rp_metrics_sidecar.sh
scripts/inter_rp/inter-rp.env.example
```
如需从本机独立开关远端200上的 Routinator 或 rpki-client，使用：
```bash
scripts/inter_rp/control_remote200_rp.sh status all
scripts/inter_rp/control_remote200_rp.sh stop routinator
scripts/inter_rp/control_remote200_rp.sh start routinator
scripts/inter_rp/control_remote200_rp.sh restart rpki-client
```
默认远端为 `root@43.110.128.200`，可通过 `REMOTE_HOST=...` 覆盖；脚本只管理指定 RP 的 loop 和当前子进程，不会自动影响另一个 RP。
关键指标：
- `inter_rp_run_wall_seconds{rp="ours-rp|routinator|rpki-client"}`
- `inter_rp_run_max_rss_bytes{rp="...",kind="aggregate_peak"}`
- `inter_rp_vrps{rp="..."}`：按 `(ASN, IP Prefix, Max Length)` 去重。
- `inter_rp_vaps{rp="..."}`：按 `(Customer ASN, Providers)` 去重，Routinator 使用 `--enable-aspa` JSON 输出转换，rpki-client 使用 `-j` JSON 输出转换。
- `inter_rp_ccr_digest_match{left="ours-rp",right="rpki-client",state="overall|mfts|vrps|vaps|tas|rks"}`
- `inter_rp_sync_age_seconds`
Grafana dashboard：
- <http://localhost:3000/d/ours-rp-inter-rp/ours-rp-inter-rp>

505
monitor/grafana/dashboards/ours-rp-inter-rp.json Normal file
View File

@@ -0,0 +1,505 @@
{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "none",
"decimals": 0
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"expr": "inter_rp_service_last_reload_success",
"legendFormat": "reload",
"refId": "A"
}
],
"title": "Metrics Reload OK",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 0
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 0
},
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"expr": "inter_rp_sync_age_seconds",
"legendFormat": "sync age",
"refId": "A"
}
],
"title": "Remote200 Sync Age",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 0
},
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"expr": "inter_rp_parse_errors",
"legendFormat": "errors",
"refId": "A"
}
],
"title": "Parse Errors",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "none",
"decimals": 0
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 0
},
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"expr": "inter_rp_ccr_digest_match{state=\"overall\"}",
"legendFormat": "overall",
"refId": "A"
}
],
"title": "Ours vs rpki-client CCR Match",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 4
},
"id": 5,
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"expr": "inter_rp_run_wall_seconds",
"legendFormat": "{{rp}}",
"refId": "A"
}
],
"title": "Wall Time by RP",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "bytes",
"min": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 4
},
"id": 6,
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"expr": "inter_rp_run_max_rss_bytes{kind=\"aggregate_peak\"}",
"legendFormat": "{{rp}}",
"refId": "A"
}
],
"title": "Max RSS Aggregate Peak by RP",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "none",
"decimals": 0,
"min": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 12
},
"id": 7,
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"expr": "inter_rp_vrps",
"legendFormat": "{{rp}}",
"refId": "A"
}
],
"title": "VRPs by RP (unique ASN/Prefix/MaxLen)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "none",
"decimals": 0,
"min": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 12
},
"id": 8,
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"expr": "inter_rp_vaps",
"legendFormat": "{{rp}}",
"refId": "A"
}
],
"title": "VAPs / ASPAs by RP (unique Customer/Providers)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "none",
"decimals": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 20
},
"id": 9,
"options": {
"showHeader": true,
"sortBy": []
},
"targets": [
{
"expr": "inter_rp_ccr_digest_match{left=\"ours-rp\",right=\"rpki-client\"}",
"format": "table",
"instant": true,
"legendFormat": "{{state}}",
"refId": "A"
}
],
"title": "CCR Digest Match States",
"type": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "none",
"decimals": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 20
},
"id": 10,
"options": {
"showHeader": true,
"sortBy": []
},
"targets": [
{
"expr": "inter_rp_vrps_diff",
"format": "table",
"instant": true,
"legendFormat": "vrps {{left}}-{{right}}",
"refId": "A"
},
{
"expr": "inter_rp_vaps_diff",
"format": "table",
"instant": true,
"legendFormat": "vaps {{left}}-{{right}}",
"refId": "B"
}
],
"title": "Output Count Diffs (VRP/VAP unique)",
"type": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 28
},
"id": 11,
"options": {
"legend": {
"calcs": [
"lastNotNull"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"expr": "inter_rp_artifact_age_seconds",
"legendFormat": "{{rp}}",
"refId": "A"
}
],
"title": "Artifact Age by RP",
"type": "timeseries"
}
],
"preload": false,
"refresh": "10s",
"schemaVersion": 40,
"tags": [
"rpki",
"inter-rp"
],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Ours RP Inter-RP",
"uid": "ours-rp-inter-rp",
"version": 1
}

81
monitor/grafana/dashboards/ours-rp-soak-overview.json
View File

@@ -15,7 +15,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"decimals": 0,
"unit": "none"
},
"overrides": []
},
@@ -147,7 +148,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"decimals": 0,
"unit": "none"
},
"overrides": []
},
@@ -238,7 +240,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"decimals": 0,
"unit": "none"
},
"overrides": []
},
@@ -265,10 +268,15 @@
},
"targets": [
{
"expr": "ours_rp_vrps",
"legendFormat": "VRPs",
"expr": "ours_rp_vrps{kind=\"total\"}",
"legendFormat": "VRPs raw",
"refId": "A"
},
{
"expr": "ours_rp_vrps{kind=\"unique\"}",
"legendFormat": "VRPs unique",
"refId": "D"
},
{
"expr": "ours_rp_vaps",
"legendFormat": "VAPs",
@@ -290,7 +298,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"decimals": 0,
"unit": "none"
},
"overrides": []
},
@@ -332,7 +341,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"decimals": 0,
"unit": "none"
},
"overrides": []
},
@@ -376,7 +386,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"decimals": 0,
"unit": "none"
},
"overrides": []
},
@@ -420,7 +431,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"unit": "none",
"decimals": 0
},
"overrides": []
},
@@ -449,8 +461,8 @@
"pluginVersion": "11.3.1",
"targets": [
{
"expr": "ours_rp_vrps",
"legendFormat": "VRPs",
"expr": "ours_rp_vrps{kind=\"total\"}",
"legendFormat": "VRPs raw",
"refId": "A"
}
],
@@ -464,7 +476,8 @@
},
"fieldConfig": {
"defaults": {
"unit": "short"
"unit": "none",
"decimals": 0
},
"overrides": []
},
@@ -557,6 +570,50 @@
],
"title": "Output Stage Durations",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "Prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "bytes",
"decimals": 2
},
"overrides": []
},
"gridPos": {
"x": 0,
"y": 24,
"w": 24,
"h": 8
},
"id": 14,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"expr": "ours_rp_run_max_rss_bytes",
"legendFormat": "Max RSS",
"refId": "A"
}
],
"title": "Max RSS Over Time",
"type": "timeseries"
}
],
"refresh": "5s",

7
monitor/prometheus/prometheus.yml
View File

@@ -11,3 +11,10 @@ scrape_configs:
labels:
rp: ours-rp
source: artifact-sidecar
- job_name: ours-rp-inter-rp-metrics
metrics_path: /metrics
static_configs:
- targets:
- host.docker.internal:9557
labels:
source: inter-rp-sidecar