add periodic snapshot reset

This commit is contained in:
2026-06-29 16:47:47 +08:00
parent f9f91f072a
commit fe8d4fcb14
10 changed files with 371 additions and 11 deletions

View File

@@ -26,6 +26,8 @@ Defaults:
- `INTERVAL_SECS=600`
- `TAL_INPUT_MODE=file-live-ta`
- `LIVE_TA_REFRESH_BEFORE_SNAPSHOT=1`
- `PERIODIC_SNAPSHOT_RESET=0`
- `PERIODIC_SNAPSHOT_MAX_DELTAS=100`
- `HOST_DATA_DIR=/var/lib/ours-rp-arm64`
- `SOAK_RESTART_POLICY=unless-stopped`
- `MONITOR_PLATFORM=linux/arm64`
@@ -69,6 +71,31 @@ Default host directory:
Each `runs/run_XXXX/` directory contains `report.json`, `result.ccr`, `input.cir`, `vrps.csv`, `vaps.csv`, `stage-timing.json`, logs and metadata.
## Periodic Snapshot Reset
New knobs:
```bash
PERIODIC_SNAPSHOT_RESET=0
PERIODIC_SNAPSHOT_MAX_DELTAS=100
```
Semantics:
- disabled by default, keeping previous behavior unchanged;
- when enabled, one successful snapshot is followed by at most `N` successful delta runs;
- after the threshold is reached, the next run is forced to snapshot;
- before that forced snapshot, only the active `state/db` is reset, while `runs/`, `logs/`, `state/rsync-mirror`, `.env`, and Prometheus/Grafana data are preserved;
- after a successful forced snapshot, the old DB staging is deleted so disk usage does not keep growing elsewhere.
Check the latest `run-meta.json` for:
- `sync_mode`
- `snapshot_reason`
- `periodic_snapshot_delta_count`
- `periodic_snapshot_forced`
- `reset_db_cleanup_status`
## Common Commands
```bash

View File

@@ -26,6 +26,8 @@ vim .env
- `INTERVAL_SECS=600`
- `TAL_INPUT_MODE=file-live-ta`
- `LIVE_TA_REFRESH_BEFORE_SNAPSHOT=1`
- `PERIODIC_SNAPSHOT_RESET=0`
- `PERIODIC_SNAPSHOT_MAX_DELTAS=100`
- `HOST_DATA_DIR=/var/lib/ours-rp-arm64`
- `SOAK_RESTART_POLICY=unless-stopped`
- `MONITOR_PLATFORM=linux/arm64`
@@ -69,6 +71,31 @@ GRAFANA_ADMIN_PASSWORD=admin
`runs/run_XXXX/` 中包含每轮 `report.json`、`result.ccr`、`input.cir`、`vrps.csv`、`vaps.csv`、`stage-timing.json`、日志和元数据。
## 定期 snapshot reset
新增配置:
```bash
PERIODIC_SNAPSHOT_RESET=0
PERIODIC_SNAPSHOT_MAX_DELTAS=100
```
语义:
- 默认关闭,行为与旧版本一致;
- 开启后,一次成功 snapshot 后最多连续执行 `N` 个成功 delta;
- 达到阈值后,下一轮强制跑 snapshot;
- 强制 snapshot 前只重置 active `state/db`,保留 `runs/`、`logs/`、`state/rsync-mirror`、`.env`、Prometheus/Grafana 数据;
- 强制 snapshot 成功后旧 DB staging 会被删除,避免磁盘只是换目录继续增长。
可通过最新 `run-meta.json` 中的以下字段确认:
- `sync_mode`
- `snapshot_reason`
- `periodic_snapshot_delta_count`
- `periodic_snapshot_forced`
- `reset_db_cleanup_status`
## 常用命令
```bash

View File

@@ -48,15 +48,47 @@ Important checks:
- latest run status;
- metrics, Prometheus and Grafana endpoints.
`status.sh` also prints:
- `periodic_snapshot_reset`
- `periodic_snapshot_max_deltas`
## Upgrade
Extract the new package, reuse the existing `.env` and `HOST_DATA_DIR`, then run:
Extract the new package into a new directory and explicitly reuse the existing `.env` through the upgrade script:
```bash
./upgrade.sh
./upgrade.sh --reuse-env-from /path/to/old-installer/.env
```
Upgrade does not delete runtime data.
If the new package directory already has a `.env`, the upgrade script keeps it.
Upgrade does not delete:
- `runs/`
- `logs/`
- `state/rsync-mirror`
- runtime configuration referenced by `.env`
- Prometheus / Grafana data
To validate periodic forced snapshot behavior, temporarily set:
```bash
PERIODIC_SNAPSHOT_RESET=1
PERIODIC_SNAPSHOT_MAX_DELTAS=2
```
Then confirm the latest `run-meta.json` contains:
```bash
snapshot_reason=periodic_snapshot_delta_limit
```
After validation, restore:
```bash
PERIODIC_SNAPSHOT_MAX_DELTAS=100
```
## Cleanup

View File

@@ -48,15 +48,47 @@
- 最新 run 状态;
- metrics、Prometheus、Grafana endpoint。
`status.sh` 还会显示:
- `periodic_snapshot_reset`
- `periodic_snapshot_max_deltas`
## 升级
把新安装包解压到新目录后,可以复用旧 `.env` 和 `HOST_DATA_DIR`。执行:
把新安装包解压到新目录后,推荐通过升级脚本显式复用旧 `.env`:
```bash
./upgrade.sh
./upgrade.sh --reuse-env-from /path/to/old-installer/.env
```
升级不会删除运行数据。
如果新目录已经存在 `.env`,升级脚本会保留它,不覆盖。
升级不会删除以下数据:
- `runs/`
- `logs/`
- `state/rsync-mirror`
- `.env` 对应的运行配置
- Prometheus / Grafana 数据
验证定期 forced snapshot 时,可临时设置:
```bash
PERIODIC_SNAPSHOT_RESET=1
PERIODIC_SNAPSHOT_MAX_DELTAS=2
```
然后检查最新 `run-meta.json` 应出现:
```bash
snapshot_reason=periodic_snapshot_delta_limit
```
验证完成后恢复:
```bash
PERIODIC_SNAPSHOT_MAX_DELTAS=100
```
## 清理

View File

@@ -66,3 +66,18 @@ For finite tests, set:
```bash
SOAK_RESTART_POLICY=no
```
## How to Confirm a Periodic Forced Snapshot
Check the latest run metadata:
```bash
latest="$(find ${HOST_DATA_DIR}/runs -maxdepth 1 -type d -name 'run_*' | sort | tail -1)"
jq '{run_id,sync_mode,snapshot_reason,periodic_snapshot_delta_count,periodic_snapshot_forced,reset_db_cleanup_status}' "$latest/run-meta.json"
```
For a threshold-triggered reset you should see:
- `sync_mode: "snapshot"`
- `snapshot_reason: "periodic_snapshot_delta_limit"`
- `periodic_snapshot_forced: true`

View File

@@ -66,3 +66,18 @@ GRAFANA_ADMIN_PASSWORD=admin
```bash
SOAK_RESTART_POLICY=no
```
## 如何确认触发了定期 forced snapshot
检查最新 run metadata:
```bash
latest="$(find ${HOST_DATA_DIR}/runs -maxdepth 1 -type d -name 'run_*' | sort | tail -1)"
jq '{run_id,sync_mode,snapshot_reason,periodic_snapshot_delta_count,periodic_snapshot_forced,reset_db_cleanup_status}' "$latest/run-meta.json"
```
阈值触发时应看到:
- `sync_mode: "snapshot"`
- `snapshot_reason: "periodic_snapshot_delta_limit"`
- `periodic_snapshot_forced: true`