add periodic snapshot reset
This commit is contained in:
@@ -26,6 +26,8 @@ Defaults:
|
||||
- `INTERVAL_SECS=600`
|
||||
- `TAL_INPUT_MODE=file-live-ta`
|
||||
- `LIVE_TA_REFRESH_BEFORE_SNAPSHOT=1`
|
||||
- `PERIODIC_SNAPSHOT_RESET=0`
|
||||
- `PERIODIC_SNAPSHOT_MAX_DELTAS=100`
|
||||
- `HOST_DATA_DIR=/var/lib/ours-rp-arm64`
|
||||
- `SOAK_RESTART_POLICY=unless-stopped`
|
||||
- `MONITOR_PLATFORM=linux/arm64`
|
||||
@@ -69,6 +71,31 @@ Default host directory:
|
||||
|
||||
Each `runs/run_XXXX/` directory contains `report.json`, `result.ccr`, `input.cir`, `vrps.csv`, `vaps.csv`, `stage-timing.json`, logs and metadata.
|
||||
|
||||
## Periodic Snapshot Reset
|
||||
|
||||
New knobs:
|
||||
|
||||
```bash
|
||||
PERIODIC_SNAPSHOT_RESET=0
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=100
|
||||
```
|
||||
|
||||
Semantics:
|
||||
|
||||
- disabled by default, keeping previous behavior unchanged;
|
||||
- when enabled, one successful snapshot is followed by at most `N` successful delta runs;
|
||||
- after the threshold is reached, the next run is forced to snapshot;
|
||||
- before that forced snapshot, only the active `state/db` is reset, while `runs/`, `logs/`, `state/rsync-mirror`, `.env`, and Prometheus/Grafana data are preserved;
|
||||
- after a successful forced snapshot, the old DB staging is deleted so disk usage does not keep growing elsewhere.
|
||||
|
||||
Check the latest `run-meta.json` for:
|
||||
|
||||
- `sync_mode`
|
||||
- `snapshot_reason`
|
||||
- `periodic_snapshot_delta_count`
|
||||
- `periodic_snapshot_forced`
|
||||
- `reset_db_cleanup_status`
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
|
||||
@@ -26,6 +26,8 @@ vim .env
|
||||
- `INTERVAL_SECS=600`
|
||||
- `TAL_INPUT_MODE=file-live-ta`
|
||||
- `LIVE_TA_REFRESH_BEFORE_SNAPSHOT=1`
|
||||
- `PERIODIC_SNAPSHOT_RESET=0`
|
||||
- `PERIODIC_SNAPSHOT_MAX_DELTAS=100`
|
||||
- `HOST_DATA_DIR=/var/lib/ours-rp-arm64`
|
||||
- `SOAK_RESTART_POLICY=unless-stopped`
|
||||
- `MONITOR_PLATFORM=linux/arm64`
|
||||
@@ -69,6 +71,31 @@ GRAFANA_ADMIN_PASSWORD=admin
|
||||
|
||||
`runs/run_XXXX/` 中包含每轮 `report.json`、`result.ccr`、`input.cir`、`vrps.csv`、`vaps.csv`、`stage-timing.json`、日志和元数据。
|
||||
|
||||
## 定期 snapshot reset
|
||||
|
||||
新增配置:
|
||||
|
||||
```bash
|
||||
PERIODIC_SNAPSHOT_RESET=0
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=100
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- 默认关闭,行为与旧版本一致;
|
||||
- 开启后,一次成功 snapshot 后最多连续执行 `N` 个成功 delta;
|
||||
- 达到阈值后,下一轮强制跑 snapshot;
|
||||
- 强制 snapshot 前只重置 active `state/db`,保留 `runs/`、`logs/`、`state/rsync-mirror`、`.env`、Prometheus/Grafana 数据;
|
||||
- 强制 snapshot 成功后旧 DB staging 会被删除,避免磁盘只是换目录继续增长。
|
||||
|
||||
可通过最新 `run-meta.json` 中的以下字段确认:
|
||||
|
||||
- `sync_mode`
|
||||
- `snapshot_reason`
|
||||
- `periodic_snapshot_delta_count`
|
||||
- `periodic_snapshot_forced`
|
||||
- `reset_db_cleanup_status`
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
|
||||
@@ -48,15 +48,47 @@ Important checks:
|
||||
- latest run status;
|
||||
- metrics, Prometheus and Grafana endpoints.
|
||||
|
||||
`status.sh` also prints:
|
||||
|
||||
- `periodic_snapshot_reset`
|
||||
- `periodic_snapshot_max_deltas`
|
||||
|
||||
## Upgrade
|
||||
|
||||
Extract the new package, reuse the existing `.env` and `HOST_DATA_DIR`, then run:
|
||||
Extract the new package into a new directory and explicitly reuse the existing `.env` through the upgrade script:
|
||||
|
||||
```bash
|
||||
./upgrade.sh
|
||||
./upgrade.sh --reuse-env-from /path/to/old-installer/.env
|
||||
```
|
||||
|
||||
Upgrade does not delete runtime data.
|
||||
If the new package directory already has a `.env`, the upgrade script keeps it.
|
||||
|
||||
Upgrade does not delete:
|
||||
|
||||
- `runs/`
|
||||
- `logs/`
|
||||
- `state/rsync-mirror`
|
||||
- runtime configuration referenced by `.env`
|
||||
- Prometheus / Grafana data
|
||||
|
||||
To validate periodic forced snapshot behavior, temporarily set:
|
||||
|
||||
```bash
|
||||
PERIODIC_SNAPSHOT_RESET=1
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=2
|
||||
```
|
||||
|
||||
Then confirm the latest `run-meta.json` contains:
|
||||
|
||||
```bash
|
||||
snapshot_reason=periodic_snapshot_delta_limit
|
||||
```
|
||||
|
||||
After validation, restore:
|
||||
|
||||
```bash
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=100
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
|
||||
@@ -48,15 +48,47 @@
|
||||
- 最新 run 状态;
|
||||
- metrics、Prometheus、Grafana endpoint。
|
||||
|
||||
`status.sh` 还会显示:
|
||||
|
||||
- `periodic_snapshot_reset`
|
||||
- `periodic_snapshot_max_deltas`
|
||||
|
||||
## 升级
|
||||
|
||||
把新安装包解压到新目录后,可以复用旧 `.env` 和 `HOST_DATA_DIR`。执行:
|
||||
把新安装包解压到新目录后,推荐通过升级脚本显式复用旧 `.env`:
|
||||
|
||||
```bash
|
||||
./upgrade.sh
|
||||
./upgrade.sh --reuse-env-from /path/to/old-installer/.env
|
||||
```
|
||||
|
||||
升级不会删除运行数据。
|
||||
如果新目录已经存在 `.env`,升级脚本会保留它,不覆盖。
|
||||
|
||||
升级不会删除以下数据:
|
||||
|
||||
- `runs/`
|
||||
- `logs/`
|
||||
- `state/rsync-mirror`
|
||||
- `.env` 对应的运行配置
|
||||
- Prometheus / Grafana 数据
|
||||
|
||||
验证定期 forced snapshot 时,可临时设置:
|
||||
|
||||
```bash
|
||||
PERIODIC_SNAPSHOT_RESET=1
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=2
|
||||
```
|
||||
|
||||
然后检查最新 `run-meta.json` 应出现:
|
||||
|
||||
```bash
|
||||
snapshot_reason=periodic_snapshot_delta_limit
|
||||
```
|
||||
|
||||
验证完成后恢复:
|
||||
|
||||
```bash
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=100
|
||||
```
|
||||
|
||||
## 清理
|
||||
|
||||
|
||||
@@ -66,3 +66,18 @@ For finite tests, set:
|
||||
```bash
|
||||
SOAK_RESTART_POLICY=no
|
||||
```
|
||||
|
||||
## How to Confirm a Periodic Forced Snapshot
|
||||
|
||||
Check the latest run metadata:
|
||||
|
||||
```bash
|
||||
latest="$(find ${HOST_DATA_DIR}/runs -maxdepth 1 -type d -name 'run_*' | sort | tail -1)"
|
||||
jq '{run_id,sync_mode,snapshot_reason,periodic_snapshot_delta_count,periodic_snapshot_forced,reset_db_cleanup_status}' "$latest/run-meta.json"
|
||||
```
|
||||
|
||||
For a threshold-triggered reset you should see:
|
||||
|
||||
- `sync_mode: "snapshot"`
|
||||
- `snapshot_reason: "periodic_snapshot_delta_limit"`
|
||||
- `periodic_snapshot_forced: true`
|
||||
|
||||
@@ -66,3 +66,18 @@ GRAFANA_ADMIN_PASSWORD=admin
|
||||
```bash
|
||||
SOAK_RESTART_POLICY=no
|
||||
```
|
||||
|
||||
## 如何确认触发了定期 forced snapshot
|
||||
|
||||
检查最新 run metadata:
|
||||
|
||||
```bash
|
||||
latest="$(find ${HOST_DATA_DIR}/runs -maxdepth 1 -type d -name 'run_*' | sort | tail -1)"
|
||||
jq '{run_id,sync_mode,snapshot_reason,periodic_snapshot_delta_count,periodic_snapshot_forced,reset_db_cleanup_status}' "$latest/run-meta.json"
|
||||
```
|
||||
|
||||
阈值触发时应看到:
|
||||
|
||||
- `sync_mode: "snapshot"`
|
||||
- `snapshot_reason: "periodic_snapshot_delta_limit"`
|
||||
- `periodic_snapshot_forced: true`
|
||||
|
||||
Reference in New Issue
Block a user