add lifecycle run state
This commit is contained in:
@@ -46,6 +46,9 @@ FAILURE_SNAPSHOT_RESET=1
|
||||
# 0: keep existing behavior.
|
||||
# 1: after one successful snapshot, allow at most N successful delta runs;
|
||||
# the next run is forced to snapshot and active state/db is rebuilt from empty.
|
||||
# Lifecycle run state is persisted independently at:
|
||||
# ${HOST_DATA_DIR}/state/run-lifecycle-state.json
|
||||
# It is not affected by run retention or state/db reset.
|
||||
PERIODIC_SNAPSHOT_RESET=0
|
||||
PERIODIC_SNAPSHOT_MAX_DELTAS=100
|
||||
DB_STATS_EXACT_EVERY=0
|
||||
|
||||
@@ -86,6 +86,8 @@ Semantics:
|
||||
- when enabled, one successful snapshot is followed by at most `N` successful delta runs;
|
||||
- after the threshold is reached, the next run is forced to snapshot;
|
||||
- before that forced snapshot, only the active `state/db` is reset, while `runs/`, `logs/`, `state/rsync-mirror`, `.env`, and Prometheus/Grafana data are preserved;
|
||||
- the counter is persisted independently in `HOST_DATA_DIR/state/run-lifecycle-state.json`, so it does not depend on retained `runs/` history;
|
||||
- if that lifecycle file is corrupt, it is backed up as `run-lifecycle-state.json.corrupt.<timestamp>.<pid>` before best-effort bootstrap from retained runs;
|
||||
- after a successful forced snapshot, the old DB staging is deleted so disk usage does not keep growing elsewhere.
|
||||
|
||||
Check the latest `run-meta.json` for:
|
||||
|
||||
@@ -86,6 +86,8 @@ PERIODIC_SNAPSHOT_MAX_DELTAS=100
|
||||
- 开启后,一次成功 snapshot 后最多连续执行 `N` 个成功 delta;
|
||||
- 达到阈值后,下一轮强制跑 snapshot;
|
||||
- 强制 snapshot 前只重置 active `state/db`,保留 `runs/`、`logs/`、`state/rsync-mirror`、`.env`、Prometheus/Grafana 数据;
|
||||
- 周期计数保存在独立 lifecycle 文件 `HOST_DATA_DIR/state/run-lifecycle-state.json`,不依赖 `runs/` 保留窗口;
|
||||
- lifecycle 文件损坏时会先备份为 `run-lifecycle-state.json.corrupt.<timestamp>.<pid>`,再从当前保留 run 尽力 bootstrap;
|
||||
- 强制 snapshot 成功后旧 DB staging 会被删除,避免磁盘只是换目录继续增长。
|
||||
|
||||
可通过最新 `run-meta.json` 中的以下字段确认:
|
||||
|
||||
@@ -84,6 +84,13 @@ Then confirm the latest `run-meta.json` contains:
|
||||
snapshot_reason=periodic_snapshot_delta_limit
|
||||
```
|
||||
|
||||
And inspect the independent lifecycle state:
|
||||
|
||||
```bash
|
||||
jq '{last_run,last_success_snapshot,successful_deltas_since_snapshot,state_health}' \
|
||||
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
|
||||
```
|
||||
|
||||
After validation, restore:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -84,6 +84,13 @@ PERIODIC_SNAPSHOT_MAX_DELTAS=2
|
||||
snapshot_reason=periodic_snapshot_delta_limit
|
||||
```
|
||||
|
||||
并检查独立 lifecycle 状态:
|
||||
|
||||
```bash
|
||||
jq '{last_run,last_success_snapshot,successful_deltas_since_snapshot,state_health}' \
|
||||
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
|
||||
```
|
||||
|
||||
验证完成后恢复:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -81,3 +81,19 @@ For a threshold-triggered reset you should see:
|
||||
- `sync_mode: "snapshot"`
|
||||
- `snapshot_reason: "periodic_snapshot_delta_limit"`
|
||||
- `periodic_snapshot_forced: true`
|
||||
|
||||
To confirm delta counting still advances after retained runs are pruned, also inspect:
|
||||
|
||||
```bash
|
||||
jq '{last_success_snapshot,successful_deltas_since_snapshot,recent_runs_count:(.recent_runs|length)}' \
|
||||
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
|
||||
```
|
||||
|
||||
## Lifecycle State File Is Corrupt
|
||||
|
||||
The script backs up the corrupt file before best-effort bootstrap from retained runs:
|
||||
|
||||
```bash
|
||||
ls -1 "${HOST_DATA_DIR}/state"/run-lifecycle-state.json.corrupt.*
|
||||
jq . "${HOST_DATA_DIR}/state/run-lifecycle-state.json"
|
||||
```
|
||||
|
||||
@@ -81,3 +81,19 @@ jq '{run_id,sync_mode,snapshot_reason,periodic_snapshot_delta_count,periodic_sna
|
||||
- `sync_mode: "snapshot"`
|
||||
- `snapshot_reason: "periodic_snapshot_delta_limit"`
|
||||
- `periodic_snapshot_forced: true`
|
||||
|
||||
如需确认 retain 裁剪后仍在累计 delta,可继续检查:
|
||||
|
||||
```bash
|
||||
jq '{last_success_snapshot,successful_deltas_since_snapshot,recent_runs_count:(.recent_runs|length)}' \
|
||||
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
|
||||
```
|
||||
|
||||
## Lifecycle 状态文件损坏
|
||||
|
||||
脚本会先备份损坏文件,再从当前保留的 `runs/` 尽力 bootstrap:
|
||||
|
||||
```bash
|
||||
ls -1 "${HOST_DATA_DIR}/state"/run-lifecycle-state.json.corrupt.*
|
||||
jq . "${HOST_DATA_DIR}/state/run-lifecycle-state.json"
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user