add lifecycle run state

This commit is contained in:
2026-06-29 23:16:19 +08:00
parent 0610a8291c
commit 860bad2113
9 changed files with 405 additions and 57 deletions

View File

@@ -46,6 +46,9 @@ FAILURE_SNAPSHOT_RESET=1
# 0: keep existing behavior.
# 1: after one successful snapshot, allow at most N successful delta runs;
# the next run is forced to snapshot and active state/db is rebuilt from empty.
# Lifecycle run state is persisted independently at:
# ${HOST_DATA_DIR}/state/run-lifecycle-state.json
# It is not affected by run retention or state/db reset.
PERIODIC_SNAPSHOT_RESET=0
PERIODIC_SNAPSHOT_MAX_DELTAS=100
DB_STATS_EXACT_EVERY=0

View File

@@ -86,6 +86,8 @@ Semantics:
- when enabled, one successful snapshot is followed by at most `N` successful delta runs;
- after the threshold is reached, the next run is forced to snapshot;
- before that forced snapshot, only the active `state/db` is reset, while `runs/`, `logs/`, `state/rsync-mirror`, `.env`, and Prometheus/Grafana data are preserved;
- the counter is persisted independently in `HOST_DATA_DIR/state/run-lifecycle-state.json`, so it does not depend on retained `runs/` history;
- if that lifecycle file is corrupt, it is backed up as `run-lifecycle-state.json.corrupt.<timestamp>.<pid>` before best-effort bootstrap from retained runs;
- after a successful forced snapshot, the old DB staging is deleted so disk usage does not keep growing elsewhere.
Check the latest `run-meta.json` for:

View File

@@ -86,6 +86,8 @@ PERIODIC_SNAPSHOT_MAX_DELTAS=100
- 开启后,一次成功 snapshot 后最多连续执行 `N` 个成功 delta;
- 达到阈值后,下一轮强制跑 snapshot;
- 强制 snapshot 前只重置 active `state/db`,保留 `runs/`、`logs/`、`state/rsync-mirror`、`.env`、Prometheus/Grafana 数据;
- 周期计数保存在独立 lifecycle 文件 `HOST_DATA_DIR/state/run-lifecycle-state.json`,不依赖 `runs/` 保留窗口;
- lifecycle 文件损坏时会先备份为 `run-lifecycle-state.json.corrupt.<timestamp>.<pid>`,再从当前保留 run 尽力 bootstrap;
- 强制 snapshot 成功后旧 DB staging 会被删除,避免磁盘只是换目录继续增长。
可通过最新 `run-meta.json` 中的以下字段确认:

View File

@@ -84,6 +84,13 @@ Then confirm the latest `run-meta.json` contains:
snapshot_reason=periodic_snapshot_delta_limit
```
And inspect the independent lifecycle state:
```bash
jq '{last_run,last_success_snapshot,successful_deltas_since_snapshot,state_health}' \
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
```
After validation, restore:
```bash

View File

@@ -84,6 +84,13 @@ PERIODIC_SNAPSHOT_MAX_DELTAS=2
snapshot_reason=periodic_snapshot_delta_limit
```
并检查独立 lifecycle 状态:
```bash
jq '{last_run,last_success_snapshot,successful_deltas_since_snapshot,state_health}' \
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
```
验证完成后恢复:
```bash

View File

@@ -81,3 +81,19 @@ For a threshold-triggered reset you should see:
- `sync_mode: "snapshot"`
- `snapshot_reason: "periodic_snapshot_delta_limit"`
- `periodic_snapshot_forced: true`
To confirm delta counting still advances after retained runs are pruned, also inspect:
```bash
jq '{last_success_snapshot,successful_deltas_since_snapshot,recent_runs_count:(.recent_runs|length)}' \
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
```
## Lifecycle State File Is Corrupt
The script backs up the corrupt file before best-effort bootstrap from retained runs:
```bash
ls -1 "${HOST_DATA_DIR}/state"/run-lifecycle-state.json.corrupt.*
jq . "${HOST_DATA_DIR}/state/run-lifecycle-state.json"
```

View File

@@ -81,3 +81,19 @@ jq '{run_id,sync_mode,snapshot_reason,periodic_snapshot_delta_count,periodic_sna
- `sync_mode: "snapshot"`
- `snapshot_reason: "periodic_snapshot_delta_limit"`
- `periodic_snapshot_forced: true`
如需确认 retain 裁剪后仍在累计 delta,可继续检查:
```bash
jq '{last_success_snapshot,successful_deltas_since_snapshot,recent_runs_count:(.recent_runs|length)}' \
"${HOST_DATA_DIR}/state/run-lifecycle-state.json"
```
## Lifecycle 状态文件损坏
脚本会先备份损坏文件,再从当前保留的 `runs/` 尽力 bootstrap:
```bash
ls -1 "${HOST_DATA_DIR}/state"/run-lifecycle-state.json.corrupt.*
jq . "${HOST_DATA_DIR}/state/run-lifecycle-state.json"
```