20260710 rpctl统一实验标识为exp-id
This commit is contained in:
parent
87607a3991
commit
de4f40eb2c
@ -1,6 +1,6 @@
|
||||
# rpctl
|
||||
|
||||
`rpctl` 是四款 RP 本地控制 CLI，用于在实验服务器本地独立启动、停止和观察单个 RP/profile 的 all5 run。
|
||||
`rpctl` 是四款 RP 本地控制 CLI，用于在实验服务器本地独立启动、停止和观察单个 RP/profile 的 all5 实验。一个实验可包含一次 snapshot 和多次 delta 子 run。
|
||||
|
||||
## 部署
|
||||
|
||||
@ -20,12 +20,12 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
||||
./rpctl status
|
||||
./rpctl start --rp ours-pp-object-cache --interval 10m --runs 6 --rirs all5
|
||||
./rpctl start --rp routinator --interval 0 --runs 1 --rirs apnic
|
||||
./rpctl continue --run-id <successful-run-id> --runs 3 --interval 10m
|
||||
./rpctl continue --exp-id <successful-exp-id> --runs 3 --interval 10m
|
||||
./rpctl tail
|
||||
./rpctl show
|
||||
./rpctl show --failed
|
||||
./rpctl view --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl delete --run-id <old-run-id> --dry-run
|
||||
./rpctl view --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl delete --exp-id <old-exp-id> --dry-run
|
||||
./rpctl paths
|
||||
./rpctl stop
|
||||
./rpctl stop --force
|
||||
@ -50,10 +50,10 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
||||
bin/ lib/ fixtures/ scripts/
|
||||
state/current.json
|
||||
logs/runner-*.log
|
||||
runs/<profile>/<run-id>/profiles/<profile>/runs/run_0001/
|
||||
runs/<profile>/<exp-id>/profiles/<profile>/runs/run_0001/
|
||||
```
|
||||
|
||||
每个 run 目录包含 `stdout.log`、`stderr.log`、`process-time.txt`、`exit-code.txt`、`run-meta.json` 以及各 RP 产物。
|
||||
每个子 run 目录包含 `stdout.log`、`stderr.log`、`process-time.txt`、`exit-code.txt`、`run-meta.json` 以及各 RP 产物。
|
||||
|
||||
## 历史查看与删除
|
||||
|
||||
@ -64,7 +64,7 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
||||
./rpctl show --json
|
||||
```
|
||||
|
||||
`show` 会扫描 `runs/<profile>/<run-id>/`，展示历史 `run_id`、状态、start/end 时间、失败 run 列表、运行参数、产物数量和目录路径。
|
||||
`show` 会扫描 `runs/<profile>/<exp-id>/`，展示历史 `exp_id`、状态、start/end 时间、失败子 run 列表、运行参数、产物数量和目录路径。
|
||||
|
||||
默认表格为了可读性不显示完整路径；需要路径时使用：
|
||||
|
||||
@ -73,31 +73,31 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
||||
```
|
||||
|
||||
```bash
|
||||
./rpctl view --run-id <run-id>
|
||||
./rpctl view --rp ours --run-id <run-id>
|
||||
./rpctl view --run-id <run-id> --command
|
||||
./rpctl view --run-id <run-id> --json
|
||||
./rpctl view --exp-id <exp-id>
|
||||
./rpctl view --rp ours --exp-id <exp-id>
|
||||
./rpctl view --exp-id <exp-id> --command
|
||||
./rpctl view --exp-id <exp-id> --json
|
||||
```
|
||||
|
||||
`view` 用于查看指定历史 run 的摘要和每一轮子 run 明细，包括运行耗时、max RSS、VRP/VAP 数量、同步模式、exit code 和目录路径。
|
||||
`view` 用于查看指定历史实验的摘要和每一轮子 run 明细，包括运行耗时、max RSS、VRP/VAP 数量、同步模式、exit code 和目录路径。
|
||||
|
||||
```bash
|
||||
./rpctl delete --run-id <old-run-id> --dry-run
|
||||
./rpctl delete --run-id <old-run-id> --yes
|
||||
./rpctl delete --rp ours --run-id <duplicated-run-id> --yes
|
||||
./rpctl delete --exp-id <old-exp-id> --dry-run
|
||||
./rpctl delete --exp-id <old-exp-id> --yes
|
||||
./rpctl delete --rp ours --exp-id <duplicated-exp-id> --yes
|
||||
```
|
||||
|
||||
`delete` 只能删除非 running 的 run。建议先用 `--dry-run` 确认路径；真实删除必须显式加 `--yes`。删除内容包括 `runs/<profile>/<run-id>/` 和匹配的 `logs/runner-<run-id>*.log`。
|
||||
`delete` 只能删除非 running 的实验。建议先用 `--dry-run` 确认路径；真实删除必须显式加 `--yes`。删除内容包括 `runs/<profile>/<exp-id>/` 和匹配的 `logs/runner-<exp-id>*.log`。
|
||||
|
||||
如果历史 run 是中途停止的 incomplete run，且没有 `run-meta.json`、normalized 文件或原始输出文件，则 VRP/VAP、wall、RSS 会显示为 `-`，表示该 run 没有可统计产物。
|
||||
如果历史实验中最后一个子 run 中途停止，且没有 `run-meta.json`、normalized 文件或原始输出文件，则 VRP/VAP、wall、RSS 会显示为 `-`，表示该子 run 没有可统计产物。
|
||||
|
||||
## 续跑已有成功 run
|
||||
## 续跑已有成功实验
|
||||
|
||||
```bash
|
||||
./rpctl continue --run-id <successful-run-id> --runs 3 --interval 10m
|
||||
./rpctl continue --rp routinator --run-id <successful-run-id> --runs 1 --interval 0
|
||||
./rpctl continue --exp-id <successful-exp-id> --runs 3 --interval 10m
|
||||
./rpctl continue --rp routinator --exp-id <successful-exp-id> --runs 1 --interval 0
|
||||
```
|
||||
|
||||
`continue` 只接受已经成功结束的历史 run id，并在原目录下追加 `run_000N`。它不会清理 profile state，也不会给 RP 强制传 snapshot-only 参数；后续 run 是 delta 还是 snapshot 由各 RP 根据自身持久状态决定。
|
||||
`continue` 只接受已经成功结束的历史实验 `exp_id`，并在原目录下追加 `run_000N`。它不会清理 profile state，也不会给 RP 强制传 snapshot-only 参数；后续子 run 是 delta 还是 snapshot 由各 RP 根据自身持久状态决定。
|
||||
|
||||
续跑会复用原 run 的 RIR 集合，避免在同一个 state 目录里混用不同 scope。若同一个 `run-id` 在多个 profile 下存在，需要加 `--rp` 消歧。
|
||||
续跑会复用原实验的 RIR 集合，避免在同一个 state 目录里混用不同 scope。若同一个 `exp-id` 在多个 profile 下存在，需要加 `--rp` 消歧。
|
||||
|
||||
@ -118,6 +118,23 @@ def read_json(path: Path, default: Any) -> Any:
|
||||
return default
|
||||
|
||||
|
||||
def normalize_experiment_metadata(payload: Any) -> dict[str, Any]:
|
||||
if not isinstance(payload, dict):
|
||||
return {}
|
||||
normalized = dict(payload)
|
||||
if "expId" not in normalized and normalized.get("runId"):
|
||||
normalized["expId"] = normalized["runId"]
|
||||
if "experimentRoot" not in normalized and normalized.get("runRoot"):
|
||||
normalized["experimentRoot"] = normalized["runRoot"]
|
||||
normalized.pop("runId", None)
|
||||
normalized.pop("runRoot", None)
|
||||
return normalized
|
||||
|
||||
|
||||
def read_current_state() -> dict[str, Any]:
|
||||
return normalize_experiment_metadata(read_json(current_path(), {}))
|
||||
|
||||
|
||||
def write_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||
@ -407,7 +424,7 @@ def parse_latest_meta(run_root: Path, profile: str, run_seq: int) -> dict[str, A
|
||||
|
||||
|
||||
def update_current(**fields: Any) -> None:
|
||||
cur = read_json(current_path(), {})
|
||||
cur = read_current_state()
|
||||
cur.update(fields)
|
||||
cur["updatedAtUtc"] = utc_now()
|
||||
write_json(current_path(), cur)
|
||||
@ -443,24 +460,24 @@ def parse_entry_rirs(entry: dict[str, Any]) -> list[str]:
|
||||
return parse_rirs(str(value))
|
||||
|
||||
|
||||
def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs: list[str], append: bool = False) -> int:
|
||||
def run_profile(profile: str, exp_id: str, runs: int, interval_secs: int, rirs: list[str], append: bool = False) -> int:
|
||||
ensure_dirs()
|
||||
rp = PROFILES[profile]
|
||||
run_root = runs_dir() / profile / run_id
|
||||
run_root = runs_dir() / profile / exp_id
|
||||
profile_root = run_root / "profiles" / profile
|
||||
(run_root / "logs").mkdir(parents=True, exist_ok=True)
|
||||
(run_root / "reports").mkdir(parents=True, exist_ok=True)
|
||||
(run_root / "versions.json").write_text((ctl_root() / "versions.json").read_text(encoding="utf-8"), encoding="utf-8")
|
||||
now = utc_now()
|
||||
if append:
|
||||
run_info = read_json(run_root / "run-info.json", {})
|
||||
run_info = normalize_experiment_metadata(read_json(run_root / "run-info.json", {}))
|
||||
if not run_info:
|
||||
raise RuntimeError(f"cannot continue run without run-info.json: {run_root}")
|
||||
raise RuntimeError(f"cannot continue experiment without run-info.json: {run_root}")
|
||||
if run_info.get("state") != "success" or not is_zero_exit(run_info.get("exitCode")):
|
||||
raise RuntimeError(f"can only continue a successful run: run_id={run_id} state={run_info.get('state')} exitCode={run_info.get('exitCode')}")
|
||||
raise RuntimeError(f"can only continue a successful experiment: exp_id={exp_id} state={run_info.get('state')} exitCode={run_info.get('exitCode')}")
|
||||
existing_runs = completed_run_count(profile, run_root)
|
||||
if existing_runs < 1:
|
||||
raise RuntimeError(f"cannot continue run without successful child runs: {run_root}")
|
||||
raise RuntimeError(f"cannot continue experiment without successful child runs: {run_root}")
|
||||
first_run_seq = existing_runs + 1
|
||||
last_run_seq = existing_runs + runs
|
||||
run_info.update({
|
||||
@ -476,7 +493,7 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
||||
first_run_seq = 1
|
||||
last_run_seq = runs
|
||||
run_info = {
|
||||
"runId": run_id,
|
||||
"expId": exp_id,
|
||||
"profile": profile,
|
||||
"rp": rp,
|
||||
"rirs": rirs,
|
||||
@ -484,7 +501,7 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
||||
"intervalSecs": interval_secs,
|
||||
"cacheProfile": cache_profile_name(profile),
|
||||
"startedAtUtc": now,
|
||||
"runRoot": str(run_root),
|
||||
"experimentRoot": str(run_root),
|
||||
"stateRoot": str(profile_root / "state"),
|
||||
}
|
||||
write_json(run_root / "run-info.json", run_info)
|
||||
@ -494,8 +511,8 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
||||
"profile": profile,
|
||||
"pid": os.getpid(),
|
||||
"pgid": os.getpgrp(),
|
||||
"runId": run_id,
|
||||
"runRoot": str(run_root),
|
||||
"expId": exp_id,
|
||||
"experimentRoot": str(run_root),
|
||||
"intervalSecs": interval_secs,
|
||||
"runsRequested": last_run_seq,
|
||||
"continueMode": append,
|
||||
@ -557,7 +574,7 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
||||
write_group_report(run_root, interval_secs, completed_seq)
|
||||
final_state = "success" if exit_code == 0 else "failed"
|
||||
update_current(state=final_state, finishedAtUtc=utc_now(), exitCode=exit_code)
|
||||
run_info = read_json(run_root / "run-info.json", {})
|
||||
run_info = normalize_experiment_metadata(read_json(run_root / "run-info.json", {}))
|
||||
run_info.update({
|
||||
"finishedAtUtc": utc_now(),
|
||||
"state": final_state,
|
||||
@ -569,12 +586,14 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
||||
|
||||
|
||||
def cmd_runner(args: argparse.Namespace) -> None:
|
||||
code = run_profile(args.profile, args.run_id, args.runs, args.interval_secs, args.rirs.split(","), append=args.append)
|
||||
code = run_profile(args.profile, args.exp_id, args.runs, args.interval_secs, args.rirs.split(","), append=args.append)
|
||||
sys.exit(code)
|
||||
|
||||
|
||||
def current_live_status() -> dict[str, Any]:
|
||||
cur = read_json(current_path(), {})
|
||||
raw_current = read_json(current_path(), {})
|
||||
cur = normalize_experiment_metadata(raw_current)
|
||||
migrated_legacy_fields = cur != raw_current
|
||||
pid = read_pid(pid_path()) or int(cur.get("pid") or 0)
|
||||
if not cur and not pid:
|
||||
return {}
|
||||
@ -587,6 +606,8 @@ def current_live_status() -> dict[str, Any]:
|
||||
write_json(current_path(), cur)
|
||||
else:
|
||||
cur["pidAlive"] = alive
|
||||
if migrated_legacy_fields:
|
||||
write_json(current_path(), cur)
|
||||
if pid:
|
||||
cur["pid"] = pid
|
||||
return cur
|
||||
@ -601,7 +622,7 @@ def cmd_status(args: argparse.Namespace) -> None:
|
||||
if not cur:
|
||||
print("state=idle")
|
||||
return
|
||||
keys = ["state", "profile", "rp", "pid", "pidAlive", "runId", "runRoot", "currentRun", "currentMode", "lastRunSeq", "lastRunExitCode", "lastRunWallMs", "lastRunMaxRssKb", "lastRunVrps", "lastRunVaps", "lastRunDir"]
|
||||
keys = ["state", "profile", "rp", "pid", "pidAlive", "expId", "experimentRoot", "currentRun", "currentMode", "lastRunSeq", "lastRunExitCode", "lastRunWallMs", "lastRunMaxRssKb", "lastRunVrps", "lastRunVaps", "lastRunDir"]
|
||||
for key in keys:
|
||||
if key in cur:
|
||||
print(f"{key}={cur[key]}")
|
||||
@ -625,8 +646,8 @@ def cmd_paths(args: argparse.Namespace) -> None:
|
||||
print(f"logs={logs_dir()}")
|
||||
print(f"bundle={cfg.get('bundle','')}")
|
||||
cur = current_live_status()
|
||||
if cur.get("runRoot"):
|
||||
print(f"current_run_root={cur['runRoot']}")
|
||||
if cur.get("experimentRoot"):
|
||||
print(f"current_experiment_root={cur['experimentRoot']}")
|
||||
if cur.get("lastRunDir"):
|
||||
print(f"last_run_dir={cur['lastRunDir']}")
|
||||
|
||||
@ -634,7 +655,7 @@ def cmd_paths(args: argparse.Namespace) -> None:
|
||||
def refuse_if_running() -> None:
|
||||
cur = current_live_status()
|
||||
if cur.get("state") in {"starting", "running", "waiting"} and cur.get("pidAlive"):
|
||||
raise SystemExit(f"another rpctl run is active: profile={cur.get('profile')} pid={cur.get('pid')} runRoot={cur.get('runRoot')}")
|
||||
raise SystemExit(f"another rpctl experiment is active: profile={cur.get('profile')} pid={cur.get('pid')} experimentRoot={cur.get('experimentRoot')}")
|
||||
|
||||
|
||||
def cmd_start(args: argparse.Namespace) -> None:
|
||||
@ -647,18 +668,18 @@ def cmd_start(args: argparse.Namespace) -> None:
|
||||
raise SystemExit("--runs must be >= 1")
|
||||
if not (ctl_root() / "versions.json").exists():
|
||||
raise SystemExit("rpctl is not initialized; run ./rpctl init --bundle <bundle-root> first")
|
||||
run_id = args.run_id or f"{utc_id()}_{profile}_{args.runs}runs_{interval_secs}s"
|
||||
exp_id = args.exp_id or f"{utc_id()}_{profile}_{args.runs}runs_{interval_secs}s"
|
||||
runner_args = [
|
||||
sys.executable,
|
||||
str(Path(__file__).resolve()),
|
||||
"_runner",
|
||||
"--profile", profile,
|
||||
"--run-id", run_id,
|
||||
"--exp-id", exp_id,
|
||||
"--runs", str(args.runs),
|
||||
"--interval-secs", str(interval_secs),
|
||||
"--rirs", ",".join(rirs),
|
||||
]
|
||||
nohup = logs_dir() / f"runner-{run_id}.log"
|
||||
nohup = logs_dir() / f"runner-{exp_id}.log"
|
||||
out = nohup.open("ab")
|
||||
proc = subprocess.Popen(runner_args, stdout=out, stderr=subprocess.STDOUT, start_new_session=True, cwd=str(ctl_root()))
|
||||
pid_path().write_text(str(proc.pid) + "\n", encoding="utf-8")
|
||||
@ -673,14 +694,15 @@ def cmd_start(args: argparse.Namespace) -> None:
|
||||
"rp": PROFILES[profile],
|
||||
"pid": proc.pid,
|
||||
"pgid": pgid,
|
||||
"runId": run_id,
|
||||
"expId": exp_id,
|
||||
"experimentRoot": str(runs_dir() / profile / exp_id),
|
||||
"intervalSecs": interval_secs,
|
||||
"runsRequested": args.runs,
|
||||
"rirs": rirs,
|
||||
"runnerLog": str(nohup),
|
||||
"startedAtUtc": utc_now(),
|
||||
})
|
||||
print(f"started profile={profile} pid={proc.pid} pgid={pgid} run_id={run_id}")
|
||||
print(f"started profile={profile} pid={proc.pid} pgid={pgid} exp_id={exp_id}")
|
||||
print(f"log={nohup}")
|
||||
|
||||
|
||||
@ -691,9 +713,9 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
||||
raise SystemExit("--runs must be >= 1")
|
||||
if not (ctl_root() / "versions.json").exists():
|
||||
raise SystemExit("rpctl is not initialized; run ./rpctl init --bundle <bundle-root> first")
|
||||
entry = find_run_entry(args.rp, args.run_id)
|
||||
entry = find_experiment_entry(args.rp, args.exp_id)
|
||||
if entry["state"] != "success":
|
||||
raise SystemExit(f"can only continue a successful run: run_id={args.run_id} state={entry['state']}")
|
||||
raise SystemExit(f"can only continue a successful experiment: exp_id={args.exp_id} state={entry['state']}")
|
||||
profile = entry["profile"]
|
||||
rirs = parse_entry_rirs(entry)
|
||||
interval_secs = parse_interval(args.interval)
|
||||
@ -702,13 +724,13 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
||||
str(Path(__file__).resolve()),
|
||||
"_runner",
|
||||
"--profile", profile,
|
||||
"--run-id", args.run_id,
|
||||
"--exp-id", args.exp_id,
|
||||
"--runs", str(args.runs),
|
||||
"--interval-secs", str(interval_secs),
|
||||
"--rirs", ",".join(rirs),
|
||||
"--append",
|
||||
]
|
||||
nohup = logs_dir() / f"runner-{args.run_id}-continue-{utc_id()}.log"
|
||||
nohup = logs_dir() / f"runner-{args.exp_id}-continue-{utc_id()}.log"
|
||||
out = nohup.open("ab")
|
||||
proc = subprocess.Popen(runner_args, stdout=out, stderr=subprocess.STDOUT, start_new_session=True, cwd=str(ctl_root()))
|
||||
pid_path().write_text(str(proc.pid) + "\n", encoding="utf-8")
|
||||
@ -724,7 +746,8 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
||||
"rp": PROFILES[profile],
|
||||
"pid": proc.pid,
|
||||
"pgid": pgid,
|
||||
"runId": args.run_id,
|
||||
"expId": args.exp_id,
|
||||
"experimentRoot": entry["experimentRoot"],
|
||||
"intervalSecs": interval_secs,
|
||||
"runsRequested": previous_runs + args.runs,
|
||||
"continueMode": True,
|
||||
@ -734,7 +757,7 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
||||
"runnerLog": str(nohup),
|
||||
"startedAtUtc": utc_now(),
|
||||
})
|
||||
print(f"continued profile={profile} pid={proc.pid} pgid={pgid} run_id={args.run_id} add_runs={args.runs}")
|
||||
print(f"continued profile={profile} pid={proc.pid} pgid={pgid} exp_id={args.exp_id} add_runs={args.runs}")
|
||||
print(f"log={nohup}")
|
||||
|
||||
|
||||
@ -776,9 +799,9 @@ def cmd_tail(args: argparse.Namespace) -> None:
|
||||
runner_log = cur.get("runnerLog")
|
||||
if runner_log:
|
||||
candidates.append(Path(runner_log))
|
||||
run_root = cur.get("runRoot")
|
||||
if run_root:
|
||||
candidates.append(Path(run_root) / "logs" / "driver-progress.log")
|
||||
experiment_root = cur.get("experimentRoot")
|
||||
if experiment_root:
|
||||
candidates.append(Path(experiment_root) / "logs" / "driver-progress.log")
|
||||
candidates.extend(sorted(logs_dir().glob("runner-*.log"), key=lambda p: p.stat().st_mtime if p.exists() else 0, reverse=True))
|
||||
path = next((p for p in candidates if p.is_file()), None)
|
||||
if path is None:
|
||||
@ -1030,8 +1053,8 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
||||
for profile_dir in sorted([p for p in runs_dir().iterdir() if p.is_dir()]):
|
||||
profile = profile_dir.name
|
||||
for run_root in sorted([p for p in profile_dir.iterdir() if p.is_dir()]):
|
||||
run_id = run_root.name
|
||||
run_info = read_json(run_root / "run-info.json", {})
|
||||
exp_id = run_root.name
|
||||
run_info = normalize_experiment_metadata(read_json(run_root / "run-info.json", {}))
|
||||
reports_csv = run_root / "reports" / "per_run_metrics.csv"
|
||||
rows = load_csv_rows(reports_csv)
|
||||
profile_run_root = run_root / "profiles" / profile / "runs"
|
||||
@ -1107,7 +1130,7 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
||||
first_cmd = run_root / "profiles" / profile / "runs" / "run_0001" / "command.txt"
|
||||
command = read_text_strip(first_cmd)
|
||||
state = "unknown"
|
||||
if current.get("runRoot") == str(run_root) and current.get("state") in {"starting", "running", "waiting", "stopping", "stopped", "success", "failed", "exited"}:
|
||||
if current.get("experimentRoot") == str(run_root) and current.get("state") in {"starting", "running", "waiting", "stopping", "stopped", "success", "failed", "exited"}:
|
||||
state = str(current.get("state"))
|
||||
elif failed_runs:
|
||||
state = "failed"
|
||||
@ -1121,7 +1144,7 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
||||
runs_requested = first_non_empty(run_info.get("runsRequested"))
|
||||
rirs = ",".join(run_info.get("rirs", [])) if isinstance(run_info.get("rirs"), list) else first_non_empty(run_info.get("rirs"))
|
||||
cache_profile = first_non_empty(run_info.get("cacheProfile"), cache_profile_name(profile))
|
||||
if current.get("runRoot") == str(run_root):
|
||||
if current.get("experimentRoot") == str(run_root):
|
||||
interval = first_non_empty(current.get("intervalSecs"), interval)
|
||||
runs_requested = first_non_empty(current.get("runsRequested"), runs_requested)
|
||||
current_rirs = current.get("rirs")
|
||||
@ -1130,7 +1153,7 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
||||
entries.append({
|
||||
"profile": profile,
|
||||
"rp": PROFILES.get(profile, profile),
|
||||
"runId": run_id,
|
||||
"expId": exp_id,
|
||||
"state": state,
|
||||
"startedAt": started,
|
||||
"endedAt": ended,
|
||||
@ -1146,11 +1169,11 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
||||
"maxRssKb": max(max_rss_values) if max_rss_values else "",
|
||||
"vrps": vrps,
|
||||
"vaps": vaps,
|
||||
"runRoot": str(run_root),
|
||||
"experimentRoot": str(run_root),
|
||||
"commandSummary": parse_command_summary(command),
|
||||
"command": command,
|
||||
})
|
||||
entries.sort(key=lambda item: (item.get("startedAt") or "", item.get("runRoot") or ""), reverse=True)
|
||||
entries.sort(key=lambda item: (item.get("startedAt") or "", item.get("experimentRoot") or ""), reverse=True)
|
||||
return entries
|
||||
|
||||
|
||||
@ -1185,10 +1208,10 @@ def cmd_show(args: argparse.Namespace) -> None:
|
||||
if args.json:
|
||||
print(json.dumps(entries, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return
|
||||
headers = ["profile", "run_id", "state", "runs", "params", "start", "end", "failed", "wall", "vrps", "vaps"]
|
||||
headers = ["profile", "exp_id", "state", "runs", "params", "start", "end", "failed", "wall", "vrps", "vaps"]
|
||||
rows = [[
|
||||
profile_label(e["profile"]),
|
||||
e["runId"],
|
||||
e["expId"],
|
||||
e["state"],
|
||||
e["runCount"],
|
||||
format_run_params(e),
|
||||
@ -1202,31 +1225,31 @@ def cmd_show(args: argparse.Namespace) -> None:
|
||||
if args.paths:
|
||||
headers.append("path")
|
||||
for row, entry in zip(rows, entries):
|
||||
row.append(entry["runRoot"])
|
||||
row.append(entry["experimentRoot"])
|
||||
if not rows:
|
||||
print("no runs found")
|
||||
print("no experiments found")
|
||||
return
|
||||
print_table(rows, headers)
|
||||
|
||||
|
||||
def find_run_entry(profile_or_alias: str | None, run_id: str) -> dict[str, Any]:
|
||||
def find_experiment_entry(profile_or_alias: str | None, exp_id: str) -> dict[str, Any]:
|
||||
profile = normalize_profile(profile_or_alias) if profile_or_alias else None
|
||||
entries = discover_run_entries()
|
||||
matches = [entry for entry in entries if entry["runId"] == run_id and (profile is None or entry["profile"] == profile)]
|
||||
matches = [entry for entry in entries if entry["expId"] == exp_id and (profile is None or entry["profile"] == profile)]
|
||||
if not matches:
|
||||
raise SystemExit(f"run not found: run_id={run_id}" + (f" profile={profile}" if profile else ""))
|
||||
raise SystemExit(f"experiment not found: exp_id={exp_id}" + (f" profile={profile}" if profile else ""))
|
||||
if len(matches) > 1:
|
||||
profiles = ", ".join(entry["profile"] for entry in matches)
|
||||
raise SystemExit(f"run_id is ambiguous; specify --rp. matches: {profiles}")
|
||||
raise SystemExit(f"exp_id is ambiguous; specify --rp. matches: {profiles}")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def is_running_entry(entry: dict[str, Any]) -> bool:
|
||||
run_root = str(Path(str(entry.get("runRoot"))).resolve())
|
||||
run_root = str(Path(str(entry.get("experimentRoot"))).resolve())
|
||||
cur = current_live_status()
|
||||
if cur.get("runRoot"):
|
||||
if cur.get("experimentRoot"):
|
||||
try:
|
||||
current_root = Path(str(cur.get("runRoot"))).resolve()
|
||||
current_root = Path(str(cur.get("experimentRoot"))).resolve()
|
||||
target_root = Path(run_root)
|
||||
if (
|
||||
current_root == target_root
|
||||
@ -1258,21 +1281,21 @@ def safe_run_root(path: Path) -> Path:
|
||||
if resolved == base or base not in resolved.parents:
|
||||
raise SystemExit(f"refusing to delete path outside runs dir: {resolved}")
|
||||
if len(resolved.relative_to(base).parts) < 2:
|
||||
raise SystemExit(f"refusing to delete non-run root path: {resolved}")
|
||||
raise SystemExit(f"refusing to delete non-experiment root path: {resolved}")
|
||||
return resolved
|
||||
|
||||
|
||||
def matching_runner_logs(run_id: str) -> list[Path]:
|
||||
prefix = f"runner-{run_id}"
|
||||
def matching_runner_logs(exp_id: str) -> list[Path]:
|
||||
prefix = f"runner-{exp_id}"
|
||||
return sorted(path for path in logs_dir().glob("runner-*.log") if path.name.startswith(prefix))
|
||||
|
||||
|
||||
def clear_current_if_points_to(run_root: Path) -> None:
|
||||
cur = current_live_status()
|
||||
if not cur.get("runRoot"):
|
||||
if not cur.get("experimentRoot"):
|
||||
return
|
||||
try:
|
||||
if Path(str(cur.get("runRoot"))).resolve() != run_root.resolve():
|
||||
if Path(str(cur.get("experimentRoot"))).resolve() != run_root.resolve():
|
||||
return
|
||||
except Exception:
|
||||
return
|
||||
@ -1284,13 +1307,13 @@ def clear_current_if_points_to(run_root: Path) -> None:
|
||||
|
||||
|
||||
def cmd_delete(args: argparse.Namespace) -> None:
|
||||
entry = find_run_entry(args.rp, args.run_id)
|
||||
entry = find_experiment_entry(args.rp, args.exp_id)
|
||||
if is_running_entry(entry):
|
||||
raise SystemExit(f"refusing to delete active run: run_id={args.run_id}. stop it first.")
|
||||
run_root = safe_run_root(Path(entry["runRoot"]))
|
||||
logs = matching_runner_logs(args.run_id)
|
||||
raise SystemExit(f"refusing to delete active experiment: exp_id={args.exp_id}. stop it first.")
|
||||
run_root = safe_run_root(Path(entry["experimentRoot"]))
|
||||
logs = matching_runner_logs(args.exp_id)
|
||||
if args.dry_run:
|
||||
print(f"run_root={run_root}")
|
||||
print(f"experiment_root={run_root}")
|
||||
for log in logs:
|
||||
print(f"log={log}")
|
||||
return
|
||||
@ -1306,18 +1329,18 @@ def cmd_delete(args: argparse.Namespace) -> None:
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
clear_current_if_points_to(run_root)
|
||||
print(f"deleted run_id={args.run_id}")
|
||||
print(f"run_root={run_root}")
|
||||
print(f"deleted exp_id={args.exp_id}")
|
||||
print(f"experiment_root={run_root}")
|
||||
print(f"runner_logs_deleted={deleted_logs}")
|
||||
|
||||
|
||||
def run_rows_for_entry(entry: dict[str, Any]) -> list[dict[str, str]]:
|
||||
return load_csv_rows(Path(entry["runRoot"]) / "reports" / "per_run_metrics.csv")
|
||||
return load_csv_rows(Path(entry["experimentRoot"]) / "reports" / "per_run_metrics.csv")
|
||||
|
||||
|
||||
def fallback_run_details(entry: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
profile = entry["profile"]
|
||||
root = Path(entry["runRoot"]) / "profiles" / profile / "runs"
|
||||
root = Path(entry["experimentRoot"]) / "profiles" / profile / "runs"
|
||||
details = []
|
||||
for run_dir in sorted(root.glob("run_*")):
|
||||
meta = read_json(run_dir / "run-meta.json", {})
|
||||
@ -1339,13 +1362,13 @@ def fallback_run_details(entry: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
|
||||
|
||||
def cmd_view(args: argparse.Namespace) -> None:
|
||||
entry = find_run_entry(args.rp, args.run_id)
|
||||
entry = find_experiment_entry(args.rp, args.exp_id)
|
||||
rows = run_rows_for_entry(entry) or fallback_run_details(entry)
|
||||
payload = {"summary": entry, "runs": rows}
|
||||
if args.json:
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return
|
||||
print(f"run_id={display_value(entry['runId'])}")
|
||||
print(f"exp_id={display_value(entry['expId'])}")
|
||||
print(f"profile={display_value(entry['profile'])}")
|
||||
print(f"rp={display_value(entry['rp'])}")
|
||||
print(f"state={display_value(entry['state'])}")
|
||||
@ -1362,7 +1385,7 @@ def cmd_view(args: argparse.Namespace) -> None:
|
||||
print(f"max_rss_kb={display_value(entry['maxRssKb'])}")
|
||||
print(f"vrps={display_value(entry['vrps'])}")
|
||||
print(f"vaps={display_value(entry['vaps'])}")
|
||||
print(f"run_root={display_value(entry['runRoot'])}")
|
||||
print(f"experiment_root={display_value(entry['experimentRoot'])}")
|
||||
print(f"command_summary={display_value(entry['commandSummary'])}")
|
||||
if args.command and entry.get("command"):
|
||||
print("command=" + entry["command"])
|
||||
@ -1428,14 +1451,14 @@ Typical usage:
|
||||
./rpctl status
|
||||
./rpctl show
|
||||
./rpctl show --failed
|
||||
./rpctl view --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl delete --run-id old_run_id --dry-run
|
||||
./rpctl view --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl delete --exp-id old_experiment_id --dry-run
|
||||
./rpctl tail -n 120
|
||||
./rpctl stop
|
||||
|
||||
Data model:
|
||||
- Run data stays under ./runs/<profile>/<run-id>/.
|
||||
- The first run is snapshot; later runs are delta and reuse the profile state.
|
||||
- Experiment data stays under ./runs/<profile>/<exp-id>/.
|
||||
- Each experiment starts with a snapshot; later child runs are deltas and reuse the profile state.
|
||||
- By default only one RP/profile can run at a time.
|
||||
""",
|
||||
)
|
||||
@ -1533,32 +1556,32 @@ Safety:
|
||||
help="RIR set: all5/all, or comma list from afrinic,apnic,arin,lacnic,ripe. Default: all5.",
|
||||
)
|
||||
p.add_argument(
|
||||
"--run-id",
|
||||
"--exp-id",
|
||||
default="",
|
||||
help="Optional run id used in ./runs/<profile>/<run-id>. Default: generated UTC id.",
|
||||
help="Optional experiment id used in ./runs/<profile>/<exp-id>. Default: generated UTC id.",
|
||||
)
|
||||
p.set_defaults(func=cmd_start)
|
||||
|
||||
p = sub.add_parser(
|
||||
"continue",
|
||||
help="append more runs to a successful historical run id",
|
||||
description="Continue an existing successful run id by appending additional runs without resetting RP state.",
|
||||
help="append child runs to a successful historical experiment",
|
||||
description="Continue an existing successful experiment by appending child runs without resetting RP state.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Run semantics:
|
||||
--run-id must refer to a historical run whose existing child runs all exited 0.
|
||||
The command appends run_000N directories under the same run root.
|
||||
--exp-id must refer to a historical experiment whose existing child runs all exited 0.
|
||||
The command appends run_000N directories under the same experiment root.
|
||||
rpctl does not clean profile state and does not force snapshot-only flags.
|
||||
Each RP decides delta/snapshot behavior from its persisted state and native implementation.
|
||||
The original RIR set is reused; continue does not allow changing scope mid-run.
|
||||
|
||||
Examples:
|
||||
./rpctl continue --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s --runs 3 --interval 10m
|
||||
./rpctl continue --rp routinator --run-id m18_237_routinator_all5_snapshot_delta_20260710T002726Z --runs 1 --interval 0
|
||||
./rpctl continue --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s --runs 3 --interval 10m
|
||||
./rpctl continue --rp routinator --exp-id m18_237_routinator_all5_snapshot_delta_20260710T002726Z --runs 1 --interval 0
|
||||
""",
|
||||
)
|
||||
p.add_argument("--run-id", required=True, help="Existing successful run id to append to.")
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter if the run id exists under multiple profiles.")
|
||||
p.add_argument("--exp-id", required=True, help="Existing successful experiment id to append to.")
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter if the experiment id exists under multiple profiles.")
|
||||
p.add_argument(
|
||||
"--interval",
|
||||
default="10m",
|
||||
@ -1569,14 +1592,14 @@ Examples:
|
||||
|
||||
p = sub.add_parser(
|
||||
"stop",
|
||||
help="stop active run",
|
||||
description="Stop the currently active rpctl runner process group.",
|
||||
help="stop active experiment",
|
||||
description="Stop the currently active rpctl experiment runner process group.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Default stop sends SIGTERM to the runner process group and waits briefly.
|
||||
Use --force to send SIGKILL when the RP does not exit.
|
||||
|
||||
Stopping does not delete run data.
|
||||
Stopping does not delete experiment data.
|
||||
|
||||
Examples:
|
||||
./rpctl stop
|
||||
@ -1589,12 +1612,12 @@ Examples:
|
||||
p = sub.add_parser(
|
||||
"status",
|
||||
help="show current status",
|
||||
description="Show current or last rpctl run status from local state files and PID liveness.",
|
||||
description="Show current or last rpctl experiment status from local state files and PID liveness.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Human output includes:
|
||||
state, profile, rp, pid, pidAlive, runId, runRoot,
|
||||
currentRun/currentMode, lastRun wall/RSS/counts if available.
|
||||
state, profile, rp, pid, pidAlive, expId, experimentRoot,
|
||||
current child run/mode, last child-run wall/RSS/counts if available.
|
||||
|
||||
States:
|
||||
idle no state file and no runner pid
|
||||
@ -1603,7 +1626,7 @@ States:
|
||||
waiting waiting for next interval
|
||||
stopped stopped by rpctl stop
|
||||
failed runner finished with non-zero exit code
|
||||
success all requested runs completed successfully
|
||||
success all requested child runs completed successfully
|
||||
exited state said running, but PID is gone
|
||||
|
||||
Examples:
|
||||
@ -1617,14 +1640,14 @@ Examples:
|
||||
p = sub.add_parser(
|
||||
"paths",
|
||||
help="show important paths",
|
||||
description="Print local control, state, log, bundle and run data paths.",
|
||||
description="Print local control, state, log, bundle and experiment data paths.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Useful paths:
|
||||
root rpctl installation root
|
||||
config init metadata
|
||||
state current.json, runner.pid, runner.pgid
|
||||
runs all generated run data
|
||||
runs all generated experiment data
|
||||
logs runner nohup logs
|
||||
bundle original bundle path recorded by init
|
||||
|
||||
@ -1641,8 +1664,8 @@ Example:
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
This shows rpctl runner errors and Python stack traces if the wrapper fails.
|
||||
Per-RP stdout/stderr are stored inside the current run directory.
|
||||
Use ./rpctl paths to find the latest run directory.
|
||||
Per-RP stdout/stderr are stored inside the current child-run directory.
|
||||
Use ./rpctl paths to find the latest experiment directory.
|
||||
|
||||
Examples:
|
||||
./rpctl tail
|
||||
@ -1654,11 +1677,11 @@ Examples:
|
||||
|
||||
p = sub.add_parser(
|
||||
"show",
|
||||
help="list historical run ids and summaries",
|
||||
description="List historical rpctl run roots with status, start/end time, parameters and product counts.",
|
||||
help="list historical experiment ids and summaries",
|
||||
description="List historical rpctl experiment roots with status, start/end time, parameters and product counts.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Show scans ./runs/<profile>/<run-id>/ and does not require the run to be active.
|
||||
Show scans ./runs/<profile>/<exp-id>/ and does not require the experiment to be active.
|
||||
It can identify failed/partial runs from exit codes, missing end time, or current status.
|
||||
|
||||
Examples:
|
||||
@ -1670,61 +1693,61 @@ Examples:
|
||||
""",
|
||||
)
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter, e.g. ours, ours-no-cache, routinator, rpki-client, fort.")
|
||||
p.add_argument("--failed", action="store_true", help="Only show failed, partial, incomplete or exited runs.")
|
||||
p.add_argument("--failed", action="store_true", help="Only show failed, partial, incomplete or exited experiments.")
|
||||
p.add_argument("--limit", type=int, default=50, help="Maximum rows to show. 0 means no limit. Default: 50.")
|
||||
p.add_argument("--paths", action="store_true", help="Also show full run root paths. Hidden by default to keep the table readable.")
|
||||
p.add_argument("--paths", action="store_true", help="Also show full experiment root paths. Hidden by default to keep the table readable.")
|
||||
p.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
|
||||
p.set_defaults(func=cmd_show)
|
||||
|
||||
p = sub.add_parser(
|
||||
"view",
|
||||
help="show one historical run summary",
|
||||
description="Show summary and per-run details for a specific historical run id.",
|
||||
help="show one historical experiment summary",
|
||||
description="Show summary and child-run details for a specific historical experiment id.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Use show first to find the run id. If the same run id exists under multiple
|
||||
Use show first to find the experiment id. If the same experiment id exists under multiple
|
||||
profiles, add --rp to disambiguate.
|
||||
|
||||
Examples:
|
||||
./rpctl view --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl view --rp ours --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl view --run-id m3_ours_apnic_smoke --command
|
||||
./rpctl view --run-id m3_ours_apnic_smoke --json
|
||||
./rpctl view --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl view --rp ours --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||
./rpctl view --exp-id m3_ours_apnic_smoke --command
|
||||
./rpctl view --exp-id m3_ours_apnic_smoke --json
|
||||
""",
|
||||
)
|
||||
p.add_argument("--run-id", required=True, help="Run id under ./runs/<profile>/<run-id>.")
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated run ids.")
|
||||
p.add_argument("--exp-id", required=True, help="Experiment id under ./runs/<profile>/<exp-id>.")
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated experiment ids.")
|
||||
p.add_argument("--command", action="store_true", help="Also print the first run command line.")
|
||||
p.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
|
||||
p.set_defaults(func=cmd_view)
|
||||
|
||||
p = sub.add_parser(
|
||||
"delete",
|
||||
help="delete one non-running run id and its local data",
|
||||
description="Delete a historical run root and matching runner logs. Active/running runs are refused.",
|
||||
help="delete one non-running experiment id and its local data",
|
||||
description="Delete a historical experiment root and matching runner logs. Active/running experiments are refused.",
|
||||
formatter_class=formatter,
|
||||
epilog="""
|
||||
Safety rules:
|
||||
--run-id must point to a non-running run.
|
||||
If the same run id exists under multiple profiles, add --rp.
|
||||
Use --dry-run first to print the exact run root and matching runner logs.
|
||||
--exp-id must point to a non-running experiment.
|
||||
If the same experiment id exists under multiple profiles, add --rp.
|
||||
Use --dry-run first to print the exact experiment root and matching runner logs.
|
||||
Actual deletion requires --yes.
|
||||
|
||||
Examples:
|
||||
./rpctl delete --run-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --dry-run
|
||||
./rpctl delete --run-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --yes
|
||||
./rpctl delete --rp ours --run-id duplicated_run_id --yes
|
||||
./rpctl delete --exp-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --dry-run
|
||||
./rpctl delete --exp-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --yes
|
||||
./rpctl delete --rp ours --exp-id duplicated_experiment_id --yes
|
||||
""",
|
||||
)
|
||||
p.add_argument("--run-id", required=True, help="Historical non-running run id to delete.")
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated run ids.")
|
||||
p.add_argument("--exp-id", required=True, help="Historical non-running experiment id to delete.")
|
||||
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated experiment ids.")
|
||||
p.add_argument("--dry-run", action="store_true", help="Only print the paths that would be deleted.")
|
||||
p.add_argument("--yes", action="store_true", help="Confirm deletion.")
|
||||
p.set_defaults(func=cmd_delete)
|
||||
|
||||
p = sub.add_parser("_runner", help=argparse.SUPPRESS)
|
||||
p.add_argument("--profile", required=True)
|
||||
p.add_argument("--run-id", required=True)
|
||||
p.add_argument("--exp-id", required=True)
|
||||
p.add_argument("--runs", type=int, required=True)
|
||||
p.add_argument("--interval-secs", type=int, required=True)
|
||||
p.add_argument("--rirs", required=True)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user