20260710 rpctl统一实验标识为exp-id
This commit is contained in:
parent
87607a3991
commit
de4f40eb2c
@ -1,6 +1,6 @@
|
|||||||
# rpctl
|
# rpctl
|
||||||
|
|
||||||
`rpctl` 是四款 RP 本地控制 CLI,用于在实验服务器本地独立启动、停止和观察单个 RP/profile 的 all5 run。
|
`rpctl` 是四款 RP 本地控制 CLI,用于在实验服务器本地独立启动、停止和观察单个 RP/profile 的 all5 实验。一个实验可包含一次 snapshot 和多次 delta 子 run。
|
||||||
|
|
||||||
## 部署
|
## 部署
|
||||||
|
|
||||||
@ -20,12 +20,12 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
|||||||
./rpctl status
|
./rpctl status
|
||||||
./rpctl start --rp ours-pp-object-cache --interval 10m --runs 6 --rirs all5
|
./rpctl start --rp ours-pp-object-cache --interval 10m --runs 6 --rirs all5
|
||||||
./rpctl start --rp routinator --interval 0 --runs 1 --rirs apnic
|
./rpctl start --rp routinator --interval 0 --runs 1 --rirs apnic
|
||||||
./rpctl continue --run-id <successful-run-id> --runs 3 --interval 10m
|
./rpctl continue --exp-id <successful-exp-id> --runs 3 --interval 10m
|
||||||
./rpctl tail
|
./rpctl tail
|
||||||
./rpctl show
|
./rpctl show
|
||||||
./rpctl show --failed
|
./rpctl show --failed
|
||||||
./rpctl view --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
./rpctl view --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||||
./rpctl delete --run-id <old-run-id> --dry-run
|
./rpctl delete --exp-id <old-exp-id> --dry-run
|
||||||
./rpctl paths
|
./rpctl paths
|
||||||
./rpctl stop
|
./rpctl stop
|
||||||
./rpctl stop --force
|
./rpctl stop --force
|
||||||
@ -50,10 +50,10 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
|||||||
bin/ lib/ fixtures/ scripts/
|
bin/ lib/ fixtures/ scripts/
|
||||||
state/current.json
|
state/current.json
|
||||||
logs/runner-*.log
|
logs/runner-*.log
|
||||||
runs/<profile>/<run-id>/profiles/<profile>/runs/run_0001/
|
runs/<profile>/<exp-id>/profiles/<profile>/runs/run_0001/
|
||||||
```
|
```
|
||||||
|
|
||||||
每个 run 目录包含 `stdout.log`、`stderr.log`、`process-time.txt`、`exit-code.txt`、`run-meta.json` 以及各 RP 产物。
|
每个子 run 目录包含 `stdout.log`、`stderr.log`、`process-time.txt`、`exit-code.txt`、`run-meta.json` 以及各 RP 产物。
|
||||||
|
|
||||||
## 历史查看与删除
|
## 历史查看与删除
|
||||||
|
|
||||||
@ -64,7 +64,7 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
|||||||
./rpctl show --json
|
./rpctl show --json
|
||||||
```
|
```
|
||||||
|
|
||||||
`show` 会扫描 `runs/<profile>/<run-id>/`,展示历史 `run_id`、状态、start/end 时间、失败 run 列表、运行参数、产物数量和目录路径。
|
`show` 会扫描 `runs/<profile>/<exp-id>/`,展示历史 `exp_id`、状态、start/end 时间、失败子 run 列表、运行参数、产物数量和目录路径。
|
||||||
|
|
||||||
默认表格为了可读性不显示完整路径;需要路径时使用:
|
默认表格为了可读性不显示完整路径;需要路径时使用:
|
||||||
|
|
||||||
@ -73,31 +73,31 @@ ssh root@47.77.237.41 'cd /root/rpki_4rp_ctl && ./rpctl init --bundle /root/rpki
|
|||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./rpctl view --run-id <run-id>
|
./rpctl view --exp-id <exp-id>
|
||||||
./rpctl view --rp ours --run-id <run-id>
|
./rpctl view --rp ours --exp-id <exp-id>
|
||||||
./rpctl view --run-id <run-id> --command
|
./rpctl view --exp-id <exp-id> --command
|
||||||
./rpctl view --run-id <run-id> --json
|
./rpctl view --exp-id <exp-id> --json
|
||||||
```
|
```
|
||||||
|
|
||||||
`view` 用于查看指定历史 run 的摘要和每一轮子 run 明细,包括运行耗时、max RSS、VRP/VAP 数量、同步模式、exit code 和目录路径。
|
`view` 用于查看指定历史实验的摘要和每一轮子 run 明细,包括运行耗时、max RSS、VRP/VAP 数量、同步模式、exit code 和目录路径。
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./rpctl delete --run-id <old-run-id> --dry-run
|
./rpctl delete --exp-id <old-exp-id> --dry-run
|
||||||
./rpctl delete --run-id <old-run-id> --yes
|
./rpctl delete --exp-id <old-exp-id> --yes
|
||||||
./rpctl delete --rp ours --run-id <duplicated-run-id> --yes
|
./rpctl delete --rp ours --exp-id <duplicated-exp-id> --yes
|
||||||
```
|
```
|
||||||
|
|
||||||
`delete` 只能删除非 running 的 run。建议先用 `--dry-run` 确认路径;真实删除必须显式加 `--yes`。删除内容包括 `runs/<profile>/<run-id>/` 和匹配的 `logs/runner-<run-id>*.log`。
|
`delete` 只能删除非 running 的实验。建议先用 `--dry-run` 确认路径;真实删除必须显式加 `--yes`。删除内容包括 `runs/<profile>/<exp-id>/` 和匹配的 `logs/runner-<exp-id>*.log`。
|
||||||
|
|
||||||
如果历史 run 是中途停止的 incomplete run,且没有 `run-meta.json`、normalized 文件或原始输出文件,则 VRP/VAP、wall、RSS 会显示为 `-`,表示该 run 没有可统计产物。
|
如果历史实验中最后一个子 run 中途停止,且没有 `run-meta.json`、normalized 文件或原始输出文件,则 VRP/VAP、wall、RSS 会显示为 `-`,表示该子 run 没有可统计产物。
|
||||||
|
|
||||||
## 续跑已有成功 run
|
## 续跑已有成功实验
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./rpctl continue --run-id <successful-run-id> --runs 3 --interval 10m
|
./rpctl continue --exp-id <successful-exp-id> --runs 3 --interval 10m
|
||||||
./rpctl continue --rp routinator --run-id <successful-run-id> --runs 1 --interval 0
|
./rpctl continue --rp routinator --exp-id <successful-exp-id> --runs 1 --interval 0
|
||||||
```
|
```
|
||||||
|
|
||||||
`continue` 只接受已经成功结束的历史 run id,并在原目录下追加 `run_000N`。它不会清理 profile state,也不会给 RP 强制传 snapshot-only 参数;后续 run 是 delta 还是 snapshot 由各 RP 根据自身持久状态决定。
|
`continue` 只接受已经成功结束的历史实验 `exp_id`,并在原目录下追加 `run_000N`。它不会清理 profile state,也不会给 RP 强制传 snapshot-only 参数;后续子 run 是 delta 还是 snapshot 由各 RP 根据自身持久状态决定。
|
||||||
|
|
||||||
续跑会复用原 run 的 RIR 集合,避免在同一个 state 目录里混用不同 scope。若同一个 `run-id` 在多个 profile 下存在,需要加 `--rp` 消歧。
|
续跑会复用原实验的 RIR 集合,避免在同一个 state 目录里混用不同 scope。若同一个 `exp-id` 在多个 profile 下存在,需要加 `--rp` 消歧。
|
||||||
|
|||||||
@ -118,6 +118,23 @@ def read_json(path: Path, default: Any) -> Any:
|
|||||||
return default
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_experiment_metadata(payload: Any) -> dict[str, Any]:
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
return {}
|
||||||
|
normalized = dict(payload)
|
||||||
|
if "expId" not in normalized and normalized.get("runId"):
|
||||||
|
normalized["expId"] = normalized["runId"]
|
||||||
|
if "experimentRoot" not in normalized and normalized.get("runRoot"):
|
||||||
|
normalized["experimentRoot"] = normalized["runRoot"]
|
||||||
|
normalized.pop("runId", None)
|
||||||
|
normalized.pop("runRoot", None)
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def read_current_state() -> dict[str, Any]:
|
||||||
|
return normalize_experiment_metadata(read_json(current_path(), {}))
|
||||||
|
|
||||||
|
|
||||||
def write_json(path: Path, payload: Any) -> None:
|
def write_json(path: Path, payload: Any) -> None:
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
tmp = path.with_suffix(path.suffix + ".tmp")
|
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||||
@ -407,7 +424,7 @@ def parse_latest_meta(run_root: Path, profile: str, run_seq: int) -> dict[str, A
|
|||||||
|
|
||||||
|
|
||||||
def update_current(**fields: Any) -> None:
|
def update_current(**fields: Any) -> None:
|
||||||
cur = read_json(current_path(), {})
|
cur = read_current_state()
|
||||||
cur.update(fields)
|
cur.update(fields)
|
||||||
cur["updatedAtUtc"] = utc_now()
|
cur["updatedAtUtc"] = utc_now()
|
||||||
write_json(current_path(), cur)
|
write_json(current_path(), cur)
|
||||||
@ -443,24 +460,24 @@ def parse_entry_rirs(entry: dict[str, Any]) -> list[str]:
|
|||||||
return parse_rirs(str(value))
|
return parse_rirs(str(value))
|
||||||
|
|
||||||
|
|
||||||
def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs: list[str], append: bool = False) -> int:
|
def run_profile(profile: str, exp_id: str, runs: int, interval_secs: int, rirs: list[str], append: bool = False) -> int:
|
||||||
ensure_dirs()
|
ensure_dirs()
|
||||||
rp = PROFILES[profile]
|
rp = PROFILES[profile]
|
||||||
run_root = runs_dir() / profile / run_id
|
run_root = runs_dir() / profile / exp_id
|
||||||
profile_root = run_root / "profiles" / profile
|
profile_root = run_root / "profiles" / profile
|
||||||
(run_root / "logs").mkdir(parents=True, exist_ok=True)
|
(run_root / "logs").mkdir(parents=True, exist_ok=True)
|
||||||
(run_root / "reports").mkdir(parents=True, exist_ok=True)
|
(run_root / "reports").mkdir(parents=True, exist_ok=True)
|
||||||
(run_root / "versions.json").write_text((ctl_root() / "versions.json").read_text(encoding="utf-8"), encoding="utf-8")
|
(run_root / "versions.json").write_text((ctl_root() / "versions.json").read_text(encoding="utf-8"), encoding="utf-8")
|
||||||
now = utc_now()
|
now = utc_now()
|
||||||
if append:
|
if append:
|
||||||
run_info = read_json(run_root / "run-info.json", {})
|
run_info = normalize_experiment_metadata(read_json(run_root / "run-info.json", {}))
|
||||||
if not run_info:
|
if not run_info:
|
||||||
raise RuntimeError(f"cannot continue run without run-info.json: {run_root}")
|
raise RuntimeError(f"cannot continue experiment without run-info.json: {run_root}")
|
||||||
if run_info.get("state") != "success" or not is_zero_exit(run_info.get("exitCode")):
|
if run_info.get("state") != "success" or not is_zero_exit(run_info.get("exitCode")):
|
||||||
raise RuntimeError(f"can only continue a successful run: run_id={run_id} state={run_info.get('state')} exitCode={run_info.get('exitCode')}")
|
raise RuntimeError(f"can only continue a successful experiment: exp_id={exp_id} state={run_info.get('state')} exitCode={run_info.get('exitCode')}")
|
||||||
existing_runs = completed_run_count(profile, run_root)
|
existing_runs = completed_run_count(profile, run_root)
|
||||||
if existing_runs < 1:
|
if existing_runs < 1:
|
||||||
raise RuntimeError(f"cannot continue run without successful child runs: {run_root}")
|
raise RuntimeError(f"cannot continue experiment without successful child runs: {run_root}")
|
||||||
first_run_seq = existing_runs + 1
|
first_run_seq = existing_runs + 1
|
||||||
last_run_seq = existing_runs + runs
|
last_run_seq = existing_runs + runs
|
||||||
run_info.update({
|
run_info.update({
|
||||||
@ -476,7 +493,7 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
|||||||
first_run_seq = 1
|
first_run_seq = 1
|
||||||
last_run_seq = runs
|
last_run_seq = runs
|
||||||
run_info = {
|
run_info = {
|
||||||
"runId": run_id,
|
"expId": exp_id,
|
||||||
"profile": profile,
|
"profile": profile,
|
||||||
"rp": rp,
|
"rp": rp,
|
||||||
"rirs": rirs,
|
"rirs": rirs,
|
||||||
@ -484,7 +501,7 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
|||||||
"intervalSecs": interval_secs,
|
"intervalSecs": interval_secs,
|
||||||
"cacheProfile": cache_profile_name(profile),
|
"cacheProfile": cache_profile_name(profile),
|
||||||
"startedAtUtc": now,
|
"startedAtUtc": now,
|
||||||
"runRoot": str(run_root),
|
"experimentRoot": str(run_root),
|
||||||
"stateRoot": str(profile_root / "state"),
|
"stateRoot": str(profile_root / "state"),
|
||||||
}
|
}
|
||||||
write_json(run_root / "run-info.json", run_info)
|
write_json(run_root / "run-info.json", run_info)
|
||||||
@ -494,8 +511,8 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
|||||||
"profile": profile,
|
"profile": profile,
|
||||||
"pid": os.getpid(),
|
"pid": os.getpid(),
|
||||||
"pgid": os.getpgrp(),
|
"pgid": os.getpgrp(),
|
||||||
"runId": run_id,
|
"expId": exp_id,
|
||||||
"runRoot": str(run_root),
|
"experimentRoot": str(run_root),
|
||||||
"intervalSecs": interval_secs,
|
"intervalSecs": interval_secs,
|
||||||
"runsRequested": last_run_seq,
|
"runsRequested": last_run_seq,
|
||||||
"continueMode": append,
|
"continueMode": append,
|
||||||
@ -557,7 +574,7 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
|||||||
write_group_report(run_root, interval_secs, completed_seq)
|
write_group_report(run_root, interval_secs, completed_seq)
|
||||||
final_state = "success" if exit_code == 0 else "failed"
|
final_state = "success" if exit_code == 0 else "failed"
|
||||||
update_current(state=final_state, finishedAtUtc=utc_now(), exitCode=exit_code)
|
update_current(state=final_state, finishedAtUtc=utc_now(), exitCode=exit_code)
|
||||||
run_info = read_json(run_root / "run-info.json", {})
|
run_info = normalize_experiment_metadata(read_json(run_root / "run-info.json", {}))
|
||||||
run_info.update({
|
run_info.update({
|
||||||
"finishedAtUtc": utc_now(),
|
"finishedAtUtc": utc_now(),
|
||||||
"state": final_state,
|
"state": final_state,
|
||||||
@ -569,12 +586,14 @@ def run_profile(profile: str, run_id: str, runs: int, interval_secs: int, rirs:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_runner(args: argparse.Namespace) -> None:
|
def cmd_runner(args: argparse.Namespace) -> None:
|
||||||
code = run_profile(args.profile, args.run_id, args.runs, args.interval_secs, args.rirs.split(","), append=args.append)
|
code = run_profile(args.profile, args.exp_id, args.runs, args.interval_secs, args.rirs.split(","), append=args.append)
|
||||||
sys.exit(code)
|
sys.exit(code)
|
||||||
|
|
||||||
|
|
||||||
def current_live_status() -> dict[str, Any]:
|
def current_live_status() -> dict[str, Any]:
|
||||||
cur = read_json(current_path(), {})
|
raw_current = read_json(current_path(), {})
|
||||||
|
cur = normalize_experiment_metadata(raw_current)
|
||||||
|
migrated_legacy_fields = cur != raw_current
|
||||||
pid = read_pid(pid_path()) or int(cur.get("pid") or 0)
|
pid = read_pid(pid_path()) or int(cur.get("pid") or 0)
|
||||||
if not cur and not pid:
|
if not cur and not pid:
|
||||||
return {}
|
return {}
|
||||||
@ -587,6 +606,8 @@ def current_live_status() -> dict[str, Any]:
|
|||||||
write_json(current_path(), cur)
|
write_json(current_path(), cur)
|
||||||
else:
|
else:
|
||||||
cur["pidAlive"] = alive
|
cur["pidAlive"] = alive
|
||||||
|
if migrated_legacy_fields:
|
||||||
|
write_json(current_path(), cur)
|
||||||
if pid:
|
if pid:
|
||||||
cur["pid"] = pid
|
cur["pid"] = pid
|
||||||
return cur
|
return cur
|
||||||
@ -601,7 +622,7 @@ def cmd_status(args: argparse.Namespace) -> None:
|
|||||||
if not cur:
|
if not cur:
|
||||||
print("state=idle")
|
print("state=idle")
|
||||||
return
|
return
|
||||||
keys = ["state", "profile", "rp", "pid", "pidAlive", "runId", "runRoot", "currentRun", "currentMode", "lastRunSeq", "lastRunExitCode", "lastRunWallMs", "lastRunMaxRssKb", "lastRunVrps", "lastRunVaps", "lastRunDir"]
|
keys = ["state", "profile", "rp", "pid", "pidAlive", "expId", "experimentRoot", "currentRun", "currentMode", "lastRunSeq", "lastRunExitCode", "lastRunWallMs", "lastRunMaxRssKb", "lastRunVrps", "lastRunVaps", "lastRunDir"]
|
||||||
for key in keys:
|
for key in keys:
|
||||||
if key in cur:
|
if key in cur:
|
||||||
print(f"{key}={cur[key]}")
|
print(f"{key}={cur[key]}")
|
||||||
@ -625,8 +646,8 @@ def cmd_paths(args: argparse.Namespace) -> None:
|
|||||||
print(f"logs={logs_dir()}")
|
print(f"logs={logs_dir()}")
|
||||||
print(f"bundle={cfg.get('bundle','')}")
|
print(f"bundle={cfg.get('bundle','')}")
|
||||||
cur = current_live_status()
|
cur = current_live_status()
|
||||||
if cur.get("runRoot"):
|
if cur.get("experimentRoot"):
|
||||||
print(f"current_run_root={cur['runRoot']}")
|
print(f"current_experiment_root={cur['experimentRoot']}")
|
||||||
if cur.get("lastRunDir"):
|
if cur.get("lastRunDir"):
|
||||||
print(f"last_run_dir={cur['lastRunDir']}")
|
print(f"last_run_dir={cur['lastRunDir']}")
|
||||||
|
|
||||||
@ -634,7 +655,7 @@ def cmd_paths(args: argparse.Namespace) -> None:
|
|||||||
def refuse_if_running() -> None:
|
def refuse_if_running() -> None:
|
||||||
cur = current_live_status()
|
cur = current_live_status()
|
||||||
if cur.get("state") in {"starting", "running", "waiting"} and cur.get("pidAlive"):
|
if cur.get("state") in {"starting", "running", "waiting"} and cur.get("pidAlive"):
|
||||||
raise SystemExit(f"another rpctl run is active: profile={cur.get('profile')} pid={cur.get('pid')} runRoot={cur.get('runRoot')}")
|
raise SystemExit(f"another rpctl experiment is active: profile={cur.get('profile')} pid={cur.get('pid')} experimentRoot={cur.get('experimentRoot')}")
|
||||||
|
|
||||||
|
|
||||||
def cmd_start(args: argparse.Namespace) -> None:
|
def cmd_start(args: argparse.Namespace) -> None:
|
||||||
@ -647,18 +668,18 @@ def cmd_start(args: argparse.Namespace) -> None:
|
|||||||
raise SystemExit("--runs must be >= 1")
|
raise SystemExit("--runs must be >= 1")
|
||||||
if not (ctl_root() / "versions.json").exists():
|
if not (ctl_root() / "versions.json").exists():
|
||||||
raise SystemExit("rpctl is not initialized; run ./rpctl init --bundle <bundle-root> first")
|
raise SystemExit("rpctl is not initialized; run ./rpctl init --bundle <bundle-root> first")
|
||||||
run_id = args.run_id or f"{utc_id()}_{profile}_{args.runs}runs_{interval_secs}s"
|
exp_id = args.exp_id or f"{utc_id()}_{profile}_{args.runs}runs_{interval_secs}s"
|
||||||
runner_args = [
|
runner_args = [
|
||||||
sys.executable,
|
sys.executable,
|
||||||
str(Path(__file__).resolve()),
|
str(Path(__file__).resolve()),
|
||||||
"_runner",
|
"_runner",
|
||||||
"--profile", profile,
|
"--profile", profile,
|
||||||
"--run-id", run_id,
|
"--exp-id", exp_id,
|
||||||
"--runs", str(args.runs),
|
"--runs", str(args.runs),
|
||||||
"--interval-secs", str(interval_secs),
|
"--interval-secs", str(interval_secs),
|
||||||
"--rirs", ",".join(rirs),
|
"--rirs", ",".join(rirs),
|
||||||
]
|
]
|
||||||
nohup = logs_dir() / f"runner-{run_id}.log"
|
nohup = logs_dir() / f"runner-{exp_id}.log"
|
||||||
out = nohup.open("ab")
|
out = nohup.open("ab")
|
||||||
proc = subprocess.Popen(runner_args, stdout=out, stderr=subprocess.STDOUT, start_new_session=True, cwd=str(ctl_root()))
|
proc = subprocess.Popen(runner_args, stdout=out, stderr=subprocess.STDOUT, start_new_session=True, cwd=str(ctl_root()))
|
||||||
pid_path().write_text(str(proc.pid) + "\n", encoding="utf-8")
|
pid_path().write_text(str(proc.pid) + "\n", encoding="utf-8")
|
||||||
@ -673,14 +694,15 @@ def cmd_start(args: argparse.Namespace) -> None:
|
|||||||
"rp": PROFILES[profile],
|
"rp": PROFILES[profile],
|
||||||
"pid": proc.pid,
|
"pid": proc.pid,
|
||||||
"pgid": pgid,
|
"pgid": pgid,
|
||||||
"runId": run_id,
|
"expId": exp_id,
|
||||||
|
"experimentRoot": str(runs_dir() / profile / exp_id),
|
||||||
"intervalSecs": interval_secs,
|
"intervalSecs": interval_secs,
|
||||||
"runsRequested": args.runs,
|
"runsRequested": args.runs,
|
||||||
"rirs": rirs,
|
"rirs": rirs,
|
||||||
"runnerLog": str(nohup),
|
"runnerLog": str(nohup),
|
||||||
"startedAtUtc": utc_now(),
|
"startedAtUtc": utc_now(),
|
||||||
})
|
})
|
||||||
print(f"started profile={profile} pid={proc.pid} pgid={pgid} run_id={run_id}")
|
print(f"started profile={profile} pid={proc.pid} pgid={pgid} exp_id={exp_id}")
|
||||||
print(f"log={nohup}")
|
print(f"log={nohup}")
|
||||||
|
|
||||||
|
|
||||||
@ -691,9 +713,9 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
|||||||
raise SystemExit("--runs must be >= 1")
|
raise SystemExit("--runs must be >= 1")
|
||||||
if not (ctl_root() / "versions.json").exists():
|
if not (ctl_root() / "versions.json").exists():
|
||||||
raise SystemExit("rpctl is not initialized; run ./rpctl init --bundle <bundle-root> first")
|
raise SystemExit("rpctl is not initialized; run ./rpctl init --bundle <bundle-root> first")
|
||||||
entry = find_run_entry(args.rp, args.run_id)
|
entry = find_experiment_entry(args.rp, args.exp_id)
|
||||||
if entry["state"] != "success":
|
if entry["state"] != "success":
|
||||||
raise SystemExit(f"can only continue a successful run: run_id={args.run_id} state={entry['state']}")
|
raise SystemExit(f"can only continue a successful experiment: exp_id={args.exp_id} state={entry['state']}")
|
||||||
profile = entry["profile"]
|
profile = entry["profile"]
|
||||||
rirs = parse_entry_rirs(entry)
|
rirs = parse_entry_rirs(entry)
|
||||||
interval_secs = parse_interval(args.interval)
|
interval_secs = parse_interval(args.interval)
|
||||||
@ -702,13 +724,13 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
|||||||
str(Path(__file__).resolve()),
|
str(Path(__file__).resolve()),
|
||||||
"_runner",
|
"_runner",
|
||||||
"--profile", profile,
|
"--profile", profile,
|
||||||
"--run-id", args.run_id,
|
"--exp-id", args.exp_id,
|
||||||
"--runs", str(args.runs),
|
"--runs", str(args.runs),
|
||||||
"--interval-secs", str(interval_secs),
|
"--interval-secs", str(interval_secs),
|
||||||
"--rirs", ",".join(rirs),
|
"--rirs", ",".join(rirs),
|
||||||
"--append",
|
"--append",
|
||||||
]
|
]
|
||||||
nohup = logs_dir() / f"runner-{args.run_id}-continue-{utc_id()}.log"
|
nohup = logs_dir() / f"runner-{args.exp_id}-continue-{utc_id()}.log"
|
||||||
out = nohup.open("ab")
|
out = nohup.open("ab")
|
||||||
proc = subprocess.Popen(runner_args, stdout=out, stderr=subprocess.STDOUT, start_new_session=True, cwd=str(ctl_root()))
|
proc = subprocess.Popen(runner_args, stdout=out, stderr=subprocess.STDOUT, start_new_session=True, cwd=str(ctl_root()))
|
||||||
pid_path().write_text(str(proc.pid) + "\n", encoding="utf-8")
|
pid_path().write_text(str(proc.pid) + "\n", encoding="utf-8")
|
||||||
@ -724,7 +746,8 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
|||||||
"rp": PROFILES[profile],
|
"rp": PROFILES[profile],
|
||||||
"pid": proc.pid,
|
"pid": proc.pid,
|
||||||
"pgid": pgid,
|
"pgid": pgid,
|
||||||
"runId": args.run_id,
|
"expId": args.exp_id,
|
||||||
|
"experimentRoot": entry["experimentRoot"],
|
||||||
"intervalSecs": interval_secs,
|
"intervalSecs": interval_secs,
|
||||||
"runsRequested": previous_runs + args.runs,
|
"runsRequested": previous_runs + args.runs,
|
||||||
"continueMode": True,
|
"continueMode": True,
|
||||||
@ -734,7 +757,7 @@ def cmd_continue(args: argparse.Namespace) -> None:
|
|||||||
"runnerLog": str(nohup),
|
"runnerLog": str(nohup),
|
||||||
"startedAtUtc": utc_now(),
|
"startedAtUtc": utc_now(),
|
||||||
})
|
})
|
||||||
print(f"continued profile={profile} pid={proc.pid} pgid={pgid} run_id={args.run_id} add_runs={args.runs}")
|
print(f"continued profile={profile} pid={proc.pid} pgid={pgid} exp_id={args.exp_id} add_runs={args.runs}")
|
||||||
print(f"log={nohup}")
|
print(f"log={nohup}")
|
||||||
|
|
||||||
|
|
||||||
@ -776,9 +799,9 @@ def cmd_tail(args: argparse.Namespace) -> None:
|
|||||||
runner_log = cur.get("runnerLog")
|
runner_log = cur.get("runnerLog")
|
||||||
if runner_log:
|
if runner_log:
|
||||||
candidates.append(Path(runner_log))
|
candidates.append(Path(runner_log))
|
||||||
run_root = cur.get("runRoot")
|
experiment_root = cur.get("experimentRoot")
|
||||||
if run_root:
|
if experiment_root:
|
||||||
candidates.append(Path(run_root) / "logs" / "driver-progress.log")
|
candidates.append(Path(experiment_root) / "logs" / "driver-progress.log")
|
||||||
candidates.extend(sorted(logs_dir().glob("runner-*.log"), key=lambda p: p.stat().st_mtime if p.exists() else 0, reverse=True))
|
candidates.extend(sorted(logs_dir().glob("runner-*.log"), key=lambda p: p.stat().st_mtime if p.exists() else 0, reverse=True))
|
||||||
path = next((p for p in candidates if p.is_file()), None)
|
path = next((p for p in candidates if p.is_file()), None)
|
||||||
if path is None:
|
if path is None:
|
||||||
@ -1030,8 +1053,8 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
|||||||
for profile_dir in sorted([p for p in runs_dir().iterdir() if p.is_dir()]):
|
for profile_dir in sorted([p for p in runs_dir().iterdir() if p.is_dir()]):
|
||||||
profile = profile_dir.name
|
profile = profile_dir.name
|
||||||
for run_root in sorted([p for p in profile_dir.iterdir() if p.is_dir()]):
|
for run_root in sorted([p for p in profile_dir.iterdir() if p.is_dir()]):
|
||||||
run_id = run_root.name
|
exp_id = run_root.name
|
||||||
run_info = read_json(run_root / "run-info.json", {})
|
run_info = normalize_experiment_metadata(read_json(run_root / "run-info.json", {}))
|
||||||
reports_csv = run_root / "reports" / "per_run_metrics.csv"
|
reports_csv = run_root / "reports" / "per_run_metrics.csv"
|
||||||
rows = load_csv_rows(reports_csv)
|
rows = load_csv_rows(reports_csv)
|
||||||
profile_run_root = run_root / "profiles" / profile / "runs"
|
profile_run_root = run_root / "profiles" / profile / "runs"
|
||||||
@ -1107,7 +1130,7 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
|||||||
first_cmd = run_root / "profiles" / profile / "runs" / "run_0001" / "command.txt"
|
first_cmd = run_root / "profiles" / profile / "runs" / "run_0001" / "command.txt"
|
||||||
command = read_text_strip(first_cmd)
|
command = read_text_strip(first_cmd)
|
||||||
state = "unknown"
|
state = "unknown"
|
||||||
if current.get("runRoot") == str(run_root) and current.get("state") in {"starting", "running", "waiting", "stopping", "stopped", "success", "failed", "exited"}:
|
if current.get("experimentRoot") == str(run_root) and current.get("state") in {"starting", "running", "waiting", "stopping", "stopped", "success", "failed", "exited"}:
|
||||||
state = str(current.get("state"))
|
state = str(current.get("state"))
|
||||||
elif failed_runs:
|
elif failed_runs:
|
||||||
state = "failed"
|
state = "failed"
|
||||||
@ -1121,7 +1144,7 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
|||||||
runs_requested = first_non_empty(run_info.get("runsRequested"))
|
runs_requested = first_non_empty(run_info.get("runsRequested"))
|
||||||
rirs = ",".join(run_info.get("rirs", [])) if isinstance(run_info.get("rirs"), list) else first_non_empty(run_info.get("rirs"))
|
rirs = ",".join(run_info.get("rirs", [])) if isinstance(run_info.get("rirs"), list) else first_non_empty(run_info.get("rirs"))
|
||||||
cache_profile = first_non_empty(run_info.get("cacheProfile"), cache_profile_name(profile))
|
cache_profile = first_non_empty(run_info.get("cacheProfile"), cache_profile_name(profile))
|
||||||
if current.get("runRoot") == str(run_root):
|
if current.get("experimentRoot") == str(run_root):
|
||||||
interval = first_non_empty(current.get("intervalSecs"), interval)
|
interval = first_non_empty(current.get("intervalSecs"), interval)
|
||||||
runs_requested = first_non_empty(current.get("runsRequested"), runs_requested)
|
runs_requested = first_non_empty(current.get("runsRequested"), runs_requested)
|
||||||
current_rirs = current.get("rirs")
|
current_rirs = current.get("rirs")
|
||||||
@ -1130,7 +1153,7 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
|||||||
entries.append({
|
entries.append({
|
||||||
"profile": profile,
|
"profile": profile,
|
||||||
"rp": PROFILES.get(profile, profile),
|
"rp": PROFILES.get(profile, profile),
|
||||||
"runId": run_id,
|
"expId": exp_id,
|
||||||
"state": state,
|
"state": state,
|
||||||
"startedAt": started,
|
"startedAt": started,
|
||||||
"endedAt": ended,
|
"endedAt": ended,
|
||||||
@ -1146,11 +1169,11 @@ def discover_run_entries() -> list[dict[str, Any]]:
|
|||||||
"maxRssKb": max(max_rss_values) if max_rss_values else "",
|
"maxRssKb": max(max_rss_values) if max_rss_values else "",
|
||||||
"vrps": vrps,
|
"vrps": vrps,
|
||||||
"vaps": vaps,
|
"vaps": vaps,
|
||||||
"runRoot": str(run_root),
|
"experimentRoot": str(run_root),
|
||||||
"commandSummary": parse_command_summary(command),
|
"commandSummary": parse_command_summary(command),
|
||||||
"command": command,
|
"command": command,
|
||||||
})
|
})
|
||||||
entries.sort(key=lambda item: (item.get("startedAt") or "", item.get("runRoot") or ""), reverse=True)
|
entries.sort(key=lambda item: (item.get("startedAt") or "", item.get("experimentRoot") or ""), reverse=True)
|
||||||
return entries
|
return entries
|
||||||
|
|
||||||
|
|
||||||
@ -1185,10 +1208,10 @@ def cmd_show(args: argparse.Namespace) -> None:
|
|||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps(entries, ensure_ascii=False, indent=2, sort_keys=True))
|
print(json.dumps(entries, ensure_ascii=False, indent=2, sort_keys=True))
|
||||||
return
|
return
|
||||||
headers = ["profile", "run_id", "state", "runs", "params", "start", "end", "failed", "wall", "vrps", "vaps"]
|
headers = ["profile", "exp_id", "state", "runs", "params", "start", "end", "failed", "wall", "vrps", "vaps"]
|
||||||
rows = [[
|
rows = [[
|
||||||
profile_label(e["profile"]),
|
profile_label(e["profile"]),
|
||||||
e["runId"],
|
e["expId"],
|
||||||
e["state"],
|
e["state"],
|
||||||
e["runCount"],
|
e["runCount"],
|
||||||
format_run_params(e),
|
format_run_params(e),
|
||||||
@ -1202,31 +1225,31 @@ def cmd_show(args: argparse.Namespace) -> None:
|
|||||||
if args.paths:
|
if args.paths:
|
||||||
headers.append("path")
|
headers.append("path")
|
||||||
for row, entry in zip(rows, entries):
|
for row, entry in zip(rows, entries):
|
||||||
row.append(entry["runRoot"])
|
row.append(entry["experimentRoot"])
|
||||||
if not rows:
|
if not rows:
|
||||||
print("no runs found")
|
print("no experiments found")
|
||||||
return
|
return
|
||||||
print_table(rows, headers)
|
print_table(rows, headers)
|
||||||
|
|
||||||
|
|
||||||
def find_run_entry(profile_or_alias: str | None, run_id: str) -> dict[str, Any]:
|
def find_experiment_entry(profile_or_alias: str | None, exp_id: str) -> dict[str, Any]:
|
||||||
profile = normalize_profile(profile_or_alias) if profile_or_alias else None
|
profile = normalize_profile(profile_or_alias) if profile_or_alias else None
|
||||||
entries = discover_run_entries()
|
entries = discover_run_entries()
|
||||||
matches = [entry for entry in entries if entry["runId"] == run_id and (profile is None or entry["profile"] == profile)]
|
matches = [entry for entry in entries if entry["expId"] == exp_id and (profile is None or entry["profile"] == profile)]
|
||||||
if not matches:
|
if not matches:
|
||||||
raise SystemExit(f"run not found: run_id={run_id}" + (f" profile={profile}" if profile else ""))
|
raise SystemExit(f"experiment not found: exp_id={exp_id}" + (f" profile={profile}" if profile else ""))
|
||||||
if len(matches) > 1:
|
if len(matches) > 1:
|
||||||
profiles = ", ".join(entry["profile"] for entry in matches)
|
profiles = ", ".join(entry["profile"] for entry in matches)
|
||||||
raise SystemExit(f"run_id is ambiguous; specify --rp. matches: {profiles}")
|
raise SystemExit(f"exp_id is ambiguous; specify --rp. matches: {profiles}")
|
||||||
return matches[0]
|
return matches[0]
|
||||||
|
|
||||||
|
|
||||||
def is_running_entry(entry: dict[str, Any]) -> bool:
|
def is_running_entry(entry: dict[str, Any]) -> bool:
|
||||||
run_root = str(Path(str(entry.get("runRoot"))).resolve())
|
run_root = str(Path(str(entry.get("experimentRoot"))).resolve())
|
||||||
cur = current_live_status()
|
cur = current_live_status()
|
||||||
if cur.get("runRoot"):
|
if cur.get("experimentRoot"):
|
||||||
try:
|
try:
|
||||||
current_root = Path(str(cur.get("runRoot"))).resolve()
|
current_root = Path(str(cur.get("experimentRoot"))).resolve()
|
||||||
target_root = Path(run_root)
|
target_root = Path(run_root)
|
||||||
if (
|
if (
|
||||||
current_root == target_root
|
current_root == target_root
|
||||||
@ -1258,21 +1281,21 @@ def safe_run_root(path: Path) -> Path:
|
|||||||
if resolved == base or base not in resolved.parents:
|
if resolved == base or base not in resolved.parents:
|
||||||
raise SystemExit(f"refusing to delete path outside runs dir: {resolved}")
|
raise SystemExit(f"refusing to delete path outside runs dir: {resolved}")
|
||||||
if len(resolved.relative_to(base).parts) < 2:
|
if len(resolved.relative_to(base).parts) < 2:
|
||||||
raise SystemExit(f"refusing to delete non-run root path: {resolved}")
|
raise SystemExit(f"refusing to delete non-experiment root path: {resolved}")
|
||||||
return resolved
|
return resolved
|
||||||
|
|
||||||
|
|
||||||
def matching_runner_logs(run_id: str) -> list[Path]:
|
def matching_runner_logs(exp_id: str) -> list[Path]:
|
||||||
prefix = f"runner-{run_id}"
|
prefix = f"runner-{exp_id}"
|
||||||
return sorted(path for path in logs_dir().glob("runner-*.log") if path.name.startswith(prefix))
|
return sorted(path for path in logs_dir().glob("runner-*.log") if path.name.startswith(prefix))
|
||||||
|
|
||||||
|
|
||||||
def clear_current_if_points_to(run_root: Path) -> None:
|
def clear_current_if_points_to(run_root: Path) -> None:
|
||||||
cur = current_live_status()
|
cur = current_live_status()
|
||||||
if not cur.get("runRoot"):
|
if not cur.get("experimentRoot"):
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
if Path(str(cur.get("runRoot"))).resolve() != run_root.resolve():
|
if Path(str(cur.get("experimentRoot"))).resolve() != run_root.resolve():
|
||||||
return
|
return
|
||||||
except Exception:
|
except Exception:
|
||||||
return
|
return
|
||||||
@ -1284,13 +1307,13 @@ def clear_current_if_points_to(run_root: Path) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_delete(args: argparse.Namespace) -> None:
|
def cmd_delete(args: argparse.Namespace) -> None:
|
||||||
entry = find_run_entry(args.rp, args.run_id)
|
entry = find_experiment_entry(args.rp, args.exp_id)
|
||||||
if is_running_entry(entry):
|
if is_running_entry(entry):
|
||||||
raise SystemExit(f"refusing to delete active run: run_id={args.run_id}. stop it first.")
|
raise SystemExit(f"refusing to delete active experiment: exp_id={args.exp_id}. stop it first.")
|
||||||
run_root = safe_run_root(Path(entry["runRoot"]))
|
run_root = safe_run_root(Path(entry["experimentRoot"]))
|
||||||
logs = matching_runner_logs(args.run_id)
|
logs = matching_runner_logs(args.exp_id)
|
||||||
if args.dry_run:
|
if args.dry_run:
|
||||||
print(f"run_root={run_root}")
|
print(f"experiment_root={run_root}")
|
||||||
for log in logs:
|
for log in logs:
|
||||||
print(f"log={log}")
|
print(f"log={log}")
|
||||||
return
|
return
|
||||||
@ -1306,18 +1329,18 @@ def cmd_delete(args: argparse.Namespace) -> None:
|
|||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
pass
|
pass
|
||||||
clear_current_if_points_to(run_root)
|
clear_current_if_points_to(run_root)
|
||||||
print(f"deleted run_id={args.run_id}")
|
print(f"deleted exp_id={args.exp_id}")
|
||||||
print(f"run_root={run_root}")
|
print(f"experiment_root={run_root}")
|
||||||
print(f"runner_logs_deleted={deleted_logs}")
|
print(f"runner_logs_deleted={deleted_logs}")
|
||||||
|
|
||||||
|
|
||||||
def run_rows_for_entry(entry: dict[str, Any]) -> list[dict[str, str]]:
|
def run_rows_for_entry(entry: dict[str, Any]) -> list[dict[str, str]]:
|
||||||
return load_csv_rows(Path(entry["runRoot"]) / "reports" / "per_run_metrics.csv")
|
return load_csv_rows(Path(entry["experimentRoot"]) / "reports" / "per_run_metrics.csv")
|
||||||
|
|
||||||
|
|
||||||
def fallback_run_details(entry: dict[str, Any]) -> list[dict[str, Any]]:
|
def fallback_run_details(entry: dict[str, Any]) -> list[dict[str, Any]]:
|
||||||
profile = entry["profile"]
|
profile = entry["profile"]
|
||||||
root = Path(entry["runRoot"]) / "profiles" / profile / "runs"
|
root = Path(entry["experimentRoot"]) / "profiles" / profile / "runs"
|
||||||
details = []
|
details = []
|
||||||
for run_dir in sorted(root.glob("run_*")):
|
for run_dir in sorted(root.glob("run_*")):
|
||||||
meta = read_json(run_dir / "run-meta.json", {})
|
meta = read_json(run_dir / "run-meta.json", {})
|
||||||
@ -1339,13 +1362,13 @@ def fallback_run_details(entry: dict[str, Any]) -> list[dict[str, Any]]:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_view(args: argparse.Namespace) -> None:
|
def cmd_view(args: argparse.Namespace) -> None:
|
||||||
entry = find_run_entry(args.rp, args.run_id)
|
entry = find_experiment_entry(args.rp, args.exp_id)
|
||||||
rows = run_rows_for_entry(entry) or fallback_run_details(entry)
|
rows = run_rows_for_entry(entry) or fallback_run_details(entry)
|
||||||
payload = {"summary": entry, "runs": rows}
|
payload = {"summary": entry, "runs": rows}
|
||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True))
|
print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True))
|
||||||
return
|
return
|
||||||
print(f"run_id={display_value(entry['runId'])}")
|
print(f"exp_id={display_value(entry['expId'])}")
|
||||||
print(f"profile={display_value(entry['profile'])}")
|
print(f"profile={display_value(entry['profile'])}")
|
||||||
print(f"rp={display_value(entry['rp'])}")
|
print(f"rp={display_value(entry['rp'])}")
|
||||||
print(f"state={display_value(entry['state'])}")
|
print(f"state={display_value(entry['state'])}")
|
||||||
@ -1362,7 +1385,7 @@ def cmd_view(args: argparse.Namespace) -> None:
|
|||||||
print(f"max_rss_kb={display_value(entry['maxRssKb'])}")
|
print(f"max_rss_kb={display_value(entry['maxRssKb'])}")
|
||||||
print(f"vrps={display_value(entry['vrps'])}")
|
print(f"vrps={display_value(entry['vrps'])}")
|
||||||
print(f"vaps={display_value(entry['vaps'])}")
|
print(f"vaps={display_value(entry['vaps'])}")
|
||||||
print(f"run_root={display_value(entry['runRoot'])}")
|
print(f"experiment_root={display_value(entry['experimentRoot'])}")
|
||||||
print(f"command_summary={display_value(entry['commandSummary'])}")
|
print(f"command_summary={display_value(entry['commandSummary'])}")
|
||||||
if args.command and entry.get("command"):
|
if args.command and entry.get("command"):
|
||||||
print("command=" + entry["command"])
|
print("command=" + entry["command"])
|
||||||
@ -1428,14 +1451,14 @@ Typical usage:
|
|||||||
./rpctl status
|
./rpctl status
|
||||||
./rpctl show
|
./rpctl show
|
||||||
./rpctl show --failed
|
./rpctl show --failed
|
||||||
./rpctl view --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
./rpctl view --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||||
./rpctl delete --run-id old_run_id --dry-run
|
./rpctl delete --exp-id old_experiment_id --dry-run
|
||||||
./rpctl tail -n 120
|
./rpctl tail -n 120
|
||||||
./rpctl stop
|
./rpctl stop
|
||||||
|
|
||||||
Data model:
|
Data model:
|
||||||
- Run data stays under ./runs/<profile>/<run-id>/.
|
- Experiment data stays under ./runs/<profile>/<exp-id>/.
|
||||||
- The first run is snapshot; later runs are delta and reuse the profile state.
|
- Each experiment starts with a snapshot; later child runs are deltas and reuse the profile state.
|
||||||
- By default only one RP/profile can run at a time.
|
- By default only one RP/profile can run at a time.
|
||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
@ -1533,32 +1556,32 @@ Safety:
|
|||||||
help="RIR set: all5/all, or comma list from afrinic,apnic,arin,lacnic,ripe. Default: all5.",
|
help="RIR set: all5/all, or comma list from afrinic,apnic,arin,lacnic,ripe. Default: all5.",
|
||||||
)
|
)
|
||||||
p.add_argument(
|
p.add_argument(
|
||||||
"--run-id",
|
"--exp-id",
|
||||||
default="",
|
default="",
|
||||||
help="Optional run id used in ./runs/<profile>/<run-id>. Default: generated UTC id.",
|
help="Optional experiment id used in ./runs/<profile>/<exp-id>. Default: generated UTC id.",
|
||||||
)
|
)
|
||||||
p.set_defaults(func=cmd_start)
|
p.set_defaults(func=cmd_start)
|
||||||
|
|
||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"continue",
|
"continue",
|
||||||
help="append more runs to a successful historical run id",
|
help="append child runs to a successful historical experiment",
|
||||||
description="Continue an existing successful run id by appending additional runs without resetting RP state.",
|
description="Continue an existing successful experiment by appending child runs without resetting RP state.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Run semantics:
|
Run semantics:
|
||||||
--run-id must refer to a historical run whose existing child runs all exited 0.
|
--exp-id must refer to a historical experiment whose existing child runs all exited 0.
|
||||||
The command appends run_000N directories under the same run root.
|
The command appends run_000N directories under the same experiment root.
|
||||||
rpctl does not clean profile state and does not force snapshot-only flags.
|
rpctl does not clean profile state and does not force snapshot-only flags.
|
||||||
Each RP decides delta/snapshot behavior from its persisted state and native implementation.
|
Each RP decides delta/snapshot behavior from its persisted state and native implementation.
|
||||||
The original RIR set is reused; continue does not allow changing scope mid-run.
|
The original RIR set is reused; continue does not allow changing scope mid-run.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
./rpctl continue --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s --runs 3 --interval 10m
|
./rpctl continue --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s --runs 3 --interval 10m
|
||||||
./rpctl continue --rp routinator --run-id m18_237_routinator_all5_snapshot_delta_20260710T002726Z --runs 1 --interval 0
|
./rpctl continue --rp routinator --exp-id m18_237_routinator_all5_snapshot_delta_20260710T002726Z --runs 1 --interval 0
|
||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
p.add_argument("--run-id", required=True, help="Existing successful run id to append to.")
|
p.add_argument("--exp-id", required=True, help="Existing successful experiment id to append to.")
|
||||||
p.add_argument("--rp", default="", help="Optional RP/profile filter if the run id exists under multiple profiles.")
|
p.add_argument("--rp", default="", help="Optional RP/profile filter if the experiment id exists under multiple profiles.")
|
||||||
p.add_argument(
|
p.add_argument(
|
||||||
"--interval",
|
"--interval",
|
||||||
default="10m",
|
default="10m",
|
||||||
@ -1569,14 +1592,14 @@ Examples:
|
|||||||
|
|
||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"stop",
|
"stop",
|
||||||
help="stop active run",
|
help="stop active experiment",
|
||||||
description="Stop the currently active rpctl runner process group.",
|
description="Stop the currently active rpctl experiment runner process group.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Default stop sends SIGTERM to the runner process group and waits briefly.
|
Default stop sends SIGTERM to the runner process group and waits briefly.
|
||||||
Use --force to send SIGKILL when the RP does not exit.
|
Use --force to send SIGKILL when the RP does not exit.
|
||||||
|
|
||||||
Stopping does not delete run data.
|
Stopping does not delete experiment data.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
./rpctl stop
|
./rpctl stop
|
||||||
@ -1589,12 +1612,12 @@ Examples:
|
|||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"status",
|
"status",
|
||||||
help="show current status",
|
help="show current status",
|
||||||
description="Show current or last rpctl run status from local state files and PID liveness.",
|
description="Show current or last rpctl experiment status from local state files and PID liveness.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Human output includes:
|
Human output includes:
|
||||||
state, profile, rp, pid, pidAlive, runId, runRoot,
|
state, profile, rp, pid, pidAlive, expId, experimentRoot,
|
||||||
currentRun/currentMode, lastRun wall/RSS/counts if available.
|
current child run/mode, last child-run wall/RSS/counts if available.
|
||||||
|
|
||||||
States:
|
States:
|
||||||
idle no state file and no runner pid
|
idle no state file and no runner pid
|
||||||
@ -1603,7 +1626,7 @@ States:
|
|||||||
waiting waiting for next interval
|
waiting waiting for next interval
|
||||||
stopped stopped by rpctl stop
|
stopped stopped by rpctl stop
|
||||||
failed runner finished with non-zero exit code
|
failed runner finished with non-zero exit code
|
||||||
success all requested runs completed successfully
|
success all requested child runs completed successfully
|
||||||
exited state said running, but PID is gone
|
exited state said running, but PID is gone
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
@ -1617,14 +1640,14 @@ Examples:
|
|||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"paths",
|
"paths",
|
||||||
help="show important paths",
|
help="show important paths",
|
||||||
description="Print local control, state, log, bundle and run data paths.",
|
description="Print local control, state, log, bundle and experiment data paths.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Useful paths:
|
Useful paths:
|
||||||
root rpctl installation root
|
root rpctl installation root
|
||||||
config init metadata
|
config init metadata
|
||||||
state current.json, runner.pid, runner.pgid
|
state current.json, runner.pid, runner.pgid
|
||||||
runs all generated run data
|
runs all generated experiment data
|
||||||
logs runner nohup logs
|
logs runner nohup logs
|
||||||
bundle original bundle path recorded by init
|
bundle original bundle path recorded by init
|
||||||
|
|
||||||
@ -1641,8 +1664,8 @@ Example:
|
|||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
This shows rpctl runner errors and Python stack traces if the wrapper fails.
|
This shows rpctl runner errors and Python stack traces if the wrapper fails.
|
||||||
Per-RP stdout/stderr are stored inside the current run directory.
|
Per-RP stdout/stderr are stored inside the current child-run directory.
|
||||||
Use ./rpctl paths to find the latest run directory.
|
Use ./rpctl paths to find the latest experiment directory.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
./rpctl tail
|
./rpctl tail
|
||||||
@ -1654,11 +1677,11 @@ Examples:
|
|||||||
|
|
||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"show",
|
"show",
|
||||||
help="list historical run ids and summaries",
|
help="list historical experiment ids and summaries",
|
||||||
description="List historical rpctl run roots with status, start/end time, parameters and product counts.",
|
description="List historical rpctl experiment roots with status, start/end time, parameters and product counts.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Show scans ./runs/<profile>/<run-id>/ and does not require the run to be active.
|
Show scans ./runs/<profile>/<exp-id>/ and does not require the experiment to be active.
|
||||||
It can identify failed/partial runs from exit codes, missing end time, or current status.
|
It can identify failed/partial runs from exit codes, missing end time, or current status.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
@ -1670,61 +1693,61 @@ Examples:
|
|||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
p.add_argument("--rp", default="", help="Optional RP/profile filter, e.g. ours, ours-no-cache, routinator, rpki-client, fort.")
|
p.add_argument("--rp", default="", help="Optional RP/profile filter, e.g. ours, ours-no-cache, routinator, rpki-client, fort.")
|
||||||
p.add_argument("--failed", action="store_true", help="Only show failed, partial, incomplete or exited runs.")
|
p.add_argument("--failed", action="store_true", help="Only show failed, partial, incomplete or exited experiments.")
|
||||||
p.add_argument("--limit", type=int, default=50, help="Maximum rows to show. 0 means no limit. Default: 50.")
|
p.add_argument("--limit", type=int, default=50, help="Maximum rows to show. 0 means no limit. Default: 50.")
|
||||||
p.add_argument("--paths", action="store_true", help="Also show full run root paths. Hidden by default to keep the table readable.")
|
p.add_argument("--paths", action="store_true", help="Also show full experiment root paths. Hidden by default to keep the table readable.")
|
||||||
p.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
|
p.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
|
||||||
p.set_defaults(func=cmd_show)
|
p.set_defaults(func=cmd_show)
|
||||||
|
|
||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"view",
|
"view",
|
||||||
help="show one historical run summary",
|
help="show one historical experiment summary",
|
||||||
description="Show summary and per-run details for a specific historical run id.",
|
description="Show summary and child-run details for a specific historical experiment id.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Use show first to find the run id. If the same run id exists under multiple
|
Use show first to find the experiment id. If the same experiment id exists under multiple
|
||||||
profiles, add --rp to disambiguate.
|
profiles, add --rp to disambiguate.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
./rpctl view --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
./rpctl view --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||||
./rpctl view --rp ours --run-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
./rpctl view --rp ours --exp-id 20260709T081122Z_ours-pp-object-cache_2runs_600s
|
||||||
./rpctl view --run-id m3_ours_apnic_smoke --command
|
./rpctl view --exp-id m3_ours_apnic_smoke --command
|
||||||
./rpctl view --run-id m3_ours_apnic_smoke --json
|
./rpctl view --exp-id m3_ours_apnic_smoke --json
|
||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
p.add_argument("--run-id", required=True, help="Run id under ./runs/<profile>/<run-id>.")
|
p.add_argument("--exp-id", required=True, help="Experiment id under ./runs/<profile>/<exp-id>.")
|
||||||
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated run ids.")
|
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated experiment ids.")
|
||||||
p.add_argument("--command", action="store_true", help="Also print the first run command line.")
|
p.add_argument("--command", action="store_true", help="Also print the first run command line.")
|
||||||
p.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
|
p.add_argument("--json", action="store_true", help="Print machine-readable JSON.")
|
||||||
p.set_defaults(func=cmd_view)
|
p.set_defaults(func=cmd_view)
|
||||||
|
|
||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"delete",
|
"delete",
|
||||||
help="delete one non-running run id and its local data",
|
help="delete one non-running experiment id and its local data",
|
||||||
description="Delete a historical run root and matching runner logs. Active/running runs are refused.",
|
description="Delete a historical experiment root and matching runner logs. Active/running experiments are refused.",
|
||||||
formatter_class=formatter,
|
formatter_class=formatter,
|
||||||
epilog="""
|
epilog="""
|
||||||
Safety rules:
|
Safety rules:
|
||||||
--run-id must point to a non-running run.
|
--exp-id must point to a non-running experiment.
|
||||||
If the same run id exists under multiple profiles, add --rp.
|
If the same experiment id exists under multiple profiles, add --rp.
|
||||||
Use --dry-run first to print the exact run root and matching runner logs.
|
Use --dry-run first to print the exact experiment root and matching runner logs.
|
||||||
Actual deletion requires --yes.
|
Actual deletion requires --yes.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
./rpctl delete --run-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --dry-run
|
./rpctl delete --exp-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --dry-run
|
||||||
./rpctl delete --run-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --yes
|
./rpctl delete --exp-id 20260710T025439Z_ours-pp-object-cache_4runs_180s --yes
|
||||||
./rpctl delete --rp ours --run-id duplicated_run_id --yes
|
./rpctl delete --rp ours --exp-id duplicated_experiment_id --yes
|
||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
p.add_argument("--run-id", required=True, help="Historical non-running run id to delete.")
|
p.add_argument("--exp-id", required=True, help="Historical non-running experiment id to delete.")
|
||||||
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated run ids.")
|
p.add_argument("--rp", default="", help="Optional RP/profile filter to disambiguate duplicated experiment ids.")
|
||||||
p.add_argument("--dry-run", action="store_true", help="Only print the paths that would be deleted.")
|
p.add_argument("--dry-run", action="store_true", help="Only print the paths that would be deleted.")
|
||||||
p.add_argument("--yes", action="store_true", help="Confirm deletion.")
|
p.add_argument("--yes", action="store_true", help="Confirm deletion.")
|
||||||
p.set_defaults(func=cmd_delete)
|
p.set_defaults(func=cmd_delete)
|
||||||
|
|
||||||
p = sub.add_parser("_runner", help=argparse.SUPPRESS)
|
p = sub.add_parser("_runner", help=argparse.SUPPRESS)
|
||||||
p.add_argument("--profile", required=True)
|
p.add_argument("--profile", required=True)
|
||||||
p.add_argument("--run-id", required=True)
|
p.add_argument("--exp-id", required=True)
|
||||||
p.add_argument("--runs", type=int, required=True)
|
p.add_argument("--runs", type=int, required=True)
|
||||||
p.add_argument("--interval-secs", type=int, required=True)
|
p.add_argument("--interval-secs", type=int, required=True)
|
||||||
p.add_argument("--rirs", required=True)
|
p.add_argument("--rirs", required=True)
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user