workspace-hub-overnight-plan-monitor

# Workspace-hub overnight plan monitor

Use this when a cron/job or user asks for a status report on an overnight planning or implementation batch (for example a 10-pack under `logs/.../*.pid` and `docs/plans/overnight-prompts/.../results/*.md`, or a Hermes-launched batch under `/mnt/local-analysis/overnight-batch-*/{logs,prompts,results}`).

## Why this skill exists

A few practical quirks showed up during monitoring:

1. `read_file()` prefixes lines with `N|`, which can corrupt naive PID parsing if you just regex the first number.
2. `search_files(target='files')` may return results under `files`, not `matches`, so handle both shapes.
3. Cached/deduped file reads can be awkward for tiny PID files; direct Python file reads from `terminal`/`execute_code` are often more reliable for read-only monitoring.
4. Dossier markdowns are not uniform; extract recommendations from the `Final Recommendation` section with tolerant parsing.

## Recommended workflow

1. **List PID files and result files**
   - Use `search_files(target='files')` on the exact batch directories.
   - For file listings, read from `result.get('files', [])` first and fall back only if needed.

1a. **Verify expected worktree roots before trusting missing artifacts**
   - For explicit worktree monitoring requests, first test whether each expected root exists and is a git repo/worktree (`Path.exists()`, `.git` file/dir, or `git worktree list --porcelain`).
   - If expected worktree roots are missing, say the run is incomplete at the requested locations and list the launched paths for operator inspection.
   - Use a bounded exact-path fallback in the main checkout (for example `/mnt/local-analysis/workspace-hub/<same-relative-summary-path>` and the same prompt directory) to find central salvage artifacts, but classify them separately from the expected worktree outputs.
   - Avoid broad `rglob`/`search_files` over all of `/mnt/local-analysis` for summary names unless tightly bounded; it can time out on large workspaces.

2. **Check live processes with direct Python/OS calls**
   - Prefer a short Python script via `terminal` or `execute_code`:
     - `Path.glob('*.pid')`
     - `pf.read_text().strip()`
     - `os.kill(pid, 0)` to test liveness
   - This avoids line-number prefixes from `read_file()`.

3. **Count completed dossiers from the results directory**
   - Count `*.md` files in the exact `results/` folder.
   - Do not infer completion from prompts outside `results/`.

4. **Map dossier to terminal/issue**
   - Terminal name usually comes from the result filename stem.
   - Issue number is best extracted from the markdown metadata table row.
   - Handle both plain and emphasized field labels, for example:
     - `| Issue | #2053 |`
     - `| **Issue** | #2055 |`
     - or `| Issue | WRK-1113 |`
   - Prefer this table over generic regexes, because the body often references dependency issues too.
   - If the value is a markdown link like `[ #2063 ](url)` or `[ #2063](url)`, normalize it to just `#2063` in the report.

5. **Extract a one-line recommendation**
   - Parse from the `## ... Final Recommendation` section.
   - Normalize formatting first:
     - strip leading `###`
     - remove surrounding `**...**`
     - collapse repeated whitespace
   - Use this priority:
     1. Explicit `RECOMMENDATION:` line
     2. First status heading/line in the section (often bold or `###`)
     3. First substantive explanatory sentence or bullet after that heading
   - Skip boilerplate lead-ins when choosing the explanatory text, such as:
     - `Rationale:` / `**Rationale:**`
     - `Action required:` / `Action needed:`
     - `Required before implementation:`
     - `Pre-implementation actions required:`
   - Good outputs look like:
     - `READY AFTER LABEL UPDATE — scaffold is complete and tested`
     - `NEEDS ISSUE REFINEMENT — required source data is missing`
     - `ALREADY MOSTLY DONE — only thin wrapper/test work remains`

6. **Completion rule for reporting**
   - If all expected dossiers exist, monitoring is complete.
   - If no batch PIDs remain alive, monitoring is also complete even if some dossiers are missing.
   - Otherwise report incomplete status and identify the still-running terminals.

## Reference Python pattern

```python
from pathlib import Path
import os, re

pid_dir = Path('/mnt/local-analysis/workspace-hub/logs/Codex-2026-04-09-10pack')
result_dir = Path('/mnt/local-analysis/workspace-hub/docs/plans/overnight-prompts/2026-04-09-10claude/results')

result_files = sorted(result_dir.glob('*.md'))

alive = []
for pf in sorted(pid_dir.glob('*.pid')):
    m = re.search(r'\d+', pf.read_text().strip())
    pid = int(m.group(0)) if m else None
    is_alive = False
    if pid is not None:
        try:
            os.kill(pid, 0)
            is_alive = True
        except OSError:
            pass
    if is_alive:
        alive.append(pf.stem)
```

## Pitfalls

- Do **not** parse PID files with a regex against raw `read_file()` output unless you strip the `N|` prefix first.
- Do **not** count prompt files outside `results/` as completed dossiers.
- Do **not** grab the first `#205x`/`WRK-xxxx` in the document body; dependency references can mislead issue extraction.
- Keep the task read-only: no file writes, no repo modifications.

## Worker-claim verification rule

Do not trust a finished worker's stdout summary by itself.

A worker can claim:
- issue moved to `status:plan-review`
- plan file written
- review artifacts present

while one of those local artifacts is actually missing from the monitored worktree.

For every completed worker, verify all three surfaces before reporting success:
1. **Live GitHub issue state**
   - check labels on the issue (`status:plan-review`, `status:plan-approved`, etc.)
   - inspect the most recent comments if the worker claimed a plan-summary or blocker comment
2. **Canonical local plan artifact**
   - confirm the claimed `docs/plans/YYYY-MM-DD-issue-NNN-*.md` file actually exists in the active worktree
3. **Local review artifacts**
   - confirm the claimed `scripts/review/results/*plan-NNN-*` files actually exist in the active worktree

If GitHub state advanced but local artifacts are missing:
- classify the issue as **governance/artifact reconciliation required**
- do not report it as cleanly verified
- say explicitly: label/comment confirmed, local artifact state not confirmed

This prevents false confidence when a worker comment or completion log overstates what landed on disk.

## Implementation-batch reconciliation pattern

When the batch is an implementation/closeout batch rather than a pure plan-review dossier batch, treat stdout as a lead, not proof.

1. **Poll Hermes process sessions if no PID files exist**
   - Use `process(action='poll', session_id=...)` for launches created by `terminal(background=true)`.
   - Record `status`, `exit_code`, and the output preview.
   - A worker can exit successfully while writing its summary to an alternate sandbox path; a worker can also be superseded by another lane while its original process exits `-15`.

2. **Verify GitHub and commit surfaces for every claimed completion**
   - `gh issue view <n> --json state,stateReason,labels,comments,url`
   - `git rev-parse --verify <sha>^{commit}`
   - `git branch -a --contains <sha>` or direct remote containment checks
   - `git show --stat --name-only --oneline <sha>`
   - Classify separately: issue state, remote landed state, local checkout/worktree state.

3. **Reconcile result artifact locations**
   - Workers may be sandboxed to `/mnt/local-analysis/workspace-hub` and unable to write `/mnt/local-analysis/overnight-batch-*/results`.
   - Search for `.nightly-results/<date>-terminal-*-summary.md` in the repo checkout and batch worktrees.
   - If the user expects a central batch directory, mirror/copy the missing summaries into `overnight-batch-*/results/` and note that this is an orchestrator reconciliation action.

4. **Handle failed/no-output/max-turn lanes decisively**
   - Do not relaunch blindly.
   - First inspect the lane log, GitHub issue state, and owned-path diff/status only.
   - If the log is just `Error: Reached max turns (...)` and owned-path status/diff is empty, write a central blocker/current-state note documenting that no owned-path edits landed.
   - Then launch a narrower recovery prompt scoped to the exact owned paths, required validators, and closeout steps.
   - If the lane was intentionally superseded (for example original process exited `-15` but another path completed the issue), preserve both a blocker/supersession note and a completion summary so morning review sees why the raw process status looks failed.

5. **Handle post-reboot ghost completions**
   - After a reboot, Hermes background process state may be gone and the worker log/central summary may be empty or missing even though the implementation commit landed on `origin/main`.
   - If a claimed or discovered commit is already on the intended remote and contains exactly the owned paths, treat this as a **closeout-only reconciliation**, not a rerun trigger.
   - Re-run the prompt's read-only validation block against the committed files and check residual owned-path diffs only for the scoped paths.
   - If validation passes but the issue is still open or still labeled `status:plan-approved`, write the missing central summary artifact, post a closeout comment, close the issue as completed, and move the label to `status:done`.
   - This closeout can be safe even while unrelated interactive agents are active in the repo, provided you do not mutate git state and only touch the external results artifact plus GitHub issue state.

6. **Write an aggregate verification report**
   - Include one row per lane: process result, GitHub state, commit/artifact verification, verdict, and next action.
   - Store it in the central results directory (for example `results/post-batch-verification-YYYY-MM-DD.md`).

## Output checklist

- Result file count
- Completed terminal/issue list
- One-line recommendation per completed dossier or implementation summary
- Alive process count and alive terminal/session list
- For each completed issue: whether GitHub state was confirmed, whether claimed commits are present on the intended remote, whether claimed files appear in the commit, whether local artifacts exist
- Missing or mirrored summary artifacts, including source and destination paths
- Failed-lane classification: max-turn/no-output/superseded/blocked, with blocker path if written
- Recovery session ID/log path if a narrow recovery run was launched
- Explicit `monitoring is complete` statement when either all expected artifacts exist or no processes remain; otherwise name the exact still-running recovery sessions