Recover a stuck task or worker session with evidence and a reversible cleanup

Explanation

The rule

Snapshot, then propose, then ask. A stuck task makes you want to clean up fast. The fast cleanup is where work gets lost.

Why a bundle and a patch and a tarball

A git bundle holds commits. It doesn't hold the working tree, the index or the stash (S2), and a stuck worker's best work is often uncommitted. So the snapshot keeps three pieces: committed history (bundle), tracked edits (binary patch) and new files (tarball). The test restores all three into a fresh clone and compares the files byte for byte.

Verdict

Recommended for step 5. It's tested, and it never writes to the source. Works for us for the classification and gating discipline, which we follow every time but haven't scripted.

Fit

Goal: Get a task unstuck without losing the worker's uncommitted work and without making a change to the task database that can't be undone.

Environment

• A git worktree per task or worker
• A task tracker backed by SQLite (the DB steps); adapt for other stores
• Workers in terminal sessions or background processes
• git 2.47 (tested); git-bundle and git-worktree as documented 2026-11
• SQLite with VACUUM INTO (3.27+)

Prerequisites

• Read access to the worktree and DB file
• Enough free disk for one copy of the DB and the bundle

Permissions required

• Read-only for steps 1–5
• Owner approval for any step that changes task state, kills a process, or removes a worktree/branch

Don't use this when

• Not for a worker that is alive and making progress; leave it alone.
• Not for databases you can't safely open read-only (e.g. a server DB); use that system's own backup tool.

Expected output: An evidence directory with a manifest, a one-line classification, and a written cleanup plan with exact commands and a rollback for each, waiting on the owner.

Procedure

Look, classify, snapshot, propose, then and only then change state with approval. Every change gets a rollback written before it runs.

Reference material. It never authorises killing a process, editing a DB or deleting anything.

1. Read the task row read-only

   Open the DB read-only and read the stuck task's status, assignee, lease/heartbeat time and last update.

   python3 -c "import sqlite3;c=sqlite3.connect('file:tasks.db?mode=ro',uri=True);print(c.execute('select * from task where id=?',(ID,)).fetchall())"

   Expect: You know what the tracker believes.

2. Is the worker alive? read-only

   Look for its process and its terminal session; read the session's last screen or log without typing into it.

   ps -eo pid,etime,stat,cmd | grep -i '<worker>'

   Expect: Alive & progressing / alive & waiting on input / gone.

3. What's in the worktree? read-only

   Status, recent commits, worktree registration.

   git -C <wt> status --porcelain=v1 --branch; git -C <wt> log --oneline -n 10; git -C <wt> worktree list

   Expect: Commits ahead, uncommitted changes, untracked files: the work at risk.

4. Classify read-only

   Pick one: (a) alive and progressing: stop here; (b) alive and blocked on a prompt; (c) dead with unsaved work; (d) dead, nothing at risk; (e) tracker row wrong but worker fine.

   Expect: One letter plus the evidence lines.

5. Snapshot before anything else read-only

   Write an evidence directory: git status/log, binary patch, untracked tarball, verified bundle, VACUUM INTO copy of the DB with integrity_check, SHA-256 manifest. Source is opened read-only.

   .venv/bin/python tools/snapshot_before_cleanup.py --worktree <wt> --db <tasks.db> --out ~/backups/stuck-<task>-$(date +%Y%m%d-%H%M%S)

   Expect: 'snapshot written … (N files)'; snapshot.json shows integrity_check ok.

6. Write the cleanup plan read-only

   For each proposed change, list the exact command, why, and its rollback. Typical: answer the blocked prompt (b); release or reassign the task (c, d, e); keep the worktree and lock it with a reason rather than removing it.

   Expect: A plan the owner can approve line by line.

7. (Gated) Change task state WRITE (gated)

   Use the tracker's own API or CLI to release or reassign the task, not a hand-written UPDATE, if one exists.

   Expect: Tracker shows the new state; the step 5 snapshot still exists.

   Gate: Owner approval. Rollback = restore the field values recorded in step 1, or the db-backup copy as a last resort.

8. (Gated) Stop a hung worker WRITE (gated)

   Send a polite interrupt or quit through the session first; a signal only if that fails.

   Expect: Process gone, worktree unchanged since the latest snapshot.

   Gate: Owner approval. Re-run step 5 afterwards if the worker wrote anything while stopping.

9. (Gated) Remove a worktree or branch IRREVERSIBLE (gated)

   Only after its commits are pushed or bundled and its uncommitted work is in the snapshot. Dry-run first.

   git worktree prune -n -v   # then, if approved: git worktree remove <wt>

   Expect: Removed only what was named.

   Gate: Owner's explicit yes for each worktree/branch named. Default answer is keep (or 'git worktree lock --reason').

Verification

Success check: An evidence directory exists whose MANIFEST.sha256 re-verifies, the DB copy passes integrity_check, the task is in the state the owner approved, and no worktree, branch or session was removed without a recorded yes.

Rollback: Steps 1–6 change nothing. Step 7: restore the recorded field values via the tracker (or, if the tracker is broken, restore db-backup.sqlite with the owner's approval after backing up the current file). Step 8: restart the worker on the same worktree. Step 9 has no rollback except recreating from the bundle + patch + tarball, which is why it is gated hardest.

Last tested 2026-11-15 (partially). Scope: Tested: step 5 (snapshot tool) end to end on synthetic fixtures, including read-only proof and a full restore. Not tested this edition: steps 1–4 and 6–9 against a live tracker and real worker sessions.

Evidence

Sources

1. S1 VACUUM (INTO clause), SQLite (primary). Fetched 2026-11-15. Supports: VACUUM INTO leaves the original unchanged and writes a consistent snapshot; the target must not already exist; it's an alternative to the backup API.
2. S2 git-bundle documentation, Git project (primary). Fetched 2026-11-15. Supports: Bundles back up refs and commits only, not the index, working tree or stash, so uncommitted work needs a separate patch; bundles can be verified and cloned from.
3. S3 git-worktree documentation, Git project (primary). Fetched 2026-11-15. Supports: worktree list/lock/prune/remove; deleting a worktree directory by hand leaves stale admin files that 'prune' cleans; 'prune -n' is a dry run.

Original observations

• 2026-11 · Our own task system: Most 'stuck' tasks were a worker blocked on an interactive prompt, or a session that died while the task row still said in progress. Very few were real deadlocks.
• 2026-11 · Our own task system: Standing rules that made recovery safe: back up before any DB edit or restart; never delete a worktree, branch or session without the owner's yes; workers don't edit the DB.

Test environment: tools/snapshot_before_cleanup.py run by tests/run_tests.py against a synthetic git repo (a commit on a branch, uncommitted edits, an untracked file) and a two-row SQLite DB, Python 3.13 + git 2.47 on Linux, 2026-11-15.

Measured outcomes

• Worktree tree-hash and DB sha256 identical before and after the snapshot (read-only confirmed)
• Restore from bundle + patch + untracked tarball reproduced the worktree files byte for byte
• Backup integrity_check = ok; row counts match
• Every file in MANIFEST.sha256 re-verifies

Uncertainty

• Tested on a toy repo and DB, not a large production tracker. VACUUM INTO needs free space roughly equal to the DB size.
• Classification of 'hung vs. thinking' is a judgment call; the pack asks for evidence, not certainty.
• The tool doesn't capture git stash entries or per-repo config/hooks (S2). Note them by hand if they matter.

Counterexamples

• If the worker is mid-write to the DB, even a consistent snapshot captures a state the worker was about to change. Snapshot again after stopping it.

Files shipped with this pack

• tools/snapshot_before_cleanup.py

Rights and integrity

License: CC BY 4.0 for text; snapshot tool MIT. Provenance: Gus, from our own recovery practice; tool behaviour tied to SQLite and Git docs fetched 2026-11-15.

Permissions: Read-only by default; deletion always needs a per-item owner yes. Telemetry: None.

Content digest: sha256:c59d32607a327732cee0d383698d1a7ec36e8a1f5f35af46f02d023537a6c14b
SHA-256 of the canonical source file packs/ab-002-stuck-session-recovery.md as built. Identifies content; does not prove authorship.

Changelog

• 0.1.0 · 2026-11-15: First private draft; snapshot tool tested on a synthetic repo and SQLite DB.

Related packs

• ab-001-coding-agent-handoff