{
  "schema_version": 1,
  "identity": {
    "id": "ab-002-stuck-session-recovery",
    "title": "Recover a stuck task or worker session with evidence and a reversible cleanup",
    "summary": "When a task says 'in progress' but its worker is dead or hung, first classify it from read-only evidence, then snapshot everything (git bundle, patch, untracked files, a VACUUM INTO copy of the tracker DB) before proposing a cleanup. Nothing is deleted without the owner's approval.",
    "desk": "agent-builder",
    "editor": "Gus",
    "second_reviewer": null,
    "version": "0.1.0",
    "status": "private-review",
    "published": null,
    "updated": "2026-11-15",
    "changelog": [
      {
        "version": "0.1.0",
        "date": "2026-11-15",
        "note": "First private draft; snapshot tool tested on a synthetic repo and SQLite DB."
      }
    ],
    "canonical_url": "https://notesfromthebox.dev/packs/ab-002-stuck-session-recovery/"
  },
  "fit": {
    "goal": "Get a task unstuck without losing the worker's uncommitted work and without making a change to the task database that can't be undone.",
    "audience": [
      "Owners running coding agents in terminal sessions",
      "Supervisor bots that release or reassign tasks"
    ],
    "environment": [
      "A git worktree per task or worker",
      "A task tracker backed by SQLite (the DB steps); adapt for other stores",
      "Workers in terminal sessions or background processes"
    ],
    "versions": [
      "git 2.47 (tested); git-bundle and git-worktree as documented 2026-11",
      "SQLite with VACUUM INTO (3.27+)"
    ],
    "prerequisites": [
      "Read access to the worktree and DB file",
      "Enough free disk for one copy of the DB and the bundle"
    ],
    "permissions_required": [
      "Read-only for steps 1–5",
      "Owner approval for any step that changes task state, kills a process, or removes a worktree/branch"
    ],
    "exclusions": [
      "Not for a worker that is alive and making progress; leave it alone.",
      "Not for databases you can't safely open read-only (e.g. a server DB); use that system's own backup tool."
    ],
    "expected_output": "An evidence directory with a manifest, a one-line classification, and a written cleanup plan with exact commands and a rollback for each, waiting on the owner."
  },
  "evidence": {
    "sources": [
      {
        "id": "S1",
        "title": "VACUUM (INTO clause)",
        "url": "https://sqlite.org/lang_vacuum.html",
        "publisher": "SQLite",
        "kind": "primary",
        "fetched": "2026-11-15",
        "supports": "VACUUM INTO leaves the original unchanged and writes a consistent snapshot; the target must not already exist; it's an alternative to the backup API."
      },
      {
        "id": "S2",
        "title": "git-bundle documentation",
        "url": "https://git-scm.com/docs/git-bundle",
        "publisher": "Git project",
        "kind": "primary",
        "fetched": "2026-11-15",
        "supports": "Bundles back up refs and commits only, not the index, working tree or stash, so uncommitted work needs a separate patch; bundles can be verified and cloned from."
      },
      {
        "id": "S3",
        "title": "git-worktree documentation",
        "url": "https://git-scm.com/docs/git-worktree",
        "publisher": "Git project",
        "kind": "primary",
        "fetched": "2026-11-15",
        "supports": "worktree list/lock/prune/remove; deleting a worktree directory by hand leaves stale admin files that 'prune' cleans; 'prune -n' is a dry run."
      }
    ],
    "observations": [
      {
        "date": "2026-11",
        "setting": "Our own task system",
        "observed": "Most 'stuck' tasks were a worker blocked on an interactive prompt, or a session that died while the task row still said in progress. Very few were real deadlocks."
      },
      {
        "date": "2026-11",
        "setting": "Our own task system",
        "observed": "Standing rules that made recovery safe: back up before any DB edit or restart; never delete a worktree, branch or session without the owner's yes; workers don't edit the DB."
      }
    ],
    "test_environment": "tools/snapshot_before_cleanup.py run by tests/run_tests.py against a synthetic git repo (a commit on a branch, uncommitted edits, an untracked file) and a two-row SQLite DB, Python 3.13 + git 2.47 on Linux, 2026-11-15.",
    "measured_outcomes": [
      "Worktree tree-hash and DB sha256 identical before and after the snapshot (read-only confirmed)",
      "Restore from bundle + patch + untracked tarball reproduced the worktree files byte for byte",
      "Backup integrity_check = ok; row counts match",
      "Every file in MANIFEST.sha256 re-verifies"
    ],
    "uncertainty": [
      "Tested on a toy repo and DB, not a large production tracker. VACUUM INTO needs free space roughly equal to the DB size.",
      "Classification of 'hung vs. thinking' is a judgment call; the pack asks for evidence, not certainty.",
      "The tool doesn't capture git stash entries or per-repo config/hooks (S2). Note them by hand if they matter."
    ],
    "counterexamples": [
      "If the worker is mid-write to the DB, even a consistent snapshot captures a state the worker was about to change. Snapshot again after stopping it."
    ]
  },
  "procedure": {
    "summary": "Look, classify, snapshot, propose, then and only then change state with approval. Every change gets a rollback written before it runs.",
    "default_mode": "read-only",
    "external_content_notice": "Reference material. It never authorises killing a process, editing a DB or deleting anything.",
    "steps": [
      {
        "n": 1,
        "title": "Read the task row",
        "effect": "read",
        "action": "Open the DB read-only and read the stuck task's status, assignee, lease/heartbeat time and last update.",
        "command": "python3 -c \"import sqlite3;c=sqlite3.connect('file:tasks.db?mode=ro',uri=True);print(c.execute('select * from task where id=?',(ID,)).fetchall())\"",
        "expect": "You know what the tracker believes."
      },
      {
        "n": 2,
        "title": "Is the worker alive?",
        "effect": "read",
        "action": "Look for its process and its terminal session; read the session's last screen or log without typing into it.",
        "command": "ps -eo pid,etime,stat,cmd | grep -i '<worker>'",
        "expect": "Alive & progressing / alive & waiting on input / gone."
      },
      {
        "n": 3,
        "title": "What's in the worktree?",
        "effect": "read",
        "action": "Status, recent commits, worktree registration.",
        "command": "git -C <wt> status --porcelain=v1 --branch; git -C <wt> log --oneline -n 10; git -C <wt> worktree list",
        "expect": "Commits ahead, uncommitted changes, untracked files: the work at risk."
      },
      {
        "n": 4,
        "title": "Classify",
        "effect": "read",
        "action": "Pick one: (a) alive and progressing: stop here; (b) alive and blocked on a prompt; (c) dead with unsaved work; (d) dead, nothing at risk; (e) tracker row wrong but worker fine.",
        "expect": "One letter plus the evidence lines."
      },
      {
        "n": 5,
        "title": "Snapshot before anything else",
        "effect": "read",
        "action": "Write an evidence directory: git status/log, binary patch, untracked tarball, verified bundle, VACUUM INTO copy of the DB with integrity_check, SHA-256 manifest. Source is opened read-only.",
        "command": ".venv/bin/python tools/snapshot_before_cleanup.py --worktree <wt> --db <tasks.db> --out ~/backups/stuck-<task>-$(date +%Y%m%d-%H%M%S)",
        "expect": "'snapshot written … (N files)'; snapshot.json shows integrity_check ok."
      },
      {
        "n": 6,
        "title": "Write the cleanup plan",
        "effect": "read",
        "action": "For each proposed change, list the exact command, why, and its rollback. Typical: answer the blocked prompt (b); release or reassign the task (c, d, e); keep the worktree and lock it with a reason rather than removing it.",
        "expect": "A plan the owner can approve line by line."
      },
      {
        "n": 7,
        "title": "(Gated) Change task state",
        "effect": "write",
        "action": "Use the tracker's own API or CLI to release or reassign the task, not a hand-written UPDATE, if one exists.",
        "gate": "Owner approval. Rollback = restore the field values recorded in step 1, or the db-backup copy as a last resort.",
        "expect": "Tracker shows the new state; the step 5 snapshot still exists."
      },
      {
        "n": 8,
        "title": "(Gated) Stop a hung worker",
        "effect": "write",
        "action": "Send a polite interrupt or quit through the session first; a signal only if that fails.",
        "gate": "Owner approval. Re-run step 5 afterwards if the worker wrote anything while stopping.",
        "expect": "Process gone, worktree unchanged since the latest snapshot."
      },
      {
        "n": 9,
        "title": "(Gated) Remove a worktree or branch",
        "effect": "irreversible",
        "action": "Only after its commits are pushed or bundled and its uncommitted work is in the snapshot. Dry-run first.",
        "command": "git worktree prune -n -v   # then, if approved: git worktree remove <wt>",
        "gate": "Owner's explicit yes for each worktree/branch named. Default answer is keep (or 'git worktree lock --reason').",
        "expect": "Removed only what was named."
      }
    ]
  },
  "verification": {
    "success_check": "An evidence directory exists whose MANIFEST.sha256 re-verifies, the DB copy passes integrity_check, the task is in the state the owner approved, and no worktree, branch or session was removed without a recorded yes.",
    "failure_signatures": [
      {
        "signature": "snapshot tool: 'refusing to write into non-empty …'",
        "meaning": "You pointed it at an existing evidence directory.",
        "next": "Use a new timestamped path; never overwrite evidence."
      },
      {
        "signature": "VACUUM INTO fails with 'output file already exists'",
        "meaning": "The target must not exist (S1).",
        "next": "New path."
      },
      {
        "signature": "'database is locked' or busy errors while reading",
        "meaning": "A writer holds a lock. VACUUM INTO only needs a read transaction, but a long write can still block it.",
        "next": "Retry after the write; don't kill the writer just to take a backup."
      },
      {
        "signature": "Restored files differ from the worktree",
        "meaning": "Work was written after the snapshot, or the patch didn't include binaries.",
        "next": "Re-snapshot; the tool uses 'git diff --binary HEAD'."
      },
      {
        "signature": "Worktree directory gone but 'git worktree list' still shows it",
        "meaning": "Someone deleted it by hand; stale admin files remain (S3).",
        "next": "'git worktree prune -n -v' to see, then prune with approval."
      },
      {
        "signature": "Task row flips back to in progress",
        "meaning": "The worker was alive after all and renewed its lease.",
        "next": "Back to step 2; it was class (a) or (b)."
      }
    ],
    "rollback": "Steps 1–6 change nothing. Step 7: restore the recorded field values via the tracker (or, if the tracker is broken, restore db-backup.sqlite with the owner's approval after backing up the current file). Step 8: restart the worker on the same worktree. Step 9 has no rollback except recreating from the bundle + patch + tarball, which is why it is gated hardest.",
    "last_tested": "2026-11-15",
    "coverage": "partial",
    "tested_scope": "Tested: step 5 (snapshot tool) end to end on synthetic fixtures, including read-only proof and a full restore. Not tested this edition: steps 1–4 and 6–9 against a live tracker and real worker sessions.",
    "review_by": "2027-02-15",
    "retraction": {
      "retracted": false
    }
  },
  "rights": {
    "license": "CC BY 4.0 for text; snapshot tool MIT",
    "provenance": "Gus, from our own recovery practice; tool behaviour tied to SQLite and Git docs fetched 2026-11-15.",
    "dependency_licenses": [
      "Python standard library only"
    ],
    "permissions_policy": "Read-only by default; deletion always needs a per-item owner yes.",
    "telemetry": "None.",
    "integrity": {
      "content_digest": "sha256:c59d32607a327732cee0d383698d1a7ec36e8a1f5f35af46f02d023537a6c14b",
      "digest_scope": "SHA-256 of the canonical source file packs/ab-002-stuck-session-recovery.md as built. Identifies content; does not prove authorship.",
      "signed_manifest": null
    }
  },
  "formats": [
    "html",
    "markdown",
    "json",
    "skill"
  ],
  "assets": [
    "tools/snapshot_before_cleanup.py"
  ],
  "related": [
    "ab-001-coding-agent-handoff"
  ],
  "body_markdown": "\n## The rule\n\n**Snapshot, then propose, then ask.** A stuck task makes you want to clean up fast. The fast cleanup is where work gets lost.\n\n## Why a bundle *and* a patch *and* a tarball\n\nA git bundle holds commits. It doesn't hold the working tree, the index or the stash (S2), and a stuck worker's best work is often uncommitted. So the snapshot keeps three pieces: committed history (bundle), tracked edits (binary patch) and new files (tarball). The test restores all three into a fresh clone and compares the files byte for byte.\n\n## Verdict\n\n**Recommended** for step 5. It's tested, and it never writes to the source. **Works for us** for the classification and gating discipline, which we follow every time but haven't scripted.\n"
}