# Agents - Parallel Orchestration and Delegation Fan-out subagent work fails in predictable ways: shared files collide, retry storms mask missing outputs, and environment faults stop every child at once. Bulk knowledge work parallelizes safely when a mechanical pre-pass builds compact bundles for the LLM workers, roles split across lead, worker, reviewer, and consistency checker, and every scheduled run carries a watchdog. ## Key points - Parallel subagents on a shared repository need file-ownership boundaries declared before launch. Without them, concurrent edits to shared tracking files (`PROGRESS.md`, `TODO.md`) create merge conflicts the coordinator must resolve. - Fan-out economics are retry-dominated. In one mass fan-out, a voter prompt ran up to 22 times, and of 431 spawned sessions only about one third produced usable output. Retry storms mask which outputs exist, so check every artifact for completion. - A 13-of-13 failure rate across two dispatches, with zero model, tool, or diagnostic activity, is a systematic environment fault, not a concurrency race. The right pivot is plain background jobs for mechanical lanes and in-session work for the rest. - Four parallel subagent builds drove the load average to 387-550 on 16 cores. Subagent timings became meaningless, and three tasks hit the 30-minute timeout. Cap parallelism with `-p=4` and re-measure on an idle machine. Worker timings are never the evidence. - Model choice is configuration, not an orchestrator decision. A model constant hardcoded in a workflow script sent 40 child runs through one expensive model in one day. - Extension-provided tools are not part of the builtin tool set. Smoke-test them in a background child before a routine is built on them. - Workflow APIs limit payload size: one lane API caps its lane inventory at 64 KiB. Copying shared rules and acceptance criteria into every stage task made the wave fail validation before any child started. Move that content into gitignored packet files and reduce each stage task to a two-line pointer. - Fanout specifics: the builtin agent role is `reviewer`, not `general-purpose`. The scripted all-runs form accepts per-child `output` fields. The parallel form rejects them. - Schedules are stored per project directory. Every open session arms the same timers, and whichever fires first receives the result. Create recurring runs as session-only schedules (`sessionOnly: true`) owned by the session that stays open. - Remote dispatch disables SSH agent forwarding and copies no local credentials. Installing the runtime on the remote machine does not make model access ready. - Agent managers address different layers. dmux manages where agents run (parallel isolated worktrees, TUI). Another manager controls what the agents do and how they coordinate. The two are complementary, not competing. - Run two model reviews as parallel background tasks, read both output files, and merge the findings. This is faster than sequential review, and the disagreements are worth investigating. - Per-run caches replace `rm -rf` when the harness blocks it: `export GOCACHE=$(mktemp -d /tmp/<prefix>-gocache.XXXXXX)`, reset scratch checkouts with `git checkout -- . && git clean -fdxq -- <paths>`, and empty finished caches with `go clean -cache`. - A team-lead, worker, reviewer, and consistency-checker split over a shared task list parallelizes bulk edits while it keeps a quality gate. - The mechanical and LLM split is load-bearing. A collector script buckets all sessions by date without an LLM and builds a compact redacted bundle. The synthesis subagent sees only that bundle plus the contract. The orchestrator context stays clean across a full-history backfill. - A scheduled capture that runs as one long-lived session fails silently. One fire never terminated, later fires never started, and no sections were written for three weeks. A stuck running session blocks the whole schedule. Long-lived scheduled runs need a watchdog, not only a cron line. ## How to apply 1. Declare file ownership per subagent before a fan-out. 2. Smoke-test extension tools in a background child before the workflow starts. 3. After a mass fan-out, check every expected output file for existence and completeness. 4. Cap build parallelism (`-p=4`) and re-measure timings on an idle machine. 5. Keep shared rules and acceptance criteria in gitignored packet files, two lines per stage task. 6. Keep model selection in configuration, not in a workflow-script constant. 7. Create scheduled runs with `sessionOnly: true` from the session that stays open. Add a watchdog for a stuck running session. 8. When `rm -rf` is blocked, use per-run caches: fresh `GOCACHE`, `git checkout` plus `git clean`, and `go clean -cache`. ## Observations - [multi-agent] Absent file-ownership boundaries make parallel subagents collide on shared tracking files. #multi-agent (source: agent sessions, 2026-05) - [process] Retry storms after a mass fan-out mask which outputs exist. Check every artifact. #fanout (source: agent sessions, 2026-07) - [tooling] A uniform failure across all dispatches, with no model or tool activity, indicates an environment fault, not a race. #subagents (source: agent sessions, 2026-08) - [cost] A hardcoded model constant in a workflow script turns one mistake into 40 paid child runs. #cost-control (source: agent sessions, 2026-10) - [scheduling] A stuck running session blocks a whole schedule. Long-lived scheduled runs need a watchdog. #scheduling (source: agent sessions, 2026-10) ## Relations - relates_to [[resources/documentation/Agents/Agents - One Workstream per Task]] (isolation of concurrent agent work) - relates_to [[resources/documentation/Agents/Agents Workflow]] (the daily agent loop) - relates_to [[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark]] (timings are evidence only on an idle machine) ## Sources - Personal agent-session notes, February 2026 to November 2026.