Manage context effectively in large codebase exploration

A single Claude session has a finite context window, and code exploration is unusually good at filling it: every Grep hit, every full-file Read, and every directory listing lands in the transcript and stays there. As that history grows, two failure modes appear. First is the 'lost in the middle' effect, where the model reliably attends to the start and end of a very long input but skips details buried in the middle. Second, and more diagnostic for this task, is context degradation: in an extended session the model starts giving inconsistent answers to the same question and begins referencing 'typical patterns' or how code 'usually' looks, rather than citing the specific classes, files, and functions it discovered earlier in the same session.

That shift from concrete to generic is the exam's tell-tale signature. If an agent that earlier named RefundService.process() starts saying things like 'a typical refund handler would validate the order first,' it has effectively lost the earlier discovery even though the tokens are technically still in the window. Recognizing this symptom is what tells you the problem is context management, not a bad prompt or a weak model.

The critical corollary is that a bigger context window does not fix this. Moving to a higher-tier model or a larger window buys headroom but does not restore attention quality over a long, noise-heavy transcript. The durable solutions all share one idea: keep the working context lean and move authoritative findings out of the transcript and onto disk or into fresh, well-seeded contexts.

A scratchpad file is a plain file (for example NOTES.md or findings/refund-flow.md) where the agent records the key findings it must not lose: which class owns the refund flow, where the auth check lives, the exact file and line of an entry point. Because the file lives outside the conversation, it survives context boundaries: even after history is compacted, summarized, or a new session starts, the durable conclusions are one Read away.

The pattern has two halves that both matter. The agent writes findings as it discovers them, and then it references the scratchpad when answering later questions instead of trusting recollection. This is exactly what counteracts context degradation: rather than asking 'what did we decide about the refund flow?' and getting a vague 'typical pattern' answer, the agent re-reads its own recorded fact. In Claude Code you enable this by instructing the agent (often in CLAUDE.md or the task prompt) to maintain and consult a scratchpad throughout the investigation.

# NOTES.md  (exploration scratchpad)
## Refund flow
- Entry: api/refunds.py -> RefundController.create()
- Core logic: services/refund_service.py RefundService.process()
- Auth gate: middleware/verify_customer.py (runs BEFORE controller)
- Gotcha: async client added 2024-Q3; old sync path still in tests

The scratchpad is authoritative state, not scratch notes to be discarded. Treating it as the source of truth, and re-reading it, is the whole point.

The single most effective structural technique is to delegate a specific, self-contained investigation to a subagent so the verbose output never touches the main agent's context. When the main agent needs to answer 'find all test files' or 'trace every dependency of the refund flow,' it spawns a subagent (via the Task tool, or Claude Code's Explore subagent) that does the noisy work: dozens of Grep and Read calls that would otherwise bloat the main transcript. The subagent burns its own isolated context on that work and returns only a distilled summary. The main agent stays lean and preserves its role as the high-level coordinator that assembles overall understanding.

This is the same hub-and-spoke idea as multi-agent orchestration (tasks 1.2 and 1.3), applied specifically to protect context during exploration. The Explore subagent is purpose-built for this: it isolates the verbose discovery phase and hands back a summary rather than a raw transcript. Remember that subagents do not inherit the parent's context automatically, so you must pass the specific question and any needed prior findings explicitly in the subagent's prompt.

Main agent (lean, coordinating)
  -> Task: "Find all test files and report their directories and
            the modules they cover. Return a summary table only."
  <- summary (10 lines) instead of 40 file reads

The payoff is that the main agent's window fills with distilled conclusions, not raw file contents, so it can coordinate a large exploration far longer before degrading.

Large explorations are naturally multi-phase: map the structure, then trace a specific flow, then plan changes. If you let one long session run through all phases, the early phases' verbose output crowds out the later phases. The disciplined pattern is to summarize the key findings from each phase before starting the next, and inject that summary into the initial context of the next phase (or the next batch of subagents).

Concretely: after a 'map the repository' phase, distill it to a short structural summary ('monorepo, three services, refund logic in service B, tests colocated'), then start the 'trace the refund flow' phase seeded with just that summary rather than the full discovery transcript. Each phase begins with a clean, high-signal context instead of dragging forward the noise of everything that came before. This pairs naturally with scratchpad files: the phase summary is often written to and read from the scratchpad.

This is closely related to the session-management guidance in task 1.7. When prior tool results are stale or simply voluminous, starting a fresh, summary-seeded context is more reliable than resuming and replaying raw output. Here the same logic applies within one investigation, phase to phase, not just across days.

Claude Code exposes the /compact command for exactly the situation where an exploration session has filled with verbose discovery output. /compact replaces the accumulated conversation history with a model-generated summary of it, freeing context while preserving the thread of what has been learned and decided. You reach for it when you notice the context filling up (or the early signs of degradation) but you still want to keep going in the same session.

Distinguish /compact from /clear. /clear wipes the conversation entirely and starts clean with no memory of the session, which is right when you are switching to an unrelated task. /compact keeps a summarized carryover, which is right mid-investigation when the details are verbose but the conclusions still matter. Compaction is lossy, so it is a complement to scratchpad files, not a replacement: anything that absolutely must survive belongs in a file, because a summary may drop a specific line number or class name.

A healthy long-session loop looks like: explore, record durable findings to the scratchpad, and when the transcript is heavy with raw output, /compact to shrink it, relying on the scratchpad for anything the summary might have blurred.

Long-running multi-agent explorations need to survive interruptions (a crash, a timeout, an intentional pause) without redoing hours of investigation. The pattern is structured state persistence: each agent exports its state (findings, progress, open questions) to a known location on disk, and the coordinator writes or reads a manifest that indexes those exports. On resume, the coordinator loads the manifest and injects the relevant state back into each agent's prompt, so work continues from the last durable checkpoint rather than from scratch.

The manifest is the recovery contract. It records where each agent's state lives and what phase the work reached, so a restarted coordinator can reconstruct the system deterministically instead of hoping the SDK preserved anything. It does not, by default: agent state is not automatically durable across a crash, and resuming a session does not rebuild multi-agent state for you. You have to design the export-and-reload flow explicitly.

// manifest.json  (loaded by the coordinator on resume)
{
  "phase": "trace-dependencies",
  "agents": {
    "structure-map": "state/structure.json  (complete)",
    "refund-trace":  "state/refund.json     (in-progress)"
  }
}

This extends the scratchpad idea from a single agent's notes to a whole coordinated system's recoverable state: everything important lives at a known path, and the coordinator rehydrates from it on resume.