AI Context Windows: Build a Working Set, Not a Data Dump

A large context window creates room for instructions, conversation, retrieved documents, tool definitions and results. It does not prove that the model will use every item reliably, preserve the right facts after compaction or leave enough space to finish the answer. Filling the window because capacity exists is the AI equivalent of loading every file into working memory and hoping relevance will organise itself.

The operational decision is narrower: what must be present for this step, what can be referenced outside the prompt, what may be compressed, and how much capacity must remain for the model to reason, call tools and respond? Treat the answer as a governed working set. Give each class of context an admission rule, provenance, budget and eviction rule, then test the assembled request at the lengths and positions it will encounter in production.

Maximum context is not effective context

A context-window limit is a capacity boundary. Effective context is the smaller amount and arrangement over which a model performs the required task at an acceptable level. The two should not be treated as synonyms.

The 2024 TACL paper Lost in the Middle tested multi-document question answering and key-value retrieval. Performance often changed when the same relevant information moved within a long prompt, with weaker results when it appeared in the middle. The experiment does not establish that every current model has the same curve, but it establishes the right control: vary evidence position while holding the task and evidence constant.

NVIDIA's open RULER benchmark broadens the test beyond a single hidden fact. It generates configurable tasks covering retrieval, multiple needles, multi-hop tracing and aggregation, then evaluates them at increasing sequence lengths. Its maintained results also show why a model's advertised window is not a release criterion for a specific application. A team needs its own effective-length curve for its own model, prompt assembly and task distribution.

This is an application-layer issue, distinct from the model-internal memory mechanisms described in our Kimi K3 memory architecture explainer. A model may technically accept a very long sequence while the application still supplies distracting, stale, contradictory or poorly positioned material.

Account for the complete request before admission

Do not budget only the documents visible to a user. Anthropic's current context-window documentation provides a useful concrete example: system instructions, messages, tool results, images, documents, tool definitions and the new output all count towards the window, while cached input still occupies context even when its billing treatment differs.

Provider accounting differs, so calculate against the exact API and model rather than a character estimate. Reserve output capacity before admitting optional evidence. Also retain operational headroom for a retry, a larger-than-usual tool result or a provider-side representation that tokenises differently from a local estimate. The size of that reserve is a measured deployment parameter, not a universal percentage.

Give every request a ledger like this:

Record both requested and actual token use by class. A request rejected before inference because it would cross the budget is a controlled outcome; silent truncation of policy, evidence or output capacity is not.

Make context admission a policy decision

Retrieval rank alone should not decide what enters the window. A highly similar document can belong to the wrong tenant, be superseded, carry untrusted instructions or repeat facts already represented more authoritatively. The enterprise RAG quality audit separates corpus, permissions, retrieval, ranking and grounded answer quality; context assembly should preserve those distinctions instead of flattening every retrieved chunk into anonymous text.

For each admitted item, carry a compact manifest with:

• a stable source or artifact identifier and content version;
• tenant, case and permission scope;
• source type and authority, not just semantic similarity;
• creation, retrieval and expiry times where freshness matters;
• the task question or decision the item is expected to support;
• trust status, including whether the content may contain instructions;
• transformation history, such as OCR, chunking, filtering or summarisation; and
• a reason for admission and a lower-priority candidate it displaced.

This manifest belongs in ordinary application state even if only a short label reaches the model. It lets the system rebuild the context, explain why a source was present and invalidate derived material when the original changes. The W3C PROV data model distinguishes entities, activities, usage, generation and derivation; those concepts are a sound basis for an AIEngine context manifest, although the specific schema above is our implementation interpretation rather than a W3C requirement.

Keep stable instructions early and clearly delimited. Group evidence by the question it answers, retain source labels beside passages, and make conflicts explicit rather than relying on order to imply which version wins. If content is untrusted, mark it as data and prohibit it from redefining tools, policy or task authority.

Compaction is a lossy state migration

Compaction can extend a long-running workflow by replacing older turns or results with a shorter representation. It also creates a new derived artifact. A fluent summary can omit a negative constraint, unresolved disagreement, exact amount, source qualification or pending commitment while sounding complete.

Treat compaction like migrating state between systems:

• define a schema for what the compacted state must preserve;
• keep policy and permission controls outside the lossy summary;
• retain source identifiers, versions and exact references for material claims;
• preserve unresolved questions, rejected options and pending side effects;
• compare the compacted state with the pre-compaction record using deterministic checks where possible; and
• create a fresh session when accumulated assumptions make a summary harder to validate than a clean reconstruction.

Tool-heavy agents need selective reduction before whole-conversation summarisation. Anthropic's tool-context guidance distinguishes on-demand tool discovery, programmatic tool chains, caching and removal of stale tool results. The durable lesson is provider-neutral: loading fewer schemas reduces baseline context; caching can reduce repeated cost but not occupied capacity; and bulky intermediate results should stay outside the working set once their decision-relevant facts are captured.

The full result should remain addressable through a protected artifact ID. That enables the privacy-safe reconstruction described in our AI incident replay guide without forcing every raw document, trace or API payload into every subsequent prompt.

Worked example: a 64K [contract](/industries/legal)-review step

Consider a hypothetical contract-review assistant deployed with a 64K-token request boundary. The number is illustrative; it is not a claim that 64K is optimal or that all models behave alike at that length.

The team reserves 8K for the answer and structured issue table, plus 4K of recovery headroom. It admits 6K of tested system policy and output schema, 4K of matter state and reviewer instructions, up to 32K of clause-level evidence, and up to 10K of tool definitions and fresh results. The budget is therefore closed before retrieval begins.

A search returns 70K tokens across the signed contract, an expired playbook, current fallback clauses, email commentary and duplicate OCR. The assembler does not truncate the concatenation. It rejects the expired playbook as governing policy, collapses duplicate OCR by source hash, keeps the relevant signed clauses verbatim, retrieves the current playbook sections, and represents email commentary as attributed but non-authoritative context. Overflow candidates remain outside the prompt with stable IDs.

After several review turns, the workflow compacts. Exact clause text and policy controls remain source-addressed; the summary carries issues accepted or rejected, reviewer instructions, open questions and the artifact IDs behind each conclusion. A validator checks that every open issue and pending action survived. If the contract version changes, the system invalidates the derived working set and rebuilds it instead of appending the new draft to old conclusions.

This design can still fail. Retrieval may miss an important clause, the summary may preserve a conclusion but lose its qualification, or the model may mishandle a conflict despite clear labels. The control is not confidence in the context builder; it is the ability to observe and test each boundary.

Choose long context, retrieval or a fresh session deliberately

These patterns can be combined. Retrieval can feed a long-context synthesis; a fresh session can start from validated compacted state; tool schemas can be discovered only when the next step makes them eligible. The decision should follow evidence dependency, not the largest window available.

Test the assembled context, not an empty-window demo

NIST's AI RMF Measure function calls for documented, repeatable testing under conditions similar to deployment and ongoing production monitoring. For context engineering, the deployment condition is the fully assembled prompt, including permissions, history, tools, retrieved evidence and reserved output—not a clean benchmark question sent alone.

Build a release matrix that varies one factor at a time:

• context fill level, including the high end expected in production;
• relevant evidence at the beginning, middle and end;
• topically similar distractors and exact duplicates;
• conflicting sources with explicit authority and version differences;
• missing evidence that should produce abstention or a retrieval request;
• reordered documents and tool definitions;
• large, malformed and partially failed tool results;
• compaction immediately before a critical decision; and
• model, tokenizer, retrieval and prompt-assembly versions.

Use representative tasks with known source requirements. Score task outcome, required-evidence coverage, citation correctness, constraint retention, abstention and side-effect proposals separately. Keep blind acceptance items away from daily prompt tuning using the method in our AI eval-leakage guide.

Monitor the working set as a production component

Track input tokens by class, generation reserve at dispatch, rejected and evicted candidates, compaction frequency, manifest completeness, stale-source invalidations, tool-result volume and task performance by fill band. Add two end-to-end measures: whether every material claim can still resolve to its admitted source, and whether every pending action and constraint survives a compaction boundary.

Prompt caching needs its own cost metric but must not be reported as context reduction. The distinction matters for capacity planning and authority too: our AI cache safety guide explains why reuse also needs tenant, policy and provenance boundaries.

Revisit the budget when the model, tokenizer, toolset, retrieval policy, source mix or task changes. The final gate is concrete: can the team name why each item is present, what was excluded, how much output space remains, which transformations occurred and how quality changes as the same evidence moves or the window fills? If not, a bigger context window has increased capacity without establishing control.