What is a context window?

A context window is the amount of text, measured in tokens, that a language model can consider or “remember” at any one time (IBM). Anthropic’s definition is wider: all the text the model can reference when generating a response, including the response itself (Claude docs).

It is working memory, not training. The docs say it is different from the large corpus the model was trained on (Claude docs). Google uses a similar picture: “An analogy for the context window is short term memory” (Gemini docs).

Tokens, not words

The length is counted in tokens. IBM says there is “no fixed word-to-token exchange rate,” so the same text can produce different token counts across models, and formatting such as special characters and line breaks also uses space. To check a request before sending it, Anthropic offers a token counting API, and every response reports usage in its usage field (Claude docs).

What counts toward the window

Anthropic lists what counts: the system prompt, every message (including tool results, images and documents), and your tool definitions. The output counts too, including extended thinking (Claude docs).

Providers differ on the details, as of November 2026:

Window sizes change often. Check the provider’s model page rather than trusting a table, including this one.

Bigger is not automatically better

Anthropic says more context “isn’t automatically better”: as token count grows, accuracy and recall degrade, which it calls context rot. That makes curating what is in context as important as how much space is available (Claude docs).

Position matters too. Liu and colleagues (submitted 6 July 2023) tested multi-document question answering and key-value retrieval and found that “performance is often highest when relevant information occurs at the beginning or end of the input context.” Accuracy fell when the relevant text sat in the middle, even for models built for long contexts (paper). The study covers models from 2023, so it shows a pattern to test for, not a fixed property of today’s models.

Google reports a similar limit for retrieval, which matters for any RAG pipeline. With one fact to find, “you can get ~99% on a single query”, but with multiple pieces of information to find, “the model does not perform with the same accuracy” (Gemini docs).

Cost grows as well. IBM says compute requirements scale quadratically with sequence length, so doubling the input tokens needs 4 times the processing power.

What happens at the limit

Behavior is provider specific. On the Claude API, input that alone exceeds the window returns a 400 invalid_request_error (“prompt is too long”). On Claude 4.5 and newer models, if input plus max_tokens exceeds the window, the request is accepted, and if generation then reaches the limit it stops with stop_reason: "model_context_window_exceeded" (Claude docs). Chat apps may behave differently: claude.ai can manage the window on a rolling first-in, first-out basis.

Ways to stay inside it

• Summarize earlier turns. Anthropic’s server-side compaction summarizes earlier parts of a conversation on the server, in beta for Claude 4.6 and later as of November 2026 (Claude docs).
• Clear stale material. Claude’s context editing offers tool result clearing and thinking block clearing.
• Cache repeated prefixes to cut cost, remembering they still use space.
• Put the key information where the model uses it best, which the 2023 study suggests is near the start or end. For long inputs, Google advises putting the query at the end, after the other context (Gemini docs).
• Write tighter prompts. See what is prompt engineering and few-shot prompting, since every example you add spends window space.

Agents hit the limit sooner because tool results and history pile up over many steps. See how to build AI agents.

See what fills the window

To find out where tokens go, record them per call. HoneyHive’s tracing concepts describe model events that store the input messages and parameters and the output, with example metadata that includes prompt_tokens, completion_tokens, total_tokens and cost. A session groups the events of a conversation, so you can see how input size grew turn by turn. More on this in LLM observability.

FAQ

What is a context window in an LLM?

It is the amount of text, in tokens, that the model can consider or "remember" at any one time (IBM). Anthropic adds that it includes the response itself (Claude docs).

Is the context window the same as the training data?

No. Anthropic's docs say it is different from the corpus the model was trained on and acts as a "working memory" for the model.

Does a bigger context window always give better answers?

No. Anthropic says that as token count grows, accuracy and recall degrade, a phenomenon it calls context rot (Claude docs). A 2023 study found models used information in the middle of long inputs less well than information at the start or end.

What happens when you exceed the context window?

It depends on the provider. On the Claude API, input alone that exceeds the window returns a 400 error ("prompt is too long") (Claude docs). IBM says the text must otherwise be truncated or summarized.

Is the context window measured in words?

No, in tokens. IBM notes there is no fixed word-to-token exchange rate, so the same text can produce different counts on different models.