Tokens and Context

An AI bill rarely grows because a team suddenly started asking more questions. More often it grows because every individual question became more expensive. Models charge by the token, meaning the chunks of text that go into the model and come back out of it, and most organizations have no idea how many tokens a single request actually carries. Without that number, optimization is guesswork. The tool for producing it sits on the provider’s side: Anthropic exposes a separate function that counts tokens before a request is even sent.

Why context inflates on its own

Three mechanisms account for most of the growth.

• A conversation with a model is stateless. The model remembers nothing from the previous turn, so the application resends the entire history with every new question. The tenth message costs several times what the first one did, even though the user typed exactly as much.
• Instructions repeat on every call. A system instruction holding a role description, rules, examples and tool definitions can easily run to several thousand tokens, and it travels with every request, including the ones a single sentence would have answered.
• Whole documents get pasted in. Someone drops in an 80-page policy to ask about one clause. The bill covers 80 pages, and it does so again with every follow-up question about the same document. Context windows now reach 1 million tokens¹, but a large window still has to be paid for. You pay for whatever you actually put in it.

The three mechanisms stack, and that is the heart of the problem. History grows with every turn, the system instruction rides along with each of them, and a pasted document is duplicated together with the history. The bill therefore grows faster than linearly, even though the team asks exactly as many questions as it did a month ago.

Prompt caching

When a large part of the prompt is identical every time, providers will hold it on their side. On the Anthropic API (application programming interface) a cache read costs 10% of the standard input price², while writing an entry costs 1.25 times that price for a five-minute lifetime, or twice for a one-hour one (as of August 2026). With the shorter lifetime the mechanism breaks even on the second call against the same prefix; with the one-hour option it takes at least three. Other major providers follow similar logic, though entry lifetimes and write premiums differ between platforms, so each price list needs checking on its own terms.

Technique	Implementation effort	When it pays back
Prompt caching	Hours	From the second call against the same prefix
Shortening the system instruction	Hours	Immediately, on every request
Routing tasks to cheaper models	Days	At steady volume of routine work
Batch processing	Days	When the answer need not be immediate
Summarising conversation history	Weeks	On long sessions with recurring context

There is a catch here that breaks more implementations than everything else combined. Matching works on the prefix, character by character. Put the current date, a session identifier or a user’s name near the top of the system prompt and nothing after it will cache. One change invalidates everything that follows it. Hence a simple design rule: stable content first, variable content last.

Two limits are worth knowing in advance. Very short prompts are never cached at all, and the threshold depends on the model, falling roughly between 512 and 4096 tokens. The second point is the one most often confused: the discount applies to input only. Tokens the model generates cost what they always did, so any savings estimate stretched across output is inflated.

Summarize rather than replay

In long conversations, and in agents that run dozens of steps, history grows faster than the value it carries. A sensible pattern looks like this: the last few turns stay verbatim, earlier ones become a compact summary, and raw tool results are dropped once they stop being relevant. Some providers, Anthropic among them, offer this as a built-in feature; some teams write their own logic. The effect is the same either way, in that the cost per turn stops rising in step with the length of the conversation.

The trade-off is explicit. A summary loses detail, so you have to decide what must never be lost in a given process, such as commitments made to a client or case numbers, and exempt it from compression.

Retrieve instead of pasting

When questions concern a fixed body of knowledge, pasting all of it stops paying off very quickly. The alternative is to search for the relevant passages and give the model only what bears on the question. This is usually called RAG (retrieval-augmented generation).

With a handful of documents there is nothing to discuss, so paste away. With several hundred documents and steady traffic the difference becomes an order of magnitude. It is worth calculating the threshold for your own case rather than adopting someone else’s. The gain is not purely financial either: shorter, better-targeted context also improves answer quality, because the model is not hunting for a needle in a haystack.

Routing and a ceiling on response length

Output tokens typically cost several times more than input tokens, and models have a marked tendency toward verbosity unless told otherwise. Two simple constraints work immediately: a hard length limit set in the call itself, and an instruction describing the expected shape of the answer. A classification task does not need a paragraph of reasoning when a label will do.

Model choice sits alongside this. Routine work such as formatting, classification and short extractions does not require a frontier model. If the result is not needed immediately, look at the batch queue: on the Anthropic API, asynchronous processing costs half the standard price and most jobs finish within an hour. For overnight recalculations or bulk document processing, that is the simplest saving available.

Measurement comes first

All of these techniques share one precondition. Without cost broken down by process and by prompt, all anyone sees is a single number on an invoice, and that number does not say where to look. The minimum useful set of metrics covers input and output tokens per call, the share of input served from cache, reported by the Anthropic API in a dedicated response field, and cost split by use case. When the cached share stubbornly reads 0% despite a correct configuration, it almost always means something variable has crept into the top of the prompt.

Where optimization stops

Two boundaries deserve to be set deliberately. The first is about quality: aggressive context trimming degrades answer accuracy, so every change needs validating against a test set rather than against an impression. The second concerns data. Content placed in a cache is held on the provider’s side for the lifetime of the entry, which in regulated organizations has to be reconciled with information classification and the retention policy that follows from GDPR.

The order of work looks much the same in most cases: measure first, then cache, then cap output length and route by task, and only then rebuild how knowledge reaches the model. The first two steps tend to be the cheapest and usually deliver the larger share of the effect.

Footnotes

1. Anthropic, model overview including context window sizes: https://platform.claude.com/docs/en/about-claude/models/overview. ↩

2. Anthropic, prompt caching pricing: reads at 0.1x, writes at 1.25x for the five-minute buffer and 2x for the one-hour buffer: https://platform.claude.com/docs/en/build-with-claude/prompt-caching. ↩