How to Cut the Cost of Tokens and Context

“The number of AI requests hasn’t changed, but the bill has doubled. Where does the difference come from?” Most often each individual request has become more expensive. Models bill by tokens, the pieces of text going into and out of the model, and most organizations do not know how many tokens they send per request. Without that knowledge optimization is guesswork. Providers offer tools for counting tokens before a request is sent, such as Anthropic in its API (application programming interface).

Why context grows by itself

Context windows now reach 1M tokens¹, but you pay for everything you put in them. The three mechanisms stack: history grows with every turn, the instructions ride along with each one and a pasted document is repeated along with the history.

Prompt caching

If a large part of the prompt is identical every time, providers let you store it on their side. In the Anthropic API a cache read costs 10% of the normal input price.² Writing a cache entry costs 1.25 times the input price with a five-minute lifetime or 2 times with a one-hour lifetime.³ With the shorter variant the mechanism breaks even on the second call with the same prompt beginning; with the one-hour variant it needs at least three. Other providers follow similar logic, but entry lifetime and the write surcharge differ.

• 10%of the normal input price for a cache read
• 1.25×the input price for a write with the five-minute buffer
• 2×the input price for a write with the one-hour buffer

Matching works on the beginning of the prompt, character by character. If the start of the system prompt contains today’s date, a session ID or the user’s name, everything after it stops being cached, because one change invalidates everything that follows. The design rule: fixed content goes first, variable content last.

Two limits are worth knowing in advance. Very short prompts are not cached at all, and the threshold depends on the model, falling roughly between 512 and 4096 tokens. The discount covers only input, so any savings estimate stretched to tokens generated by the model is inflated.

Techniques by implementation effort

Summarizing history

In longer conversations and in agents that take dozens of steps, history grows faster than the value it carries. The last few turns stay in the original, earlier ones are replaced with a concise summary and raw tool-call results are removed once they are no longer needed. Some providers, including Anthropic, offer this as a ready feature. The cost per turn then stops growing with the length of the conversation.

A summary loses detail, so you need to decide what in a given process must not be lost, such as agreements with a customer or case numbers, and exclude it from shortening.

Retrieval instead of pasting

When questions concern a fixed body of knowledge, instead of pasting everything you retrieve fragments and give the model only what relates to the question. This is RAG (retrieval-augmented generation). With a few documents, pasting is enough. With hundreds of documents and regular traffic the cost difference reaches an order of magnitude, and a shorter, more relevant context also improves answer quality.

Limits, routing and batches

Without guidance models write at length. Two limits work immediately: a hard output length limit set in the call and an instruction describing the expected form of the result. A classification does not need a paragraph of justification; a label is enough.

Routine tasks such as formatting, classification and short extractions do not need a flagship model. If the result is not needed right away, there is the batch queue: in the Anthropic API asynchronous processing costs half the standard price.² For overnight recalculations and bulk document processing it is the simplest saving there is.

Measure first

Without a cost breakdown by process and prompt you see only one number on the invoice. The minimum set of indicators:

1. Input and output tokens per call.
2. The share of cache reads, reported in the Anthropic API in a separate response field.
3. Cost broken down by use case.

The limits of optimization

Aggressively cutting context lowers answer accuracy, so every change is checked on a test set, not on impressions. Content in the cache is stored on the provider’s side for the entry’s lifetime, which in regulated sectors needs to be aligned with information classification and the retention policy that follows from GDPR.

The rollout order is the same in most cases: measurement first, then caching, then output limits and routing and finally a redesign of how knowledge is supplied. The first two steps are the cheapest and usually deliver most of the effect. We measure costs and set up an environment with these techniques as part of the AI Accelerator.

Frequently asked questions