Kishore Kanthan
Article · · 4 min read

How to cut LLM cost with prompt caching

How a stable prompt prefix turns repeat agent calls into cache hits, what breaks the cache, and how to measure it.

The first call writes the stable prefix to the cache. Later calls read it, and only the new part costs in full.

TL;DR

  • Put everything that does not change at the start of the prompt, and everything that does at the end.
  • If your provider asks for it, mark where the stable part ends. The provider can then store that part and reuse it.
  • Any change inside the stable part, even one character, starts a new cache.
  • Measure the share of input tokens read from the cache. If you do not measure it, you do not know it works.

Why repeat calls cost so much

An AI agent sends a lot of text that it has sent before. Each call carries the same instructions, the same tool definitions and often the same reference documents. Only the last part, the new question or the latest tool result, is new.

Without caching, the model reads all of that text again on every call. You pay for all of it every time, by the token: a small piece of text, such as a word or part of a word. In a long agent session, the repeated part is usually most of the bill.

How prompt caching works

Most large model providers can store the start of a prompt, called the prefix, after reading it once. When the next request starts with exactly the same text, the provider reuses the stored work. It does not read that text again.

Two rules matter:

  • The match is on the prefix. The cache only helps if your request starts with the same text, in the same order. A match in the middle of the prompt does not count.
  • Cached reads are cheaper, cache writes are not. Reading from the cache costs a small part of the normal input price. Writing to it usually costs a little more than a normal read. A prefix that is written but never read again costs you more, not less.

Stored prefixes also expire after a short time without use. A cache that is hit every few seconds stays warm. One that is hit once an hour does not.

Put the stable part first

Order the prompt from the part that changes least to the part that changes most:

  1. System instructions and rules.
  2. Tool definitions.
  3. Long reference material, such as documents or a schema.
  4. The conversation so far.
  5. The new message.

The end of the stable part is called a cache breakpoint. The drawing shows one breakpoint, after the documents.

Three calls with one cache breakpoint. The first call writes the prefix. Later calls read it, so only the part after the breakpoint costs full price.

Mark the cache breakpoints

Some providers cache automatically. Others ask you to mark where the cacheable prefix ends. Each mark is called a cache breakpoint. These providers often limit how many marks one request can carry.

Put a mark at the end of each block that is reused at a different rate. A common pattern is one mark after the tools and reference material, and one after the conversation so far. The next turn then reuses everything up to the latest mark. With only one mark, as in the drawing above, the conversation costs full price on every call.

What breaks the cache

Any change to the prefix starts a new cache. So does a change of model or settings, and so does a long pause. The usual causes are small and easy to miss:

  • A timestamp, a request ID or today’s date in the system prompt.
  • Tool definitions built in a different order on each call, for example from a map with no fixed order.
  • A small edit to the instructions, made for one request only.
  • A different model, or different settings that the provider counts as part of the match.
  • A long pause between calls, so the stored prefix expires.

Put anything that must change, such as the current date, after the last breakpoint.

One value that changes at the start breaks the whole prefix. Move it after the breakpoint, and the prefix stays in the cache.

Measure the hit rate

Most APIs report three numbers for each response:

  • Input tokens read from the cache.
  • Input tokens written to the cache.
  • Input tokens not cached at all.

Log these three numbers for every call.

From them you can work out the one number that matters:

cache hit rate  = input tokens read from the cache / all input tokens
all input tokens = read from the cache + written to the cache + not cached

Watch it per agent and per kind of task. A sudden drop usually means something new has crept into the prefix. A rate that never rises usually means the prefix is too short to cache, or calls are too far apart.

What I would do first

Log the cache numbers before changing anything. Then change one thing at a time. First, fix the order of the prompt. Next, remove anything that changes from the prefix. Last, add breakpoints. Check the hit rate after each step, so you know which change paid off.