Prompt Cache benchmark: same call volume, 62% lower bill
We benchmarked prompt cache hit rates and cost impact on real workloads, with the full data.
Background
Anthropic and OpenAI both rolled out prompt cache through late 2024: the first request is billed at the cache-write price (typically 1.25x the input price), and later cache hits are billed at the cache-read price (typically 10% of the input price).
In theory the savings are big. But between theory and reality sits the “cache hit rate.” We ran a month-long benchmark using ZetaAPI’s own customer-support agent and codegen pipeline.
Test scenarios
- Scenario A: support agent — fixed system prompt (~3,200 tokens), varying user questions, accumulating dialogue. ~18k calls/day.
- Scenario B: codegen pipeline — a project-context prefix (~12k tokens) refreshed on repo changes; consecutive calls on the same repo reuse the prefix. ~4k calls/day.
Both scenarios tag the system prompt with cache_control: { type: 'ephemeral' }.
Results
| Scenario | Cache hit rate | Raw input cost | With cache | Savings |
|---|---|---|---|---|
| Support agent | 88% | $300/day | $103/day | 66% |
| Codegen pipeline | 71% | $643/day | $279/day | 57% |
| Combined | 82% | $943/day | $382/day | 59.5% |
Adding the start-of-month reconciliation, the full-month savings came to 62%.
What kinds of requests have high hit rates?
From the data, the factors that determine hit rate, by weight:
- Prefix stability — when the system prompt stays unchanged for a long time, hit rate is 80%+ outright
- Call density — Anthropic’s ephemeral cache has a 5-minute TTL; the denser the calls, the more likely they land in the same TTL window
- Prefix length — prefixes shorter than 1,024 tokens don’t even reach the cache threshold
A counterintuitive finding
We assumed codegen would have a higher hit rate than support — the project context is “more stable.” But in practice codegen breaks the cache on every commit, and its real hit rate was 17 percentage points lower.
Caching is a time-dimension optimization, not a semantic-dimension one. Change a single token in a prompt and it’s a brand-new prompt, unrelated to the previous one.
Advice for integrators
- Make the system prompt an unchanging template and push the dynamic parts into user messages
- Batch jobs should be submitted together — don’t split them into waves across TTLs
- Place the
ephemeraltag at the end of the longest unchanging prefix, not at the end of the whole system prompt
We’ll add per-API-key cache-hit-rate stats in the console so integrators can judge for themselves whether caching is on.