| Layer | How a hit happens | You see it as | Typical save |
|---|---|---|---|
| OpenAI | Automatic longest identical prefix, usually 1,024+ tokens. Explicit breakpoints on GPT-5.6+. | cached_tokens | ~50–90% off cached input; writes extra on newest models |
| Claude | Automatic top-level cache_control, or up to 4 explicit breakpoints. Writes only at markers. | cache_read_input_tokens | Reads at 10% of input; 5-minute TTL, optional 1 hour |
| Self-hosted | vLLM / SGLang prefix KV cache on GPU. Hash of the token prefix. Same left-edge rule. | Faster prefill, not a bill | Latency, not an API discount |
| Your app | Semantic cache: embed the query, return a previous answer if close enough. | Zero model call | 100% — but it is a different product |
This talk is about the first three — KV prefix reuse. Do not mix them up with “we cached the chatbot reply.”