Prompt Caching for AI APIs: When It Saves Money—and When It Costs More

Last reviewed: August 30, 2026. API interfaces and product settings change; verify current official documentation before production deployment.

LLM prompt caching can reduce repeated input processing, latency, and cost when many requests share a stable prefix. It can also add cache-write cost without meaningful hits when prompts change constantly or traffic is too sparse. Measure the complete workload before calling caching a saving.

What prompt caching actually reuses

Prompt caching reuses matching prompt prefixes processed recently under provider-specific rules. It is not a semantic database and does not return a previous final answer. The model still processes uncached input and generates new output. Exact eligibility, minimum sizes, retention, breakpoints, and billing differ by provider and model, so current official documentation is the source of truth.

Put stable content before dynamic content

Place durable system instructions, long reference material, tool definitions, and reusable examples before per-user context and the current question. Changing one early token can invalidate the matching prefix after it. Do not reorder tools or serialize equivalent JSON inconsistently between requests. A canonical request builder often improves cacheability more than adding cache flags.

Understand write and read economics

A cache write can cost more than ordinary input while a cache hit is discounted. The break-even point depends on write price, hit price, prefix size, number of reuses, retention, and baseline input price. Calculate net cost for a representative time window instead of quoting a universal percentage. Include retries and failed requests if they are billed.

Measure cache fields in usage

Record uncached input tokens, cache-write tokens, cached or cache-read tokens, output tokens, model, route, prompt version, and task outcome. Aggregate by prompt family rather than only by user. A high hit rate can still be unprofitable if the cached prefix is tiny or if output and tool costs dominate the workload.

Choose implicit or explicit behavior deliberately

Some models automatically select cacheable prefixes; newer options may allow explicit breakpoints or modes. Explicit control is useful when the application knows which blocks are stable, but it adds configuration that must be tested. Avoid writing multiple overlapping cache points without evidence that they improve net cost.

const usage = {
  uncachedInput: 12000,
  cacheWrites: 8000,
  cacheReads: 32000,
  output: 4000,
};

// Multiply each token class by the current model-specific rate.
// Compare total cost per successful task with the no-cache baseline.

Avoid tenant data leakage

Caching must respect the provider's isolation model, but applications should still separate tenants in request construction, access control, and observability. Never move user-specific secrets into a shared stable prefix merely to improve hits. Follow retention and zero-data-retention documentation for the exact provider and model.

Use caching in agent workflows

Agents repeatedly send system instructions, tool schemas, repository guidance, and conversation state, making them natural candidates. However, changing tool lists or replaying growing histories can reduce matching. Expose only relevant tools, keep stable definitions canonical, compact history carefully, and measure cost per completed task rather than cost per individual call.

Know when application caching is better

If the same input should produce the same approved output, an application cache can avoid the model call entirely. Prompt caching is better when the prefix repeats but the user request or desired generation changes. Retrieval caches, embedding caches, and tool-result caches solve other problems. Use the narrowest cache with a clear invalidation rule.

Run a controlled experiment

Choose a prompt family, record a baseline, canonicalize the stable prefix, enable the supported cache mode, and replay representative traffic. Compare task quality, cache reads and writes, total input and output cost, time to first token, and error rate. Test cold-start behavior and retention expiry instead of measuring only warm traffic.

Evaluate caching through Lofee

Use a dedicated key and inspect whether the selected Lofee route exposes the provider's cache controls and usage breakdown required by your application. OpenAI-compatible does not guarantee identical prompt-caching semantics across model families. When usage fields differ, normalize them in your application without inventing data and document which routes have been validated.

A practical 30-minute diagnosis workflow

Begin by freezing changes and recording one failing request with its timestamp, safe endpoint, model, application-key fingerprint, status, structured error, request ID, latency, and retry count. Reproduce it with the smallest possible input and no optional tools. Compare the failing environment with one known-good environment, changing only one variable at a time: credential, Base URL, endpoint family, model, SDK version, streaming, then tool configuration. Check the provider status page when the failure appears suddenly across unrelated workloads. Do not rotate keys, switch models, change proxies, and increase retries simultaneously; that destroys the evidence needed to identify the cause. Once the minimal call works, add production features back individually and record which change reintroduces the failure.

Build a repeatable test matrix

Create automated tests for authentication, a short non-streaming response, a long streamed response, cancellation, structured output, one tool call, a controlled 4xx error, a simulated 5xx error, and a timeout. Run the matrix against every model and route the application officially supports. Store sanitized response fixtures so parsers can be tested without spending tokens or depending on a live service. Include a quality check, because a technically valid fallback can still fail the business task. Re-run the suite after SDK upgrades, model alias changes, editor updates, gateway changes, and provider deprecation notices. A dated capability matrix is more useful than a one-time claim that an endpoint is compatible.

Monitor the result after the fix

For LLM prompt caching, monitor request volume, success rate, error classes, retry amplification, time to first token, total latency, input and output usage, resolved model, route, and cost per successful task. Break dashboards down by environment and application key so one noisy client does not hide the rest. Alert on changes from the workload’s own baseline instead of choosing arbitrary global thresholds. Review the first hour and first day after a fix, then convert the diagnosis into a short runbook with owner, rollback step, and links to official documentation. Remove temporary debug logging once the evidence has been captured, especially if it could include prompts, file paths, or user data.

Prevent the same issue from returning

Move endpoint, model, timeout, and feature settings into reviewed configuration rather than scattering them through source code and individual laptops. Validate required variables at startup, reject unknown models, and expose a safe configuration summary that never includes secrets. Assign every application key and route an owner, environment, purpose, and rotation date. Subscribe to provider release and deprecation notices, but promote changes only after representative evaluations. Keep a tested rollback path and make emergency switches visible in logs and dashboards. Finally, review whether the original alert detected the customer impact early enough; if not, improve the signal while the incident evidence is still fresh.

LLM prompt caching: final production checklist

• Use the documented Base URL, credential type, endpoint and model ID.
• Start with a minimal reproducible request before enabling tools or agents.
• Classify errors before retrying and keep retries inside a total deadline.
• Log request IDs, route, model, latency and token usage without secrets.
• Test streaming, cancellation, failure recovery and rollback.
• Verify every gateway-specific feature instead of assuming complete compatibility.

Frequently asked questions

Official sources

• OpenAI prompt caching
• OpenAI pricing
• Anthropic prompt caching
• Anthropic pricing

This article provides technical guidance, not a guarantee of compatibility, availability, pricing, or security certification.