What sets the cost of an embedded model: 61,977 production calls

Between 29 May and 27 August 2026 our system made 61,977 calls to language models in production — not in testing, but on live user requests. Every call is logged: surface, provider, language, input and output tokens, response time, and whether the request was served from cache.

What is absent here: money. The cost field is not populated for our two main providers — 31,796 calls are recorded as zero. That is a gap in our accounting rather than a free service, and we will not present those zeros as expenditure. So what follows concerns tokens and cache share — quantities that are reliably measured.

That is no great loss, though. The main lever on cost is not the price per token but the share of requests that never reach the provider at all. That we measured precisely.

Of 61,977 calls, 30,119 (48.6%) were served from cache and never reached a provider. By month:

The share rose from 23% to 52% in three months, by a simple mechanism: a cache works better the more requests have already passed through it. The key is built from request content, so every new user asking about an already-checked subject gets an answer for free.

The practical conclusion for anyone embedding a model: caching yields more than choosing a cheaper model. Moving to a model half the price halves the bill once. A cache with a growing hit rate reduces it continuously, and by the third month ours had removed half the load.

In tokens that is roughly 37 million never sent and never paid for — estimated from the average call size in our sample.

This is perhaps the most useful observation. The caching mechanism is identical across surfaces, and the result differs eightyfold:

The difference comes down to how repetitive the input is. For articles the input is the article text itself, identical for every reader — hence 82%. For a header check the input is one site’s full header set at one moment; it almost never repeats — hence 1%.

From which a design rule follows: before embedding a model, look at how repetitive the input is. If it is unique per request, caching will not help and savings must come elsewhere — a shorter prompt, a smaller model, or handing part of the work to ordinary code.

Three observations.

Input is three times larger than output. Roughly three and a half request tokens per response token. Since most providers price input below output but not by a factor of three and a half, shortening the prompt usually pays better than shortening the answer — while it is the answer that gets optimised more often.

A response takes seconds, not milliseconds. Averages of 3.9 and 5.4 seconds are two orders of magnitude slower than our ordinary tools, where the median is 95 ms. Embedding a model in an interface means assuming the answer arrives in seconds, and showing the user that it will.

A caveat on the OpenAI timing: our requests to it pass through an intermediate node outside Russia, and part of those 5.4 seconds is network path rather than model work. This column must not be used to compare providers directly.

Russian calls number 60,545 and English 1,432 — a ratio of 42 to 1. That reflects the audience rather than model quality, but it has a direct consequence for cost: the bulk of the load falls on the provider serving Russian, and its rates set the bill almost entirely.

Our language split was not made for price: Russian requests go to a provider with infrastructure in Russia, English to a foreign one. That is a data-handling requirement, not an optimisation.

The overall conclusion from three months of operation: the cost of an embedded model is set by architecture, not by rates. Cache share, input repetitiveness and prompt length produce multiples; the choice between models of comparable class produces fractions.

A caveat on applicability: this is one product with a particular set of tasks. The 48.6% cache share is ours, not an industry figure; a product with unique input on every request will see it near zero.