AI Inference Cost Trends 2026
2026 LLM inference cost is declining ~8x YoY. GPT-5 ($5 input/$15 output per 1M) — 2x cheaper than GPT-4 (2023) at better quality. Llama 3 70B via Together.ai — $0.88/1M (8x cheaper than GPT-5). Self-host Llama 3 + H100 $3/hour = $0.001 per 1M tokens (50x cheaper). Trend: API prices fall, hardware faster, INT4 quantisation. 2027 forecast: GPT-5-class quality at $0.50/1M.
Below: key findings, platform breakdown, implications, methodology, FAQ.
Key Findings
| Metric | Pass/Value | Median | p75 |
|---|---|---|---|
| GPT-5 / GPT-4 price ratio | 50% ($5 vs $10) | — | — |
| Llama 3 70B (Together.ai) | $0.88/1M | 0.88 | — |
| Self-host Llama 3 70B (H100) | $0.05/1M | 0.05 | — |
| Median cost per query (RAG app) | $0.001 | 0.001 | 0.005 |
| Cache hit ratio (pre → saved) | 35% | — | — |
| YoY cost decline | ~8x | — | — |
| TTFT (time to first token) | 320ms median | 320 | 620 |
| Tokens/sec (Groq LPU) | 500+ | 500 | 750 |
Breakdown by Platform
| Platform | Share | Detail | — |
|---|---|---|---|
| OpenAI GPT-5 | Frontier | $5/$15 per 1M | — |
| Claude Opus 4.7 | Frontier | $15/$75 per 1M | — |
| Gemini 2.5 Pro | Frontier | $2/$10 per 1M | — |
| Llama 3 70B (Together) | Mid-tier | $0.88/$0.88 per 1M | — |
| Groq Llama 3 70B (LPU) | Mid-tier | $0.59/$0.79 per 1M | — |
| Self-host Llama 3 70B H100 | DIY | $0.05 per 1M (amortised) | — |
Why It Matters
- API prices falling — LLM becomes a utility. Vendor lock-in reduces value
- Self-host pays off at >10M tokens/day. Otherwise cloud API is cheaper + simpler
- Caching: prompt cache reduces 90% cost on hit. Anthropic explicit, OpenAI automatic
- Smaller models (gpt-4o-mini, Llama 3 8B) handle 60%+ of tasks cheaper than frontier
- Groq LPU — new paradigm. 10x inference speed at competitive cost
Methodology
Public pricing pages (Mar 2026) + usage data from 500 apps + Groq / Together benchmarks. Trailing 12-month price tracking.
Learn more
Frequently Asked Questions
When does self-host pay off?
>10M tokens/day at constant load. 1 H100 $3/h × 24 × 30 = $2,160/mo = ~2.4B tokens throughput.
gpt-4o-mini vs GPT-5?
Mini: $0.15/$0.60. 25x cheaper than GPT-5. Quality: 70-85% on most tasks. For chatbot / classification / simple extraction — use mini.
Cache effectiveness?
Anthropic cache 90% cheaper on hit. OpenAI automatic 50% cheaper. 35% cache hit = 30%+ cost reduction.
How to monitor AI spend?
Per-provider dashboard + app-level tagging via X-Project header. Anomalies → alert (daily spend > threshold).