Open-Weights vs API: The Real Cost Comparison
AI pricing guide · updated 2026-08-07
Open weights are free to download and not free to run. Replacing an API bill with your own hardware converts a variable cost into a fixed one, and whether that is cheaper comes down to how busy you keep the machine.
Two different cost shapes
An API charges per token. Send nothing and pay nothing; send ten times more and pay ten times more. Cost tracks usage exactly.
Self-hosting charges for capacity. A GPU costs the same whether it runs at full load or sits idle, so the effective price per token falls as utilisation rises and becomes very high when the machine is quiet.
The comparison is therefore not "open weights versus API" but "your utilisation versus their price". Cohere's Command A+ illustrates the choice cleanly: Apache 2.0 weights that run on as few as two H100 GPUs, or the same model on Cohere's hosted API at $2.50 and $10.00 per million tokens.
What self-hosting actually costs
GPU time is the visible line, whether rented hourly or bought outright. Around it sit costs that estimates routinely omit: engineering time to deploy and keep a serving stack current, monitoring, capacity planning for peaks, and the idle hours between them.
Utilisation is the number that decides everything. A machine busy most of the day can beat API pricing substantially at scale. The same machine serving sporadic traffic loses badly, because you pay for the silence.
Reasons that are not about cost
Several teams self-host at a higher price on purpose. Data residency and regulatory constraints can require that inference happen inside a specific boundary. Weights in your possession cannot be deprecated out from under a running product, which matters when a vendor retires a model you depend on. Fine-tuning on proprietary data is easier when the model is yours.
Those are legitimate reasons to accept a worse per-token price, and they are worth separating from the cost argument rather than bundled into it.
How to decide
Measure your monthly token volume and your traffic shape first. Multiply the volume by a candidate API rate for the baseline, then price the GPU capacity needed to serve your peak, not your average, and add a realistic figure for engineering time.
Steady high volume favours self-hosting. Spiky or modest volume favours an API. And a middle path exists that is frequently the right answer: run an open-weights model through an inference provider, which gives you per-token pricing on an open model without operating the hardware yourself.
Frequently asked questions
Is self-hosting an open-weights model cheaper than using an API?
It depends almost entirely on utilisation. A GPU costs the same idle as busy, so steady high-volume traffic can beat API rates substantially while sporadic traffic loses badly. Price the capacity needed for your peak load, not your average, and include engineering time.
What are the hidden costs of self-hosting an LLM?
Idle GPU hours, engineering time to deploy and maintain a serving stack, monitoring, and capacity provisioned for peaks that runs unused the rest of the time. The GPU invoice is the visible part; these usually decide whether the comparison actually works.
Are open-weights models free?
The weights are free to download under their licence. Running them is not: you pay for hardware instead of tokens. Cohere's Command A+ shows both routes, with Apache 2.0 weights that run on as few as two H100 GPUs and a hosted API at $2.50 and $10.00 per million tokens.
Is there a middle option?
Yes, and it suits most teams: run an open-weights model through an inference provider. You get per-token pricing on an open model, avoid operating hardware, and keep the option of self-hosting later because the weights remain available.
See it in the data
Related guides
Terms used in this guide
Published by Tokenando. Last updated 2026-08-07. Figures in this guide are computed from our own pricing index and dated where they can move; see the methodology and corrections policy.