Pre-training

During pre-training, a model repeatedly predicts the next token across a corpus assembled from web text, books, code, and licensed data, adjusting billions of parameters until the predictions become accurate. This is where general language ability and world knowledge form, and it consumes the overwhelming majority of a model's training compute.

Frontier pre-training runs occupy thousands of accelerators for months, with estimated costs in the tens to hundreds of millions of dollars per run — a capital barrier that keeps the number of labs training at the frontier small.

Pre-training alone produces a base model that merely continues text. Post-training — instruction tuning and preference optimization — turns it into a usable assistant. For API buyers, pre-training cost matters indirectly: it is amortized into per-token prices and explains why frontier rates carry a premium.