Tokenization

Most modern models use variants of byte-pair encoding (BPE), which builds a vocabulary of common character sequences from training data. Frequent words become single tokens, while rare words are split into several smaller pieces. The same approach extends to code, and multimodal models apply analogous schemes to convert images or audio into token-like units.

Each model family ships its own tokenizer, so the same text produces different token counts on different models. A prompt that is 1,000 tokens on one provider can be 1,100 on another, which matters when comparing per-token prices: the cheaper rate does not always mean the cheaper bill.

Providers publish tokenizer libraries and counting endpoints, and using them to measure real workloads is far more reliable than the 4-characters-per-token rule of thumb, especially for non-English text and structured data.