Multimodal model

Most frontier models are now natively multimodal on the input side, accepting images and often audio or video alongside text. Output multimodality (generating images, speech, or video) is sometimes built into the same model and sometimes served by dedicated companion models.

Non-text inputs are billed by converting them into token equivalents: an image is sliced into patches or tiles that count as input tokens, and audio is metered per second or per token depending on the provider. A single high-resolution image can cost as much as several pages of text, which matters for pipelines that process screenshots or documents at volume.

Multimodality changes workload design: tasks that once required a separate OCR or speech-to-text step can go straight into the model, trading pipeline complexity for token spend.