RLHF

The classic pipeline has two stages: humans compare pairs of model responses and their preferences train a reward model, then reinforcement learning optimizes the language model to score highly against that reward. OpenAI's InstructGPT work and ChatGPT made the technique famous, and it is the reason chat models feel cooperative rather than like raw text predictors.

The family has since broadened: direct preference optimization (DPO) skips the separate reward model, RLAIF replaces human raters with AI feedback, and reinforcement learning on verifiable rewards — checking math answers or running code — drives modern reasoning training.

Preference training is a meaningful share of post-training budgets because human judgment is expensive to collect at scale. Its fingerprints are also commercially visible: differences in helpfulness, verbosity, and refusal behavior across providers largely trace to their preference-training choices, and verbosity in particular shows up directly on output-token bills.