Direct preference optimization: aligning a language model to human preferences by training directly on pairs of preferred and rejected answers with a simple classification-style loss. It targets the same objective as RLHF without training a separate reward model or running reinforcement learning.
Card 1 of 292. Answer: Direct preference optimization: aligning a language model to human preferences by training directly on pairs of preferred and rejected answers with a simple classification-style loss. It targets the same objective as RLHF without training a separate reward model or running reinforcement learning.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Rafailov and colleagues showed in 2023 that the KL-regularized objective RLHF optimizes has a closed-form best policy, so the reward can be written in terms of the policy itself: β·log(π(y|x) / π_ref(y|x)). Putting that into the usual preference model gives the DPO loss, −log σ(β × [log-ratio of the preferred answer − log-ratio of the rejected one]), where each log-ratio compares the model being trained with a frozen reference, usually the supervised fine-tuned model.
Example with β = 0.1: at the start the model equals the reference, both log-ratios are 0, and the loss is ln 2 ≈ 0.693. After training raises the preferred answer's log-probability by 2 nats and lowers the rejected one's by 1, the margin is 0.1 × (2 + 1) = 0.3 and the loss falls to −ln σ(0.3) ≈ 0.554.
Follow-ups: β plays the role of RLHF's KL penalty, and a smaller β lets the model drift further from the reference; DPO trains offline on a fixed set of pairs, with no sampling during training, which makes it cheaper and more stable than PPO; Meta used it in Llama 3's post-training; and variants include IPO, KTO, which works from thumbs-up and thumbs-down labels instead of pairs, and ORPO, which drops the reference model.
