Behaviour, not knowledge. Supervised fine-tuning on demonstrations, then a reward model trained on human preference pairs, then policy optimisation against it with a KL penalty to stay near the SFT model. DPO skips the reward model.
Card 1 of 292. Answer: Behaviour, not knowledge. Supervised fine-tuning on demonstrations, then a reward model trained on human preference pairs, then policy optimisation against it with a KL penalty to stay near the SFT model. DPO skips the reward model.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
RLHF changes how a base model behaves, not what it knows: a model that continues any text becomes an assistant that follows instructions and writes answers people prefer. RLHF, reinforcement learning from human feedback, tunes the model toward a learned score of human preference.
A reward model, scoring the preferred answer r₁ and the rejected one r₂, treats σ(r₁ − r₂) as the probability people prefer the first, so scores of 2.0 and 0.5 give σ(1.5) ≈ 0.82 and a loss of −ln 0.82 ≈ 0.20. PPO then maximizes r(x, y) − β·log(π(y|x) / π_SFT(y|x)), reward minus a KL penalty against the supervised fine-tuned model. In InstructGPT (Ouyang and colleagues, 2022), labelers preferred a 1.3B RLHF model to 175B GPT-3.
Without the KL term the policy games the reward model with long, flattering or repetitive answers: reward hacking. The GPT-4 technical report showed post-training made the pretrained model's well-calibrated probabilities less so. DPO (Rafailov and colleagues, 2023) optimizes the same objective directly on preference pairs, with no reward model or sampling loop. None of these efficiently adds facts.
