Reinforcement Learning from Human Feedback
What is it?
RLHF is a training method that fine-tunes a model using human preferences — people rank the model's answers, and the model learns to produce responses people prefer.
Explain like I'm 5
Why was it created?
A base language model predicts likely text, not helpful or safe text. RLHF was developed to align models with what humans actually want — helpful, honest, and harmless responses.
Where is it used?
- Chat assistants
- Making models follow instructions
- Reducing harmful or off-topic output
- Tuning tone and helpfulness
Why should developers care?
RLHF is the step that turns a raw next-word predictor into a usable assistant, so it underlies the chat models most developers build on.
How does it work?
Humans rank multiple model responses to the same prompt. Those rankings train a 'reward model' that scores answers. The language model is then optimized with reinforcement learning to maximize that reward, nudging it toward preferred responses.
Real-world example
Given a question, the model drafts several answers; annotators rank them; the model learns to favor the clear, helpful style people ranked highest.
Common use cases
- Instruction following
- Safety and refusal behavior
- Aligning tone and formatting
- Preference-based fine-tuning
Advantages
- Aligns output with human preferences
- Improves helpfulness and safety
- Captures fuzzy goals hard to write as rules
- Works on top of a pretrained model
Disadvantages
- Expensive — needs human labelers
- Reward model can be gamed
- Can over-refuse or become bland
- Hard to reproduce consistently
When should you use it?
When you need a model to behave like a helpful, aligned assistant rather than just predict text.
When should you avoid it?
When a task is fully specified by rules or when simpler prompting/fine-tuning already meets the goal.
Alternatives
Related terms
Interview questions
Beginner
- What kind of feedback does RLHF use?
- Why isn't a base language model already a good assistant?
Intermediate
- What is the role of the reward model in RLHF?
- How can a reward model be 'gamed'?
Senior
- How does DPO simplify the RLHF pipeline?
- What failure modes appear when over-optimizing a reward model?
Common misconceptions
- "RLHF teaches the model new facts" — it mainly shapes behavior and style, not knowledge.
- "More RLHF is always better" — over-optimizing can make models evasive or bland.
Fun facts
- InstructGPT showed a small RLHF-tuned model could be preferred over a much larger untuned one.
- The reward model is itself a trained neural network.
Timeline
- 2017 — Deep RL from human preferences demonstrated
- 2022 — InstructGPT popularizes RLHF for chat models
Learning resources
Quick summary
RLHF fine-tunes a model on human preference rankings via a reward model, turning a raw text predictor into a helpful, aligned assistant.
Cheat sheet
- Humans rank answers → reward model
- Model optimized to maximize reward
- Turns base model into an assistant
- Costly; reward can be gamed