Top news
Investing in the public benefit OfficialAnthropic NewsOct 18Popular Musk Pledges SpaceX Shares to ‘Every Kid in America’ WebBloomberg TechnologyOct 19Popular Websites are closing their doors to AI agents WebThe Verge AIOct 17 This is the new ChatGPT Voice, powered by GPT Live OfficialOpenAI on YouTubeOct 18 How AI giants will reshape global philanthropy WebAxios AIOct 19 Gemini 4 Argon Really Surprised Me WebTwo Minute PapersOct 19 Google Gemini 4 agents just beat Minecraft with no human intervention WebEngadget AIOct 19 Who could it be? 🧐 OfficialGoogle on YouTubeOct 18 Anthropic gives outside AI watchdog “employee-like” access WebArs Technica AIOct 18 What are self-replicating ‘AI worms’, and why is Anthropic worried about them? WebThe Guardian AIOct 18
GlossaryTrainingIntermediate

Reinforcement learning from human feedback

Also written: RLHF, Direct Preference Optimization, DPO, RLVR, Feedback loops, Reward-Aligned Learning, Reinforcement learning with human feedback

Training AI models using human preferences as a reward signal to guide behavior and improve responses.

In plain terms

Reinforcement learning from human feedback, or RLHF, is a way to train AI models. Instead of just giving the AI lots of text to learn from, people help teach it by rating its answers. This feedback tells the AI what kind of responses are good and which ones are not.

Imagine you are teaching a chatbot to be helpful. You ask it a question, and it gives you two answers. You then tell the AI which answer you liked better. The AI uses this information to learn how to give better answers in the future. This process helps make AI models more useful and safer.

The expert version

Reinforcement Learning from Human Feedback (RLHF) is a machine learning technique used to align AI models, particularly large language models (LLMs), with human preferences and values. It involves using human judgments as a reward signal to fine-tune a pre-trained model. The process typically includes collecting human preference data (e.g., ranking different model outputs), training a reward model to predict these preferences, and then using reinforcement learning to optimize the LLM against this reward model.

RLHF is crucial for making AI systems more helpful, honest, and harmless. However, it can be resource-intensive and may lead to issues like sycophancy, where models favor agreement over truthfulness, as noted in recent Anthropic research. Tradeoffs include the cost of human annotation and the potential for the reward model to be misspecified.

Why it matters now

Recent news highlights how companies are moving beyond text generation to use human feedback for more complex AI decision-making and actions. This approach is crucial for developing trustworthy AI systems that can automate tasks, act as assistants, and even make scientific discoveries by aligning AI behavior with human values.

Where it shows up 30 items, newest first
How each company uses it
Anthropic Claude
Anthropic's research touches on AI safety, implying a need for methods to govern powerful systems, which often involves human feedback for alignment.
Meta Meta AI
Meta AI is exploring AI agents that will act for users, a capability likely refined through human feedback to ensure appropriate actions.
Microsoft Copilot
Microsoft's CARE-X research uses 'Reward Aligned Learning,' a concept closely related to RLHF, to improve the clinical accuracy of radiology AI.
OpenAI ChatGPT
OpenAI is a key player, with its models powering applications like Fyxer's AI executive assistant, which relies on trust built through human feedback, and research into AI decision-making beyond text.
Related terms

Reinforcement learningFine-tuningLarge language modelsAI alignmentReward modelingSupervised fine-tuning

Often confused with
Supervised fine-tuning
This involves training a model on a dataset of input-out
Direct Preference Optimization (DPO)
A newer method that achieves similar align
Reinforcement learning
A broader machine learning paradigm where agents learn b
Origin

The concept of using human feedback to guide machine learning models has roots in various areas, but RLHF as a specific technique for LLMs gained prominence with work from OpenAI and DeepMind around 2017-2020. Source ↗

Try this about 5 minutes

To see RLHF in action, try interacting with a chatbot like ChatGPT or Claude. Ask it a question, and if it provides multiple answers or a response you want to refine, look for options to 'regenerate' or provide feedback. Repeatedly asking clarifying questions or providing feedback on its answers helps the model learn from your preferences.

Try it: Playbook entries that use thisPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Questions people ask

Is RLHF the only way to train AI?
No, RLHF is one of many training methods. Other common methods include supervised fine-tuning on large datasets and unsupervised pre-training.
Does RLHF guarantee AI will be perfectly safe?
RLHF significantly improves safety and alignment, but it is not a perfect solution. AI models can still exhibit unintended behaviors or biases.

First seen on Subvolts Sep 4, 2020. Updated Oct 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.