Training Language Models to Follow Instructions (InstructGPT)

A raw language model is good at predicting text, but that is not the same as being helpful. It might ignore your instruction or answer in an unhelpful way. InstructGPT addresses this by adding a layer of human feedback to the training.

People review the model's answers and indicate which ones are better. The model is then adjusted to produce more of the kind of answers humans prefer. This technique is known as reinforcement learning from human feedback, or RLHF.

Rather than only learning from raw text, the model learns from human judgments about quality. Reviewers rank responses, those rankings train a model of what people like, and that preference model then guides the language model toward more helpful, honest, and on-target replies.

The result is a model that actually does what you ask. A smaller model tuned this way can feel more useful than a larger one that was only trained to predict text, because following the instruction is what people care about.

This alignment recipe is what turned powerful but unwieldy language models into assistants people can talk to. ChatGPT and the wave of chat assistants that followed are built on this idea.

For builders, the takeaway is that capability and helpfulness are different things, and human feedback is a practical way to bridge them. Much of what makes modern AI feel cooperative comes from this step.