A plain summary, so you can get the gist here without leaving.
In 2020, OpenAI researchers found that the quality of a language model improves in smooth, predictable curves as you give it more compute, more data, and more parameters, so you can forecast how good a model will be before you build it.
What it is
This paper studies how three ingredients shape a model's performance: the amount of computing power used to train it, the amount of text it trains on, and the number of parameters, which are the adjustable internal settings the model learns.
The surprising finding is regularity. As you increase these ingredients, the model's error drops along clean mathematical curves rather than jumping around. The relationships held across a wide range of sizes.
The core idea
Bigger and more, within reason, means better, and the improvement is predictable. If you know the curve, you can estimate how much better a model will get if you double the compute or the data, before spending the money to train it.
That predictability turns model building into something closer to engineering than guesswork. It lets teams plan where to invest and reason about the returns from each extra unit of scale.
Why it matters
Scaling laws gave the field confidence to invest in much larger models, because the gains were forecastable rather than a gamble. A lot of the recent leap in AI capability traces back to taking these curves seriously.
For builders, the lesson is practical. Capability often comes from scale, and you can reason quantitatively about the tradeoffs between size, data, and compute instead of relying on intuition alone.
- Published in 2020 by OpenAI.
- Model error falls in smooth, predictable curves as scale grows.
- The three levers studied are compute, data, and parameter count.
- Lets teams forecast a model's quality before training it.
- Gave the field confidence to invest in much larger models.
OpenAI
New to this? Come build with us.
Reading is good. Building with people is better. Our drop-ins are free and open to total beginners.