Training Compute-Optimal LLMs (Chinchilla)

This work, which produced a model nicknamed Chinchilla, revisits how to spend a fixed training budget. The two main things you can scale are model size, the number of parameters, and data, the amount of text the model reads. The question is how to balance them.

DeepMind's answer challenged the prevailing habit. Many earlier models had grown very large but had not been trained on proportionally enough data. Chinchilla showed that rebalancing toward more data, with a smaller model, used the same compute more wisely.

For a given amount of compute, there is a sweet spot between size and data. Going too big without enough text wastes the budget, because the model has more capacity than it has examples to learn from.

Chinchilla, a smaller model trained on much more data, outperformed larger models trained on less, while using a comparable budget. The lesson is that data and size should grow together in a balanced way, not size alone.

This reshaped how teams plan training runs. Smaller, well-fed models are cheaper to run afterward and can match or beat bloated ones, which matters for both cost and accessibility.

For builders, it is a reminder that more parameters is not automatically better. Thinking about the balance between model size and training data leads to models that are both stronger and more efficient to use.