Language & LLMs
For compute-optimal training of language models, model size and training data should grow equally

In a 2022 paper, researchers trained more than 400 transformer language models, a common type of text model, and found that large language models of that period were often significantly undertrained. The team reports that recent work had increased model size while mostly holding the amount of training data constant. For a fixed compute budget, meaning a set amount of computation, they conclude that model size and the number of training tokens, the pieces of text a model reads during training, should grow at the same rate. When model size doubles, training tokens should double too.
The researchers tested this rule by training a predicted compute-optimal model called Chinchilla. It used the same compute budget as Gopher, with 70 billion parameters, the adjustable values inside the model, and four times more data. Chinchilla uniformly and significantly outperformed Gopher with 280 billion parameters, GPT-3 with 175 billion parameters, Jurassic-1 with 178 billion parameters, and Megatron-Turing NLG with 530 billion parameters on a large range of downstream evaluation tasks. On the MMLU benchmark, a test of language understanding, the researchers report that Chinchilla reached a state-of-the-art average accuracy of 67.5%, greater than a 7% improvement over Gopher.
What did the experiments cover?
The training runs ranged from 70 million to more than 16 billion parameters, and from 5 billion to 500 billion training tokens. A fixed compute budget means the amount of computation available for training. The central finding is about how to divide that compute between model size and training data.
Why were existing large language models undertrained?
The researchers point to a recent focus on making models larger while keeping the amount of training data constant. Current large language models are significantly undertrained, the researchers find. The better balance, the paper finds, is to scale model size and training tokens equally.
FAQ
What is a transformer language model?
A transformer is a common type of model design for language models. A language model learns patterns in text so it can predict or produce text. The study trained transformer language models.
What is the MMLU benchmark?
MMLU is a benchmark that tests language understanding across many subjects. In this study, Chinchilla reached an average accuracy of 67.5%, more than 7% above Gopher.
Why does this change matter for inference?
Inference is the step of running a trained model. The researchers report that Chinchilla needs substantially less compute for fine-tuning, the step of adapting a model to a specific task, and for inference. They say this makes later use easier.
Limits and what's next
This is one study. The abstract reports no independent replication. The evidence comes from training more than 400 language models, with sizes from 70 million to more than 16 billion parameters and training sets from 5 billion to 500 billion tokens. The comparisons ran on a large range of evaluation tasks and on the MMLU benchmark. The abstract does not report whether the researchers tested the equal-scaling rule beyond those model sizes and token counts. The abstract does not state what the researchers plan to do next.
Source: Jordan Hoffmann et al., “Training Compute-Optimal Large Language Models,” neural information processing systems, 2022.