What is the name for the empirical power-law relationships showing that a language model's loss falls predictably as model size, dataset size, and training compute increase?
Show answer
Scaling laws
Neural scaling laws (Kaplan et al., 2020) found that test loss decreases as a smooth power law in parameters, data, and compute, making capability gains forecastable in advance. They reframed progress as largely a matter of scaling the right quantities together.
Also asked as:
- What term describes the predictable power-law decrease in test loss as compute, data, and model parameters are increased?