Which 2022 DeepMind result showed that large language models were badly undertrained, recommending that model size and the number of training tokens be scaled in roughly equal proportion for a given compute budget?

Show answer

Chinchilla

The Chinchilla paper (Hoffmann et al., 2022) found compute-optimal training needs far more data relative to parameters than prior practice, roughly 20 tokens per parameter. A 70B Chinchilla trained this way outperformed the much larger 280B Gopher, shifting the field toward smaller, data-richer models.

Also asked as:

Which DeepMind model's training results suggested that for optimal performance, model size and training tokens should be scaled in equal proportion?

Would you remember this in 6 months?

Trivia Forge uses spaced repetition so facts stick. Free, no signup needed.

Turn on Machine Learning practice →

More Machine Learning trivia

Which 2023 method aligns language models directly from preference data with a simple classification-style loss, removing the separate reward model and reinforcement-learning loop used by RLHF?
Which activation function is defined by the formula f(x) = max(0, x)?
Which activation function is most commonly used in the final layer of a multi-class classification neural network to produce a probability distribution?
Which adversarial architecture consists of two networks, a Generator and a Discriminator, competing in a zero-sum game?
Which 1996 clustering algorithm is density-based and can identify outliers as 'noise' points?
Which clustering algorithm iteratively assigns points to the nearest cluster center and then updates the center based on the mean of assigned points?

See all 98 →