What breakthrough 2017 paper introduced the Transformer architecture, which is built around self-attention?

Show answer

Attention Is All You Need

This seminal paper from Google researchers replaced traditional recurrent layers with a mechanism called self-attention, allowing models to process entire sequences in parallel. The title reflects the radical idea that attention mechanisms alone are sufficient for state-of-the-art performance without the need for complex architectures. It paved the way for modern large language models like GPT and BERT by significantly improving training efficiency.

Also asked as:

Which 2017 research paper proposed the self-attention mechanism as a replacement for recurrence and convolutions in sequence modeling?

Would you remember this in 6 months?

Trivia Forge uses spaced repetition so facts stick. Free, no signup needed.

Turn on Machine Learning practice →

More Machine Learning trivia

What core mechanism allows Transformer models to process all parts of an input sequence simultaneously while weighing their relative importance?
What does the optimization algorithm name 'Adam' stand for?
What does the 'PAC' stand for in Leslie Valiant's mathematical framework for analyzing the complexity of learning algorithms?
What evaluation metric is defined as the harmonic mean of precision and recall?
What general term describes the failure mode in which a model's deployment input distribution differs from its training distribution, degrading performance, and which sequential imitation learners induce on themselves?
What heuristic is commonly used in K-means clustering to determine the optimal number of clusters by looking for a 'kink' in the cost function?

See all 98 →