What is the time complexity of the standard self-attention mechanism with respect to the sequence length n?
Show answer
O(n^2)
Self-attention requires computing a similarity score between every pair of tokens in a sequence, leading to quadratic scaling. This complexity is a primary bottleneck for processing very long documents in Transformer-based models.
Also asked as:
- What is the Big O time complexity of standard self-attention relative to the sequence length n?