What is the computational complexity of the self-attention mechanism in a standard Transformer relative to the sequence length L?
Show answer
O(L^2)
Because every token in a sequence attends to every other token, the cost of computing the attention matrix grows quadratically with sequence length. This scaling makes processing very long sequences a significant challenge for vanilla Transformers.
Also asked as:
- How does the computational cost of the self-attention mechanism scale with respect to the sequence length L in a vanilla Transformer?