What core mechanism allows Transformer models to process all parts of an input sequence simultaneously while weighing their relative importance?
Show answer
Self-attention
Self-attention (specifically Scaled Dot-Product Attention) calculates a weighted representation of the input by comparing each element with every other element. This allows the model to capture long-range dependencies more effectively than traditional RNNs.
Also asked as:
- What mechanism in Transformers weighs the relative importance of different elements in an input sequence simultaneously?