Attention Mechanism
The attention mechanism is the part of a transformer model that lets it weigh how relevant every other word in a sequence is to the word it's currently processing, capturing context and long-range relationships.
Key takeaways
- The attention mechanism lets a model weigh how relevant every other token is to the one it's currently processing.
- It's the core innovation behind the transformer architecture introduced in 2017.
- Self-attention relates tokens within one sequence; cross-attention relates tokens across two different sequences.
- Attention lets models capture long-range relationships that earlier step-by-step architectures struggled with.
- Attention is what made it practical to train the large, highly capable language models used today.
What is the attention mechanism?
The attention mechanism is the core innovation behind the transformer architecture. For every word, or token, in a sequence, it computes a relevance score against every other token, letting the model decide which parts of the input matter most for understanding or generating the current word, regardless of how far apart they are in the text.
How attention works
Attention assigns each token three roles, often described as a query, a key, and a value, and uses the mathematical similarity between queries and keys to decide how much weight to give each value when building a representation of the current token. In practice, this lets a model correctly connect, for example, a pronoun to the specific name it refers to several sentences earlier.
Self-attention vs cross-attention
Self-attention compares tokens within the same sequence to each other, which is how a model understands relationships within a single passage of text. Cross-attention compares tokens from two different sequences, such as connecting an image to a caption describing it, which is common in multimodal models that need to relate two different types of input.
Why the attention mechanism matters
Before attention-based models, earlier architectures processed text step by step and struggled to retain relationships between distant words. Attention lets a model consider an entire sequence at once and directly weigh long-range relationships, which is the key capability that made today's large, highly capable language models possible.
Frequently asked
- What does the attention mechanism do?
- It lets a model determine how relevant every other part of an input is to the part it's currently processing, which is how transformers capture context and relationships across a sequence.
- What is self-attention?
- Self-attention is when a model compares tokens within the same sequence to each other, letting it understand relationships within a single passage of text.
- Is attention the same as a transformer?
- No. Attention is the core mechanism within a transformer architecture, but a transformer also includes other components, like feed-forward layers, built around that attention mechanism.