Start with the big idea
Attention lets a position combine information from other permitted positions. The model learns which context matters for producing each representation.
Understand it in more detail
For each position, linear projections produce query, key and value vectors. Scores are computed using QKᵀ, scaled by the square root of key dimension, masked where necessary and normalized using softmax. The resulting weights combine value vectors.
Multi-head attention performs several such calculations with separate projections and mixes their results. Positional information is required because attention alone does not inherently encode word order.
The original 2017 Transformer used an encoder-decoder structure. Many modern text generators instead use decoder-only causal transformers. Bidirectional encoders can attend in both directions, while causal language models prevent attention to future positions. Encoder-decoder models may add cross-attention to encoder outputs.
Worked example
In 'The robot moved the box because it was heavy', an attention mechanism can incorporate information from other tokens when representing 'it'. This is an illustration of possible dependencies, not a measured attention map from a trained model.
A misconception to avoid
An attention heatmap is not a complete causal explanation of a model's reasoning. The architecture diagram of an original encoder-decoder Transformer does not describe every modern LLM.
Experiment: where can attention look?
Explore a five-token example. Select a query position and switch between causal masking and unrestricted attention. Percentages come from hand-authored example scores, not a trained model or a measured attention map. This is one simplified attention head, not an explanation of a model's reasoning.
Weights for the selected query
Predict: When the fourth token is the query, which future positions must have zero weight under a causal mask? Why do allowed attention weights add to 100% before rounding?
Check your understanding
Think first, then open each answer to check your explanation. These self-checks do not record a score.
1. Why apply a causal attention mask?
To prevent a prediction position from reading future tokens during autoregressive processing.
2. Do all transformer-based LLMs have both an encoder and a decoder?
No. Many use decoder-only architectures.
Try it yourself
Draw a four-token causal attention matrix. Shade only the positions each token can access, then contrast it with a bidirectional matrix.
Explain it back: Summarize the main idea in your own words and name one limitation or risk.
Continue learning
Read an independent reference: https://arxiv.org/abs/1706.03762. This link is for reference, not a requirement to create an account. External websites may have different privacy and access policies.
Educational pilot. Terminology, age suitability and activities are subject to expert review. These pages do not send your answers to a server.