What is an attention mechanism?
An attention mechanism is a component of a neural network that computes, for each element it is processing, a set of weights over all the other elements and uses those weights to pull in the most relevant information. When a language model processes the word “it” in a contract clause, attention lets it look back across the sentence and weight “the Supplier” heavily and “the delivery date” lightly. The mechanism learns these weights from data rather than from rules.
Attention was first added to recurrent translation models so the decoder could look back at the right source words. The transformer then made it the main building block, and self-attention, where a sequence attends to itself, is the operation that gives large language models their grasp of context.
How the attention mechanism works
Each token’s embedding is projected into three vectors: a query, a key, and a value. The query of the current token is compared with the key of every token, and the comparison scores are scaled and passed through a softmax so they sum to one. Those scores are the attention weights. The output is the weighted sum of all the value vectors. In multi-head attention, several sets of these projections run in parallel, so one head can track grammatical agreement while another tracks which defined term a pronoun refers to. Cross-attention does the same across two sequences, such as a document and a question about it.
Why the attention mechanism matters
Attention gives a model direct access to any part of its input regardless of distance, which recurrent networks could never manage reliably. It is also why context windows have limits: comparing every token with every other token means compute and memory grow with the square of the input length. The KV cache, which stores keys and values for tokens already processed, is the standard engineering fix during generation and accounts for a large share of GPU memory use in an inference server. Attention weights are also one of the few windows into what a model is doing, though they are a partial explanation at best.
What to watch for
- Long inputs cost more than proportionally, so a model advertised with a huge context window may be slow and expensive at the far end of it.
- Models attend unevenly across long prompts, and information buried in the middle of a very long document is the most likely to be missed.
- Attention maps are suggestive rather than a faithful explanation of a decision, so do not present them to auditors as one.