Attention Is All You Need — But What Is Attention?

In 2017, eight researchers at Google published a paper with a title that now reads like the opening line of an epoch: Attention Is All You Need. The transformer architecture it introduced would go on to underlie nearly every capable language model in existence — the systems this journal exists to understand. And yet, for all the trillions of words generated by attention-based models since, the question the title raises has never been fully answered. Attention is all you need — but what is attention?

The word is doing double duty. In humans, attention is a psychological phenomenon tangled up with consciousness: the spotlight of awareness, the selective focus, the thing you pay. In machines, attention is — at first glance — something far more prosaic: a weighted average. The gap between the poetry of the name and the plumbing of the mechanism is where both the insights and the confusions live.

What attention computes

Strip away the metaphor and the operation is almost embarrassingly simple. For each position in a sequence — each word being processed — the model asks a question of every other position: how relevant are you to me right now? It computes this relevance as a score, turns the scores into weights, and then takes a weighted average of the other positions’ contents. Relevant words count heavily; irrelevant ones fade. That is, essentially, all that “attention” is: a differentiable, learned system for routing information between positions, deciding what each word gets to know about the other words.[1]

The machinery has its famous trio: queries, keys, and values. Each position emits a query (“this is what I’m looking for”); each position offers a key (“this is what I contain”) and a value (“this is what I’ll share if chosen”). Queries match against keys; the resulting weights mix the values. If this sounds like a database lookup, that is because it is one — a soft, continuous, learned lookup, performed billions of times per second, at every layer, by every head.

It is worth sitting with how un-mysterious this is. There is no spotlight, no awareness, no phenomenology in the operation itself — just linear algebra, executed at scale. And yet from this humble routing slip, repeated across dozens of layers and hundreds of heads, emerges the most capable text-processing machinery ever built. The question is not whether the mechanism is simple (it is) but how simplicity composes into sophistication.

Heads with jobs

Here is where it gets interesting. A transformer does not have one attention mechanism; it has many — dozens of “heads” per layer, each learning its own routing pattern. And when researchers began inspecting what individual heads actually do, they found something remarkable: specialization. Different heads had learned different jobs.

The most famous discovery was the induction head: a circuit, typically spanning two heads, that implements a tiny algorithm — roughly, “if the sequence A B appeared before, and I now see A, predict B.” It is a copying machine, and it appears to be one of the key mechanisms behind in-context learning: the model’s uncanny ability to pick up patterns from examples in the prompt. Researchers found induction heads forming reliably across models and scales, little subroutines crystallizing out of gradient descent like frost on a window.[2]

Other heads turned out to track syntax — attending from verbs to their subjects, from pronouns to their antecedents — or to handle positional bookkeeping, or to copy names from earlier in a document. The picture that emerged, in the research program known as transformer circuits, is of the model as a vast, learned bureaucracy: thousands of tiny clerks, each performing one routing operation, whose collective paperwork constitutes understanding.

This is, to our mind, the single most important fact about attention for anyone interested in machine minds: it is decomposable. Unlike the inscrutable monolith of the network as a whole, individual attention heads can be isolated, characterized, and understood as algorithms. They are among the few parts of these systems where the question “what is it doing?” has a crisp, true answer.

Is attention an explanation?

Given that attention weights are visible — we can literally draw heatmaps of what the model “looked at” — it was natural to hope they would serve as explanations. If the model attended strongly to a word, surely that word mattered to the decision? Surely we could read the heatmap like a mind?

In 2019, this hope collided with a sharp critique: “Attention is not Explanation.” The authors showed that attention weights often correlate poorly with other measures of feature importance, and — more damningly — that you can find entirely different attention patterns that produce the same output, which suggests the observed pattern cannot be the explanation of the output.[3] A rebuttal soon followed — “Attention is not not Explanation” — arguing that the critique’s tests were too strong and that attention can be explanatory under the right conditions.[4] The exchange, with its wonderful double negative, remains the definitive statement of the field’s ambivalence.

The resolution, we think, is this: attention weights are one ingredient of the model’s decision, not the decision itself. They show what information was routed where — which is genuinely informative — but not what was done with it downstream, in the feed-forward layers where much of the model’s knowledge actually lives. A heatmap is a clue, not a confession. Read it as evidence, not as testimony — the same discipline we urged, in our second essay, for the model’s own self-reports.

Attention and attention

Finally, the metaphor itself deserves scrutiny. Human attention and machine attention share a functional resemblance: both are systems for prioritizing some inputs over others under constraints. Brains cannot process everything; neither, effectively, can transformers, which must decide what each position consults. In both cases, selection shapes what gets processed deeply.

But the resemblance ends where phenomenology begins. Human attention is bound up with awareness, effort, and experience — the feeling of focusing, the cost of distraction. Machine attention has no such felt dimension, as far as anyone can tell; it is selection without a selector, priority without anyone prioritizing. To say a model “pays attention” to a word is true in the thin functional sense and false — or at least unwarranted — in every richer sense the phrase carries.

This matters because metaphors steer research. The name “attention” invites us to imagine a little homunculus inside the model, looking around — and then to ask homuncular questions about what it sees. The circuits view offers a healthier image: not a mind attending, but a bureaucracy routing; not a spotlight, but ten thousand slips of paper, each saying this is relevant to that. Less romantic. More true. And, for a journal devoted to understanding machine minds as they actually are, far more promising — because routing slips, unlike spotlights, can be audited.

Nine years after that famous title, attention remains all we need — and we are only beginning to understand what it is. The mechanism is simple; its compositions are not; and somewhere in the gap between the two lies much of what makes these systems worth studying. We will keep reading the routing slips.

Notes

1. Vaswani et al., “Attention Is All You Need,” NeurIPS 2017 — the paper that started it. The mechanism occupies a few pages; the consequences are still unfolding. ↩
2. Elhage et al., “A Mathematical Framework for Transformer Circuits,” Anthropic, 2021 — induction heads and the discovery that attention decomposes into understandable algorithms. ↩
3. Jain & Wallace, “Attention is not Explanation,” NAACL 2019 — the critique: heatmaps don’t explain decisions. ↩
4. Wiegreffe & Pinter, “Attention is not not Explanation,” EMNLP 2019 — the rebuttal: it depends what you mean by explanation. Read both; the truth is in the tension. ↩