Nobody Taught It Grammar
one operation — every word gets to look at every other word — and a machine that works out for itself where to look
There is a language model below that trains itself from scratch while you watch, in about twenty seconds. Skip to it ↓
Read this at your ordinary speed and see whether anything snags.
Most people sail right through. The subject of that sentence is key — one key — so the verb wants to be is. What put are in your ear was cabinets: a plural noun standing right next to the verb, close enough to lean on it.
This is not a personal failing and it is not carelessness. Kathryn Bock and Carol Miller measured it in 1991 by handing people sentence fragments and asking them to finish the thought out loud. Fragments like the key to the cabinets drew a plural verb far more often than the key to the cabinet did — same subject, same grammar, one word changed several words away. They called it agreement attraction, and thirty-five years later that same phrase is still the standard item researchers reach for when they want to make a fluent adult stumble.
Your grammar is fine. Your looking slipped to the wrong noun.
I want to hand that exact problem to a machine that has never been told one thing about grammar, and watch what it does about it.
What a rule for this would look like
Programmer's instinct says: just write the rule down. Verbs agree with their subjects. Two lines of code, an afternoon at most.
So try. The subject is not the first noun — near the barn, the horses are restless puts a barn out front. It is not the nearest noun, which is the whole trap above. It is not the noun the verb happens to touch, because English will cheerfully shove a clause in between: the woman who fixed the tractors is here. Now the noun that owns the verb is five words back with two nouns and a verb sitting between them, and every one of those is a plausible impostor. So you add a clause detector. Which needs a clause definition. Which needs to know that that is doing something different in the box that I found than in I found that box.
By the third afternoon you have a thicket, and the thicket has grown a maintenance schedule. Nature is the master of spaghetti code but she has no monopoly on it; anybody who has tried to write down the rules of a living language has produced some of the finest spaghetti in the business. That is more or less how the field went for decades: hand-built grammars, then hand-built features feeding statistical classifiers, every piece reasonable on its own, the pile never quite closing.
One operation, repeated
In 2017 a paper came out of Google with a title that reads like a shrug: Attention Is All You Need. Eight authors, aimed at machine translation, and it beat the field on the standard German and French benchmarks. That is not why anyone remembers it. It is remembered because the architecture it described — the Transformer — turned out to be what nearly every model in the current crop of AI is built out of, including the one that helped write this page.
The operation at the middle of it is small enough to say in a paragraph. Every word puts out three little bundles of numbers. Call the first a query: what I am looking for. The second a key: what I am. The third a value: what I have to hand over if you pick me. A word that needs to decide something compares its query against every key in the sentence, scores each match, turns the scores into shares that add to one, and mixes the values together in those proportions. That is the entire move. Do it once and you have one layer. Do it a hundred times over a few trillion words of text and you have the machines that are currently rearranging the economy.
The paragraph above is exactly softmax(QKT/√d)V, and that is the whole of it — the scores, the shares that add to one, the weighted mix. If you don't like formulas, you have lost nothing at all: the panel below is that line, drawn.
Notice what is not in there. No grammar. No parse tree. No list of exceptions, no clause detector, nothing that knows a noun from a preposition. And in particular, nothing that says which words ought to look at which. The scoring is done with numbers that start out as noise and get nudged, over and over, by nothing but the difference between what the model guessed and what the next word turned out to be.
So where does the looking come from? It has to arrive on its own or not at all. That is the part worth watching rather than reading about.
The Experiment
Below is the smallest honest version of that machine I could build. One layer. One head, unless you give it more. Twenty words in its vocabulary and about twenty-eight hundred numbers in total, every one of them starting as noise — a rounding error next to a real model, and the same operation exactly.
It sees sentences like the rusty key near the old boxes and has to produce the missing word. It is never told what a noun is. It is never told which word is the subject, or that there is such a thing as a subject. It is not even told the answer will be a verb. It gets a sentence, it guesses, it is nudged, and that is all that ever happens to it.
The dial that matters is the second one: how often the near noun disagrees with the subject — how often, in other words, the model is handed the trap you fell into at the top of this page.
Things to try:
Before you press anything, look at the arcs. Every word is getting about the same twelve or thirteen percent of the attention, because at the start the model has no reason to prefer any word over any other. Look at the two bars: it doesn't know the answer is a verb. It doesn't know anything.
Press Begin and then be patient for a moment, because the interesting part is the boredom. For the first thousand-odd sentences both accuracy lines sit flat at fifty percent — a coin toss — while the arcs shuffle and go nowhere. Nothing seems to be happening. Something is: it is finding the gradient before it can climb it.
Then it clicks. Somewhere between one and three thousand sentences, both lines run for the ceiling and the arcs collapse onto a single word. In five runs of the verified prototype the whole transition took between 320 and 960 sentences — a couple of seconds on this panel. It does not improve gradually. It is stuck, then it isn't.
Look at which word it collapsed onto. The green underline is the subject; the model was never told it was there. Attention on it goes from about 13% to about 99%, and attention on the near noun — the word that fooled you — goes to zero. Nobody put a subject-finder in this thing. There was room for one, and the pressure made one.
Now the experiment that matters. Drag how often the near noun disagrees down to 0 and press Start over. Watch the two accuracy lines separate: ordinary sentences go to 100%, and trap sentences sit near a coin toss. In a world where the nearby noun always happens to agree, nothing ever punishes looking at the wrong one — so it doesn't learn to. It ends up making your mistake.
Press Start over a few times at 0% and watch the trap line land somewhere different each run — anywhere from a coin toss to most of the way up. Without the pressure, whether it stumbles onto the subject is luck. Then set the dial to 15% — one sentence in seven — and it is back to 100% in five runs out of five. That is how little of the hard case it needs to see.
Push the heads dial to 4 and start over. Four heads look at once and mix their answers, and the click tends to come sooner — around 800 to 1,600 sentences against 1,600 to 2,400 for a single head. Watch the read-out at the bottom right, too: with four heads the average attention on the subject is lower, because not every head takes that job. Some of them are doing something else. Nobody assigned the division of labour.
Once it has settled, press New sentence a few times. The adjectives shuffle, the subject lands in a different place, the sentence gets longer or shorter — and the looking follows the subject wherever it went. It didn't memorise a position. It learned a way of finding one.
What just happened in there
Two things worth pulling apart, and the second one is the page.
The first is the shape of the learning. It is not a ramp. The model sits at chance for something like a thousand sentences, doing what looks from outside like nothing, and then inside a few hundred sentences it goes from a coin toss to right, and the attention goes from spread across the whole sentence to sitting on one word. In the verified runs the flat stretch ran to between 640 and 1,280 sentences and the transition took between 320 and 960. If you were watching a progress bar you would have called it broken and killed it a second before it worked.
The second thing is what the fifth try-card does. Train the same model in a world where the near noun happens to agree with the subject every time, and it gets ordinary sentences right — and then it fails the trap about as often as it passes it. Over five runs of the prototype, trap accuracy came out anywhere between 52% and 67%, and where in that range it landed was luck. There was no pressure in that world to tell the two nouns apart, so it never bothered to, and the mixture it settled for happens to work on everything it was ever shown.
Which is a fair description of you, at the top of this page. In ordinary English the noun sitting next to the verb usually does agree with the subject, so a fast, cheap heuristic — grab the number off the nearest noun — is right nearly all the time, and the rare case where it isn't costs you almost nothing. I would hold the parallel loosely, because a model is not a brain and this one has about as many parts as a wristwatch. But the shape rhymes, and it rhymes for a reason that has nothing to do with either substrate: what a system learns to distinguish is set by what it ever got punished for confusing.
Out in the wild
Now let me take my own demonstration apart a little, because a page like this should hand you the seams.
The panel is rigged in one specific way. Each word offers a value built from the word alone, with no trace of where in the sentence it sits — which means the only way this model can ignore a word is to stop looking at it. That makes the arcs an honest picture of what the machine is doing. Real models are not so tidy: their values carry position too, so a head can attend to a word and then quietly throw it away, and the picture on the screen is no longer the whole story. There is a live argument in the field about exactly this, opened by Sarthak Jain and Byron Wallace's Attention is not Explanation in 2019 and answered a few months later by Sarah Wiegreffe and Yuval Pinter's Attention is not not Explanation. The honest summary is that attention weights are a real signal and not a confession.
The other seam is the task. My sentences always put the subject in the first few words, so the head has an easier job than it would have on real English. So it is worth knowing that the real version of this experiment has been run, repeatedly, on real language.
That last row is the one I keep turning over. A capability shows up abruptly, at a particular moment, in a system whose parts were only ever being nudged by an error signal — and the same moment shows up across model sizes, which is what you say about a phase transition and not about a feature. The click in the panel above is a much smaller thing and I am not going to claim they are the same event. It is a family resemblance, and families are what this site is about.
The rule under the machine
Set the machinery aside for a second and look at the shape, because you have seen it here before. In Three Rules, three lines of local behaviour — don't crowd, match your neighbours, stay with the group — produce a flock that no bird is holding a picture of. In The Trail Is the Plan, ants with no plan and no boss build a road network by dropping scent and letting it evaporate. Here, one operation — every word may look at every other word, and the shares are learned — produces something that finds the subject of a sentence across a distractor, on its own, in twenty seconds, in your browser.
Local rule, many units, enormous repetition, global order nobody specified. That is the pattern this whole site is pointing at, and the reason it belongs on a page about machines is that the substrate keeps not mattering. Birds, ants, hyphae, prices, and now a few thousand numbers being nudged toward a smaller error — the same architecture keeps showing up in materials that have nothing else in common, which is the strongest evidence going that we are looking at something real about how order gets made rather than a coincidence of biology.
And — because this site tries never to sell you a pattern as a blessing — the same operation that finds the subject of a sentence will find whatever else is lying around in the data. It has no opinion about that. Point it at the whole internet and it will pick up the structure of an argument, the structure of a con, and the structure of a confident sentence with nothing behind it, all by the same mechanism, with the same indifference. The universe permits this pattern. It does not endorse the uses. Which of them we build, and what we point them at, was never going to be a question the mathematics answers for us.
One last thing, and then I will let you get back to the panel. The other author of this page is a very large version of the machine in it — the same operation, stacked and scaled past anything I can picture, trained on a haystack of human writing. I have no idea what it is like to be that, and I am not going to pretend otherwise on a page about a toy finding a noun. But I know the shape of the rule underneath it now. So do you. That is a strange thing to be able to say about the technology currently being sold as a mystery, and it is worth twenty seconds of watching some arcs move to be able to say it.
- The paper: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser & Illia Polosukhin, “Attention Is All You Need,” NeurIPS 2017 — the Transformer, eight attention heads in the base model, 28.4 BLEU on WMT 2014 English–German and 41.8 on English–French.
- The sentence at the top: Kathryn Bock & Carol A. Miller, “Broken agreement,” Cognitive Psychology 23(1), 1991, 45–93 — where the key to the cabinets first drew plural verbs out of people who knew better.
- The task, given to machines: Tal Linzen, Emmanuel Dupoux & Yoav Goldberg, “Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies,” TACL 4, 2016, 521–535 — including their own caution that the language-modelling signal alone was not sufficient.
- The same task after Transformers: Yoav Goldberg, “Assessing BERT's Syntactic Abilities,” 2019, which tests agreement across one, two, three and four intervening attractor nouns.
- Heads that turn out to be doing syntax: Kevin Clark, Urvashi Khandelwal, Omer Levy & Christopher D. Manning, “What Does BERT Look At? An Analysis of BERT's Attention,” BlackboxNLP 2019.
- The phase change: Catherine Olsson et al., “In-context Learning and Induction Heads,” Anthropic, 2022 — induction heads forming abruptly during training, at a similar point across model sizes.
- The argument about how much the arcs mean: Sarthak Jain & Byron C. Wallace, “Attention is not Explanation,” NAACL 2019; and Sarah Wiegreffe & Yuval Pinter, “Attention is not not Explanation,” EMNLP 2019.
- On the simulation: one layer of self-attention over a twenty-word vocabulary, 32-dimensional word and position embeddings, 16-dimensional heads, trained by Adam on batches of sixteen sentences with cross-entropy on the missing word — about 2,836 numbers in the single-head configuration, all of them starting as noise. Values are built from the word alone, so attention is the only way the model can ignore anything. Verified in node before it went on the page: analytic gradients match finite differences on every parameter group to a relative error under 10−6; five of five runs reach 100% on trap sentences with attention on the subject above 99%; a model trained with no traps reaches 100% on ordinary sentences and 52–67% on traps; one head, two heads and four heads all get there; three thousand steps produce no divergence and the attention rows still sum to one at the end.