A small model, read from the inside.

Six findings, each with its control.

1. The model decides that it will answer long before it decides what. Read out at intermediate depth, a question such as “The capital of France is” decodes to a literal blank — ·____ — from about depth 14 to depth 22, often with more than 90% probability. The answer then appears within one or two blocks.
2. The look-up is a single hand-off. Activation patching shows the country's identity sitting on the country token through block 19 and on the final token from block 20 on. Attention does the moving; the MLPs of the next few blocks turn “France” into “Paris”.
3. The last block is an editor, not an author. It rotates the residual stream more than any block except the first, removes the giant first-token signal, and lowers the model's confidence: answers held at 0.80 probability at depth 26 leave the model at 0.40.
4. Calendars are rings and numbers are a line — and both are there early. Months and weekdays sit in calendar order around a ring already in the embedding table. Two-digit numbers do not exist in the embedding table at all (the tokenizer splits them into digits); blocks 0 and 1 assemble them, and by depth 3 the hundred numbers lie in order along a line.
5. Not every human ordering is a geometry. Temperature words (freezing … scorching) and size words show no ordered layout that beats shuffled labels. We report that as found.
6. Concept directions are portable. A direction computed from twelve words in one fixed sentence picks out unseen members of the same family inside ordinary sentences with 0.87–1.00 accuracy at every block but the last (0.61); chance is 0.067.
7. Copying is a small, separable circuit. Previous-token heads in blocks 0–2, the first induction head in block 3, the rest in blocks 16–24. Removing the 16 strongest costs ordinary text 0.047 nats per token and destroys in-context copying.

“Depth” below counts residual-stream states: depth 0 is the embedding, depth n is the state after block n − 1, depth 28 is what the model actually outputs. “Block” counts blocks from 0.

A blank, then the answer.

The logit lens asks a simple question: if the model had to stop at this depth and speak, what would it say? Pick a prompt and read down the table. The early rows are noise. Around depth 8–12 the model is restating the topic — “Cities”, “currency”, “located”. Then, for every question-shaped prompt, comes a long stretch where the most likely next token is an empty slot. Only in the low twenties does the slot get filled, and it happens fast.

The placeholder changes with the kind of continuation expected. A list (“Monday, Tuesday, Wednesday,”) holds ·etc where a question holds ·____; a prompt that wants a quoted word holds an opening quotation mark. The code prompt has no placeholder phase at all and locks its answer by depth 16. We read this as the model first settling the type of what comes next and then, separately, its identity. Runner-up tokens at the moment of resolution are frequently the same concept in Chinese (巴黎 beside Paris, 冷 beside cold, 周四 beside Thursday): what arrives is a concept, and the language is chosen afterwards.

This is not a quirk of a few examples. Across the 93 prompts the model answers correctly, a blank placeholder is the single most likely token at depth 20 in 100.0% of them. On ordinary running text the same read produces a placeholder at 0.64% of positions at that depth (maximum over all depths: 0.83%), so the blank is tied to the question format, not to the lens.

One behavioural aside the table hides: asked for the past tense of an irregular verb, this model gives the past participle — eat → eaten, go → gone, see → seen, take → taken, write → written — in 9 of 16 cases. It is wrong in a perfectly consistent way.

Where the fact moves.

A readout shows what is decodable; it cannot show what is used. For that we ran the model on “The capital of Japan is” while copying in, at one block and one token position, the internal state from “The capital of France is”, and measured how much of the shift from Tokyo to Paris that single transplant buys (66 prompt pairs, 22 countries).

The picture is unusually clean. Through block 19 the country token holds 0.905 of the answer and the final token 0.099; after block 20 it is 0.191 and 0.809. Nothing is gained by patching “The”, “capital” or “of” anywhere. The attention panel lights up only for windows that contain block 20, and the MLP panel at the final position peaks just after (0.594 for the window centred on block 22) — attention fetches the subject, then MLPs compute the attribute. That is the depth at which the blank in Finding 1 gets filled.

The first rows of the attention and MLP panels are a second result. Transplanting only block 0's writes at the country token is enough to restore essentially the whole answer, even though the token embedding underneath still says “Japan”. In this model the embedding is tiny (norm 1.50) and block 0 overwrites it (norm 18.1, cosine to the embedding 0.055): the working identity of a token is written by block 0, not read from the table.

Down to single blocks and single heads

Repeating the experiment one block at a time, and then one attention head at a time, for three different kinds of question:

One caution about that last column. Head L20H13 is the strongest single mover — copying its output alone restores most of the answer — but it is not individually necessary: replacing it with its average leaves capital-city accuracy at 0.83 (from 1.00), the largest drop of any single head, while the other heads in its block leave accuracy at 0.96–1.00. The look-up rides mainly on one head but the network routes around its loss. Attribution names the carrier; it does not prove a bottleneck.

Three regimes, and an editor at the end.

Block 0 replaces the input (cosine 0.055 with the embedding). Blocks 1–19 accumulate: each changes the state a little (cosine 0.87–0.96), the state grows roughly 1.20× per block, and an intermediate read agrees with the final output less than 11% of the time. Blocks 20–26 resolve: agreement climbs from 15% to 72%. Block 27 edits: cosine 0.531, an MLP write of norm 3,297 against a stream of 3,299, entropy back up from 1.16 to 2.40 nats. The same block is where concept directions stop generalising (Finding 5) and where calendar order is weakest.

The anchor token

Qwen3 has no beginning-of-text token, so whatever token comes first is drafted into the job. Block 2 writes a state of norm 14,392 onto it — almost all of it in two of the 2,048 coordinates — while every other token is near 31. That state is then carried through 24 further blocks varying by less than 1.8%, and the final block takes it back down to 1,183. Attention follows: heads put 8% of their weight on the first token in block 2 and 70% in block 3, and stay between 39% and 81% from there on. It is a parking place for attention that has nothing to say, built in one block and dismantled in one block.

How many dimensions it uses

Measured over 30,362 tokens at full width, between 1,698 and 1,884 of the 2,048 directions are needed to hold 99% of the variance at every depth: the model does not live in a small subspace. The participation ratio — a softer count of how evenly variance is spread — runs 26–235, peaks at depth 9, and collapses in the last three states as the stream is reshaped around the prediction.

Walk the manifolds.

Each point is one word. Its position is the model's internal state for that word, projected onto a flat plane. To keep ourselves honest the plane is fitted on three carrier sentences and the points you see come from three different sentences; and every ordering score is compared with the same score under 2,000 random relabelings. Drag the slider to move through depth, or press play.

Months and weekdays are rings in calendar order from the embedding table onward. The month ring holds (score ≥ 0.89) until depth 20 and then loosens to about 0.65 — the same depth at which, in Finding 2, final-position states start being rewritten toward the next token.

Numbers are the clearest case of a manifold being built. The tokenizer splits “42” into “4” and “2”, so at depth 0 the state at the last digit knows the units (1.00) and nothing about the tens (0.00). One block later the tens digit is identifiable 44% of the time, after two blocks 92%, and a held-out number's value can be read linearly with R² 0.993. From depth 3 the hundred numbers lie in order along one direction. Underneath the line there is periodic structure with periods 10, 2 and 5 — the same periods reported for larger models — though a period of 10 is partly unavoidable here because the final digit token repeats every ten numbers. Tens-digit identity dips through the middle of the network (to 0.77) and returns to 0.99 at depth 20.

What is not there. Seven temperature words from “freezing” to “scorching” show no ordered layout at any depth (best score 0.393 against 0.331 for shuffles; held-out R² below zero). Size words are marginal, life stages inconclusive with six items. Letters are ordered through the middle of the network (R² up to 0.85 at depth 9) and lose it late. With seven items these tests are weak, and different carrier sentences might do better; but we looked, and a line is not what we saw.

Concepts as directions — and whether they travel.

Fifteen families of twelve words each (animals, metals, emotions, instruments …), each word placed in the same neutral sentence. Family membership is linearly readable at every block (0.861–0.989 leave-one-out accuracy, chance 0.067) — unsurprising, since the word itself is in view. The view below is fitted on half of each family and shows only the other half.

The harder question is whether such a direction means anything outside the sentence it was made in. We took each family's direction and applied it, untouched, to 31 words the directions had never seen — surgeon, papaya, mandolin, sleet — inside ordinary sentences, asking which of the fifteen directions each word's state points along.

Steering, with a null

Reading a direction is one thing; using it is another. We built a sentiment direction from eight positive and eight negative sentences, added it to the residual stream of different, neutral prompts (“I thought the film was”), and measured the log-odds of positive over negative next words. Every dose was also run with twelve random directions of identical size.

At block 14 the sentiment direction moves the log-odds from -0.92 to 7.96 across the dose range; random directions of the same size stay between 1.65 and 1.96. The effect is present from block 6, and grows with depth (largest span at block 26) — later interventions have fewer blocks left to be corrected in.

The copying circuit.

Feed the model 48 random tokens twice. The first time through it can do nothing (13.60 nats per token); the second time it predicts almost perfectly (0.22). The attention patterns show how: a handful of heads look back to the token after the previous occurrence of the current one.

11 of 448 heads put more than half their attention on the induction target; they sit in blocks 3, 6, 16, 18, 19, 20, 21, 22, 24. The earliest is in block 3, directly above previous-token heads in blocks 0, 1, 2, 15 — the textbook two-step circuit — and most of the rest share the band of blocks that performs the look-up in Finding 2.

Silencing the top 16 raises loss on repeated text to 4.03 nats; the worst of twenty random sets of 16 reaches 1.87. On ordinary text the same 16 heads cost 0.047 nats per token, less than random heads do (0.091): they are specialists.

What we got wrong on the way.

What this does and does not show.

Methods. Logit lens, activation patching between matched prompts, attention-pattern scores on repeated random tokens, mean-ablation, difference-of-means directions, linear projection with held-out carriers, label-permutation tests. All are published, standard techniques; none is ours.

Limits. One model, one snapshot, one seed. Prompt sets are small (12–100 items) and hand-written; the ordering tests on six- and seven-item sets have little power. The logit lens is a read, not a mechanism — the placeholder finding is supported by the causal timing in Finding 2 but we have not intervened on the placeholder itself. Patching used one fact template for the block-window panels. Attention scores describe where heads look, not what they write; the ablation is the causal part. Nothing here has been replicated on another model family, and nothing here is a claim about safety.

Receipt

Every number in the text above is computed by the page builder from these capture files; the hashes identify the exact files this page was built from.