Two tiny transformers

July 17, 2026 · ml, transformers

I spent the last week working through Karpathy’s “Let’s build GPT” and typing out a character-level transformer from scratch: embeddings, positional encodings, the attention blocks, the training loop. Nothing about it is novel — that’s the point. There’s a real difference between having watched someone explain multi-head attention and having debugged your own.

The model is tiny — 128-dimensional embeddings, a context of 32 characters — and it trains on the ~1MB tiny Shakespeare dataset. At initialization it produces noise:

l me son?dEiP ,ydh,cGPkaF ?SV
kkdQOi'

After training, this:

ISABELLA:
Let, and wilts; let see of you'l come thou,
Which which'd computs fallain, unly mrobe noisur

It’s nonsense, but it’s Shakespeare-shaped nonsense: character names in caps with a colon, line breaks in the right places, words that feel almost Elizabethan. Everything from “what a letter is” upward was learned from scratch. For calibration, the notebook also has a bigram baseline — seeing how much better the transformer gets from the same data is most of the lesson.

Then I pointed the same architecture at a different problem: arithmetic on 6-digit numbers, one character at a time. Instead of prose the training data is generated equations like 483920+7=, and the model has to complete the answer. One trick that matters: the answer is written back to front, so the model produces the least significant digit first — the same order you’d do it on paper, which means the carry it needs is always already in context. I like this task because, unlike Shakespeare, there is an exact right answer — the model can’t hide behind style. It has to learn something like an algorithm, carries and all, purely from next-character prediction.

Notebooks: nanogpt.ipynb and addition_lm.ipynb.