Andrej Karpathy released MicroGPT this week. 200 lines of Python. Zero dependencies. A complete GPT-2 implementation that you can read in one sitting.
The code includes dataset loading, a character-level tokenizer, an autograd engine, the full GPT-2 transformer architecture, an Adam optimizer, a training loop, and inference. One file. One afternoon to understand it.
Karpathy's quote: "This script is the culmination of multiple projects (micrograd, makemore, nanoGPT) and a decade-long obsession to simplify LLMs to their bare essentials."
The Numbers
This isn't a toy. The architecture mirrors GPT-2. Train it on Shakespeare and it generates Shakespeare. Train it on code and it generates code. The fundamentals are the same ones running billion-parameter models.
What's Inside
Six components, each doing one thing:
1 Dataset Loading
Reads a text file. Converts characters to integers. Splits into train/val. That's it. No HuggingFace, no data loaders, no preprocessing pipelines. Raw text in, integers out.
2 Character Tokenizer
Maps characters to integers and back. No BPE, no SentencePiece, no subword complexity. Character-level keeps the focus on architecture. You can swap in a real tokenizer later.
3 Autograd Engine
Tracks operations, builds computation graphs, computes gradients via backpropagation. The same engine from micrograd, now embedded directly. When you understand this, you understand how neural networks learn.
4 GPT-2 Architecture
Token embeddings, position embeddings, transformer blocks (attention + MLP), layer norms, final linear projection. The same structure as GPT-2. Scaled down, but structurally identical.
5 Adam Optimizer
Updates weights using momentum and adaptive learning rates. The same optimizer behind every modern language model. Implemented from scratch — no torch.optim.
6 Training Loop
Sample batches, forward pass, compute loss, backward pass, update weights. Print loss every N steps. The loop that trains every neural network.
Why This Matters for Builders
Build on top of LLMs? Understanding the internals changes how you think about them.
Context windows aren't magic
Position embeddings define context length. The model learns positional patterns during training. Extend the window? You need new position embeddings and fine-tuning. MicroGPT makes this obvious — block_size controls everything.
Attention is matrix multiplication
Queries, keys, values. Dot products. Softmax. That's attention. The "magic" is learned weights that decide what to attend to. When you see the math, you understand why certain prompts work better than others.
Temperature is softmax scaling
Higher temperature = flatter probability distribution = more randomness. Lower temperature = sharper distribution = more deterministic. One parameter, one division. The mystique disappears when you read the code.
Training is loss minimization
Cross-entropy loss measures how wrong the predictions are. Backprop computes how to adjust weights. Adam applies the adjustments with momentum. Repeat billions of times. That's training.
The Education Stack
MicroGPT sits at the top of Karpathy's education stack:
- micrograd — Autograd engine in 100 lines. Understand backpropagation.
- makemore — Character-level language models. Understand generation.
- nanoGPT — GPT-2 in PyTorch. Understand transformers at scale.
- MicroGPT — Everything combined, no dependencies. Understand the whole system.
Each project strips away one layer of abstraction. MicroGPT removes them all.
What You Can Do With It
Clone it. Run it. Break it. Modify it.
- Trace a forward pass — Add print statements. Watch activations flow through layers.
- Visualize attention — Export attention weights. See what the model attends to.
- Train on your data — Replace Shakespeare with code, emails, or product descriptions.
- Experiment with architecture — Change layer count, head count, embedding size. See what breaks.
- Add features — Implement dropout. Add weight decay. Build your own BPE tokenizer.
# Clone and run
git clone https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95
python microgpt.py
The Bigger Picture
We're building an AI system that runs 24/7. It scrapes leads, writes content, monitors analytics, sends outreach. The models are black boxes — until they aren't.
Understanding how transformers work changes how you prompt them. You think in terms of attention patterns, token distributions, position dependencies. You debug systematically instead of guessing.
MicroGPT won't help you fine-tune GPT-4. But it'll make you a better builder. The fundamentals compound.
Time investment: 2-3 hours to read the code. Another 2-3 to trace execution. A lifetime of better intuition about how these systems work.
Links
- MicroGPT Gist — The 200 lines
- Karpathy's Blog Post — Full writeup
- micrograd — The autograd engine
- nanoGPT — The PyTorch version
Read the code. Run the code. The best way to understand LLMs is to build one from scratch — and Karpathy just gave you the blueprint.