This page provides a high-level introduction to nanochat: its purpose as a minimal full-stack LLM training system, the single-complexity-dial design philosophy, and the Time-to-GPT-2 leaderboard concept that drives development. For detailed instructions on installation and running your first training, see Getting Started For in-depth documentation of specific subsystems, consult the relevant sections (e.g., Base Model Pretraining Model Architecture Data Pipeline).
Sources: README.md1-6
nanochat is a minimal experimental harness for training large language models from scratch. It is designed to be:
The repository enables you to train your own ChatGPT-like model and interact with it through either a command-line interface (scripts/chat_cli.py) or web interface (scripts/chat_web.py).
| Stage | Script | Output | Description |
|---|---|---|---|
| Tokenization | scripts/tok_train.py | tokenizer.pkl | Trains BPE tokenizer with 32,768 vocab using rustbpe nanochat/tokenizer.py42-61 |
| Base Pretraining | scripts/base_train.py | Base checkpoint | Trains transformer from scratch using scaling laws README.md6 |
| SFT | scripts/chat_sft.py | Chat checkpoint | Adapts base model for conversation using Task Mixture nanochat/tokenizer.py140-146 |
| RL (Optional) | scripts/chat_rl.py | Aligned checkpoint | Further aligns model using GRPO/SimPO |
| Evaluation | scripts/base_eval.py scripts/chat_eval.py | Metrics | Measures CORE score, val_bpb, task accuracy README.md12-24 |
| Inference | scripts/chat_cli.py scripts/chat_web.py | Interactive chat | Deploys model for user interaction using Engine nanochat/tokenizer.py140-146 |
Sources: README.md1-6 nanochat/tokenizer.py9-21 nanochat/tokenizer.py42-61
nanochat's defining design principle is the single complexity dial: the --depth argument (number of transformer layers) automatically determines all other hyperparameters to produce compute-optimal models README.md6-8 Users specify only the model size they want; the system calculates:
n_embd), number of attention heads (n_head)dmodel_lr_scale)This is implemented through scaling law formulas in scripts/base_train.py that map depth to optimal hyperparameters README.md6 The result is that sweeping --depth produces a miniseries of compute-optimal models at various scales README.md88-98
| Depth | Approximate Capability | Parameters | Training Time (8xH100) |
|---|---|---|---|
| 12 | GPT-1 scale | ~124M | ~5 minutes |
| 16 | Intermediate | ~220M | ~15 minutes |
| 20 | Approaching GPT-2 | ~343M | ~45 minutes |
| 24-26 | GPT-2 | ~475-600M | ~1.65 - 1.8 hours |
The philosophy eliminates the need for exhaustive hyperparameter tuning: any improvement to the codebase must work across all depths, ensuring principled changes rather than single-model overfitting README.md6 dev/LEADERBOARD.md51
Sources: README.md6 README.md88-98 dev/LEADERBOARD.md51
The primary development focus is the Time-to-GPT-2 leaderboard, which tracks wall-clock time to train a model that exceeds GPT-2's CORE score of 0.256525 on an 8xH100 node README.md10-24
| # | Time (hours) | val_bpb | CORE | Description | Date | Commit |
|---|---|---|---|---|---|---|
| 0 | 168.00 | - | 0.2565 | Original OpenAI GPT-2 (1.6B) | 2019 | - |
| 4 | 2.02 | 0.71854 | 0.2571 | NVIDIA ClimbMix dataset | Mar 4 2026 | 324e69c |
| 5 | 1.80 | 0.71808 | 0.2690 | Autoresearch round 1 | Mar 9 2026 | 6ed7d1d |
| 6 | 1.65 | 0.71800 | 0.2626 | Autoresearch round 2 | Mar 14 2026 | a825e63 |
total_training_time) excluding evaluation/logging README.md24 dev/LEADERBOARD.md49To participate, run runs/speedrun.sh (which implements the current SOTA), verify core_metric > 0.256525, and submit a PR with improved training time README.md10-26
Sources: README.md10-25 dev/LEADERBOARD.md5 dev/LEADERBOARD.md49
The following diagram shows the main entry points and how they orchestrate the core subsystems:
Entry Points and Scripts Architecture
Sources: README.md43-60 runs/speedrun.sh1-80 nanochat/tokenizer.py34-35
The complete pipeline from raw data to deployed chat interface follows these stages:
Complete Training Pipeline from Data to Deployment
Each stage is independent: base pretraining checkpoints can be evaluated directly, or used to warm-start SFT.
Sources: runs/speedrun.sh43-80 README.md20
The nanochat/ directory contains the implementation modules:
| Module | Primary Classes/Functions | Responsibility |
|---|---|---|
gpt.py | GPT, GPTConfig | Transformer architecture with Flash Attention 3 support. |
engine.py | Engine | Inference engine with KV cache and sampling strategies nanochat/tokenizer.py140-146 |
dataloader.py | make_dataloader | BOS-aligned packing and distributed data loading. |
tokenizer.py | RustBPETokenizer | BPE tokenizer using rustbpe for training and tiktoken for inference nanochat/tokenizer.py34-36 |
optim.py | MuonAdamW, DistMuonAdamW | Hybrid optimizer (Muon for matrices, AdamW for rest). |
checkpoint_manager.py | save_checkpoint, load_checkpoint | Checkpoint I/O with rank-aware saving dev/LEADERBOARD.md31 |
common.py | compute_init | DDP setup and hardware-specific precision selection. |
Sources: README.md1-60 nanochat/tokenizer.py34-61 dev/LEADERBOARD.md31
nanochat uses an explicit precision system rather than PyTorch's autocast. The global COMPUTE_DTYPE variable is auto-detected based on hardware:
| Hardware | Default COMPUTE_DTYPE | Rationale |
|---|---|---|
| CUDA SM 80+ (A100, H100) | bfloat16 | Native BF16 tensor cores |
| CUDA SM < 80 (V100, T4) | float32 | No BF16 support; requires manual override for FP16 |
| CPU / MPS | float32 | No reduced-precision tensor cores |
Optional FP8 training is available via the --fp8 flag on Hopper+ GPUs, using torchao for tensorwise scaling dev/LEADERBOARD.md24 dev/LEADERBOARD.md117
Sources: dev/LEADERBOARD.md24 dev/LEADERBOARD.md117
nanochat makes several architectural choices that distinguish it from typical LLM frameworks:
rustbpe for training the BPE vocabulary and tiktoken for high-performance inference nanochat/tokenizer.py34-36Muon for internal transformer matrices to accelerate training via orthogonalization dev/LEADERBOARD.md117<|bos|>, <|user_start|>, and <|python_start|> to support chat and tool-use capabilities nanochat/tokenizer.py9-21Refresh this wiki
