💻nanoGPT
The simplest, fastest repository for training/finetuning medium-sized GPTs.
⭐ Hugely popular: 62k stars, gaining about 223 a week
View on GitHub ↗repo profile
momentum
durability
since we covered it
why it's a big deal
- Simplicity and Clarity: The codebase is concise, with approximately 300 lines each for the training loop and model definition, facilitating easy understanding and customization.
- Performance Optimized: Capable of reproducing GPT-2 (124M) on OpenWebText in about 4 days using a single 8xA100 40GB node.
- Flexible Training: Supports training from scratch or fine-tuning existing models, accommodating various project needs.
under the hood
- Built with PyTorch: Leverages PyTorch for model implementation, ensuring compatibility and leveraging its robust ecosystem.
- Configurable Architecture: Allows users to adjust model parameters such as layers, heads, and embedding sizes to suit specific requirements.
- Data Preparation Tools: Includes scripts for preparing datasets like OpenWebText and Tiny Shakespeare, streamlining the training process.
our take from PR#11, 2025-06-25
star history
- PR#11 42k 2025-06-25
- now 62k + 20k since first covered
curve is sampled from GitHub's star history, plus our own daily readings since we covered it; the dashed stretch is before we first covered it, the solid line since. figures at coverage are the numbers we printed then (approx.), current count is live.
understory
Better known than its recent output, coasting a little on attention.
- output, commits & releases
- clout, star velocity
output = commits & releases; clout = star velocity, both 0 to 100 monthly indices; the gap where output runs above clout is the understory. The understory →
covered in
-
A Minimalist Framework for Training and Fine-Tuning GPT Models
similar projects
compare these →- 🖥️ Open WebUI
2.4× the stars
User-friendly AI Interface (Supports Ollama, OpenAI API,...)
149k ACTIVE - 🧪 autoresearch
AI agents running research on single-GPU nanochat training automatically
94k ACTIVE - 🦙 LLaMA-Factory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
74k ACTIVE
comments
Sign in with GitHub to add your blip on nanoGPT.