Neural Networks: Zero to Hero
I worked through Andrej Karpathy's Neural Networks: Zero to Hero course, building neural networks and language models from scratch in code rather than through high-level libraries.
What I did
- Implemented backpropagation from first principles, starting with a scalar autograd engine.
- Built language models of increasing capability from scratch, working up from bigram models and multi-layer perceptrons to a transformer-based GPT.
- Worked in Python and PyTorch throughout, implementing each component myself before comparing against library equivalents.
What I took away
- A ground-up understanding of how modern language models are trained and how they work internally.
- A stronger technical foundation for following AI safety research and debate.
The code is on GitHub.