dabochen/spreadsheet-is-all-you-need: A nanoGPT pipeline packed in a spreadsheet
A nanoGPT pipeline packed in a spreadsheetThis is a project that I did to help ... understand how GPT works.
It is pretty fun to play with, especially when you are trying to figure out what exactly is going on inside a transformer.
This helped me to visualize the entire structure and the data flow.
All the mechanisms, calculations, matrices inside are fully interactive and configurable.
It contains all the transformer components including:
- embedding
- layer norm
- self attention
- projection
- MLP
- softmax
- logits
It is based on Andrej Karpathy's NanoGPT structure which includes roughly 85000 parameters.
It is clearly a very small size, but it is both complex enough for me to understand how it works, and also not too big to crash my computer. In contrast to chatgpt, this project is a character based prediction system, meaning that each token is a character, and to reduce the complexity, only letter A/B/C are being tokenized.
- Andrej Karpathy's youtube tutorial "Let's build GPT":
https://www.youtube.com/watch?v=kCc8FmEb1nY - Andrej Karpathy's NanoGPT project:
https://github.com/karpathy/nanoGPT - Brendan Bycroft's 3D visualization of transformers:
https://bbycroft.net/llm - 3Blue1Brown's LLM course:
https://youtu.be/eMlx5fFNoYc?si=k40zeuPdM_4cB88o
