dabochen/spreadsheet-is-all-you-need: A nanoGPT pipeline packed in a spreadsheet
A nanoGPT pipeline packed in a spreadsheetThis is a project to help ... understand how GPT works.
It is pretty fun to play with, especially when you are trying to figure out what exactly is going on inside a transformer.
This helped to visualize the entire structure and the data flow.
All the mechanisms, calculations, matrices inside are fully interactive and configurable.
It contains all the transformer components including:
- embedding
- layer norm
- self attention
- projection
- MLP
- softmax
- logits
It is based on Andrej Karpathy's NanoGPT structure which includes roughly 85000 parameters.
It is clearly a very small size, but it is both complex enough to understand how it works, and also not too big to crash computer. In contrast to chatgpt, this project is a character based prediction system, meaning that each token is a character, and to reduce the complexity, only letter A/B/C are being tokenized.
- Andrej Karpathy's youtube tutorial "Let's build GPT":
https://www.youtube.com/watch?v=kCc8FmEb1nY - Andrej Karpathy's NanoGPT project:
https://github.com/karpathy/nanoGPT - Brendan Bycroft's 3D visualization of transformers:
https://bbycroft.net/llm - 3Blue1Brown's LLM course:
https://youtu.be/eMlx5fFNoYc?si=k40zeuPdM_4cB88o
No comments:
Post a Comment