Large Language Model From Scratch
This project showcases a custom-built Large Language Model (LLM) developed completely from scratch using a Transformer-based architecture. It covers the full pipeline including tokenization, model design, training, and text generation.
The model is trained on a large-scale dataset of 471 million tokens and contains approximately 49 million parameters, making it lightweight yet capable of generating meaningful text outputs.
This project is mainly focused on learning and experimentation, providing a deep understanding of how modern language models work internally.
Github Link: [ View ]
Custom LLM Built from Scratch
Overview
A lightweight Large Language Model (LLM) built completely from scratch, trained on 10M tokens with a 50M parameter architecture.
- Dataset Size: 471 Million Tokens
- Model Size: 49 Million Parameters
- Architecture: Transformer
- Purpose: Learning + Experimentation
Tech Stack
- Python
- PyTorch
- tiktoken
- NumPy, Pandas
- Matplotlib
Model Details
- Transformer Decoder Architecture
- Multi-Head Self Attention
- Positional Encoding
- Feed Forward Layers
- Residual Connections & LayerNorm
Features
- Built completely from scratch
- Custom tokenizer pipeline
- Efficient training loop
- Text generation support
- Lightweight and fast
Training Process
- Tokens: 471M
- Parameters: 49M
- Optimizer: Adam / AdamW
- Loss Function: CrossEntropyLoss
- Task: Next Token Prediction
Example Output
Input: "A little girl went to the woods"
Output: A little girl went to the woods and saw the famous animals playing racing... (truncated)