Stable Diffusion From Scratch

A complete implementation of Stable Diffusion including VAE, Conditional/Unconditional Diffusion, ControlNet, and Latent Diffusion architectures.

PyTorchStable Diffusion

What I Built

This is a complete from-scratch implementation of Stable Diffusion, one of the most popular text-to-image generation models. The project covers the entire pipeline from data processing to model training and inference.

Text Prompt → CLIP Encoder → Latent Space → UNet (Diffusion) → VAE Decoder → Generated Image

Core Components Built From Scratch

VAE (Variational Autoencoder)

Compresses images to latent space and reconstructs them with high fidelity

Conditional Diffusion

Generates images conditioned on text prompts or class labels

Unconditional Diffusion

Pure image generation without any conditioning

ControlNet

Adds spatial conditioning using edges, poses, or segmentation maps

Latent Diffusion

Diffusion process in compressed latent space for efficiency

Model Specifications

Latent Space Dimension: 4x64x64

UNet Channels: 320, 640, 1280, 1280

Attention Heads: 8

Text Encoder: CLIP (77 tokens)

Training Steps: 500k+

Batch Size: 32

Learning Rate: 1e-4

Noise Scheduler: DDPM / DDIM

Tech Stack

PyTorchDiffusersTransformersOpenCVNumPyMatplotlibHugging Face Hub

Training Details

DatasetLAION-5B / Custom Dataset

Image Resolution512x512

Training HardwareNVIDIA A100 (60GB)

Key Features

→Text-to-Image Generation

→Image-to-Image Translation

→Inpainting & Outpainting

→ControlNet Support (Canny, Depth, Pose, Scribble)

→Custom Checkpoint Loading

→CLIP Guidance & Prompt Weighting

→Negative Prompt Support

→Multi-GPU Training