Character-Level Transformer
I built a small autoregressive Transformer from scratch in PyTorch while following Andrej Karpathy's educational GPT implementation. I trained it on Shakespeare's Coriolanus and published the checkpoint and model notes on Hugging Face.
This is an educational model, not a production language model. Its value is the hands-on implementation of causal self-attention, masking, residual connections, normalization, training, and evaluation.
Technologies used
Architecture
- 6 Transformer blocks
- 6 attention heads
- 384-dimensional embeddings
- 256-character context
- 4× feed-forward expansion
- ReLU activations, LayerNorm, residual connections, and dropout
Training result
- Training loss: 0.8848
- Validation loss: 1.5879
- Character-level next-token prediction
Stack
- Python and PyTorch
- Causal multi-head self-attention
- Language-model training and evaluation