Transformer-Bert-PyTorch

This repo contains my minimalist PyTorch implementation of Transformer and BERT.

Transformer Anatomy

In my transformer anatomy note, I give a detailed description on each component of the transformer with clean math formula, including

• Overall architecture
• Input output conventions
• Multi-head attention with marks
• Encoder anatomy
• Decoder anatomy

References

https://github.com/codertimo/BERT-pytorch
https://nlp.seas.harvard.edu/2018/04/03/attention.html
https://github.com/huggingface/transformers