Attention Mechanism and Transformers

Overview

You will learn the basics of Attention Mechanism and Transformers, implementing a Vision Transformer

Goals

• Learn about Attention Mechanism and Transformers

• Why Attention and not Convolutions?

• Patch Embeddings (Conversion of Images to “sequences”)

• Understanding CLS tokens and Positional Embeddings

• How to build a single Attention Head (Q, K, V)

• Expanding Single Headed Attention to MultiHeaded Attention

• The purpose of Layer Normalization

Lecture Materials

• Slides

• Attention

• Transformers

• Vision Transformer

Homework Assignment

• None

Supplemental Readings

• Attention Is All You Need

• Neural Machine Translation By Jointly Learning To Align and Translate

• Transformer: A Novel Neural Network Architecture for Language Understanding (Jakob Uszkoreit, 2017) - The original Google blog post about the Transformer paper, focusing on the application in machine translation.

• The Illustrated Transformer (Jay Alammar, 2018)

• Attention? Attention! (Lilian Weng, 2018)

• Illustrated: Self-Attention (Raimi Karim, 2019)

• The Transformer family (Lilian Weng, 2020)