Attention & Multi-Head Attention
Attention lets each word look at other words in the sentence to figure out what relates to what.
Words don't stand alone
To understand a sentence, the model has to connect words. When it sees the word "it", it needs to figure out what "it" points to. Attention is how the transformer decides which words matter for understanding another word.
Click a word to see what it looks at
Tap any word, then try the word "it". The lines show which words it pays attention to.
Every word can look around
Attention helps each word look at other words in the sentence. This helps the model understand relationships.
For example, "it" may need to look back at "cat" to understand who was hungry.
Looking in several ways at once
Multi-head attention means the model looks at the sentence in multiple ways at the same time. Each "head" focuses on a different kind of relationship.
- • Attention helps words connect to related words.
- • A word can look at many other words.
- • Stronger attention means a stronger relationship.
- • Multi-head attention looks at the sentence in different ways.
- • Attention helps the transformer understand context.