Word Order & Positional Encoding
The same words in a different order can mean something completely different. A transformer needs to know where each word sits, that's what positional encoding is for.
Order carries meaning
A transformer looks at all the tokens together, but on their own the tokens don't say who did what. "Dog bites man" and "Man bites dog" use the exact same words, yet they mean opposite things. The only difference is the order.
Swap the words, watch the meaning flip
The positions stay the same (1, 2, 3). Only the words move. Press the button and see how the meaning changes.
The words are the same, but the meaning changes because the order changes.
Adding a "where" to every token
Positional encoding is extra position information added to each token. It helps the transformer know which token came first, second, third, and so on.
With vs without position
Loose words. The order is unclear.
Ordered sentence. The meaning is clearer.
- • Word order changes meaning.
- • Tokens alone are not enough.
- • Transformers need position information.
- • Positional encoding adds order information.
- • This helps the model understand the sentence better.