Residual Connections & Layer Normalization
Two quiet helpers that let transformers go deep: one keeps earlier information, the other keeps values stable.
Keep the good, stay steady
Each transformer step changes a token. But changing too much can lose useful details, and values can get shaky. Residual connections carry earlier information forward, and layer normalization keeps the numbers balanced so the next step works smoothly.
Send a token through, with and without residual
Watch the token "cat" pass through one step. Toggle the residual connection to see the difference.
A shortcut for earlier information
A residual connection takes the token's original information and adds it back after the step. So the model keeps what it already knew and gains the new processed result.
This helps the model avoid losing useful details as tokens pass through many steps.
Bring the values into balance
After the residual connection, some values can be very high and others very low. Layer normalization balances them so the next step can process them more easily.
The full path through one step
- • Residual connections carry earlier information forward.
- • They help the model avoid losing useful details.
- • Layer normalization keeps values balanced.
- • These steps make deep transformer layers more stable.
- • They help information move smoothly through the model.