Thinking Step (Feed Forward Network)
After attention connects the words, each token gets a quiet thinking step that refines what it means.
Attention connects, thinking refines
Attention helps words find each other. But after that, every token still needs to be tidied up. The Feed Forward Network takes each token, one at a time, and improves its internal meaning before the next step.
Pick a token and watch it get refined
Sentence: "The cat chased the mouse." Choose a word to follow it through the thinking step.
A per-token thinking step
The Feed Forward Network is a processing step inside a transformer layer. It works on each token on its own and improves that token's internal meaning.
Attention decides which words matter to each other. The thinking step then takes each token and makes it more useful for the next layer.
Before vs after the thinking step
Token has a basic meaning.
= small animal
Token meaning becomes more useful for the next layer.
= thing being chased
- • Attention finds relationships between words.
- • The Feed Forward Network processes each token after attention.
- • It improves the token's internal representation.
- • It prepares the token for the next transformer step.
- • It is like a thinking / refining step inside the model.