What are Encoder and Decoder?
Two transformer roles: the encoder reads and understands, the decoder generates output.
Reading vs writing
A transformer has two kinds of parts. An encoder reads input and builds an understanding of it. A decoder uses that understanding to write output, one token at a time. Let's watch both in action.
Run it step by step
Type a short sentence, then press Step. Watch the encoder read each word, build an understanding, then the decoder generate the output one word at a time.
It reads and understands
An encoder reads input text and creates a useful representation of it, a compact understanding the model can work with.
It generates output
A decoder generates output step by step. It uses the tokens it has written so far to predict the next token.
- • Encoder means read and understand.
- • Decoder means generate output.
- • Some models use only encoders.
- • Some models use only decoders.
- • Some models use both encoder and decoder.
An encoder helps a model understand input. A decoder helps a model generate output. Different transformer models use these parts in different ways.