Pretraining
Pretraining is the first major learning stage. The model studies a large amount of text and learns general patterns in language, long before it is given any specific job.
Feed the data, build a base model
A model starts out knowing nothing. Press Start Pretraining and watch large text data flow into the model. As it reads, it slowly picks up patterns and becomes a base model.
Click a pattern to see an example
Pretraining is not about memorising facts. The model soaks up broad, general patterns. Tap each card below to see what that looks like in practice.
Two stages, two kinds of model
Pretraining creates the base model. Fine-tuning later makes it more useful for specific tasks.
- • Pretraining is the first major training stage.
- • The model learns from large amounts of data.
- • It learns general language and knowledge patterns.
- • Pretraining creates a base model.
- • Fine-tuning can later specialize the base model.
Pretraining gives an AI model its general foundation. It learns broad patterns from large data before being adapted for more specific tasks.