Inside an LLM

Tokens

Tokens are the small pieces AI breaks text into before reading it. Every word, space, and punctuation costs tokens, and that's what determines your ChatGPT limits and API costs.

This is a simple learning tokenizer, not the exact tokenizer used by real AI models.

Interactive Playground

💡 Try these →

Live Visualization

0 tokens

Type something above to see tokens appear ✨

💡

Key Takeaway

Before AI can compare text, it first breaks text into smaller pieces called tokens.

Previous: How an AI Model Works Next: Context Window