Back to Home
Inside an LLM
Tokens
Tokens are the small pieces AI breaks text into before reading it. Every word, space, and punctuation costs tokens, and that's what determines your ChatGPT limits and API costs.
This is a simple learning tokenizer, not the exact tokenizer used by real AI models.
Interactive Playground
💡 Try these →
Live Visualization
0
tokens
Type something above to see tokens appear ✨
Statistics
0
Characters
0
Words
0
Tokens
Words
Mode
How It Works
📝
Text
→
⚙️
Tokenize
→
🏷️
Tokens
→
🔢
Count
🎓
Beginner Tips
4 tips
Tokens ≠ Words. A single word can become 2–3 tokens. Try "unbelievable" in Subwords mode to see it split up.
Spaces matter. Switch to Characters mode and notice that spaces become their own tokens, AI counts every character.
Fewer tokens = cheaper. AI APIs charge per token. Writing concise prompts saves money in production.
Try emoji or numbers. Type "GPT-4 🚀 costs $0.03/1K" to see how special characters tokenize differently.
💡
Key Takeaway
Before AI can compare text, it first breaks text into smaller pieces called tokens.