tiktoken
Fast BPE tokenizer: count tokens exactly for OpenAI models.
About
tiktoken is OpenAI's fast byte-pair-encoding tokenizer: the exact tokenization used by GPT-4o, o1, and the embedding models, exposed as a tiny Python/Rust library. It's the standard way to count tokens before an API call (cost estimation, context-window budgeting, chunking text for RAG) — 179M PyPI downloads/month. 19.4k stars, MIT. Note: encodings are OpenAI-model-specific; token counts don't transfer to Anthropic/Google/Meta models (use their tokenizers or approximations instead).
What it can do
- **Exact counts**: matches OpenAI's server-side tokenization
- **All encodings**: cl100k_base, o200k_base, and legacy model encodings
- **Fast**: Rust core, fast enough for large-scale preprocessing
- **No network**: fully offline tokenization
Setup
```bash
pip install tiktoken
```
```python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, world!")
print(len(tokens), "tokens")
```
No API key needed — pure local computation.
Prerequisites
Python 3.8+ (Rust core available separately). No API key or network needed.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: 19,443 GitHub stars (log-scaled vs catalog peers); blended with ~178,983,368 monthly package downloads; last push 57d ago; 2 releases/12mo; 143 open issues. Manual review: exact token counts for OpenAI models, 19.4k stars, 179M PyPI/mo; OpenAI-specific encodings don't transfer to other providers. License: MIT.
★ 19,443 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.6/10
- Maintenance · recency, releases, issue load
- 6.9/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 9.0/10
- · 19,443 GitHub stars (log-scaled vs catalog peers)
- · blended with ~178,983,368 monthly package downloads
- · last push 57d ago; 2 releases/12mo; 143 open issues
- · manual review: exact token counts for OpenAI models, 19.4k stars, 179M PyPI/mo; OpenAI-specific encodings don't transfer to other providers
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.