tiktoken is a fast BPE tokeniser library for use with OpenAI models, with a Python API to encode text into tokens and decode tokens back to text. It is useful for counting or converting between model tokens and raw text, and it also provides an extension mechanism for adding custom tokeniser encodings.
Project status
- Active maintenance: The repo shows recent activity, with the latest listed update 0.13.0 on 2026-05-15 and an upstream push on 2026-05-24, so it does not appear to be dormant.
- Update cadence: Update intervals appear to be on the order of several months (for example, 0.12.0 on 2025-10-06 to 0.13.0 on 2026-05-15, roughly 7 months), and the most recent changes are within about 2 months of today (2026-07-19).
AI summary generated
Recent updates
0.13.0
Release 0.13.0 focuses on performance improvements in the byte pair encoding (BPE) path and upgrades Rust/Python dependencies. The repo also adjusts Python-side behavior around very large inputs and updates how blobfile data is loaded.
0.12.0
Release 0.12.0 updates the Python wheels and Rust/PyO3 integration to support free-threaded Python and adds new model identifier support. The diff shows multiple Python binding and file-loading behavior changes alongside dependency bumps.
Features0.11.0
Version 0.11.0 updates tiktoken to support newer OpenAI model families (including GPT-5) and refreshes the Rust/Python build stack (Rust edition update and pyo3 bump). The core tokenization logic and error handling were also adjusted, including a fix related to how special tokens are handled when producing NumPy buffers.
BreakingFeatures0.9.0
Release 0.9.0 adds support for o1 and o3 model prefixes, a new Encoding.encode_to_numpy API, and improved error messages when loading invalid BPE vocabulary files. The changes also include delayed or conditional imports and some Rust/PyO3 refactoring to structure Python bindings behind a feature flag.
Features0.8.0
No publisher-provided release notes were included for version 0.8.0, so the documented changes are effectively missing. The code diff shows a mix of packaging updates (new wheel build matrix, dependency bumps), model/encoding support expansion (o1- and chatgpt-4o- prefixes), and internal Rust changes around encoding/decoding and input handling.
0.7.0
Release 0.7.0 adds support for the gpt-4o model by introducing a new o200k_base encoding and mapping it in the Python model registry. The repo also reports performance improvements, alongside build pipeline updates for wheel generation.
Features0.6.0
Release 0.6.0 focuses on improving tokenization performance and robustness when downloading model files, with additional support for pickling Encoding objects. It also extends model name mappings to support more OpenAI model aliases and improves regex efficiency used by the tokenizers.
Features0.5.2
Version 0.5.2 updates the project packaging/build setup to include Python 3.12 wheels and bumps PyO3 to 0.20. Runtime behavior is adjusted around the default cache directory so that permission errors do not fail tokenization. The published release notes field is empty, but CHANGELOG.md contains a few documented items that largely match the code changes.
Features0.5.1
Release 0.5.1 bumps the crate and Python package versions and introduces a new API helper to map a model name to its underlying encoding name string. The change also refactors the internal model-to-encoding mapping variables and makes a small adjustment in the OpenAI extension’s GPT-2 special token configuration.
Features0.5.0
Release 0.5.0 adds an educational BPE implementation module, expands model to encoding mappings, and introduces a new Encoding.decode_with_offsets API. The actual diff also includes Rust dependency bumps (including pyo3) and changes to plugin error handling in the encoding registry. No publisher release notes were provided for 0.5.0, so the documented changes are effectively missing and developers must rely on the code diff.
BreakingFeatures