Tokie is a Rust tokenizer library with Python bindings, designed to load HuggingFace tokenizers and tokenize text quickly. It supports major tokenizer algorithms (BPE, WordPiece, SentencePiece, and Unigram) and provides fast encode, decode, counting, batch encoding, and save/load for tokenizers.
Project status
- Actively maintained: The GitHub source has recent updates (latest push on 2026-05-28), and there was a short, same-day sequence of version increments (0.0.10-alpha.1 → 0.0.10-alpha.2 → v0.0.10), suggesting ongoing work rather than a long-abandoned project.
- Update cadence: Updates appear to have clustered on 2026-05-28, with no evidence in the provided log of further updates after that date (from 2026-05-28 to 2026-07-19, about 7 weeks).
AI summary generated
Recent updates
v0.0.10
Release v0.0.10 is presented as adding official Pyodide (PyEmscripten) support for Python 3.14 only, plus related packaging and publishing workflow changes. However, the provided code diff between 0.0.10-alpha.2 and v0.0.10 shows only version metadata updates, with no implementation changes visible.
Features0.0.10-alpha.2
Release 0.0.10-alpha.2 is a version bump from 0.0.10-alpha.1. The release notes only mention a chore-level change to update the version.
v0.0.10-alpha.1
This alpha release adds Pyodide compatibility, enabling the library to run in browser-based Python (WebAssembly) environments. It also includes a build or publishing workflow update, but no user-facing API or behavior changes are documented.
Featuresv0.0.9
Tokie v0.0.9 introduces a major internal refactor in the `pretokie` crate, replacing 7 duplicated pretokenizer iterators with a single trait-based, generic `Core` implementation parameterized by compile-time config types. The release also updates CI packaging to include Python 3.10 builds.
Featuresv0.0.8
Release v0.0.8 focuses on aligning WordPiece behavior with HuggingFace by enforcing the `max_input_chars_per_word` limit, including reading it from `tokenizer.json` and maintaining backward compatibility for older `.tkz` files. It also introduces a major pretokenizer refactor (new `pretokie` crate with hand-coded, zero-allocation iterators) to improve tokenization accuracy across multiple model families.
BreakingFeaturesv0.0.7
Release v0.0.7 focuses on improving tokenizer accuracy for several Unigram models by fixing Viterbi scoring when the model lacks `<0xXX>` byte fallback tokens. The release notes describe a targeted change to penalize `<unk>` during Viterbi decoding for those models.
v0.0.6
v0.0.6 claims perfect tokenizer accuracy (22/22 models) against HuggingFace tokenizers by addressing remaining mismatches in SentencePiece BPE, added token handling, Unigram <unk> behavior, and SentencePiece normalizer handling of format/control characters. It also adds a `bench_vs_hf` example to compare throughput versus HF tokenizers and tiktoken-rs, stating no speed regression.
Featuresv0.0.5
This release adds token byte offset tracking via new `encode_with_offsets()` APIs (Rust + Python), plus an HF-compatible `num_special_tokens_to_add()` helper. It also refactors the decoding architecture by splitting vocab byte lookup into `VocabDecoder` and introducing `DecoderType`, and removes the unused `SentencePieceBPEv2` encoder. The README and Python docs are updated with benchmarks and examples for the new functionality.
BreakingFeaturesv0.0.4
Release v0.0.4 introduces HuggingFace-style vocabulary access APIs, HuggingFace-compatible padding and truncation controls, and changes Python encoding outputs to return a structured Encoding object. It also adds a new batch pair encoding API and includes performance-focused changes (cached CPU counting, zero-copy vocabulary lookups).
BreakingFeaturesv0.0.3
v0.0.3 introduces Python bindings for tokie via PyO3 and maturin, with a simple Python API example for Tokenizer usage. The release also restructures the workspace and changes the project license to a more permissive option.
BreakingFeatures