Tokie is a Rust tokenizer library with Python bindings, designed to load HuggingFace tokenizers and tokenize text quickly. It supports major tokenizer algorithms (BPE, WordPiece, SentencePiece, and Unigram) and provides fast encode, decode, counting, batch encoding, and save/load for tokenizers.
Project status
- Active maintenance: The repo shows very recent activity (last upstream push on 2026-07-26) and multiple back-to-back updates in late July 2026 (v0.1.1 to v0.1.4), indicating the project is currently actively worked on.
- Update cadence: From the latest sequence, updates landed within about a day across v0.1.2 to v0.1.4 (all 2026-07-23 to 2026-07-24), and with a recent upstream push within the last week relative to 2026-07-30, suggesting an active development period rather than a long pause.
AI summary generated
Recent updates
v0.1.4
tokie 0.1.4 focuses on speeding up SentencePiece-Unigram (e.g., T5, XLM-RoBERTa, ALBERT) by replacing whole-string Viterbi with metaspace-delimited, per-unit Viterbi, plus memoization of Zipf-frequent metaspace units. The release notes claim byte-for-byte identical output to the previous encoder and HuggingFace, with a correctness guard to fall back to whole-string Viterbi when vocabularies contain interior metaspace tokens.
v0.1.3
tokie 0.1.3 introduces a new Rust-backed bulk file encoding API, `encode_files` and `count_tokens_files`, intended for tokenizing large corpora efficiently. It also rewrites major parts of the hot encoding path for speed, including cache-first processing, a rank-table based pair-merge core, and bulk pretokenization draining, while maintaining token-exact parity with HuggingFace. Release notes also document a `.tkz` rank-table load fix and PyO3 and NumPy binding upgrades for security.
SecurityFeaturesv0.1.2
tokie v0.1.2 focuses on “exact added-token semantics” to match HuggingFace’s per-token flags (lstrip, rstrip, normalized, single_word) and correct SentencePiece metaspace prepend behavior. The release also regenerates the published tokiers tokenizers as self-contained .tkz v13 with added/special tokens embedded, and updates the compiled-cache key to be build-discriminated to prevent stale artifacts. Code changes implement a substantial internal token-matching and normalizer model to get parity across many listed models.
Featuresv0.1.1
tokie v0.1.1 introduces .tkz format v13, which embeds added tokens and special tokens directly in the binary file so loading can avoid fetching and parsing tokenizer.json. The code also updates the local compiled-cache behavior to remove the JSON sidecar, and fixes the speed regression workflow.
BreakingFeaturesv0.1.0
tokie v0.1.0 claims exact token-for-token parity with HuggingFace tokenizers on real web text, and reports substantial performance improvements for loading, single-string latency, and batch throughput. The release notes emphasize a new .tkz format (v12 with SentencePiece precompiled charsmap), updated tokenier hub artifacts, and companion pretokie 0.1.0. It also highlights new Python/Rust APIs for faster ids-only and flat-batch encodes, plus multiple low-level performance optimizations.
BreakingFeaturesv0.0.10
Release v0.0.10 is presented as adding official Pyodide (PyEmscripten) support for Python 3.14 only, plus related packaging and publishing workflow changes. However, the provided code diff between 0.0.10-alpha.2 and v0.0.10 shows only version metadata updates, with no implementation changes visible.
Features0.0.10-alpha.2
Release 0.0.10-alpha.2 is a version bump from 0.0.10-alpha.1. The release notes only mention a chore-level change to update the version.
v0.0.10-alpha.1
This alpha release adds Pyodide compatibility, enabling the library to run in browser-based Python (WebAssembly) environments. It also includes a build or publishing workflow update, but no user-facing API or behavior changes are documented.
Featuresv0.0.9
Tokie v0.0.9 introduces a major internal refactor in the `pretokie` crate, replacing 7 duplicated pretokenizer iterators with a single trait-based, generic `Core` implementation parameterized by compile-time config types. The release also updates CI packaging to include Python 3.10 builds.
Featuresv0.0.8
Release v0.0.8 focuses on aligning WordPiece behavior with HuggingFace by enforcing the `max_input_chars_per_word` limit, including reading it from `tokenizer.json` and maintaining backward compatibility for older `.tkz` files. It also introduces a major pretokenizer refactor (new `pretokie` crate with hand-coded, zero-allocation iterators) to improve tokenization accuracy across multiple model families.
BreakingFeaturesv0.0.7
Release v0.0.7 focuses on improving tokenizer accuracy for several Unigram models by fixing Viterbi scoring when the model lacks `<0xXX>` byte fallback tokens. The release notes describe a targeted change to penalize `<unk>` during Viterbi decoding for those models.
v0.0.6
v0.0.6 claims perfect tokenizer accuracy (22/22 models) against HuggingFace tokenizers by addressing remaining mismatches in SentencePiece BPE, added token handling, Unigram <unk> behavior, and SentencePiece normalizer handling of format/control characters. It also adds a `bench_vs_hf` example to compare throughput versus HF tokenizers and tiktoken-rs, stating no speed regression.
Featuresv0.0.5
This release adds token byte offset tracking via new `encode_with_offsets()` APIs (Rust + Python), plus an HF-compatible `num_special_tokens_to_add()` helper. It also refactors the decoding architecture by splitting vocab byte lookup into `VocabDecoder` and introducing `DecoderType`, and removes the unused `SentencePieceBPEv2` encoder. The README and Python docs are updated with benchmarks and examples for the new functionality.
BreakingFeaturesv0.0.4
Release v0.0.4 introduces HuggingFace-style vocabulary access APIs, HuggingFace-compatible padding and truncation controls, and changes Python encoding outputs to return a structured Encoding object. It also adds a new batch pair encoding API and includes performance-focused changes (cached CPU counting, zero-copy vocabulary lookups).
BreakingFeaturesv0.0.3
v0.0.3 introduces Python bindings for tokie via PyO3 and maturin, with a simple Python API example for Tokenizer usage. The release also restructures the workspace and changes the project license to a more permissive option.
BreakingFeatures