Chonkie is a lightweight ingestion library for building fast, efficient, and robust RAG pipelines, with multiple chunking approaches like token, sentence-based, recursive, semantic similarity, and LLM-assisted chunking. It provides a Python API (including Pipeline workflows that fetch, chunk, refine, embed, and send to a vector database) and can also be run as a self-hosted REST API service with reusable pipeline configurations.
Project status
- The source appears actively maintained, with recent GitHub activity (last push on 2026-07-17) and multiple versioned updates in 2026 (v1.7.0 on 2026-07-07, v1.6.8 on 2026-06-01).
- The apparent updates cadence is steady, roughly every 2 to 5 weeks in mid-2026 (late May alpha, then stable 1.6.8 in early June, then 1.7.0 in early July).
AI summary generated
Recent updates
v1.7.0
Chonkie v1.7.0 adds LiteParse integration for broader document parsing and publicly exposes ComponentRegistry for easier extension. The release notes also emphasize a full documentation redesign, migrated to GitHub Pages, along with various dependency maintenance updates.
Featuresv1.6.8
v1.6.8 includes wrapper support for Gemini embedding dimension handling, a dependency update set, and a fix related to how non-ASCII text is serialized during JSON exporting. Code inspection also shows the library is now releasing as 1.6.8 (not the alpha build string) and adjusts JSON serialization to write UTF-8 characters directly rather than using escaped \uXXXX sequences.
Featuresv1.6.8-alpha.1
v1.6.8-alpha.1 adds support for specifying custom embedding output dimensions when using Gemini embeddings through the Catsu embedding adapters. It also extends CI to build and test PyEmscripten wheels (Pyodide compatibility). The release notes mention an idna dependency bump, but the actual diff shows additional direct dependency version changes.
Featuresv1.6.7
v1.6.7 adds MistralOCR support to extract text from images and PDFs, and extends the overlap refinery with a new overlap strategy named "justified". It also introduces a new Pipeline stage concept (Vision) so OCR can run between fetch and chef steps, and tweaks CodeChunker language downloading behavior.
Featuresv1.6.6
v1.6.6 focuses on a major CodeChunker update, with release notes claiming large latency and indexing-time improvements. The notes also include several dependency bumps (urllib3, langchain-core, mako, authlib) and a turbopuffer requirement relaxation. However, the code diff shows additional behavioral and packaging changes that are not fully described in the release notes.
v1.6.5
v1.6.5 adds support for agentic skills/plugins via skills.sh, including README and docs updates to help developers install and use the official skills. It also introduces CI workflows for lazy import checking and TruffleHog secret scanning, and it switches the tokenizer implementation to use tokie as the primary backend.
Featuresv1.6.4
This release is primarily a fix for users installing chonkie without the optional table dependencies. The library now avoids importing pandas at module import time and instead loads it lazily when table conversion helpers are used.
v1.6.3
Chonkie v1.6.3 adds a new LanceDB handshake and improves metadata traceability (including adding a filename key) along with markdown and table chunking support. Internally, it also refactors delimiter-based splitting into a shared implementation and introduces chunk metadata propagation into downstream chunk/handshake behavior.
Featuresv1.6.2
v1.6.2 adds a new TeraflopAIChunker, plus a fix to per-chunk overlap calculations when context_size is a float. It also improves CodeChunker by validating that the provided tree-sitter language is supported, and then bumps several dependencies.
Featuresv1.6.1
Chonkie v1.6.1 is a patch release centered on dependency updates and import-related cleanup. The code diff also includes several small metadata handling and typing-related adjustments across multiple vector backends, plus a dependency list restructure to include httpx.