Chunk is a fast text chunking library that splits input text at semantic boundaries like periods and newlines, with configurable chunk sizes and delimiters. It provides a Rust `chunk` API and includes related basic operations used by the chonkie core, with additional Python and JavaScript/Wasm packages mentioned in the project.
Project status
- The
feyninc/chunkGitHub source appears actively maintained, with recentupdatesclose to today (last push 2026-05-28) and continued versioned changes through v0.10.2. - Apparent update cadence: v0.10.0 on 2026-03-29, v0.10.1 the next day (2026-03-30), then v0.10.2 on 2026-05-28, suggesting bursts of development followed by a slower interval.
AI summary generated
Recent updates
v0.10.2
Release v0.10.2 publishes PyEmscripten (Pyodide) wheels alongside the regular Python wheels, and includes CI-related workflow updates. The code changes also introduce feature-gated NumPy functionality and modify the published Python package metadata for Python version and NumPy dependency handling.
Featuresv0.10.1
v0.10.1 exposes multi-byte pattern splitting in both Python and JavaScript via a new `patterns` option (composable with `delimiters`), aligning binding-layer behavior with the underlying v0.10.0 functionality. It also fixes a JavaScript wrapper issue where `consecutive` and `forwardFallback` were not passed through in the non-`pattern` code path.
Featuresv0.10.0
v0.10.0 introduces multi-byte pattern based chunk splitting, adding a new `.patterns(&[&str])` API on both `Chunker` and `OwnedChunker`. When patterns are set, the implementation uses a hybrid strategy (SIMD memmem for small pattern sets, Aho-Corasick for larger sets) and supports composing patterns with `.delimiters()` for mixed ASCII and multi-byte punctuation.
Featuresv0.9.3
v0.9.3 adds Node.js specific WASM initialization by detecting the runtime and switching from the existing fetch based initialization to filesystem based initialization. It also adjusts the initSync invocation to use the new wasm-bindgen object parameter form.
Featuresv0.9.2
Release v0.9.2 adds a new `merge_splits` function to the WASM bindings. The implementation lives in `packages/wasm/src/lib.rs` and exposes a new wasm-exported API that merges segments based on token counts and a token budget.
Featuresv0.9.1
Release v0.9.1 primarily addresses npm publishing so that the generated WASM artifacts are included in the published package output. The code diff also includes version bumps across the Rust crates, Python package metadata, and the WASM package.json, plus a new benchmark example file.
v0.9.0
v0.9.0 introduces a new Savitzky-Golay based signal processing module for semantic chunking, and adds NumPy array support in the Python bindings for improved performance. The code changes also include new multi-pattern splitting utilities. However, several public API and behavior changes appear that are not described in the release notes.
Featuresv0.8.0
v0.8.0 introduces token-aware merging of split segments via a new merge_splits API, intended for RecursiveChunker to merge smaller segments into token-budgeted chunks. The release adds bindings for Rust, Python, and JavaScript, including support for an optional whitespace token join adjustment.
Featuresv0.7.0
Release v0.7.0 release notes describe a new delimiter-based splitting feature plus added split_offsets APIs for Python and JavaScript, along with a Rust crate refactor. However, the provided code diff shows only version number updates across Rust, Python, and WASM packaging, plus an update to the version bump script, with no actual implementation changes included in the diff.
Featuresv0.6.0
v0.6.0 introduces delimiter-based splitting via the new Rust API `split_at_delimiters`, plus corresponding JavaScript and Python wrappers. The Rust crate was refactored into separate modules (`chunk.rs`, `split.rs`, `delim.rs`) and the delimiter logic was optimized (single-pass `min_chars` merging and allocation pre-sizing).
Features