js-tokens is a tiny JavaScript tokenizer that uses regular expressions to split a JavaScript source string into an iterable stream of token objects. It is useful for tooling or analysis that needs lexer-level tokenization, including handling invalid JavaScript without throwing (it yields an “Invalid” token when it cannot classify a character).
Project status
- Actively maintained: The upstream last push is 2025-12-08, and there is a recent version bump to v10.0.0 (2025-12-08), indicating ongoing maintenance rather than abandonment.
- Update cadence: Updates appear to be on the order of months to about a year between tags (for example, v9.0.1 in 2024-11 to v10.0.0 in 2025-12).
AI summary generated Today
Recent updates
v10.0.0
5 months agoRelease v10.0.0 primarily migrates the package from CommonJS to ESM. The provided publisher release notes are empty, so most user-visible changes are only visible by inspecting the code diff and repository documentation files.
v9.0.1
11/22/2024v9.0.1 makes a targeted fix to the regular expression literal tokenizer (notably around character class edge cases like `/a[/]/`). The code diff also includes a larger repository/tooling shift to ESM (`package.json` `type` change), and the test stack is migrated from Jest to Vitest with updated CI Node versions.
Breakingv9.0.0
2/8/2024Release v9.0.0 was published, but no release notes were provided by the publisher. As a result, there is insufficient information to determine what changed, whether there are breaking changes, or what developers should migrate.