FTS5 ICU Tokenizer for SQLite

Showcase
,
1

FTS5 ICU Tokenizer for SQLite:

This project, rewritten in Zig from C, tokenizes text input before indexing it with SQLite’s FTS5. This enables full-text search on non-space-tokenized languages like Thai.

Supported Zig versions

Zig 0.16.0

AI / LLM usage disclosure

I’ve rewritten much of my previous C code to Zig using various AI models and iterations. The reason for this rewrite is detailed in the README.md but in short, converting UTF-8 to UTF-16 for ICU and then convert it back to UTF-8 for SQLite in C is quite cumbersome.

1 Like