# FTS5 ICU Tokenizer for SQLite

**URL:** <https://ziggit.dev/t/fts5-icu-tokenizer-for-sqlite/17228>\
**Category:** Showcase\
**Tags:** llm, zig-0-16\
**Created:** [August 14, 2026, 3:16pm UTC](https://ziggit.dev/t/fts5-icu-tokenizer-for-sqlite/17228 "2026-08-14T15:16:45Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![cwt](https://ziggit.dev/user_avatar/ziggit.dev/cwt/32/9919_2.png) [@cwt](https://ziggit.dev/u/cwt)\
**Post date:** [August 14, 2026, 3:16pm UTC](https://ziggit.dev/t/fts5-icu-tokenizer-for-sqlite/17228/1 "2026-08-14T15:16:45Z")

</div>

# FTS5 ICU Tokenizer for SQLite:

This project, rewritten in Zig from C, tokenizes text input before indexing it with SQLite’s FTS5. This enables full-text search on non-space-tokenized languages like Thai.

> **[GitHub - cwt/fts5-icu-tokenizer: FTS5 ICU Tokenizer for SQLite (mirror)](https://github.com/cwt/fts5-icu-tokenizer)**
>
> FTS5 ICU Tokenizer for SQLite (mirror)

### Supported Zig versions

Zig 0.16.0

### AI / LLM usage disclosure

I’ve rewritten much of my previous C code to Zig using various AI models and iterations. The reason for this rewrite is detailed in the README.md but in short, converting UTF-8 to UTF-16 for ICU and then convert it back to UTF-8 for SQLite in C is quite cumbersome.