[model2vec-zig]: static text embeddings compiled into a single binary
I was inspired by MinishLab’s Model2Vec work, and wanted to push it into the application layer. That led me to build model2vec-zig, with the goal of compiling their potion embedding models into a single static binary, which Zig is especially well-suited for. This is useful if you want semantic search/ranking inside a Zig CLI or daemon without Python, ONNX, a model server, or network calls.
The part I think is most interesting is the 4-bit quantization result. I read Google’s TurboQuant paper and applied it to the potion models: each embedding row gets rotated by a fixed random orthonormal matrix, then stored as 4-bit values with one scale per row. The rotation is never stored anywhere. Since the rotation itself preserves cosine similarity, the runtime never needs to know it happened. This takes the 129 MB retrieval-tuned potion model down to 16.4 MB. After 4-bit quantization, I measured the remaining drift on MTEB across two models on retrieval and STS: 0.0020 NDCG@10 at worst, 0.0005 or less on the other three suites. Before trusting those quality numbers, I first reproduced MinishLab’s published MTEB scores per-task on the same harness. Mean pooling averages away per-token quantization noise, which is why static embedders compress this well. Full tables and methodology are in docs/turboquant.md in the repo.
Supported Zig versions
0.16.0
AI / LLM usage disclosure
I used LLMs heavily for the implementation and eval harness, under my direction. Rather than relying on my review alone, I verified the work against external references: output vectors match the Python implementation to under 1e-5, the i8 quantizer’s output is byte-identical to the reference quantizer’s, and MinishLab’s published MTEB scores were reproduced per-task. The evaluation harness is included as scripts/mteb_eval.py in the repo.