ZENFMT:
Hi Everyone,
I would like to share.
A universal document conversion library, CLI, server, Python package, and WebAssembly module built entirely in Zig.
The project is directly inspired by John MacFarlane’s pandoc, who has been working on this project for over 20 years. This taught a beautiful idea how essential is a single well structured intermediate AST can be for handling different isolated formats.
zenfmt is a small attempt to explore that idea in Zig. It focuses on mechanical sympathy, explicit loss reporting, compact memory layouts, and zero-dependency execution. It supports 16 input formats across 5 primary document families, converting them to Markdown.
| Family | Supported Formats |
|---|---|
| Word Processing | docx / docm, legacy binary doc, odt, rtf |
| Spreadsheets | xlsx / xlsm, xlsb, legacy binary xls, ods, csv / tsv |
| Presentations | pptx / pptm / ppsx, legacy binary ppt / pps / pot, odp |
| Publishing & Web | epub, pdf (native Zig text extraction), html |
| Markup & Plain Text | markdown, plain text |
Inputs are detected by content signature (ZIP central-directory part names, OpenDocument/EPUB mimetype entries, CFB directory streams, %PDF, {\rtf), using file extensions as an initial hint.
The Problem
Every document format is a lossy projection onto a shared semantic model. When converting a .docx file or a .pdf to Markdown, information is inevitably dropped (underlines, cell grid coordinates, slide layouts, font variations).
Traditional document processing stacks (like Apache Tika, Docling, or Python/Java wrappers around LibreOffice) handle this in one of two ways:
- Silent Loss: Discarding metadata without telling the user what was lost.
- Runtime Overhead: Booting heavy runtimes (JVMs, Python interpreter stacks, or full headless browser engines) that pay a 50–500 ms startup penalty and consume hundreds of megabytes of RAM before processing the first byte.
We wanted to answer three simple engineering questions:
- Can we store a rich document AST with zero per-node pointer overhead and zero heap allocation per node?
- Can we make format lowering declarative, deterministic, and provably optimal using compile-time meta-programming?
- Can a full office document converter compile to a freestanding < 2 MB WebAssembly module with zero host imports that runs entirely client-side in a browser worker?
In zig, the answer to all three turned out to be a resounding yes.
How Zig Made This Possible
1. Flat Struct-of-Arrays (SoA) Storage
Following the precedent of Zig’s own compiler (std.zig.Ast), zenfmt does not store document nodes as heap-allocated tree objects connected by pointers. Instead, the document forest is stored in a single arena as a preorder Struct-of-Arrays (std.MultiArrayList) indexed by 32-bit typed indices (enum(u32)).
pub const BlockRow = struct {
tag: BlockTag,
payload: PayloadIndex,
attrs: OptionalAttrsIndex,
inlines: InlineRange,
subtree_len: u32,
};
Because every node stores its total subtree_len, structural properties become trivial:
- Child Traversal: Sibling hopping is simply i + subtree_len (no parent pointers or linked lists).
- Subtree Contiguity: Every subtree occupies a contiguous slice [i, i + subtree_len).
- Zero-Allocation Filters: A transform filter that leaves a subtree untouched copies the range with a single memcpy per array column.
2. Comptime Schema Tables & Zero-Cost Capabilities
In zenfmt, format definitions and writer capabilities carry no runtime lookup cost. A single comptime schema table defines the node set:
pub const BlockSchemaRow = struct {
tag: BlockTag,
payload: type,
children: ChildKind,
placement: Placement,
};
pub const block_schema: []const BlockSchemaRow = &.{ ... };
From this single table, Zig’s comptime constructs:
- All AST node views and placement predicates.
- Structural tree validators.
- Exhaustive visitor dispatch switches.
- Writer capability matrices (validating at compile time that every writer declares exact, degraded, or refused behavior for every node tag).
3. Sparse Stand-off Annotations (Facets)
To prevent rich format data (like DOCX tracked revisions, XLSX cell formulas, or PDF page coordinates) from inflating plain flow conversions, zenfmt uses Stand-off Annotations (Facets).
Facets (such as StyleFacet, LayoutFacet, GridFacet, RevisionFacet) are stored in append-only side tables keyed by a lazy EntityId. If a document attaches no facets (e.g. converting plain text or Markdown), the facet tables remain completely unallocated i.e. costing 0 bytes per node.
Head-to-Head Benchmarks
We benchmarked zenfmt against pandoc, anydoc, and Docling across a corpus of 16 real-world documents from public sources (specifications, thesis presentations, 25,000-row CSVs, multi-page PDFs, and complex DOCX/XLSX files).
Measurements take the median of 5 runs per file; process wall clock, CPU time, and peak RSS are recorded via wait4 rusage.
Summary Ratios (Geometric Mean over Shared Corpus Files)
| Comparator | Shared Files | Wall Clock Speedup | CPU Time Ratio | Peak Memory (RSS) |
|---|---|---|---|---|
| zenfmt vs. Docling | 5 | 190.4x faster | 205.5x | 47.5x smaller |
| zenfmt vs. Pandoc | 6 | 18.2x faster | 16.5x | 16.6x smaller |
| zenfmt vs. anydoc | 14 | 6.9x faster | 7.9x | 10.1x smaller |
Ratios represent Competitor / zenfmt (higher numbers mean zenfmt is faster or uses less memory).
Key Takeaways from the Data:
- Startup Floor: Interpreted or heavy runtime competitors (Docling/Python, Tika/JVM) pay a 40ms to 2,000ms startup floor before reading a single byte. zenfmt converts typical documents (e.g. a 33 KiB .docx) in 4.5ms with a peak memory footprint of 3.4 MB.
- Memory Scaling: On large inputs (like an 850 KiB HTML page), Pandoc’s garbage-collected heap reaches 426 MB, whereas zenfmt’s single arena allocation peaks at 23.4 MB.
- The CSV Case (data.csv): zenfmt measures every column across 25,000 rows to output width-aligned GFM table pipes (O(rows × cols) pass). Even with column width calculation enabled, zenfmt converts the file in 53.9ms vs anydoc’s 83.1ms and Pandoc’s 829.4ms.
Replacing Heavy Server Stacks & Running Anywhere
One of our primary goals was to simplify deployment pipelines that currently rely on complex Java (Apache Tika) or Python microservices.
- Single-Binary REST & Admin Server (zenfmt serve):
The native zenfmt executable embeds a high-performance HTTP microservice, OpenAPI 3.1 documentation, streaming multipart parsing, rate limiting, and an admin UI (DaisyUI) in a single < 25 MB executable. It requires no external dependencies, Python scripts, or JVM installations. - Freestanding WebAssembly (wasm32-freestanding):
By compiling against wasm32-freestanding with zero host imports, the entire document conversion engine runs inside the visitor’s browser via Web Worker threads. No document data ever leaves the user’s machine. - Typed Python SDK (pip install zenfmt):
Bundles the native C bridge via CFFI, releases the Python GIL during conversion, and returns structured diagnostic reports and in-memory ensembles.
Try It Out
- Live WASM Playground & Multi-Language Docs
- Design Records (ZDS 0001–0016): In the repository to read the architectural RFCs that govern the project.
Supported Zig versions
0.16
AI / LLM usage disclosure
Used LLM extensively for development.
Thanks to entire zig team and community for building such a beautiful language eco-system and kind community.
Please be kind, we have used LLM extensively to build it and also learn the language (still learning).