Ochi - logging database

Showcase
1

[Ochi]:

It’s my journey into database.
I work in enterprise and everyone is ranting about observability cost, as a result all the money savings makes us debugging longer and migrating from one provider to another.

So I built a datastore to make ingestion and querying of logs more efficient.

9 months on track, just a beginning.

Design points

  1. does support any block device (generally speaking hdd, ssd, a regular disk drive) instead of s3. It feels important to hold storage and compute next to each, it leads to more predictable cost and better performance
  2. it uses a custom column oriented file format. Although I respect headless architecture and PAX in general since it gave us Arrow, Parquet and others, but being restricted since the beginning won’t lead me to a better design. However, long term I want to support headless architecture to give a wider usage.
  3. Multi tenancy out of the box, they are stored in different data blocks.
  4. No mmap usage, although I might consider it later for specific workloads that require sequential reads such as merge tables.
  5. No WAL, Clickhouse inspired me on that move so SS Tables working fine to me, the solution accepts a risk to loose last second data.
  6. Niche integer compression (bitpacking) is used (timestamps processing is a visible part of workload in the profiler) via SIMD, described by FastLane and implemented in zig by steelcake github user, gives good performance to move this problem a bit away

What’s next

Many things are not designed well, but I’m it was fantastic to learn so much about databases engines and there are many things to come after:

  • page buffer pool, apparently every (decent) database pre-allocate memory in advance to provide predictable performance
  • background tasks scheduler to balance ingestion and tables merging (it’s now very primitive set of semaphores), currently it’s not even possible to run it on a single CPU machine (terrible I know, very high prio)
  • mmap or different file descriptors for merging in order to keep page cache / fadvise different for different file descriptors
    And many more.

The end goal

During the development I was missing pyroscoep for Zig and in general was lacking performance detection tools, so it inspired me to use the Ochi engine to build pprof backend for zig and build a long term profile store + traces to have an alternative to the solutions, I wasn’t happy with the existing trace + profile visualization solution to profile Ochi.
Profile are just traces with a stricter schema and traces are just logs with point lookup by id.

Glad to hear any feedback and wish to see more databases on Ziggit.

Appreciation

Appreciate zig team for making it happen.
I tried starting with rust, but learning it for 2 weeks didn’t let me to start, while with zig after the first week of reading a zig book I just started writing the code and I could see executing, the bad one, but it was running.

Supported Zig versions

One can build it from source using zig-0.16.

AI / LLM usage disclosure

  • generated Heap data structure and it’s rebalancing (0 manual code in this file, but I plan to replace it to hand crafted loser tree)
  • catching a crash/bug I couldn’t understand I asked to write a unit test that reproduces it, lately I learned to use lldb quite a bit so it’s getting better
  • my lovely wife used LLM to generate a website which I find amazing artifact of her creativity
  • grafana datasource is a complete slop, I’m on it right after this post to start doing a better version of it

I don’t want to share my opinion on LLM, but want to note it was very hard to use it since I barely understood what I was building and I do know every line there.
Large design shakes still happen so it’s not very good at generating large chunks.

3 Likes