Project: bridge server to stream Postgres CDC to a NATS message broker

Showcase
1

I am working on a project that captures Postgres CDC and forwards the changes to a NATS message broker.

It’s essentially a lightweight Dezebiul-lik solution, but with a much smaller footprint (≈15 MB image, ≈7 MB RAM).

The project is already fairly advanced, though it likely still needs polishing and—most importantly—feedback and challenges from others.

While the project is not a pur Zig implementation. - I vendor the NATS client nats.c and libpq because pg.zig did not cover all my needs - but the architectural ideas might still be of interest.

For example, it uses a SPSC ring buffer.

Would anyone here be interested in this topic?

2 Likes
2

  • take a look on list on zig nats clients

  • pay attention that nats.c is the porting of go code (you will have (1)zig wrapper for (2)C code based on (3)go design)

  • if you can directly use jetstream instead of kv - use it

welcome to the club :clap:

1 Like
3

Ah!! thks, I don’t know why I did not see this.. yours looks good. I still use a KV store though. Maybe I can just use a “singleton” stream instead? MaxMsgsPerSubject: 1, Discard: Old, MaxAge: 0

4

In very interested. I’ve been planning to do something like that, just not as a generic framework.

5

I use zig-messagepack library by default to encode to NATS if you connect a backend to NATS, but you can switch to JSON if you connect via a JS client so that you can populate a WASM db (SQLite or PGLIte directly).

EDIT

For the NATS client, I need more than a sync `nats.publish`, I need the non-blocking `nats.publishAsync` and then `nats.flush` to ensure delivery for PG to prune the WAL with the last LSN. I also need the reconnection mechanism.

So maybe https://github.com/lalinsky/nats.zig is a better fit for me?

6

you can check what I use: https://github.com/ndrean/PG-CDC-NATS-BRIDGE

Be indulgent…, I am not a Zig master

7

go ahead

btw i don’t like automatic re-connect on lib layer, better to implement it on your application layer

8

I had a closer look to https://github.com/lalinsky/nats.zig. I think it has everything! It will be much nicer than using nats.c. Good!