ZRoaring: roaring bitmaps in zig

That seems to be a good idea. I’ve done this along with adding automatic compaction when blocks are shrunk and now were around 90% of croaring’s speed on this benchmark :slight_smile:.

Of course that means 2 allocations instead of one and a slightly less happy deinit. But I can accept that.

zig build bench -Doptimize=ReleaseFast
$ zig build bench -Doptimize=ReleaseFast
--------------------------------------------------
overall:
--------------------------------------------------
CRoaring: 11430 ops 20.698ms   0.55MB ops/sec
ZRoaring: 11430 ops 22.894ms   0.50MB ops/sec

                                  ratio -- 0.90 πŸ₯”

--------------------------------------------------
individual ops: speed=ops/sec
--------------------------------------------------
op                 cr speed zr speed #    ratio
--------------------------------------------------
clear              1.62MB   35.25MB  240  21.82 ⚑
run_optimize       0.61MB   0.60MB   365  0.99  πŸ‘πŸ»
shrink_to_fit      4.17MB   1.20MB   330  0.29  πŸ’©
portable_serialize 0.70MB   0.94MB   105  1.34  ⚑
frozen_serialize   2.31MB   2.28MB   85   0.99  πŸ‘πŸ»
minimum            32.69MB  34.56MB  580  1.06  πŸ‘πŸ»
maximum            14.55MB  7.65MB   585  0.53  πŸ’©
add                12.09MB  10.37MB  2200 0.86  πŸ₯”
rank               10.29MB  9.69MB   25   0.94  πŸ‘πŸ»
select             8.72MB   8.43MB   30   0.97  πŸ‘πŸ»
contains           22.46MB  22.67MB  580  1.01  πŸ‘πŸ»
add_many           2.52MB   2.86MB   915  1.13  ⚑
add_range_closed   2.37MB   2.29MB   1545 0.96  πŸ‘πŸ»
contains_range     16.02MB  17.40MB  570  1.09  ⚑
range_cardinality  6.67MB   5.15MB   5    0.77  πŸ₯”
remove             11.65MB  11.98MB  630  1.03  πŸ‘πŸ»
and                0.34MB   0.41MB   420  1.22  ⚑
or                 196.69kB 219.31kB 305  1.12  ⚑
xor                157.98kB 180.81kB 375  1.14  ⚑
andnot             217.86kB 0.33MB   320  1.50  ⚑
lazy_or            161.72kB 239.00kB 250  1.48  ⚑
or_inplace         0.32MB   0.34MB   390  1.05  πŸ‘πŸ»
and_inplace        140.05kB 142.59kB 60   1.02  πŸ‘πŸ»
is_subset          134.90kB 144.89kB 180  1.07  ⚑
equals             26.55MB  6.86MB   30   0.26  πŸ’©
and_cardinality    0.74MB   1.05MB   40   1.42  ⚑
or_cardinality     0.79MB   0.93MB   45   1.18  ⚑
xor_cardinality    0.95MB   0.94MB   35   0.99  πŸ‘πŸ»
andnot_cardinality 0.81MB   0.96MB   20   1.18  ⚑
jaccard_index      0.88MB   1.06MB   40   1.20  ⚑
or_many            19.59kB  12.23kB  130  0.62  πŸ’©

I’ve just pushed some minor optimizations which bring the overall benchmark i’ve been sharing up around 1% to 91% from 90% last time.

More importantly, I’ve improved the overall benchmark a bit. It now uses clear (which is an alias to deinit) instead of clear_retaining_capacity.

As a result clear is no longer like 20x faster :sweat_smile: . And the overall benchmark is now a more honest comparison with CRoaring.

Overall plot:

$ zig build bench -Doptimize=ReleaseFast
$ zig build bench -Doptimize=ReleaseFast
----------------------------------------------------
overall:
----------------------------------------------------
CRoaring: 23450 ops 46.171ms   0.508MB ops/sec
ZRoaring: 23450 ops 50.605ms   0.463MB ops/sec

                                 ratio -- 0.912 πŸ₯”

----------------------------------------------------
individual ops: speed=ops/sec
----------------------------------------------------
op                   cr speed zr speed #    ratio
----------------------------------------------------
clear                1.62MB   1.66MB   530  1.03  πŸ‘πŸ»
run_optimize         0.46MB   0.40MB   780  0.88  πŸ₯”
shrink_to_fit        4.21MB   3.08MB   700  0.73  πŸ’©
portable_serialize   3.07MB   2.32MB   210  0.75  πŸ₯”
portable_deserialize 216.75kB 0.42MB   860  1.94  ⚑
frozen_serialize     3.03MB   2.39MB   170  0.79  πŸ₯”
minimum              27.32MB  26.72MB  970  0.98  πŸ‘πŸ»
maximum              21.53MB  18.53MB  950  0.86  πŸ₯”
add                  12.05MB  12.03MB  4470 1.00  πŸ‘πŸ»
rank                 9.69MB   9.69MB   50   1.00  πŸ‘πŸ»
select               8.72MB   4.50MB   60   0.52  πŸ’©
contains             23.15MB  22.07MB  800  0.95  πŸ‘πŸ»
add_many             2.52MB   3.11MB   1900 1.24  ⚑
add_range_closed     2.46MB   2.21MB   3210 0.90  πŸ₯”
contains_range       19.16MB  18.89MB  1050 0.99  πŸ‘πŸ»
range_cardinality    8.33MB   5.52MB   10   0.66  πŸ’©
remove               12.06MB  12.43MB  1360 1.03  πŸ‘πŸ»
and                  0.26MB   0.27MB   930  1.04  πŸ‘πŸ»
or                   201.05kB 189.69kB 610  0.94  πŸ‘πŸ»
xor                  155.58kB 183.67kB 750  1.18  ⚑
andnot               212.07kB 0.43MB   640  2.04  ⚑
lazy_or              168.66kB 226.22kB 500  1.34  ⚑
or_inplace           0.31MB   0.33MB   780  1.08  ⚑
and_inplace          147.68kB 154.56kB 120  1.05  πŸ‘πŸ»
is_subset            134.64kB 139.47kB 360  1.04  πŸ‘πŸ»
equals               28.99MB  20.62MB  60   0.71  πŸ’©
and_cardinality      0.66MB   0.77MB   80   1.16  ⚑
or_cardinality       0.75MB   0.76MB   90   1.03  πŸ‘πŸ»
xor_cardinality      0.93MB   0.91MB   70   0.98  πŸ‘πŸ»
andnot_cardinality   0.70MB   0.82MB   40   1.17  ⚑
jaccard_index        0.80MB   0.79MB   80   0.98  πŸ‘πŸ»
or_many              20.50kB  11.80kB  260  0.58  πŸ’©
1 Like

Excited to see zroaring has pulled even with CRoaring on the overall ReleaseFast benchmark i’ve been sharing.

This is the result of 2 recent changes:

  • dropping the resizable-struct dependency for a normal extern struct with faster, static offset calculations.
  • moving Block storage from Bitmap to Container like CRoaring does. while this complicates memory management it makes for fewer required copies and scales better as Bitmaps grow large.

I’d still like to find a a solution with less pointers chasing in deinit, maybe storing Blocks together somehow. But for now I’m happy with the perf improvements.

$ zig build bench -Doptimize=ReleaseFast
----------------------------------------------------
overall:
----------------------------------------------------
CRoaring: 29210 ops 147.571ms  197.937kB ops/sec
ZRoaring: 29210 ops 146.932ms  198.798kB ops/sec

                                 ratio -- 1.004 πŸ‘πŸ»

----------------------------------------------------
individual ops: speed=ops/sec
----------------------------------------------------
op                   cr speed zr speed #    ratio
----------------------------------------------------
clear                1.02MB   0.79MB   830  0.78  πŸ₯”
run_optimize         229.32kB 179.44kB 1170 0.78  πŸ₯”
shrink_to_fit        3.58MB   4.61MB   980  1.29  ⚑
portable_serialize   0.49MB   0.40MB   530  0.82  πŸ₯”
frozen_serialize     0.57MB   0.49MB   380  0.87  πŸ₯”
minimum              18.34MB  18.73MB  280  1.02  πŸ‘πŸ»
maximum              15.22MB  11.95MB  210  0.79  πŸ₯”
add                  10.72MB  8.39MB   4780 0.78  πŸ₯”
rank                 8.03MB   7.09MB   410  0.88  πŸ₯”
select               4.36MB   3.34MB   400  0.77  πŸ₯”
contains             13.53MB  14.10MB  300  1.04  πŸ‘πŸ»
add_many             2.44MB   2.08MB   2290 0.85  πŸ₯”
add_range_closed     2.52MB   1.86MB   3540 0.74  πŸ’©
contains_range       9.65MB   8.80MB   330  0.91  πŸ₯”
range_cardinality    11.89MB  12.43MB  390  1.05  πŸ‘πŸ»
remove               11.95MB  12.52MB  1610 1.05  πŸ‘πŸ»
and                  224.95kB 232.43kB 1210 1.03  πŸ‘πŸ»
or                   109.21kB 117.00kB 880  1.07  ⚑
xor                  86.01kB  102.43kB 1070 1.19  ⚑
andnot               103.16kB 132.21kB 950  1.28  ⚑
lazy_or              98.76kB  91.53kB  840  0.93  πŸ₯”
or_inplace           200.11kB 207.22kB 1160 1.04  πŸ‘πŸ»
and_inplace          125.61kB 100.96kB 320  0.80  πŸ₯”
is_subset            28.69kB  30.32kB  710  1.06  πŸ‘πŸ»
equals               2.94MB   7.80MB   320  2.65  ⚑
and_cardinality      0.45MB   0.42MB   420  0.94  πŸ‘πŸ»
or_cardinality       0.45MB   0.42MB   400  0.93  πŸ‘πŸ»
xor_cardinality      0.74MB   0.68MB   280  0.91  πŸ₯”
andnot_cardinality   0.36MB   0.33MB   240  0.92  πŸ₯”
jaccard_index        0.72MB   0.63MB   290  0.88  πŸ₯”
or_many              13.47kB  12.67kB  610  0.94  πŸ‘πŸ»
portable_deserialize 44.32kB  67.20kB  330  1.52  ⚑
statistics           2.43MB   2.08MB   300  0.86  πŸ₯”
flip                 163.00kB 118.95kB 450  0.73  πŸ’©

Vind wasm binary size went down a bit too:

$ ~/.../zig/vind]$ zig build wasm -Droaring=zig && ls -lh zig-out/wasm/vind.wasm
... 113K Jul  6 17:29

Vind bench seems around the same here:

$ zig build bench -Droaring=c
timings
  build         686.287 ms         72856 docs/s
  finalize        7.506 ms       6660895 docs/s
  serialize       5.171 ms    1379200769 bytes/s
  deserialize   700.814 ms         71346 docs/s
  total        1399.778 ms
...
  combined total 2165.707 ms
$ zig build bench -Droaring=zig
timings
  build         699.478 ms         71482 docs/s
  finalize        7.497 ms       6668907 docs/s
  serialize       5.262 ms    1355371566 bytes/s
  deserialize   717.759 ms         69661 docs/s
  total        1429.997 ms
...
  combined total 2184.581 ms
4 Likes

Excellent numbers! Thank you for the update! :slight_smile:

1 Like