# 300 MiB/s Zig lexer, fixing an edge case in the grammar

**URL:** <https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514>\
**Category:** Brainstorming\
**Tags:** language\
**Created:** [April 7, 2025, 2:18pm UTC](https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514 "2025-04-07T14:18:38Z")\
**Posts on this page:** 4\
**Page:** 2

<div class="post-metadata">

**Author:** ![mnemnion](https://ziggit.dev/user_avatar/ziggit.dev/mnemnion/32/2478_2.png) [@mnemnion](https://ziggit.dev/u/mnemnion)\
**Post date:** [April 15, 2025, 3:11pm UTC](https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514/21 "2025-04-15T15:11:02Z")

</div>

> [@andrewrk](#):
>
> Of course, it’s also reasonable to argue that it’s not worth a worse syntax for slightly more lexing perf.

In the absence of solid data, my stance is that a bottleneck due to this lexing quirk hasn’t been established.

I would probably advance that argument if such a thing were established, which as of now it has not been.

Mainly I wanted to establish some rhetorical distance between “possible optimization available” and “grammar bug”.

In the absence of attention paid to efficiency in parsing, you get C++. No one wants that. But I’m sure that switching Zig to s-expressions would make parsing faster, and almost no one wants that either.

---

<div class="post-metadata">

**Author:** ![mnemnion](https://ziggit.dev/user_avatar/ziggit.dev/mnemnion/32/2478_2.png) [@mnemnion](https://ziggit.dev/u/mnemnion)\
**Post date:** [April 15, 2025, 3:55pm UTC](https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514/22 "2025-04-15T15:55:49Z")

</div>

> [@prokoprandacek](#):
>
> The current implementation has to backtrack in this situation

Y’know, it doesn’t actually have to:

```zig
.int_period => {
    switch (self.buffer[self.index + 1]) {
        '_', 'a'...'d', 'f'...'o', 'q'...'z', 'A'...'D', 'F'...'O', 'Q'...'Z', '0'...'9' => {
            self.index += 2;
            continue :state .float;
        },
        'e', 'E', 'p', 'P' => {
            self.index += 1;
            continue :state .float_exponent;
        },
        else => {},
    }
},

```

I don’t actually know if this is better, but it might be worth benchmarking, maybe? The effect is going to be very small if there is one.

In fact I’d want to check first just to see if optimized versions generate different machine code at all.

---

<div class="post-metadata">

**Author:** ![mnemnion](https://ziggit.dev/user_avatar/ziggit.dev/mnemnion/32/2478_2.png) [@mnemnion](https://ziggit.dev/u/mnemnion)\
**Post date:** [April 15, 2025, 4:05pm UTC](https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514/23 "2025-04-15T16:05:42Z")

</div>

Here it is as a patch against Zig master, if anyone is interested in playing around with it.

I don’t have the time to build zig today and run the tests so apologies if it’s simply broken in some way I can’t perceive.

```diff
diff --git a/lib/std/zig/tokenizer.zig b/lib/std/zig/tokenizer.zig
index 09c79d3562..7ea1003284 100644
--- a/lib/std/zig/tokenizer.zig
+++ b/lib/std/zig/tokenizer.zig
@@ -1068,18 +1068,19 @@ pub const Tokenizer = struct {
                 }
             },
             .int_period => {
- self.index += 1;
- switch (self.buffer[self.index]) {
+ switch (self.buffer[self.index + 1]) {
                     '_', 'a'...'d', 'f'...'o', 'q'...'z', 'A'...'D', 'F'...'O', 'Q'...'Z', '0'...'9' => {
- self.index += 1;
+ self.index += 2;
                         continue :state .float;
                     },
                     'e', 'E', 'p', 'P' => {
+ self.index += 1;
                         continue :state .float_exponent;
                     },
- else => self.index -= 1,
+ else => {},
                 }
             },
+
             .float => switch (self.buffer[self.index]) {
                 '_', 'a'...'d', 'f'...'o', 'q'...'z', 'A'...'D', 'F'...'O', 'Q'...'Z', '0'...'9' => {
                     self.index += 1;

```

---

<div class="post-metadata">

**Author:** ![squeek502](https://ziggit.dev/user_avatar/ziggit.dev/squeek502/32/409_2.png) [@squeek502](https://ziggit.dev/u/squeek502)\
**Post date:** [April 15, 2025, 5:24pm UTC](https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514/24 "2025-04-15T17:24:08Z")

</div>

> [@prokoprandacek](#):
>
> I have to say that the results so far are very discouraging so I was lacking the motivation to invest more time into this.

Sorry, I didn’t mean to discourage you, just wanted to suggest that more apples-to-apples benchmarking is needed.

It’d be helpful if you could provide instructions on exactly how you benchmarked your implementation and how others can reproduce your results (you mentioned `Sema.zig`, but your `BM_Lex` function uses an embedded `tracy.zig`?). As of now, I’m not sure exactly how to run your benchmark or what benchmark you’re actually using.

It’d also be good if you could run the Zig benchmark code in my comment that you replied to and post your results (remember to compile with `-OReleaseFast`).

[Previous page](https://ziggit.dev/t/300-mib-s-zig-lexer-fixing-an-edge-case-in-the-grammar/9514.md?page=1)