Does your tokenizer do realloc for tokens like Zig’s tokenizer or one big malloc of tokens from file size but the tokenizer scans the file once?
I do have this discusssion idea where their is no realloc for tokens, and the malloc tokens are smaller but at the cost of tokenizer scanning the file twice, first pass (Faster than second pass) scans ONLY for tokens length, then one malloc that tokens length, and the second pass does the usual parsing.