Bm25er - Search local files like you're searching the web (BM25 implementation)

[vincentd]

First thing I noticed:

lenght: u32

Doesn’t affect the logic, but it still hurts to see it.

Some parts have way too much going on in a single line. For example:

return @log(1 + (((@as(f32, @floatFromInt(doc_counter)) - @as(f32, @floatFromInt(n)) + 0.5) / (@as(f32, @floatFromInt(n)) + 0.5))));

Good practice is to break it up into simpler calculations. It’s easier to read, and it’s obvious which calculations happen before other ones.

const a: f32 = @as(f32, @floatFromInt(doc_counter));
const b: f32 = @as(f32, @floatFromInt(n));
const c: f32 = @as(f32, @floatFromInt(n));
const d: f32 = (a - b + 0.5) /  (c + 0.5);
return @log(1 + d);

Note: a/b/c/d are for example, you should try to use names that help describe what the equation is doing in each step.

It’s easier to see patterns this way, so you can simplify the equation:

const a: f32 = @as(f32, @floatFromInt(doc_counter));
const b: f32 = @as(f32, @floatFromInt(n));
const d: f32 = (a - b + 0.5) /  (b + 0.5);
return @log(1 + d);

You can also remove the @as() as it can now infer the type from the variable:

const a: f32 = @floatFromInt(doc_counter);
const b: f32 = @floatFromInt(n);
const d: f32 = (a - b + 0.5) /  (b + 0.5);
return @log(1 + d);

Easier to read, easier to modify, and you can add comments next to each step!

Also, if one of the operations causes an error at runtime it’s easier to find what’s causing it, because the line numbers are different.

.

@splinterofchaos

On binary files: I’ve heard of a technique for detecting if you’re reading a binary file where you check the first 128 or so bytes to see if they’re all valid, printable characters. As far as I know there isn’t support in the standard for non-ASCII but just checking std.ascii.isAscii() would be okay.

I believe the way grep does it is by scanning the first buffer-load (e.g 32kB) it reads for null characters, or invalid UTF-8 encoding (e.g. overlong characters)

I think you could use the functions in std.unicode to do that. I might give that a try actually - I have a crude zig version of grep which I made as an example in a library.

EDIT:

This seems to work - for zig-0.16.0

/// First ensures the buffer is full (or EOF), then returns true
/// if the buffer consists entirely of UTF-8 codepoints.
///
/// Useful for distinguishing text files from binary files.
fn checkFileIsValidUtf8(reader: *std.Io.Reader) Io.Reader.Error!bool {
    const peek = reader.peek(reader.buffer.len) catch |err|
        switch (err) {
            Io.Reader.Error.EndOfStream => try reader.peekGreedy(0),
            else => return err,
        };
    // std.log.debug("Checking utf-8 validity for first {} bytes", .{peek.len});
    return std.unicode.utf8ValidateSlice(peek);
}