LinearFifo replacement with Reader and Writer

Help
1

In some old code I’m trying to update, I was using LinearFifo and a mutex to stream data between two threads.

One thread handles IO, does std.posix.poll(), reads from stdin, locks, pushes into the LinearFifo, unlocks then signals the second thread.

The second thread runs a game event loop, it receives the signal, locks, reads from LinearFifo then unlocks.

I haven’t looked much at how std.Io works yet. Is there an example of how to implement a ring buffer using a Reader and Writer?
Is there some kind of Pipe which connects a Reader and Writer to give me a Fifo?
How do I go about getting the same functionality?

From the 0.15.1 release notes:

std.fifo.LinearFifo is removed due to being poorly designed. This data structure was unnecessarily generic due to accepting a comptime enum parameter that determined whether its buffer was heap-allocated with an Allocator parameter, passed in as an externally-owned slice, or stored in the struct itself. Each of these different buffer management strategies describes a fundamentally different data structure.

Furthermore, most of its real-world use cases are subsumed by New std.Io.Writer and std.Io.Reader API which are both ring buffers.

2

Use std.Io.Reader.fixed, the reading thread just takes a *std.Io.Reader, the writing thread writes data to the reader. You still need the mutex and signals ofc, the interfaces have no concept of thread safety.

It’s only a little different from implementing a normal custom reader, implementing the vtable functions aren’t useful since you don’t want an underlying source, hence Reader.fixed. Instead, it’s given the data externally.

I’m assuming you parse the data from stdin, otherwise you can just use a normal file reader.

3

Then, am I right in thinking I want something like q:Reader.fixed() available to both threads:

var bufa:u8[4096];
const q = std.io.Reader.fixed(&bufa);

In my consumer thread:

// wait for signal ...
// lock
var bufc: [4096]u8 = undefined;
const count = try stdin_reader.readSliceShort(&bufc); // ?
...
// unlock

But what should the producer thread do?
It looks like I can make another reader which pulls from stdin.

var bufb:u8[4096];
var stdinreader = std.fs.File.stdin().reader(&bufb);

var fds = [_]std.posix.pollfd{
	.{
		.fd = std.fs.File.stdin().handle,
		.events = std.posix.POLL.IN,
		.revents = undefined,
	}
};

while(true) {
	const ready = try std.posix.poll(&fds, 1000);
	if (ready) {
		// lock
		const count = try stdin_reader.readSliceShort(&bufc); // ?
		// ???
		// unlock
	}
}

Do, I then write code to copy from stdin_reader to q?
Or, is there some pipe() call bind them?

Or, something entirely different?

4

You might be interested in https://github.com/ziglang/zig/pull/24968 as well (it’s not in 0.15.x though).

3 Likes
5

you would

while (true) {
    // fills the buffer as much as possible with one read
    try stdin_reader.fillMore()
    // get buffered data
    const buffered = stdin_reader.buffered();
    // ensure there is space for the data
    // it is unclear if this errors if you request more
    // space than is available or not
    try q.rebase(buffered.len);
    // calculate the amount of data that can be copied
    const copy_len = @min(buffered.len, q.buffer.len - q.end);
    // copy the data
    @memcpy(q.buffer[q.end..][0..copy_len], buffered[0..copy_len]);
    // update que data length
    q.end += copy_len;
    // update stdin read length
    stdin_reader.toss(copy_len);
}

Didn’t test this, I probably got something wrong :sweat_smile:
You’d probably want a wrapper which would look very similar to the queue @squeek502 linked

I havent used reader or writer to replace a fifo before, just know its possible in most cases.

1 Like