Good practice or not? Using a var as a cache

Explain
1

Hello! I am relatively new to idea of memory management on backend perspective (coming from JS world :slight_smile:) and am wondering whether or not this may be a good practice or not for caching small data to reduce API calls?

My concept is to create a struct which contains the API data and has a timestamp. When the function to get the API data is invoked, it first checks the var and if the timestamp is not stale (e.g. less than 5 mins ago) it just returns that data.

Here’s an example:

const ApiData = struct {
    some_data: []const u8,
    timestamp: i64,
};

var cached_api_data: ?ApiData = null;

pub fn getApiData() ApiData {
    if (cached_api_data) |cd| {
        const now = std.time.milliTimestamp();
        if (now - cd.timestamp < 300000) {
            return cd;
        }
    }

    const new_api_data = fetchApiData();
    cached_api_data = ApiData{
        some_data = new_api_data,
        timestamp: std.time.milliTimestamp(),
    };
    return cahced_api_data;
}

This implementation works as expected so far, but wondering if anyone with more advanced knowledge/wisdom might be able to point out some obvious flaws I may be missing.

Thanks in advance!

2 Likes
2

If you are fine with the data being potentially old (and it seems you are), you may want to think about whether you would want to update the data in the background once the 5 minutes are over.

Basically if you mostly care about latency, you might want to return the old data until the ApiFetch (that would run every 5 minutes if the function is being called regularly) has returned the data. With the current implementation you would have very low latency for 5 minutes then a massive spike in latency until the fetch returns and so on.

But it depends on what you are doing whether that is important.

Personally I might write the code a bit more like this:

const CachedData = struct {
    data: ?[]const u8 = null,
    timestamp: i64 = undefined,
    
    pub fn ensureData(self:*CachedData) !void {
        if (self.data != null) {
            const delta = std.time.milliTimestamp() - self.timestamp;
            if (delta < 300000) return;
        }
        self.data = try fetchApiData();
        self.timestamp = std.time.milliTimestamp();
    }
    pub fn getData(self:*CachedData) ![]const u8 {
        try self.ensureData();
        return self.data.?;
    }
};

var cached_api_data: CachedData = .{};
// const data = try cached_api_data.getData();

I think another big question is how the fetchApiData can fail and if we need to pass specific arguments to that. I also think if you have many of those caches you might want to add a reset method to that and keep pointers to all of them in a list so that you could implement a function that resets all of them.

1 Like
3

Hello and welcome to the forum!
I myself am not a professional so I would appreciate a second opinion.

Global variables are considered a code smell unless absolutely necessary. I think it would be better to create a struct thats holds this variable. This would also allow you to have multiple instances of the api.

The code would be something like:

const std = @import("std");

const Api = struct {
    const Data = struct {
        some_data: []const u8,
        timestamp: i64,
    };

    cached_data: ?Data = null,

    pub fn getData(self: *Api) Data {
        if (self.cached_data) |cd| {
            const now = std.time.milliTimestamp();
            if (now - cd.timestamp < 300000) {
                return cd;
            }
        }

        self.cached_data = Data{
            .some_data = fetchApiData(),
            .timestamp = std.time.milliTimestamp(),
        };
        return self.cached_data;
    }
};

fn fetchApiData() []const u8 {
    return "This is data";
}

test {
    _ = Api;
}
4

I actually prefer this code over the one I wrote myself lol, but Im glad we came to the same conclusion in terms of globals

5

I think globals are fine and often better than Singleton like stuff (I really dislike c++ for making those popular in its community), but I agree that using globals (or in this case container scoped variables) gets easier by bundling up things with a struct and thus reducing the surface area / having related things closer together.

I think it also makes it easier to see, whether there are many slightly different variations of these caches and then think whether those differences could be expressed via fields with different data to have one common type.

1 Like
6

Yea this is where my knowledge ends though, all I could understand, I can agree with, but I am unfamiliar with singletons, I’ve only heard of em before.

1 Like
7

The real problem with Singletons is when you have a shared one among multiple endpoints.

Really, in simple cases, we won’t see the complexity with globals arising. There’s just not enough surface area to create conflicts. In examples like this, you can’t really make an argument for or against because it’s easy to reason about and you’ll probably have just one instance of something in trivial examples anyway.

When you get to bigger applications that may have a different view of the state of the Singleton in different places, that’s when you hit problems. Even for getting it setup, you can get into what’s often referred to as an initialization fiasco. It can also be complex because of the introduction of locks and thread safety mechanisms.

Practically speaking, a Singleton is something that’s supposed to have one instance for the lifetime of a program. You can get further into the weeds, but you can typically expect that to be the case. I’m sure someone could raise an example of a Singleton that goes through state changes (may even be de-initialized and re-initialized at points), but you don’t have two instances at the same time.

Wikipedia isn’t too bad on this one: Singleton pattern - Wikipedia

1 Like
8

I’ll also point out that a premier place to see the complexity with globals is in SQL databases. Take a table, for example, that is shared among multiple processes. They are (in my mind) one of the most clear examples of a “global variable” that you’ll see. They’re truly global, unannounced (code just uses them as if they already exist), and they have very complex locking/safety mechanisms applied to them.

9

The data structure usually used for this kind of application is an LRU cache. Here’s one in Zig, I can’t speak for the code there, just the first thing which turned up in a search.

The additional complication here is that Zig doesn’t manage memory for you, so you’ll need to establish an ownership policy for data in the application. An LRU makes that relatively easy: set up the LRU with its own Allocator, and write the program such that the cache owns the memory.

So every fetch goes through the cache, and only the cache frees data, when it decides to evict. That way the server code never frees everything, and the LRU can be freed in one action at the end of the program.

This is only strictly correct if the data in the cache never changes (every request is immutable), like a static site server. If your source of truth can change, now you have cache invalidation, and that’s one of the two hard problems.

If the source of truth changes rarely, but all at once, you can just evict the entire cache whenever that happens.

The first link goes into other cache replacement policies, LRU is just a good balance of simplicity and generality. “It’s stale after five minutes” is another cache replacement policy, use what works for your application.

Also, I agree with other posters: use an instance of a struct here, not a global. For one thing, globals are hard to get rid of if it turns out you need more than one instance, and that usually happens.

10

To re-state this in slightly stronger terms, a Singleton should mean that it’s an error for more than one to exist. Not “I happen to need only one of these, I’ll make a Singleton” but “more than one of these will make my program incorrect, I have to use a Singleton here”.

1 Like
11

This should come with a warning actually: be sure you aren’t evicting and freeing data while the server code is using it! An LRU policy usually prevents this, because the timestamp updates every time data is asked for, but if you combine that with a recency policy, you can have a race. That won’t happen in single threaded code with synchronous system calls, but if and when you start adding concurrency of any sort, this race condition is a risk.

12

This is a great point. Thinking about my use-case, I think this is acceptable. Basically, I self-host a blog with very low traffic and I have a widget which fetches the local weather conditions where the server is located. I don’t want to call the API every 5 minutes as it is very likely that there is nobody on the site. But, I don’t want to call the API every request because someone browsing through a few pages can rack up API calls. So striking a balance here.

This is really interesting concept, thank you for sharing this!

Yes, this is getting to where I am learning the most. For a browser-based app I would generally just store the data in the local storage or cookies, and then evaluate the timestamp when reading the local storage. But for Zig where I am actually running the server, memory management becomes a necessary thought and is one where I’m enjoying learning. Thanks for sharing this concept of LRU as well as explaining with some detail into how/why/when etc.

Really appreciate all of the helpful insights and discussion from this simple question I had. Thanks to all here! I have a lot of learning ahead and am on a more prouctive path thanks to all of your guidance.

2 Likes