After saying this, I found myself wondering: do we need eager dependencies at all? Is there some advantage to having eager dependencies that I’m not seeing? Or could the build system consolidate around lazy dependencies.
We don’t have eager compilation, after all. It would be nice to be able to opt-in to eager analysis actually, but that’s a separate aside, I don’t think anyone wants the compiler burning clock cycles creating object code which won’t get used in the program.
My initial thought is that the same basic logic applies to dependencies: it might be useful to eagerly fetch them on an opt-in basis, so that offline builds would work if the code using the dependency didn’t add it right away. But it would be fine if, from the build system’s perspective, they were all lazy.
No we don’t need eager dependencies.
We need lazy dependencies for platform specific code, for testing specific code and other reasons.
There is no need to mark dependencies as lazy – all of them are lazy.
There is no need to have a different API for getting lazy and eager dependencies – either you need a dependency and must fetch it or you don’t need it and can skip its fetch.
But this problem is for zig core team to solve, and the solution is really easy.
Naturally I’m inclined to agree! But if I’m making some wrong assumption somewhere, this is a good way to tease that out.
I would barely call it a problem, and I do think being able to trigger an eager fetch has some utility. But if the build system can get away with one kind of dependency where it currently has two, then it probably should.
I agree with the general premise that there’s no need for eager dependencies at all and that the build system would probably be better of if everything was always lazy (with some optional means of pre-fetching all dependencies if needed). The main problem with the current implementation of lazy dependencies is that they aren’t lazy enough.
If your build.zig uses if (b.lazyDependency("foo", .{}) |foo_dep| {} to resolve a dependency, the resolution that forces Zig to fetch the package and recompile the build runner happens immediately after the “configure” phase and not during the “make” phase. This means that if you have
a default install step that requires no dependencies, and
a test step that requires some lazy dependency,
and you invoke zig build, you will still fetch the dependency, even though it’s not needed.
This problem extends to packages meant for consumption as a dependency; if the modules/artifacts you expose don’t require any dependencies, but you have some test or example steps that do, all users consuming your package will still need to download the unneeded test/example depdencies. As I understand it, this is the problem that @FObersteiner was running into in the original post.
It would be much better if the resolve/fetch/recompile mechanism happened during the “make” phase, when a step requiring some not-yet-fetched dependency is encountered. This would require rethinking and redesigning certain aspects of the build system. For example, b.dependency would need to defer checking whether a dependency actually exists until when the relevant step is processed by the build runner, and functions like std.Build.Dependency.artifact/module would need to return some kind of LazyArtifact/LazyModule which starts off unresolved but gets resolved and assigned its relevant data when first needed (similar to LazyPath). We could then likely consolidate b.dependency and b.lazyDependency because they would both be one and the same.
But even with those changes, one problem that remains is lazily @importing the build.zig of a dependency. b.lazyImport, which I helped implement and which is to @import what b.lazyDependency is to b.dependency, makes it possible to conditionally import a dependency during the “configure” phase based on some user-specified option, but the API is a bit of an awkward hack and it has the same “not truly lazy” problem as mentioned above where you might end up fetching the dependency even when the particular step the user invoked doesn’t need it. And unlike the runtime APIs, it wouldn’t be possible to consolidate @import and b.lazyDependency because the result of an @import needs to be known at compile time (for obvious reasons).
On the other hand, I’m not sure if lazily @importing a dependency is that much of a common use case that it needs to work elegantly. This type of usage is mainly used by “SDK-like” packages that are often a core element of an application and unlikely to be needed conditionally, and if std.Build.Dependency was additionally enhanced with the ability to export arbitrary simple values (version numbers, strings, etc.; some sort of LazyValue mechanism is desperately needed for e.g. config headers), the few non-SDK-like projects that currently need the @import mechanism would have a good lazy runtime alternative.
One of the sneaky reasons I branched this thread off was to lure someone in who thoroughly understands this corner of the build system. Documentation is lacking, understandable, things are in flux, but I’ve been cargo-culting like almost anyone else here. Thanks for helping us understand this stuff.
My ideal is optimal laziness. It costs at most one cache miss when code which needs a dependency is reached, and… I’ll cover why I think this is good in a second.
I think this can be late-bound at the cost of starting over sometimes. I don’t know if that’s actually a good idea or not, I think it is though.
I’ve seen transitive dependency become a problem over and over. I can’t actually think of a software ecosystem where it doesn’t rear its ugly head. There’s always a dance to try and get optional enhancements which work with a specific package, but only if you’re using that package, and those systems generally work poorly.
But all of those languages, so far as I know, work on a principle of eager compilation, using a later tree-shaking stage or link-based elimination to winnow the code down. The Zig thing where it refuses to even look at code except to make sure it parses, until something calls it, is borderline unique, and it really could tame all that complexity.
My test case here is I’m working on a library, and I’m going to add a few functions which take an mvzr regex. But the library itself doesn’t need a regex at all, unless you call one of those two or three functions. Ideally, until the instant the build system tries to compile those functions, it doesn’t fetch that dependency, unless the user does something like zig fetch --all (which does not exist).
That seems achievable if we’re willing to have a build fail because a dependency has to be fetched, and then started over. I think the effort expended would pay off, though I could be wrong. Something like: the build system optimistically (or pessimistically if you prefer) stubs any dependency which isn’t hashed into the cache as a trap, and if code hits that trap, then compilation stops, it signals back to the build system, which reconstructs everything using that dependency.
Maybe it should finish Sema and then fail, it would be tedious if there were fifteen dependencies and the code actually uses all of them, and the build took sixteen tries to succeed. Even for a first time build, that isn’t ideal.
It would be a good outcome though, I’m fairly convinced of that. If someone offers a module a PR which integrates some other beast of a codebase with some optional features, it’s easier to say yes to that if you know that users who don’t employ those features are never going to download them to begin with.
That is what I experienced. I had this warning in the build.zig (which shouldn’t have been there in the first place, I’ve found a better solution), and it was triggered no matter what build step was selected. The warning was part of the “build examples” step, so it made no sense for it to appear if I ran the test step for instance.
Regarding dependencies in general, following @dimdin and your advice, I have “protected” them with an option - which works fine I think if you have a library that can optionally build a binary, but it would probably be awkward if it’s needed for tests.