Comptime and Incremental Compilation

[mlugg]

@mnemnion

Continuing the discussion from Comptime unique IDs:

@Sze

Global counters at comptime if they work somehow, are a bug that you are exploiting. Using such bugs, are counter to the efforts of getting Zig to a state where it can incrementally compile things and compile more things in parallel.

Worth pointing out a thread about this very topic already exists, with a relevant issue about why, even though you can probably make this work in 0.13, it should be considered a latent bug to do this.

As far as I’m aware, there is no longer a way to do this. If you find one, please do say.

The issue you link is essentially solved, albeit with slightly different semantics. That issue proposes some stricter restrictions which we may or may not apply, but that’s not related to incremental compilation.

@mnemnion

My hope is that, as developer attention turns from building out the stage2 compiler, to adding incremental compilation, that will be an opportunity to start specifying comptime semantics in detail. It’s Zig’s signature move, after all, but the way it works right now is rather ad-hoc.

I fear we’re facing a trilemma here, where the horns are:

• Zig has comptime, which lets you do things while compiling which only a Turing-complete language gets you. To put it more directly, Zig has comptime, a feature which lets you use Zig while compiling (with some wiggle room for necessary differences in semantics).
• Zig has incremental compilation, and eventually, hot loading: you can work on a Zig program while it’s running, and the compiler will a) only recompile what it has to and b) patch the new object code into the runtime.
• Zig’s incremental compilation is deterministic and consistent: it always results in the same program as a full compilation, modulo details which don’t affect execution.

You seem to think the interactions between these features are unsolved problems. The relevant changes in the 0.12.0 release cycle have all but solved this problem. We have modeled incremental compilation in such a way that no more major language changes are probably necessary, with the possible exception of removing usingnamespace.

@mnemnion

My bias is that I consider a powerful comptime essential, it’s easily the #1 reason I’ve committed to the language and intend to make heavy use of it for many years. But while I see the appeal of incremental compilation and hot loading, and would love to see that added to the language, I don’t need it or especially want it, and certainly don’t want it to cripple comptime.

Even ignoring the aforementioned fact that no more big changes to comptime are necessary, I’d also like to note that Andrew has explicitly stated his willingness to change the language if necessary to make incremental compilation viable. To be blunt, if they’re necessary, these changes are going to happen.

Also, global comptime mutable state was always a bug. It worked in often unexpected ways, and made compiling Zig code non-deterministic. This “feature” is not coming back.

@mnemnion

The characteristic trait of a Turing complete language is that it isn’t decidable. This is, to put it mildly, a barrier to deterministic incremental compilation of code which has comptime components.

Why do you deem this a problem? Decidability is completely tangential to incremental compilation.

@mnemnion

It’s a heavy-duty and gnarly manifestation of cache invalidation.

There’s some way to thread this needle, but I don’t think we know what that is yet. I don’t expect that limiting mutable state to particular scopes will turn out to be sufficient, which might mean it isn’t necessary either.

We do know what it is; we’ve done it. The incremental compilation model in the Zig compiler should work correctly under the current semantics.

@mnemnion

There’s two paths forward here: one is taking the time and energy to specify comptime semantics, to decide what they even should be. The other is to barrel ahead with incremental compilation without doing that, and put out fires caused by comptime in a piecemeal way, declaring the result to be the semantics of comptime.

As discussed above, we can and will change the language as needed to make incremental compilation work. From this perspective, nailing down the precise semantics before we’re sure of what works would be a ridiculous move.

@mnemnion

I think maintaining a unique global counter is representative of an entire class of behavior which comptime should enable. That raises a whole host of questions even before incremental compilation enters the picture.

Removing it from the language before even trying to answer those questions would be a mistake.

Global comptime counters are not and will never be an intended feature of Zig. We are not going to design a language where the behavior of the program at compile-time is hugely implementation-defined (and in fact non-deterministic once we thread semantic analysis). This matter is entirely settled and has been for a long time.