I’m running up against two changes in Zig 0.17.0 that I didn’t see in the release notes:
@Vector types are no longer allowed in extern structs/unions or as parameters to functions that have callconv(.winapi), due to not having a defined memory layout.
Pointers to @Vector fields are now aligned in such a way that you can’t pass them to a function that accepts a *f32 pointer.
These are both very hard for me to change in my project so now I’m looking for excuses to refactor my vector math to use arrays of floats rather than @Vector types. So I’m curious if I am abusing @Vectors in my current implementation of vector math. There’s no need to read the whole file, the first Vector2Type function is enough to understand what I am doing. I originally did this thinking that there would be a performance advantage to using @Vector when doing vector calculations since all elements of the vector can be calculated at the same time, but I’m not convinced that there would be a big difference anymore. If I end up refactoring I can obviously compare the performance, but since that’s a somewhat time consuming task I wanted to get some more opinions on this before I go down that route.
There’s a reasonable chance your previous code with @Vector was already suboptimal compared to an array. When you use @Vector, the compiler does not do any optimizations. It fully trusts you to write optimal code. The problem is that a lot of naive implementations of vector operations are suboptimal compared to what the compiler can do. When you write a loop over an array, the compiler detects the naive implementation and improves it. When you use @Vector, that doesn’t happen.
I had a similiar issue, but after learning about vector - array coercion I think it makes more sense the way they have it than what I did originally, here’s a migration guide, just store [N]T instead of @Vector(N, T) and use coercion when you want to do anything with SIMD.
I checked the assembly in my project, the only difference is that LLVM will use
vmovdqu instead of vmovdqa, which is basically the same command but with different layout assumptions. AFAIK that’s the same performance.
If you want to force the vmovdqa then you can just use [*]align(vectorAlignment)[8]u32 Compiler explorer example
Edit: TLDR - store your data in arrays instead of vectors and pretend they are vectors using coercion.
This is wrong. Most optimizations act differently on an array vs. a vector (I haven’t looked into this that deep, but I could imagine that e.g. sroa does not work on vectors), but some optimizations work with vectors (e.g. turning vec * splat(2) into vec << splat(1)).
It does feel to me like this is a very hairy part of the language. I personally don’t think declaring something as a vector should tell the compiler which instructions it should use to process the data. It’s saying this data structure should be interpreted as a mathematical vector, just as @Complex() would (if we had one) say treat these values as a complex number or @Matrix would say treat it as a matrix. It’s then up to the compiler to decide how to do the work efficiently. SIMD or not.
I think there’s a proposal to separate SIMD semantics from vector semantics, but I can’t put my finger on it now. I think it’s desperately needed though.
There are many areas for which zig is used, that use the word vector for something (simd vectors, vectors in the mathematical sence, people coming from c++ might think vectors are arraylists). Currently in zig a @Vector is lowered by the llvm backend to the llvm vector type which is very much simd related. AFAIK other backends either scalarize @Vectors or also use them for simd (spirv might be an exception, but I’m not sure).
One problem with the current state is, that authors of e.g. linear algebra libs assume @Vector is the correct type to use, which is sometimes true, but also often wrong.
I don’t understand what you mean with “mathematical vector” vs “simd vector”
simd is often use to speed up math heavy code ?
in my experience it helps the compiler generates better code, because eg when the input is []Vector(8,f32) instead of []f32 it skips the épilogue that handles odd sizes which never happen in practice.
SIMD vector are instruction where the registers are big, like cache line big. Basic use is loop unrolling, more advanced space includes parsing. IMO, in the context of zig, it’s about forcing to use those special CPU instructions around those very large registers without platform specific assembly.
Mathematical vector would be related to geometry/graphics/linear algebra: matrix multiplication, dot product, cross product, scale (scalar * vector) and add (vector1 + vector2). IMO, in the context of zig, it would primarily be about ergonomics, but it would also compile to SIMD instructions very often.
The point OP made is that you might do a simple loop on @Vector(4, f32). Using the @Vector(4, f32) might stop the compiler from unrolling the loop and essentially operating on @Vector(8, f32) or even @Vector(16, f32). It’s good that it’s this way, because you might want to benchmark, but if you just want to go fast, leaving the data in an array leaves more room for the compiler to generate faster code (ironically, not in 0.17).
My point is that @Vec(8, f32) already matches the semantic of linear algebra. What more do you want?
The argument “let the compiler optimize” is really not convincing to me. You just linked Hashimoto blog post that highlight what you can achieve with simple Simd code. And simdjson is another category which is just leagues ahead of what LLVM could ever dream to “auto vectorize”
I’m genuily curious. I wrote some linear algebra code in Zig this year using Vector and thought it was great (a part from @reduce who generates horrible code). What am I missing?
I feel bad drifting this thread, and also I might’ve misrepresented the OP, because it’s something somebody else said.
I think the opening write up is better than what I can come up with, e.g. swizzling, part of GLSL.std.450 as std/builtins (including length). It’s mostly about ergonomics, in std/userspace. Maybe no new builtin at all.
It is prohibited to use a math operator on a mixture of scalars (individual numbers) and vectors.
It basically does a Hadamard product and not a dot product which I guess most would expect if they multiply two mathematical vectors. Not to mention that there are other products defined for mathematical vectors, like also with scalars which are explicitly disallowed in Zig.
And yeah dividing, sqrt, bit-shifts and so on are not really possible with mathematical vectors.
I think there’s a proposal to separate SIMD semantics from vector semantics, but I can’t put my finger on it now. I think it’s desperately needed though.
What are the “vector semantics” that are different from “simd semantics” ?
In pytorch when you do A * B where A and B are vectors you get the same thing than in zig when you do vec_a * vec_b so it’s totally reasonable and expected.
dot is kinda weird because LLVM/simd isn’t that good at reduction. As a result you want to accumulate individual using Simd instructions: vec_acc += vec_a * vec_b. But yes there could be a std.math.dot(Vec: type, x: []const Vec, y: []const Vec) std.meta.Child(Vec). Does that warrant a new type ?
All of this seems disjoint from my actual problems, like how do I get my zig code to use vdpbf16ps without inline asm ? or how do I force this intermediate results to stay in SIMD registers without ever being copied back on the stack or regular registers ?
I up this too, with the fact that many instructions are only available consistently through inline assembly or llvm_intrinsics. And Zig ought to be able to allow people to express them, without reaching for inline assembly. For example in one project I’ve made, I had to use @"llvm.x86.ssse3.pshuf.b.128" to get the correct instruction. Because Zig wasn’t able to generate it, despite the fact that the lut associated, was comptime generated, and comptime known.
This approach worked well and ended up being a pretty small refactror. I didn’t notice any impact on the total performance of the application, but that makes sense since the actual calculations are still happening on vectors.
I am bit surprised that coercion worked though. If the original issue was that vectors don’t have a guaranteed layout in memory but arrays do, how come an array can be coerced into a vector without running into the same error? I was under the impression that coercion was just reinterpreting the same memory as a different type with no actual changes to the shape of the memory. Does the coercion being present force the compiler to give the vector the same layout as the array while a vector on its own could have various layouts depending on what the compiler choses? The docs touch on this by saying that coercion between vectors and arrays works via an implicit @bitCast which works because vectors have well-known bit layout, but that they don’t have a well-known byte layout. I don’t understand how that is possible when bits are smaller than bytes, if the bit layout is known I would expect the byte layout to be known also. What am I missing here?
I am not the best at explaining these things, I will try to show an example and hopefully that works
To give an example of different bit layout vs memory layout consider a [4]u6 , the “byte” layout as you see is 4 bytes, so that would be 32 bits.
However, the “logical bit layout” is 24 bits, because for each of the u6 there’s 2 bits of padding.
If we have for example a @Vector(2, u12) then I honestly have no idea what the memory layout of that thing is, but we can understand that it’s logical bit layout is 24 bits, hence bitcasting works. Does that make sense?
How is that implemented is I guess the mystery where you need to know target-specific information, so that’s something that the compiler can easily do because it knows what it’s targeting, AFAIK this would be very difficult to write by hand.
Related devlog on @bitCast
What I personally like about this, is that the new bitcast is something that’s very hard to implement in userland, whereas for a use case for the old bitcast (for example when dealing with extern structs) we can hand roll the old behaviour.
// Userland implementation of old bitcast
fn bitCast(T: type, value: anytype) T {
const N = @sizeOf(T);
if (@sizeOf(@TypeOf(value)) != N) @compileError("Bitcast must have equal sizes");
var ans: T = undefined;
const ansBytes: *[N]u8 = @ptrCast(&ans);
ansBytes.* = @as(*const [N]u8, @ptrCast(&value)).*;
return ans;
}
// Example usage is to convert from a extern struct {a: f32, b: f32} to a [2]f32
// The new bitcast won't allow it
I want to say another thing about the performance point. I haven’t tested It, but I think we can confidently speculate based on the hardware that what I am about to say is true.
I think there might be even use cases where the array approach would perform better than the Vector approach.
There are 2 scenarios:
@Vector(N, T) has the same size as [N]T in this scenario we should observe basically identical ASM, since they are the same thing with the same memory layout, so LLVM should not produce inferior code for a [*][N]T, the only difference would be alignment (which you can change on the array) but since AFAIK aligned and unaligned loads have the same performance, that’s optional.
They have different sizes, in that case since the array is as packed as it can be, that would mean that the Vector is bigger. for example a [6]u32 vs a @Vector(6, u32) in this case the array version would perform better, because it’s more packed in memory and we will have better cache locality.
My conclusion is that there’s no situation where storing long term in arrays in memory is inferior to storing in Vectors.
Yeah, I think I get it now. For some reason I was thinking that any bit layout can be expressed in a byte layout, but that would require fractional bytes which doesn’t make any sense. I’ll have to read that devlog, but I think I assumed that @bitCast worked like the old version that you showed a userland example of.
Thanks for taking the time to explain it, I’m not accustomed to thinking about memory layout.