Packed structs are represented in memory using a backing integer (u24 in your case). Access to fields is essentially a convenience feature around bitshifts. This means the backing integer is subject to the same alignment rules as regular integers (u24 will align to 4 bytes). This means you will have padding in the most significant bits (highest memory address for little endian systems, lowest memory address for big endian systems).
Some more reading:
Better documentation about the memory layout of packed structs is needed, but nobody has gotten around to it yet.