Zig Autodoc : files from cache and standard lib in the tarball?

FObersteiner · April 16, 2025, 5:47pm

Hi all, quick question regarding the autodoc feature.

I noted the rather large size of the sources.tar file it creates for zdt, and seems like it contains files from the cache plus the standard library:

zdt on  dev-docs [$!] via  v0.15.0-dev.345+ec2888858
❯ tar -tf ./autodoc/sources.tar | rg cache
zdt/.zig-cache/o/ef8801f69d257a70292db7d82f8db66d/cimport.zig
zdt/.zig-cache/o/073e00f5c2234235d9da0a88c470acc9/cimport.zig
zdt/.zig-cache/o/f7fafbc9e4d92277c266ff50c5709203/cimport.zig
zdt/.zig-cache/o/7574969272d98d01ce6dc15ad005950d/tzdb_prefix.zig
zdt/.zig-cache/o/4ddd0e8ccb65d370d6250531c65a1aea/cimport.zig
zdt/.zig-cache/o/79cd4feb187fef06fea0e22482028705/cimport.zig
zdt/.zig-cache/o/cd83cc1350dae514c0bdabe80c1b1715/dependencies.zig
std/valgrind/cachegrind.zig

Why? I don’t think I can (nor would I want to) access the std lib docs via the docs of some package? Also, it’s unclear to me why those cached files went in there - they do not seem to have any relevant documentation for the package?

dimdin · April 16, 2025, 6:37pm

I think this is a bug; autodoc could exclude sources in .zig-cache.
A workaround is to move zdt.zig in lib and use lib/zdt.zig as root source.

floooh · April 16, 2025, 8:32pm

FWIW I wrote a little quick’n’dirty tool which extracts ‘just the sources’ and repackages them:

github.com/floooh/sokol-zig

tools/fixdoctar.zig

master

//! repackage the autodocs sources.tar to only contain sokol sources
//! (which reduces the size from 13 MBytes to abour 500 KBytes)
const std = @import("std");
const Allocator = std.mem.Allocator;

pub fn main() !void {
    var arena_state = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena_state.deinit();
    const arena = arena_state.allocator();

    // parse args
    const prefix = try arg(arena, "--prefix");
    const input_dir = try arg(arena, "--input");
    const output_path = try arg(arena, "--output");

    // iterate over sources.tar file, find relevant files and write to output tar file
    const inp_path = try std.fs.path.join(arena, &.{ input_dir, "sources.tar" });
    const inp_file = std.fs.openFileAbsolute(inp_path, .{}) catch |err| {
        fatal("failed to open input file '{s}' with {}", .{ inp_path, err });
    };

This file has been truncated. show original

This is then built and called in build.zig like this:

github.com/floooh/sokol-zig

build.zig

7683a155d


      
          //== DOCUMENTATION =====================================================================================================
          fn buildDocs(b: *Build, target: Build.ResolvedTarget) void {
              const lib = b.addStaticLibrary(.{
                  .name = "sokol",
                  .root_source_file = b.path("src/sokol/sokol.zig"),
                  .target = target,
                  .optimize = .Debug,
              });
              // need to invoke an external tool to inject custom functionality into a build step:
              const tool = b.addExecutable(.{
                  .name = "fixdoctar",
                  .root_source_file = b.path("tools/fixdoctar.zig"),
                  .target = b.graph.host,
              });
              const tool_step = b.addRunArtifact(tool);
              tool_step.addArgs(&.{ "--prefix", "sokol", "--input" });
              tool_step.addDirectoryArg(lib.getEmittedDocs());
              tool_step.addArg("--output");
              const sources_tar = tool_step.addOutputFileArg("sources.tar");
              tool_step.step.dependOn(&lib.step);

This file has been truncated. show original

IIRC it reduced the sources.tar from ca 13 MBytes down to 0.5 MBytes.

FObersteiner · April 16, 2025, 10:43pm

thanks, I might just shamelessly steal this, with proper attribution of course Still, I wonder why autodocs always (?) puts /builtin and /std into the tar - what’s actually creating the html version, is it this piece of code?

floooh · April 17, 2025, 8:48am

Don’t know, at first I was thinking that maybe it just follows all imports and that’s how the stdlib ends up there. But I just carefully removed all std imports from my bindings project and the sources.tar still ends up 14 MBytes in size.

It might be an intended ‘feature’ so that links to stdlib types work.

FObersteiner · April 17, 2025, 10:54am

right, I think I get it now. If you import and use something from std (or any dependency in fact), you can open it in the autodocs pages. Still unsure why there’s also stuff from the cache in there, but @dimdin’s solution worked.

I can imagine there are cases where this is useful. I’m undecided yet if I miss a flag to limit the scope to the current package - the tarball can become quiet heavy I guess, if you have many dependencies.