Atlas: Package indexing and management utility (without reinventing the world)

Brainstorming
1

TL;DR

I’ve been prototyping Atlas, a package index and CLI for Zig focused on package discovery, immutable release records, and managing build.zig.zon. It’s not a replacement for Zig’s package system, does not create its own configuration for packages, does not include a different installer, or anything like that. In essense, it’s how I think community-based package discovery/management in Zig should work.

I’m sharing it as an early idea/prototype and would like feedback from y’all. :slight_smile:

Long(-er|-ish) form:

I’ve been enjoying Zig SO MUCH, but one thing I keep missing is package discovery. I used to use CPAN (for Perl) and such systems, essentially author-driven indexing systems. As an author, you upload a package and manage its listing, and users can search and download.

There are already some forms of it:

Other than Awesome Zig (which is a curated list), these are closer to package indexes, but I wanted to take a different approach. An index that is not tied to any hosting (GitHub, CodeBerg, BitBucket, etc.), has explicit package/release ownership concepts (that authors drive), stores release records, and provides a similar utility to cpan, yum, et. al. without trying to replace any of Zig’s existing package tooling.

The basic idea:

  • build.zig.zon remains the source of truth for dependencies and zig still does fetching and installing.
  • Atlas helps discover packages, resolve package/release metadata, and handle dependency entries.
  • Packages are not assumed to live on GitHub. You can host it anywhere. (In the future, maybe on Atlas too.)
  • There are namespaces for packages, owned by users/projects/organizations.
  • Top-level “aliases” can exist, but are assigned through open community curation (project tickets).
  • Strict no AI policy, so the project is open and welcome to everyone.
  • Obvious, but free and open in every regard (client, server, governance, etc.).

I named it Atlas (as in a map of the package world).

The model is:

  • Packages live at package coordinates like namespace/package (similar to gh_user/gh_repo).
  • Releases live at release coordinates like namespace/package@version.
  • Versions are not limited to semver.
  • A release points to a source URL and Zig package hash, immutable once published.
  • The CLI can look up packages and add dependencies to build.zig.zon.
  • Top-level “aliases” to package/release coordinates, so people can use httpz instead of something like karlseguin/http.
  • These help avoid typosquatting.

I already have an early prototype working that can:

  • Register users (atlas register-user my_user)
  • Register packages (atlas register-pkg my_user/my_pkg) - add description, tags, etc.
  • Release packages (atlas release my_user/my_pkg@1.2.3 https://codeberg/...)
  • Fetch release info (atlas find my_user/my_pkg / atlas find my_user/my_pkg@2.1.1)
    • Search using package descriptions and tags not yet implemented
  • Add dependencies to build.zig.zon (atlas add my_user/my_pkg@1.1.2, uses zig fetch)
    • Supports special names (atlas add karlseguin/http --as httpz)
  • Remove dependencies from build.zig.zon (atlas rm httpz)

This is still rough, and I’m a Zig beginner (this project forcing me to learn so much). I’m not asking anyone to adopt it yet (or I would put it in “Showcase”), but I do want feedback!

I know I would want to have this tool, but I’m not sure what others think of it, and whether people have ideas and/or interested in collaborating on it.

So, what do you all think?

1 Like
2

How do you prevent somebody from registering using the name of somebody else and pointing it at their own fork?

3

How do you prevent somebody from registering using the name of somebody else and pointing it at their own fork?

Yeah, name squatting is tricky. The sites that just monitor GitHub are also susceptible to something similar: I can register on Codeberg the same ID as GitHub and once the indexes try to support Codeberg, there’s a conflict (or both appear).

I can only come up with three ways to deal with it:

  1. Make it possible for people to stamp a repo as “the real thing”, but that’s a bit tricky here. We can use generated tokens to help you verify that sze on Atlas is sze on Codeberg, but maybe the Codeberg user isn’t the authoritative user for sze. So the identity check is just bumped elsewhere. What we can do here is make sure that if sze/foo is pointed at codeberg/sze/foo, then whoever has access to it is the person who owns the sze/foo package on Atlas.
  2. Pre-allocate a bunch of them. That’s possible, but based on what? What if there’s a mistake?
  3. Use the Codeberg ticket system for authors to pose a fraud concern and remove that person’s association with the namespace.

What do you think? Do you have any good ideas of dealing with it?

4

One neat approach i’ve seen used widely with java and related packaging (maven, android apps, etc.) is using domain name ownership for “namespacing” packages, and use DNS records for validating ownership of the domain

so e.g. a package foo published by the owner of example.com would be under com.example/foo, reversed domain name parts, and then the package.

for github and codeberg, you could use the same approach, but instead of a domain name, you use a gist or a marker repo for verification, and the namespace would be io.github.username/page.codeberg.username (github/codeberg userpages). specifically avoiding com.github or org.codeberg here so that if codeberg themselves want to publish they can use the actual domain name.

1 Like
5

The part that I don’t like about it, is that it seems like a centralized index.

If it instead adopted an email-style-protocol, like email- or matrix-protocol servers, than everyone who hosts their own git repositories could host their own package-index-server right on the server they use for hosting. (And people could essentially use user@host to identify users)

That way you would have a decentralized solution where these indexing servers would collaborate on making things findable. And users could either trust some indexing server and make their user account there or host one directly on their own.

So overall I think I would want a decentralized protocol instead of one centralized server.
A hacky and possibly bloaty way to accomplish this might be to use some existing decentralized protocol and send messages over it, but looking at these existing protocols it might be possible to create a simplified version of them that only does this single thing. (For example matrix home servers seem to be quite resource hungry to setup) I am not sure whether there already is such a protocol.

Also seems a bit like rss+search/automatic mirroring.
Or another hack: publish special DNS records to discover servers that have Zig packages?

I don’t have expertise in decentralized protocols, but I think I would like to use one for something like this, better than some centralized server that then gets compromised.

6

This could work, except for meaning a person cannot move their package to a different hosting or it will break it for all users.

The idea is to separate “where you host it” from the name. However, the username is effectively the namespace. You own the namespace for you, so it gives a decentralized naming convention within a centralized index.

7

The part that I don’t like about it, is that it seems like a centralized index.

I think we see it differently. My issue is specifically missing a centralized index.

There are risks, but with a centralized index you knowing that there’s one location to find packages. That’s where you search, where authors share what they have - an authoritative, trusted place. Benefits of CPAN, npm, PyPI, and others is exactly this. You don’t tell users to “Google it” or “search GitHub” or “yeah, it’s not GitHub, it’s elsewhere.” With decentralized, you’re entering this world again and need to come up with protocols to overcome this very fundamental design problem, IMHO.

Using namespaces allows each user to have their own area so they don’t need to fight over general namespace, which improves upon CPAN, npm PyPI, and the likes.

In other words, that is what I think is needed, not what I’m trying to avoid.

8

Yes I think we fundamentally disagree, I don’t want something like npm, I see it as a supply chain attack waiting to happen.

Having different ways of finding things also makes it more difficult for attackers to trick all users at once.
If there is no decentral way to have good discovery and convenience, than I would rather not have it all.

1 Like
9

My preferred solution for that would be to use content addressed urls that identify the package, so protocol://content-hash becomes what fully identifies the content where protocol is some decentralized protocol that doesn’t use location based addressing, would also make it easier to automatically download things from mirrors which are close to you.

So basically ipfs.

10

Thanks for expanding on this. I think I understand your concern better now, and I definitely don’t want Atlas to become “another npm but for Zig.” I put effort in leaning into Zig’s perspective.

If I read you right, you’d rather avoid a shared discovery/index layer unless that layer itself can be decentralized, even if discovery becomes less convenient. Is that fair?

The distinction I’m stuck on is:

  1. Did I get the bytes I expected?
  2. How did I know which bytes/package/hash I wanted?

AFAICT, IPFS helps with fetching/mirroring content by hash. But doesn’t Zig’s package hash already cover the “did I get these bytes?” part?

The harder part seems to be discovery. If I want httpz, how do I know which source/hash that means? Does IPFS solve that, or does something still need to map httpz to package/source/hash?

Right now that mapping is basically GitHub search, Google, awesome lists, GitHub-only indexes, stars, blog posts, copied URLs, etc. That decentralizes the threat, but also leaves no coordinated place to defend users. If everyone installed packages by passing around torrent links, that would be decentralized too, but it’d be easy to get a Shrek movie (or something malicious) instead of an HTTP library.

Atlas does add a trust point, so yes, that’s a real risk. But maybe it can improve discovery first, while being designed for stronger verification later: signed releases, namespace/package keys, public ownership history, warnings on key/alias changes, maybe content-addressed mirrors.

Let’s not let “best” be the enemy of “good” (or “better”). When the friction is too much, people resolve to what works, even at their peril. I think Atlas would work better than “just search and hope for the best” that we have now.

I may still be missing how you imagine decentralized discovery working. If there’s a way to solve the name-to-package/hash mapping without just moving trust somewhere else, I’d genuinely like to understand it.

11

Have you considered just using DNS? DNS already provides an authoritative way to say “I am this address” and also you can control where that address points.

ATProtocol does something like this: Identity - AT Protocol

There’s already a forge using this idea: https://tangled.org/

I’m not saying “you should use this (atproto)” but more of “there are maybe some interesting and relevant ideas in here you might find value from”

12

That’s an interesting direction, thanks. I didn’t think of AT Protocol or Tangled.

My first read is that AT Protocol is relevant to the discovery side: portable identities, signed records, and indexers.

Something like:

@some-author.myexample publishes signed records:
  package: foo
  release: 1.2.3
  source: ...
  zig hash: ...

Seems useful for verifying “this release came from this identity.” What I don’t see yet is how it answers “is this the identity/package I meant?”

Atlas could become more of an index/client over signed author-owned records (which reduces central authority), but aliases, search ranking, typosquatting, and “which foo do I want?” still seem an issue. Could be just my lack of familiarity with AT Protocol/Tangled, though.

The way I go back to seeing it is users still don’t start with knowing the author. They look for an HTTP library or for foo. Some shared discovery layer still seems needed for aliases, ranking, warnings, and typosquatting defense. Decentralizing doesn’t fix it. Without a central authority, a malicious package can be validly signed/decentralized and still fool users.

More importantly, no matter how good it is, if publishing is too hard, authors are less likely to use the system. :confused:

13

I added Tangled just as an example of someone in the space using the protocol with a git forge, because the idea abstracts out a bit in the package hosting space too.

The protocol basically describes a structured way that distributed systems can interact (and a way to define the actions a user can do) and interoperate.

I’m not going to give a full explainer of ATProto (because I would just do a bad job), but I do think there might be something there.

tl;dr: DNS gives you identity. DNS is portable and lets your backend be portable. The protocol describes actions (through Lexicons) users can do (publish package, revoke, report, etc.) and how other people can subscribe to those actions to build their own stateful apps (that also can produce action records!). So, you build search, ranking, reporting, etc. on top of a view of the events flowing through the system.

14

yes

One part that I don’t like about centralized aliases is that they mean that one person or group decides what they are and everyone else needs to accept them, if you don’t have influence over the group that decides, you can’t improve them without re-inventing your own alias system.

That makes me think, what if everyone could publish their own set of aliases, that way they could basically become like a list of favorite packages.

But it also might be nice to accumulate different people’s sets of aliases adding your own refinements.

Maybe we could have something like a priority list of:

union {
    mapping: struct { alias:[]const u8, user_project: []const u8 },  // .{ .alias = "httpz", .user_project = "karlseguin/http" }
    delegate: []const u8,  // "fictional-embedded-group/aliases"
}

Where a user of the index software could use the default which may be the alias list of the atlas project, but they can configure it to use some other alias list (which may be partially aggregated from a bunch of other alias lists).

I think the nice thing about this would be that:

  • it would weaken the probability that a supply chain attack successfully hits all users
  • the multitude of separate aggregations could make it more likely that people notice suspicious activity
  • people can go after their own interests and adapt it to their needs, potentially exploring different ecosystem compositions/“distributions” which could lead to a nice cycle of friendly competition of trying to provide the best user experience

I guess a negative could be, that now there are a multitude of options and some of those might not be put under much scrutiny or just may bitrot and not properly maintained. But as a linux and open source fan that is something I can live with.

So I think if atlas can be built in such a way that users could easily choose their own top-level aggregation of aliases then I think it would be better. That way different groups could collaborate on who basically reviews what kinds of packages and if people aren’t agreeing with what packages get picked as the default then they can create their own.

Personally I think it doesn’t need to be a fully decentralized or peer to peer system (I think it could be cool but I don’t see it as strictly necessary), the part I would find more important is that it isn’t fully centralized, so if it had an architecture where people can self-host a server and those can then cooperate, so I think a federated style where you can participate by having a user@host would be good enough in my opinion. I am pretty weary of services like discord which are all or nothing with no option to self-host.

Being able to self-host would also be nice in case somebody wants to use it for their internal tools and then you could imagine 2 people/organizations working together and linking up their separate instances via some kind of vpn link.

I think with the priority mappings, there could possibly be some sort of update/diffing algorithm which complains or gives special attention to alias-mappings that override aliases that pointed to somewhere else? Maybe you would have to explicitly state that you are overriding a mapping and otherwise it would result in an error?

Basically encouraging people to socially-organize their mappings so they stay strictly disjoint (at least at that level of tree depth) and are only merged by more meta/high-level aggregations.

15

I think a person working with Zig that doesn’t know all the major packages, the major authors, the community forums, etc., cannot easily answer “how do I do X” and “where can I find the most recognized package for Y”. I know because I’m that person. Decentralized systems don’t solve it either. So we end up Googling and checking package indexes.

The onus in such centralized system falls to the same people as decentralized - the community. We should build something that makes it easy for good players to act on that responsibility instead of seeing bad actors abuse it. That was my intention and hope.

This was just posted today: An open-source package registry where you can browse, search and actually discover Zig libraries and applications, which tells me the need is real. It’s the third or fourth of such an implementation, still having the same drawbacks that I’m trying to resolve with my proposal.

Decentralized solutions require greater engineering effort, create a higher bar for users (which means they circumvent them), and still don’t solve some of the key issues of trust and discoverability.

I was hoping this brainstorming would receive more interest, but that hasn’t happened. So maybe this isn’t what the Zig world needs. I think I’m just gonna let go of this idea. Not every idea is a winner. :slight_smile: I learned a lot by writing the code for it, so there’s always a silver lining.

I just want to say I greatly appreciate the feedback everyone (and especially you) provided here and the willingness to engage in it at depth. Thank you.

1 Like
16

We are aware of this issue and already have a number of ideas on how to resolve it; they are currently being developed, and I expect we will release a solution within the next two weeks.

Fun fact: we’re actually developing a utility similar to yours right now :slight_smile:

1 Like
17

Why do we even need aliases?

Make the author part of the search and do search via fuzzy finding.

So if I search “http” you get all packages (and authors) with “http” in the name.

And the index then just links to the actual source.

This way you could even search for authors and get all the packages by that author.

This would also make name squatting harder since only the author “doofenshmirtz” could put any package on the index where the author part is “doofenshmirtz”.

The only “disadvantage” (which I would actually consider an advantage) would be that one can have multiple authors publish something with the same package name.

1 Like
18

So if I search “http” you get all packages (and authors) with “http” in the name.
And the index then just links to the actual source.

That is the design. When you install, you install from an author.

This way you could even search for authors and get all the packages by that author.

Correct. That’s part of it. An author has as many packages as they want, and you can search by author (seeing all their packages) or by package of an author, or packages by any author that fit your criteria.

The only “disadvantage” (which I would actually consider an advantage) would be that one can have multiple authors publish something with the same package name.

Also correct. This is where typosquatting comes in and that’s why I wanted to have aliases. The aliases are Community-curated shortcuts.

Think of cURL bindings. The ones used nowadays are jiacai2050/zig-curl. Someone forks it into jiacai2055/zig-curl. Now which do you choose? You might know, but a newcomer doesn’t, so the community points and says “what you want is jiacai2050’s version.” So the newcomer says “install zig-curl and figure out which one I want.”

The idea was that the community aliases are assigned using publicly-visible and publicly-voted-on Codeberg tickets, so everyone can see what’s going on and we have a clear history of how a certain alias wa assigned. Imagine author passes a library to someone else and you want the alias to point to the new author - history helps with that.

1 Like