Package index service

Brainstorming
,
1

At the moment there is no way to just browse packages. What could could be done about this? The way the package system works is genius already, as all that needs to be done is really just a list of links or way to find repos.

My idea would be to make some sort of convention for a tag that people are encourage to use in git hosting services. This would be very accessible and easy. The downside would be that packages hosted on e.g. Codeberg wouldnt show up for most people.

The alternative would obviously be a website, tcp server or something along the lines.

2

Interesting question… I wonder if we could do something about with #package tag? @Sze has done a lot with our tag system so I’d defer there.

First, I’m assuming that when you say:

…that you are referring to Zig packages/projects in particular.

In terms of what we can do as individuals, it’s hard to say. We have project lists that are use-ready, but those tend to be a smaller group of packages that already have recognition. If it’s supposed to expand package awareness, that’s something users would need to provide.

1 Like
3

Yes, and I agree with the idea of making a tag here too, it would be very very helpful. There is actually a package index, but its for zigmod packages and not the official system, though some projects listed there also support that. I wouldnt consider this a solution though, as any project not compatible with zigmod cant be found there (https://aquila.red/ is the website, I actually like the idea and how its made (its simplicity etc. but yeah it falls short regarding the aspects I have listed)).

4

I think @markus is talking about establishing the convention of using one specific tag on external sites like github, gitlab and codeberg, so that zig packages can be found there via that tag.

I think it could be useful, but it probably would only be used by some people, but maybe we can reach a high percentage if we promote the visibility of that tag, or maybe it could even snowball if enough people use it.

I don’t really think I like the idea of some package index service, because I think there would quickly be 3 alternative half baked competing implementations and also it would not do that much, except crawl a bunch of hosting services for existing packages. Zig specifically tries to avoid centralized package management, so you would always have to deal with getting packages from multiple sites and discovering those sites.

I think one alternative could be to just create more Awesome Lists:

One thing that may make sense is to talk to creators / curators of popular awesome lists whether there could be some way that we can write a script that basically merges and deduplicates all those lists. But I guess that would only work if those lists are similar enough and I don’t really want to create anything that pushes additional work on anybody.

So for that merging thing the question would be whether the curators are interested in merging their list with other lists, if they aren’t interested, people could still write a script themselves that does that locally, sort of building an index of awesome lists.

Theoretically such a script also could use some apis to do searches based on some tag. All of that sounds like a community coordination effort, that needs to be carried by at least a few people who actually can invest some time in that. Creating some thing once and then letting it bit-rot doesn’t really help anybody. So I think it would take some people who actually want to commit some steady time to do that.

Personally I think I would be most interested in some simple tool that just helps you in building a localized index, collected from an extendable number of sources, basically a tool with plugins to search github, gitlab, sourcehut, codeberg, etc…

2 Likes
5

Topic zig-package is used in github but not so much in codeberg.
Topic zig (and ziglang) is also heavily used but the name excludes C packages for zig.
Topic package gives results for typescript, python, lua, etc.

I prefer the zig-package for the topic name.

We can link to various repositories and we can ask the various zig awesome lists to do the same.

I’ll love to have a centralized index (for example a json file in a git repository). An entry could be added by everyone with a link to a repository that have a build.zig.zon. We can also batch insert entries periodically from the zig package topics, or from the awesome pages links.
Once we have the index we can build ui that lists, searches, have dependencies and package usage (reverse dependencies).

6

I think a centralized index should be avoided, to avoid situations like these: When "Everything" Becomes Too Much: The npm Package Chaos of 2024 - Socket

But a merge-able and fork-able list in a git repo, would be a decentralized index, I think that might be ok, would effectively allow everyone to maintain their own fork or follow someone else.
Maybe instead of json, it could use zon or ziggy.

Found this:

The packages seem really old, but maybe the site could be freshened up and populated with newer packages? Probably would also need a way to search for supported/wanted zig versions?

2 Likes
7

I don’t really think I like the idea of some package index service, because I think there would quickly be 3 alternative half baked competing implementations and also it would not do that much, except crawl a bunch of hosting services for existing packages. Zig specifically tries to avoid centralized package management, so you would always have to deal with getting packages from multiple sites and discovering those sites.

While I dont disagree with you here, its also just the nature of anything decentralized / federated. Personally, I dont really see an issue with this otherwise, one service will simply become more popular and probably stand out as the norm, though that would still mean one would have to use several of them to leverage all packages of the ecosystem.

Personally I think I would be most interested in some simple tool that just helps you in building a localized index, collected from an extendable number of sources, basically a tool with plugins to search github, gitlab, sourcehut, codeberg, etc…

Easily my favourite Idea so far, this could be very simple, be just a bunch of api wrappers and some simple arg options so you can search well and one would be done with it for the most part. The only requirement would be to motivate people to use the zig-package tag on their packages.
This would obviously be the biggest effort here, but should be very possible with just something like e.g. putting it on the zig website build system guide as a convention.

2 Likes
9

I must preface by saying I’m a new guy, I don’t think my opinion is worth more than half of anyone in this branch, yet, here’s my 2 cents -

I don’t think this is indicative of centralized indices in general, rather of the policy implemented by npm in the wake of the left-pad debacle.

Had they not implemented hard dependencies by disallowing removal of packages that other packages depend upon, this would not have been possible.

Also, in my humble and very unoriginal opinion, the main task for any potential package index would be keeping track of whether the packages compile with either current release of Zig (0.11 at the moment) or the master Zig (0.12-dev-xxxxx ATM).

If a package is left untouched for several months, let alone a year or two, chances are you won’t be able to just use it.

AFAIR, I’ve read about Rust guys actively trying to compile all the published Rust code they know about, when a new Rust version comes out. The process was automated, of course. I could be wrong, since that I read about it a long time ago, in the pre-pandemic times.

Anyway, just having a list of “everything we could find” wouldn’t be of much help.

P.S. I’m really wary of external libraries, packages, etc. in any “real” project I take part in. Unless truly necessary, I stick to standard libraries as long as I can.

1 Like
10

There are only for or five large public code repositories. I think a cli tool to search all of them at once for zig packages matching a query should be pretty easy to build.

Add the ability to get a list of available releases and/or branches and corresponding tarball URLs and you have something more usefull than a centralized package index.

2 Likes
11

Thats a very good consideration. I will try to hack something together in the coming days, (or weeks, because I have a lot to do at the moment), this will be something I might look into after getting the basic functionality done

2 Likes
12

A proposal for a decentralized index for PostgreSQL, inspired by the one in Go: Contemplating Decentralized Extension Publishing

13

Hello, I’m very new to Zig but have researched the Zig package manager. I’m down to having a centralized repository due to the benefits of UX and exploring new packages. I’m willing to contribute to or develop the program if anyone has the same idea and passion. Feel free to contact me. :slight_smile:

1 Like
14

The scraper is ready! It scraps github for zig-package and zig-library.
The git central repository is ready! It is called zpm and anyone can send a pull request with a package.
The zpm repository is searchable here: https://zpm.random-projects.net/

2 Likes
15

While this is quite nice, it lacks the ability to check if the package is officially compatible (has build.zig.zon etc) and also requires updating of the repository. Ideally, the programm would scrap github codeberg and some other services itself, so it wouldnt require anyone to even interact with that github repo imo. This would have the advantage of just allowing anyone to use any package without imposing another task upon package maintainers

16

I have been trying to make the webscraping version for a few days now (very on and off, though, because of school) and cant find a way to to scrap github effectively. This would, unfortunately only leave me with the option to make a website based service, which is currently absolutely not part of my expertise and comfort zone.

Scraping codeberg, however, wasnt a great experience but worked just fine.

17

What I could, however, try to do would be something like a repository and a cli tool that lets someone conveniently submit a package to it and query it for certain keywords