There was a saying back in the naughties that “The internet routes around damage”. Basically that high availability was a solved problem by having redundancy. Then everybody centralised everything.
Feels like we’re realising that was a mistake.
Maybe the new freenet could be an alternative p2p way to share code. Here two talks:
Freenet Lives! And You Can Use It!
https://www.youtube.com/watch?v=3SxNBz1VTE0
The talk walks through four applications you can use now: River for group chat, with private rooms newly added; Delta for publishing; Atlas for discovery and search, in its first working version; and freenet-git, which supports clone, fetch and push over Freenet.
I haven’t tried it yet, but freenet seems more general purpose than just code hosting, for example with their Atlas app those repositories may be searchable/findable too.
Freenet is written in rust but the contracts are compiled to wasm, so maybe we also could build a project that makes it easy to build zig-based projects that run on freenet.
Then long-term we could maybe even build a zig specific decentralized package index, what if we had: search, comments, reviews, ratings, issues & collaboration, wikis, docs, maybe even forums, all as a decentralised app (or a few) on freenet (possibly assembled from multiple projects)?
Maybe you could even have runnable examples, or interactive tutorials, if we can eventually ship a web build of the zig compiler. (Directly play/learn ziglings instead of its readme?)
Also:
Founded in 2001, Freenet is a 501(c)(3) nonprofit organization dedicated to the development of open and democratic digital infrastructure.
Before big corporation and things being centralized on GitHub, open source software was distributed through university FTP servers and these had mirrors by other universities or even companies. I don’t think a P2P system can be used for software distribution in general, and definitely not for development dependencies. There is a mismatch of incentives. People who want to download the software would have to keep running the publicly available service on their computer, that’s just not going to happen. And if you depend on large seed boxes mirroring the content, there are easier solutions. Stuff like the Go proxy model is better, in my opinion. For Go, it’s Google hosting it, but it could also be collaborative, just like Zig download mirrors are. Caching and proxying are much easier problems with more scalable solutions than anything P2P.
Yeah, it’s a lot like BitTorrent in this sense. People may go through the trouble of downloading and running it to get something they want, but as soon as they have it, they are likely to shut it down.
This is where blockchain technology may help with incentives, although I hesitate to even mention it for it’s much deserved horrible reputation, but it would fix the incentive problem with a decentralized approach if there was some reward for hosting.
It’s a centralized package registry with extra steps, and like all other package registries, it requires people to be on call for it to be highly available. You find it nice because you don’t have to pay for it.
That sounds to me like approaching a p2p network, or are you saying that community members would host full copies of the registry? I think that might be a bit too big for people to host comfortably, and that’s another nice thing about Radicle: you only seed what you care about.
This is the case for small solo-dev projects, but for projects that are big enough, like certainly Zig, but also smaller stuff like Zine, the author would just need to put a node on a VM to ensure their project is always available. And that would also apply to the project’s dependencies, which would give resilience to the smaller projects they depend on.
That seems pretty well aligned in my opinion.
No, I like it because it’s a system I can run on my own. Whenever I was working for a company, we had caching mirror of all open source dependencies we used. And the Go model is as decentralized as one can imagine. The original source is distributed, and there is a built-in model for the proxying. So you can very easily host your mirror.
Sorry for the multiple replies.
Yes, I’m saying this. You have a bunch of companies, it’s trivial for them to run a caching proxy that hosts a few TB of data. They can afford it. If you have 10 such mirrors, you have a pretty resilient system.
From the perspective of a company running their own proxy (for their own use) that caches the packages that they depend on, I can see that making sense and I agree with the convenience/simplicity argument.
From the other perspective, that companies should be open to host publicly accessible proxies for the community that cache non-trivial quantities of data, I’m not as sold, although it’s true that if there’s many of those, then we would have redundancy so no one instance would need be highly available… except that in this setup all instances would need to hold all the data because if one goes down and the other doesn’t have it, then your build might still fail. Similarly, the various instances would need to agree on which hash is a given package supposed to have and decide what to do with disagreements.
I think this second approach really quickly ends up approximating a p2p network like Radicle once you decide to iron out all these details. Also the fact that multiple entities are involved starts making the trust problem much more realistic. Even if Google didn’t publish the log for their proxy, you could be reasonably confident that they’re not going to mess with the packages. Once you start having 5, 10, 100 different instances operated by different companies and individuals, you cannot ignore this aspect anymore.
For those who are not in the know, Zig 0.17.0-dev recently changed how dependency caching works: the global cache now stores packages as compressed archives, and each project has a local
zig-pkg/directory that contains the extracted files of the dependencies used by it. This makes vendoring trivial if that’s your goal: just check yourzig-pkg/directory into source control. (Otherwise, it’s recommended to add it to.gitignore.)
TIL Zig supports vendoring!
Next stage will be Package Managers are Evil
Zig build always supported vendoring, all you had to do in the past was place the dep in your repo and then do .path = "vendor/my-dep" in your zon file. Now it’s just more convenient.
Quite the opposite, the whole argument of my post is that vendoring and forking is literally a suboptimal strategy. One that works, but that makes terribly inefficient usage of the redundancy those practices create, while something like Radicle exploits it for the benefit of everybody.
I actually remember one project attempting to use blockchain to decentralize git: GitHub - cjb/GitTorrent: A decentralization of GitHub using BitTorrent and Bitcoin · GitHub
About fetching from radicle: I think this could be served by fetch plugins (see #14294)?
DAMN I wish I knew that
For those wanting more reading, the underlying concepts that contrast package managers include:
- named data networks: Named data networking - Wikipedia
- content addressable storage: Content-addressable storage - Wikipedia
- content addressable networks: Content-addressable network - Wikipedia
The build.zig.zon file includes a hash of the desired content, which is a form of content-based addressing.
The rad clone rad:z3WukSjzicL8WaZHFALbBwb2r8W52 is demonstrating named data networking, asking the network for a piece of content and not necessarily requesting it from a specific location / host.
A lot of discussion about modern package manager designs is moving towards content-based addressing / name data networking to address issues like dependency confusion attacks, uptime, reproducibility.
Making everyone share the load is nice in theory, but the nature of internet infrastructure doesn’t favor decentralized p2p most of the time. BitTorrent works well because it’s used to share static, non-updating files; high latency and the unavailability of less popular content are a tolerable part of its design. For frequently updated data, such as git repos or social network feeds, high latency is excruciating and data becoming unavailable because it’s unpopular is not tolerable.
I was never able to get radical to work properly, (I remember trying it twice on separate occasions). Is it finally ready?
Granted I was trying to use it privately, but still.
(post deleted by author)