Petals is from 2022. Nowadays intelligence of smaller models, quantization techs, and optimizations to run models faster on consumer GPUs have improved a lot.<p>For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that powers SillyTavern. It also has an interesting economy model of kudos.<p>[0] <a href="https://stablehorde.net/" rel="nofollow">https://stablehorde.net/</a>
The recently discussed <a href="https://meshllm.cloud/" rel="nofollow">https://meshllm.cloud/</a> is the one I've been playing with but don't have the hardware to try with a model split between nodes, which apparently is supported and just not part of the public demo.
It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately.<p>It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (Alpha Centauri ever relevant).
Not a neural network though ;) <a href="https://biologyinsights.com/do-fungi-have-a-nervous-system-the-biology-explained/" rel="nofollow">https://biologyinsights.com/do-fungi-have-a-nervous-system-t...</a>
Relevant: Why Switzerland has 25 Gbit internet and America doesn't <a href="https://news.ycombinator.com/item?id=47652400">https://news.ycombinator.com/item?id=47652400</a>
Too early too in that later (hopefully) each node can serve a full model so it is then a simpler share tit for tat of complete models.
Too early? the project has been going for years now...
Hm, looks like this is an old project (2022) associated with HuggingFace (<a href="https://huggingface.co/bigscience" rel="nofollow">https://huggingface.co/bigscience</a>).<p>Doesn't look like it's very active nowadays: <a href="https://github.com/bigscience-workshop/petals" rel="nofollow">https://github.com/bigscience-workshop/petals</a><p>Article with more details: <a href="https://techcrunch.com/2022/12/20/petals-is-creating-a-free-distributed-network-for-running-text-generating-ai/" rel="nofollow">https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...</a><p>> [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to complete larger tasks, similar to Folding@home and other distributed compute setups.
Related. Others?<p><i>Run LLMs at home, BitTorrent‑style</i> - <a href="https://news.ycombinator.com/item?id=37546810">https://news.ycombinator.com/item?id=37546810</a> - Sept 2023 (125 comments)
This is awesome. Now we also need distributed <i>training</i> of models!
I have been thinking of something on these lines but with much smaller models. The entire model has to fit on a single computer. Host owner would choose the model they prefer, perhaps because they already use it. Then it is more about utilizing the GPU for LLM requests.<p>Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be equal - I have not thought through the fairness part. Perhaps initially it starts with "create your friends group and have access to all the host nodes".
I did a triple take. Many years old but an early ground breaking project. Has some exploratoy ideas a year or two ago last I looked. Crazy to see Petals at 16 frontage (at time of writing)
Any updates here to make this work on more modern llms?
Seems a bit like GNUS [0].<p>---<p>[0]: <a href="https://www.gnus.ai/" rel="nofollow">https://www.gnus.ai/</a>
This is super cool technology, but will be abused to death
Imagine if there was some kind of way to cryptographically 'prove' your GPU is doing some kind of 'work' here and contributing to the network.<p>You could even hand out some kind of digital 'currency' to the people proportional to the amount of work their doing!
I've occasionally written comments about the difficult of achieving proof-of-useful-work mechanisms and I should probably find or write a standard post about this question.<p>In order to be as decentralized as Bitcoin and related PoW mechanisms, a proof-of-work mechanism should have<p>* the ability to create an unbounded number of instances of the problem deterministically from numeric seeds<p>* the ability to scale difficulty up and down using a difficulty parameter<p>* in a way where anyone can easily confirm that a given problem instance corresponds to a given seed, and anyone can easily confirm that a given solution correctly solves a given problem instance<p>I'm not aware of a proof-of-useful-work mechanism that meets these criteria. (I'm also not aware of an argument that it's impossible to have one. Zero-knowledge proofs might actually go far toward making it possible in the future. but it's kind of complicated.)
As we've seen, that's a solution looking for a problem.
Not gonna work, will be starved by inter node bandwidth.
You can easily build a localized, access-controlled compute grid using Apple Silicon hardware. Haven't tested but as far as I can see, the key to making it performant would be keeping full models resident on individual nodes rather than attempting to distribute the tensor processing across the network.