AI At Home Part 1: A Box Of Scraps

(jdagostino.github.io)

46 points by timmmmmmay3 hours ago

9 comments

Aurornis2 hours ago
> I got these 10,000 RPM fans because I wanted to make sure I was moving enough air, and they were the same price as slower fans. They are really loud! I wanted the motherboard to control their speed based on the GPU temperature, and this didn't work at first, so I just wore ear protection during initial setup.I think everyone remembers the first time they went from "I think I can tolerate some server fans. How loud can they be?" to "I had no idea a small 12V fan could be this loud"
- nazgulsenpai2 hours ago
 The first time I powered on a Dell Poweredge R420 without the lid on
rcarmo24 minutes ago
Those are some pretty meaty scraps, way above what you’d have on several people’s shelves…
fabiensanglard2 hours ago
Quite courageous to go with AMD. I am very curious to see Chapter 2 and how he solves the software stability that used to plague AMD AI applications.
- SwellJoe1 hour ago
 "used to" in your sentence already answers your curiosity.ROCm runs most things just fine without any ceremony or difficulty. I have a couple of the same cards as covered in the article, bought before they got expensive (I'd recommend current Radeon AI Pro 9700 cards over the V620 now, though, as they have increased in price since I bought mine), and they're at the long end of the supported chart for ROCm...nearly EOL. But, they currently work great with llama.cpp and current ROCm 7.14. They're pretty fast and stable. I also have a Strix Halo 128GB, and while the Strix Halo can run bigger models, the dedicated GPUs are quite a bit faster. And, since the best models you can run at home are probably Qwen 3.6 27B and Gemma 4 31B, and those both fit comfortably on dual 32GB GPUs, I find I use the desktop more often than the Strix Halo.Anyway, there's very little reason to spend 3x or more for the Nvidia ecosystem these days. The only exception is the Strix Halo vs the Nvidia GB10 platform. Strix Halo was a no-brainer when it was half the price, but it's risen in price to be almost the same price as an Asus GX10. In that one instance, I think the Nvidia based Asus is a better choice. The Strix Halo is almost too slow to make use of 128GB. Some MoE models are comfortable, like Laguna S2.1, so it's probable that there will someday be an MoE model that is better then Qwen 3.6 27B or Gemma 4 31B that won't run on 64GB but will run on 128GB. I don't know of one, yet, though.AMD also used to focus almost entirely on their data center line and not pay any attention to their lower end stuff for more advanced AI features, but that's been changing, and ROCm support has broadened to include almost everything AMD ships now, including the smaller embedded stuff. I think they've realized that as long as the only way to develop AI for AMD was to have a quarter million dollars worth of hardware, they would always fall behind a platform that can be developed for and tested on consumer hardware.
 - Muromec49 minutes ago
 >I'd recommend current Radeon AI Pro 9700 cards over the V620 nowThat's about 4x the price of v620, which is 450 eurobucks. Can't imagine buying 4 of those at 4x the price.
 - SwellJoe17 minutes ago
 Ah, the price has gone up on the AI Pro 9700, again, I see. So, yeah, it's a worse deal now. But, the V620 is not really a good deal, either, at $450 on eBay. It's roughly half the performance of the newer card. Usable but not blazing. And, you probably don't want to deal with cooling them; if you already have a 3D printer, you can print the shrouds for a few bucks and buy the fans for a few more bucks. But, if you don't have a 3D printer, the shrouds you can buy on eBay are for the tiny fans that have to run at ludicrous speed and have to be so fucking loud it should be illegal. I went through three different shroud and fan combos before finding a set that worked in my case with this cards (about $100 worth of experimentation, probably).I should say, though, that I actually don't recommend buying anything right now. I wrote up my setup and made recommendations (and the main recommendation was "don't"). <a href="https://swelljoe.com/post/how-i-run-local-llms/" rel="nofollow">https://swelljoe.com/post/how-i-run-local-llms/</a>The V620 at $450 or the Radeon AI Pro 9700 at ~$1400 are a good deal compared to everything else right now, but buying tokens from DeepSeek is a better deal. DeepSeek V4 Flash 0731 is better than anything you can host locally, they'll serve it to you at blistering fast speeds for pennies a day, and with 1 million token context. You could host a 2-bit quantization of it on a Strix Halo or four of these V620s, and it would run at a crawl on either one. The Strix Halo gets 9-13 t/s. Four V620s would, I guess, get two-three times that. Which is still too slow for comfortable interactive agentic use, and much slower than getting it from DeepSeek, and at two bits there is measurable loss. You're paying a lot more for self-hosted and you're getting worse models.Also, if you want four cards, you need a server-class motherboard and CPU and RAM. More money.It's all just a bad investment. Self-hosting is a bad idea if you don't already have the hardware, unless and until memory and GPU prices come down. Even at the prices I paid (before RAMpocalypse really kicked off; $2k for the Strix Halo, ~$350 for each V620), I wouldn't recommend it if you don't have a strong urge to tinker with hardware and it'll probably never pay for itself vs. buying inference from DeepSeek directly.Edit: The same shrouds for all the old Instinct cards also work for the V620, as they have the same dimensions and screw and cable layout. I tried several and ended up with this one: <a href="https://www.thingiverse.com/thing:7296707" rel="nofollow">https://www.thingiverse.com/thing:7296707</a>
- cyberax1 hour ago
 Uhh.... I have a box with two AMD Radeon AI PRO R9700 32GB.It was the most trouble-free AI setup that I have done. The driver worked right out of the box with the stock Fedora kernel, ROCM can be installed from the regular repo, and most AI software has ROCM builds by now.
hk13371 hour ago
Our definition of "Box of Scraps" is wildly different.
comandillos1 hour ago
I ended up buying 2 DGX Sparks interconnected over QSFP with the intention of getting rid (as much as I could) of any cloud-based AI provider. I'm running DS4 Flash 0731 on it and some OCR models, using Oh My Pi and OpenWebUI as my main ways to interface with the agent... and from someone that has been using Claude for a long time, I can definitely say I don't need it anymore. Not for the stuff I'm doing.
- nater50001 hour ago
 I mean, this is cool and all, but 2 DGX Sparks is like $10k, right? That's a huge upfront cost to run DS4 Flash 0731 which is definitely not comparable to higher-end Claude models. It's extra egregious when you consider how much it'd cost to run DS4 Flash 0731 via an API (how long would it take to run up a $10k bill doing what you're doing?).>and from someone that has been using Claude for a long time, I can definitely say I don't need it anymore. Not for the stuff I'm doing.I'm curious what this implies. It suggests that you were using Claude prior to this setup, so it doesn't seem like you were limited by security or local features? I can understand that someone not wanting to send their data to Anthropic (etc.) might be willing to pay for this kind of setup to accomplish that, but what was your motivation for doing this?If it matters, if I had enough money that I could blow $10k on 2 DGX Sparks without it being a significant cost I probably would for the hell of it, so I'm not digging on you if this is ultimately what this comes down to. But, as an investment, this doesn't seem like a very good deal.
 - belval34 minutes ago
 Sure but the GPUs in the article are from 2021 and cost ~500-600$ a pop. The OP bought 4 so that's 2k, let's say another 500$ for the rest of the chassis and you stand at $2.5k for a monster server that will burn electricity and warm your house.Compare that to a $5k DGX than consumes much less and has the same amount of VRAM and there is a real question as whether this is worth doing at all (well aside from the cool factor).
 - comandillos56 minutes ago
 Absolutely agreed, I don't see this as an investment leading to cost-savings in the future, at all.I see this rather as an investment in improving my capabilities and knowledge of this technology, letting me play with a 'GPU cluster', vLLM and other technologies that otherwise would require me to rent GPUs on the cloud.
_def1 hour ago
> You'll start depending on it and then it'll get taken away from you.You can not get around this, conceptually. Local inference will keep depend on updated models for quite a while. Partly because they will contain outdated training data, partly because of demands for the improved models. And it's still not clear where this will lead us. We're still in the rosey phase where people get lured in.
- ryandrake1 hour ago
 But, we know 100% that cloud-provided anything can and will get nerfed, broken, removed, or in some other way rug-pulled. It already happens all the time, and companies are getting more and more aggressive/stingy about what counts as "yours" and what counts as "bought" and what counts as "acceptable use".Yes, bringing everything local still means you need to download things, maybe over and over. But once it is on your machine, nobody can yoink it from you just because they want more money or they don't like what you're doing with it.
- zdragnar29 minutes ago
 Is there any hope that something like unsloth studio will let people retrain models continually so that, when the day comes that there are no good open models being released, something like the final generation of qwen whatever can continue being relevant into the future?
- edgyquant38 minutes ago
 Updated models isn’t the same thing as hosting them somewhere else
- hypfer42 minutes ago
 We will try anyway. No need to try to FUD people into passive acceptance of the cloud.
Muromec51 minutes ago
One piece of v620 costs about 450 eurobucks on ebay right now. Weird to see a card with no HDMI output at all.
petesergeant2 hours ago
> Around this time I also realized that I didn't have Ethernet in the garage, so I started cutting holes in the drywall at 11:00 PM.Ahh, one of those projects
- ryandrake1 hour ago
 We've all been there. I don't think I've ever owned a home where I didn't have to cut holes in the drywall to run cables.
 - Muromec46 minutes ago
 The only one place where I need ethernet and don't have it is right on the opposite wall from the electric closet that has the fiber. That's also exactly where I have my desk.Yes, I was the person drawing the floor plans and forgot the electric socket there too.