I’ve been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It’s a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use. https://github.com/mw00/project-maya

My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.

I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling

It’s quite a ride lol. There’s a group of people on the level1techs forum trying to get mi250x OAM units working on similar hardware.

I made a post about my own lab earlier this year too: https://lemmy.world/post/47291115

Chinese sellers on ebay have been making sxm2 mezzanines that use standard slimsas connections for a while. They even go up to sxm4-5. I have dual nvlinked v100s with their giant 4u tower coolers, some arctic 92mm fans and PTM for the TIM and they never break 70c under load for my personal use.

They definitely aren’t the flashiest, but local LLMs have come a long way in a short time to the point where they are quite nice to work with.

I appreciate the kind words! Until recently, my day job was CAD monkey. I wanted to consolidate the hardware I was cobbling together, found a cheap 8 GPU mining rig and some extra 2020 extrusions to play with.

The seller for the mezzanine said it followed the mATX mounting hole pattern, (it does but is ~75% in the width and doesn’t use all the points). I have a tendency to overthink designs and kind of stalled for a bit before finally taking the plunge and whipping up these struts. I used some brass heat set inserts to accept the standoffs and everything pretty much went right together lol

I am using PTM sheets and they idle at decent temps, though I did ziptie some high CFM fans behind them lol

eBay has some rad Chinese mezzanine boards for these guys too. Nvlink works and everything lol

But this is the opposite of dull. Excellent work!

I bet you used a keyboard, instead of a magnetic needle and a steady hand to code this too.

/s

For real though, nice work. 🎉

“the front fell off?!”

They’ve been temporarily removed for about a year now lol

I’m actually quite happy with it. It’s a 24gb p40 which runs some of the older LLMs quite well. Repasting with liquid metal helped thermals a ton too

It’s a Tesla P40 24gb I got off eBay a while ago. Just like AllHailTheSheep suggested, it’s a server GPU meant for compute workflows so it relies on forced air from a server rack. I repasted the die with liquid metal and added a 3d printed duct to attach the radial blower to. The thing never cracks 60c under inference which is nice. Idles around 24c

(Edit: I’m mostly using it for LLMs, but it supposedly makes a good CAD card, so I might give that a shot too)

Thank you 🥹 It’s been quite the experience lol

I thankfully snagged this one pretty cheap off eBay. It’s actually an 8-gpu mining rig frame that conveniently fit my CEB motherboard lol

My janky homelab lol. Mostly ebay secondhand Enterprise stuff and a Chinese SXM2 mezzanine board to run dual NVLinked 16gb V100s. I also have a TrueNAS Scale mini itx server running upstairs with my “arr” stack and some other useful tools.