Running services, network usage, memory usage, bandwidth, disk I/O, successful logins, whether the thing is even alive, etc…
So far my only method has been “hope and pray”.
I have the best detector for issues with my self hosted services: my wife!
I run UptimeKuma for monitoring a couple of things, but honestly, I’m pretty much my only user and my homelab is independent of the regular home network, so I don’t really care too much. I’ve been considering some simple log monitoring… when I get around to it :-)
I have a basic dashboard set up in my home assistant that I never check. Don’t even know how it works, I forgot. That’s why I really want to just put my whole setup in code. That’s why nothing is happening, the chore just seems so daunting.
So I listen for complains ringing through the house and react accordingly.
They’re in my attached garage. In the living room when it’s quiet I can hear them humming. Does that count?
Prometheus & Grafana
Use service, wait for 404 or data loss…
You guys are monitoring your servers?
Seriously. I just kinda check it out once in a while.
“can’t access this thing, server must be down”
I’ll fix it when I get home
I plan to perhaps, just maybe, I am not sure, to try self-hosting an email server. But I’ll need a good uptime.
Although I am really scared about security.I self-host my own email, have done so for years. I’ve not gotten pwned. Use an SSH key and a firewall, only expose the ports that should be public, have reasonable access controls, etc, just practise general cybersecurity—it applies to servers too.
I know everyone complains about hosting email. But I’ve not run into major problems. My emails get through spam filters and I have good uptime. It’s also a fun learning experience.
I think that you have to do quite a bit more setup on email than most other services. There’s SPF and DKIM and DMARC and so on. Plus spam filtering, setting up accounts and authentication, …
Plus email is arguably more important than anything else, since it’s kind of the lowest common denominator of the Internet; your dentist will probably send you an email to remind you of appointments, job offers will arrive via email, password resets arrive via email, and so on.
So it is a bit scary.
Having said that, as long as you’re not hosting from a residential IP addresses it is definitely achievable. You can start with one domain and use it for low-value purposes (or for accounts that allow multiple email addresses, like GitHub).
Yes, it can be a bit involved. I’d say I can spin up a mail server in 1–2 days though provided the OS is already installed and I have a domain name and port 25 isn’t blocked (that’s the main one that tends to be blocked). It may take longer for people who aren’t familiar with mail servers but it’s not some huge operation.
You can just try spinning up a mail server for fun. If it goes badly then just don’t move over your dentist appointments to it. It’s only critical infra if you use it for critical things, which you probably shouldn’t till it’s stable.
Well, it’s a good learning experience. But…… don’t. Good luck getting around the major email providers’ smart screen shit. It’s probably not gonna happen. I tried a few years ago, and even microslop couldn’t figure out why their shit screen crap wouldn’t allow my emails. It’s not in their best interest to fix it. So they don’t. But it’s worth doing it in the short term to learn how to do it. Teaches you about email, spf, dkim etc. which actually does translate to real world skills. I used what I learnt setting it up in my real life job.
Yeah, email is one of the more annoying things to self host. Google rejects mine because my ISP has all their non static IPs on block lists (even though my IP hasn’t changed once since I’ve been with them it’s still not “static”, can’t change the reverse DNS) and I’m not paying 150% more a month for a business account to get a static IP. I ended up just using a free email relay. Still had to setup SPF, DKIM, DMARC, and all that jazz still.
Yup. It’s so damn annoying hosting email. Do it once to learn, but don’t do it as a permanent solution. It takes so much maintenance to keep it going, if you even do get it working.
What maintenance do you mean? run OS updates and pay the vps bill. that’s true for any self hosted application. which specific maintenance are you referring to?
Setting up the newfangled security thing. TLS, DKIM, DMARC, SPF are the big ones over the years. Finding out you’ve been silently blackisted by a provider and hoping and praying they’ll answer your request to unblock. Managing your own spam filter.
Been running one for many years, mostly issue-free. Had to request whitelists way in the beginning, but things turned out ok.
Check out this super great guide. https://workaround.org/ispmail-trixie
I use Signoz and OTel collectors to bring in everything from Kubernetes (and also SNMP-to-OTel for the network and NAS stats.)
That covers dashboards and basic alerting. Then on top of that I have an n8n workflow that runs every morning - it pulls all the main stats out of Signoz, as well as issues and their status from my infrastructure repo in Gitlab, and feeds it all into a local LLM (Qwen-3.8-Flash-Next at the moment), which emails me a report on overall cluster health. It also automatically opens a ticket in Gitlab for anything that might need fixing.
It works really nicely, gotta say.
Gatus. Works and easy incremental set up.
I notice something stopped working, or someone in the household notices.
Then I go fix it.
And the best time to notice that the backups weren’t working is when you need access to the backup. 😅
Backups are pretty much the only thing I get notifications for. Otherwise, one app or another on my phone will complain if it loses connection to my server.
That and degraded RAID!
Backups? In this economy?
But I need to get something working. I am about to move to VaultWarden and that makes me very nervous.
I run a vaultwarden container since forever and it’s really smooth. All updates went through just fine, never had an issue with it.
Same. And I was also nervous in the beginning, but in the end if you are the only user, you’ll have your vault replicated everywhere you have the app.
Beszel & Pangolin.
Icinga and Phare Uptime
Ultime Kuma, Beszel and Ntfy
Monit, simple to setup, gets the job done, runs on my router too.
I have this on my NAS, works well
I have a robust monitoring system for my Jellyfin server, been running for the last few years. Checks in periodically, at least once per 24 hours and notifies me if it’s down. Doesn’t use any electricity, but does consume a good amount of Cheerios and mac & cheese.
Lies! It uses electricity!
Indirectly. But as another commenter pointed out, it balances out with the tax rebates.
