I always wonder if things can be simpler. When you think of a really simple DB you think of SQLite. What's the really simple K8s? Even doing a single node deployment these days seems complicate with Prometheus, Grafana, etc. etc. docker/podman compose up with quadlets and all of this stuff just seems so eh.<p>I really like the idea of something like Firebase, but it never seems to work out or just move the complexity to the vendor, which is fine, but I like knowing I can roll my own.
> What's the really simple K8s?<p>It's k3s. You drop a single binary onto the node, run it, and you have a fully functional one-node k8s cluster.
In ascending order of functionality and how much complexity you <i>need</i>:<p><pre><code> - Docker Compose running on a single server
- Docker Swarm cluster (typically multiple nodes, can be one)
- Hashicorp Nomad or K3s or other light Kubernetes distros</code></pre>
One of the issues is that the open source Helm charts (or whatever) for something like Grafana do not come out-of-the-box with good config. I spent a significant amount of time a little while ago reading blogs to get Grafana to use up to date indexing algorithms, and better settings, etc.<p>Considering these companies make money when you use their hosted solution, this is not surprising, and it just goes to show TANSTAFL.
Big question is which feature subset you want to replicate.<p>Kubernetes means everything to everyone. At its core, I think it’s being able to read/write distributed state (which doesn’t need to be etcd) and being able for all the components (especially container hosts) to follow said state. But the ecosystem has expanded significantly beyond that.
> What's the really simple K8s?<p>I think K8s couples two concepts: the declarative-style cluster management, and infrastructure + container orchestration. Keep CRDs, remove everything else, and implement the business-specific stuff on top of the CRD-only layer.<p>This would give something like DBus, except cluster-wide, with declarative features. Then, container orchestration would be an application you install on top of that.<p>Edit: I see a sibling mentioned KCP. I’ve never heard of it before, but I think that’s probably exactly what I’d like.
Of course things can be simpler.<p>Remove abstractions like CNI, CRI, just make these things built-in.<p>Remove unnecessary things like Ingress, etc, you can always just deploy nginx or whatever reverse proxy directly. Also probably remove persistent volumes, they add a lot of complexity.<p>Use some automatically working database, not separate etcd installation.<p>Get rid of control plane. Every node should be both control plane and worker node. Or may be 3 worker nodes should be control plane, whatever, deployer should not think about it.<p>Add stuff that everyone needs. Centralised log storage, centralised metric scrapping and storage, some simple web UI, central authentication. It's reimplemented in every Kubernetes cluster.<p>The problem is that it won't be serious enough and people will choose Kubernetes over simpler solutions.
Some people want their k8s logs to be centralized with non k8s logs. Standardizing log storage seems like a challenging problem. Perhaps they could add built in <i>log shipping</i>. But even then, the transfer format needs to be specified.<p>Adding an idp is pretty standard in k8s... What do you want to actually do different?
IMO this is what keeps people from building systems that might challenge kubernetes. Everyone wants to say Kuberentes is too complex, so we built something that does much less. I respect that! But I think it usually fails to grok what Kubernetes is and why it's such an interesting and vital rallying point, that's so thoroughly captured our systems-making. Let's look at the premise:<p>> <i>That’s why I like to think of Kubernetes as a runtime for declarative infrastructure with a type system.</i><p>You can go build a simple way to deploy containers or ship apps: but you are missing what I think allows Kubernetes to be such a big tent, thats a core useful platform for so many. Kubernetes works the same <i>for all types</i>, for everything you want to manage. It's the same desired state management + autonomic systems patterns, whatever you are doing. An extensible platform with a very simple common core.<p>There are other takes and other tries, but managing <i>desired</i> state for <i>any</i> kind of type is a huge win that allows many people to find their own uses for kube, that is absolutely the cornerstone to it's popularity.<p>If you do want less, the one project I'd point to that is kubernetes without the kubernetes complexity is KCP. It's just the control plane. It doesn't do anything at all. This to me is much simpler. It's not finding a narrowly defined use case to focus on, it's distilling out the general system into it's simplest parts. Rebuilding a good simple bespoke app container launching platform around KCP would be doable, and maintain the overarching principles that make Kube actually interesting.<p>I seriously think there is something deeply rotten with our striving for simplicity. I know we've all been burned, and there's so often we want to throw up our hands, and I get it. But the way out is through. I'd rather dance the dance & try to scout for better further futures, than reject & try to walk back.
Everything in infrastructure is a set of trade-offs that work in both directions.<p>If you want better monitoring, metrics, availability, orchestration, logging, and so on, you pay for it with time, money, and complexity.<p>If you can't justify that cost, you're free to use simpler tools.<p>Just because everyone sets up a Kubernetes / Prometheus / ELK stack to host a web app that would happily run on a single VPS doesn't mean you need to do the same, or that nowadays this is the baseline for running something.