The true cost of Kubernetes: People, Time and Productivity
8 min
Koyeb provides developers the fastest way to deploy full stack applications and APIs globally. Sign up today and deploy 2 services for free forever!
While writing a comparison of Kubernetes and Koyeb, we tried to determine how much operating a Kubernetes cluster really costs. This section of our comparison took us hours to write and ended up being so long that we decided to write a dedicated post about it.
Full disclaimer: At Koyeb, we're building a serverless platform and we have a purpose-built orchestration engine. We're heavy users of Kubernetes for our own control plane, which, sadly, can't run on Koyeb as it would otherwise generate circular dependencies. We also previously ran our users workloads on Kubernetes, but decided to switch to a custom stack composed of Nomad, Firecracker, and Kuma.
Kubernetes is a proven technology for container orchestration and is a great tool for running resilient applications, but the true cost is often underestimated. Our comparison explored the opportunity costs related to developer experience and managed responsibilities, this post investigates the actual financial costs of using Kubernetes.
When building a production platform, you can select different layers of abstraction. We distinguish four major abstraction layers you can pick from. If you're not an infrastructure company, we advocate building on top of cloud service provider primitives. Spoiler alert: We estimate that operating Kubernetes clusters costs at least $100k and can easily cost more than $500k annually for a ridiculously low amount of resources.
The cost of Kubernetes on Azure, GCP, and AWS
Kubernetes is an open-source technology, but open-source doesn't mean it's free to operate. The operational costs depend on how you choose to use Kubernetes. When opting to use Kubernetes, you have two main options:
Self-hosted Kubernetes
If you go with a self-hosted Kubernetes solution, you will need to budget for the nodes needed to run your workloads, but you will also need to add the costs of the nodes to run your control plane and of the team to manage the operation.
For a reliable Kubernetes production environment, you will need to create and manage nodes that host your control plane, in addition to separate worker nodes to host your application workloads. The control plane is where the main Kubernetes components needed to configure and orchestrate your workloads live. The control plane runs key processes including:
- kube-api-server to expose the Kubernetes API,
- etcd, a highly-available key value store, to store all cluster data,
- kube-schedule, which schedules newly created Pods,
- and more processes like the kube-controller-manager and cloud-controller-manager.
Technically, you could use the same nodes to host the control plane and worker nodes, but it is not recommended for production environments.
Self-hosted Kubernetes total cost
For this calculation, we decided to do the cost simulation using nodes running on Azure. We did the estimation using D2s v3 Azure VMs machines, which have 2 vCPUs and 8 GB RAM and used 3 nodes which is the minimum for redundancy purposes. With that, you should have 5.4GB of RAM per node, so ~16GB of usable RAM and 6vCPU.
Secondly, we took the average DevOps compensation in the US, $141,000 (cited by Builtin) and multiplied it by 4 to represent a team of four engineers, which is needed to properly cover 24/7 operations.
As a result, here is what the total cost of ownership (TCO) for self-hosted Kubernetes looks like:
| Expense | Annual Cost |
|---|---|
| Nodes to host the control plane of your clusters | $0.096 x 24 x 365 x 3 = $2523 |
| Nodes to host application workload | $0.096 x 24 x 365 x 3 = $2523 |
| Compensation for members on your DevOps/SRE team | $141,000 x 4 = $564,000 |
Annual total base cost: $2523 + $2523 + $564,000 = $569,046
This cost estimation doesn't include the sanity of your DevOps team when issues arise, which is, we believe, priceless.
Managed Kubernetes on Azure, GCP, or AWS
Then there is the managed Kubernetes route. With a managed Kubernetes solution, you do not need to handle creating or managing your Kubernetes clusters, nor do you need to worry about creating the control plane or installing its components like etcd, kube-apiserver, or kube-scheduler.
GKE, EKS, and AKS: The cost of the managed control plane
The 3 biggest managed Kubernetes providers are GCP, AWS, and Azure with Google Kubernetes Engine (GKE), Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). From what we found, the pricing is similar for the managed control plane of all these services at $0.10/hour.
| Managed Kubernetes Provider | Cost to run one cluster |
|---|---|
| GCP GKE | $0.10 x 24 x 365 = $876 |
| AWS EKS | $0.10 x 24 x 365 = $876 |
| Azure AKS | $0.10 x 24 x 365 = $876 |
For all these offerings, there are no automatic version updates or auto-recovery and you still need to pay for the computing resources like CPU, memory, and ephemeral storage that your worker pods consume.
Cost of the worker nodes
The cost of worker nodes also varies across cloud service providers and depends on the amount of computing resources you need as well as in which region of the world your servers are running. We multiply by 3 for redundancy purposes.
| Provider | Resources | Annual Cost |
|---|---|---|
| AWS m6gd.large EC2 instances | 2 vCPUs and 8 GB RAM | $0.0904 x 24 x 365 x 3 = $2376 |
| Azure D2s v3 VM machines | 2 vCPUs and 8 GB RAM | $0.096 x 24 x 365 x 3 = $2523 |
| GCP e2-standard-2 VMs | 2 vCPUs and 8 GB RAM | $0.067 x 24 x 365 x 3 = $1761 |
With these instances, you should have 5.4GB of RAM per node, so ~16GB of usable RAM and 6vCPU.
Managed Kubernetes total cost
You also need to account for the maintenance responsibilities not covered. You'll need at least one dedicated DevOps person to monitor this managed solution and handle the responsibilities not covered by these offerings.
| Managed Kubernetes Provider | Cost to run one cluster | Cost of worker nodes | Salary for dedicated DevOps | TCO |
|---|---|---|---|---|
| AWS EKS | $876 | $2376 | $141,000 | $144,252 |
| Azure AKS | $876 | $2523 | $141,000 | $144,399 |
| GCP GKE | $876 | $1761 | $141,000 | $143,637 |
This option buys you a lot more time and energy to focus on developing and improving your applications, but you will still be stuck with a lot of the Kubernetes responsibilities and complexity depending on the cloud service provider you use.
Serverless container solutions are a more efficient alternative
Kubernetes is Greek for helmsman or pilot, so the question becomes: is this a fleet you really want to command? If it's not your core business, we advocate you shouldn't.
We're strong believers that, in the majority of cases, a serverless container solution combined with managed databases and object storage will be way more efficient than going with Kubernetes. Kubernetes is a generalist platform that leaves a lot of operational work in the hands of the user but all modern cloud providers now offer advanced serverless container solutions.
The key benefits of serverless container solutions are:
- you have no upfront cost with no minimum number of nodes running
- the nodes supporting your containers are completely managed, you don't have to think about security upgrades or orchestration of your containers
- you keep the flexibility and portability of containers
- scaling is completely seamless
Some might argue that resources (namely RAM and CPU) are more costly with serverless solutions but this is without accounting for the management costs of Kubernetes we highlighted above.
Serverless solutions provide different levels of complexity and maturity and we would, in a totally biased manner, recommend Koyeb. If you want to go with AWS or GCP, we recommend to look at AWS Fargate and GCP Cloud Run. All of these solutions remove the hassle of managing Kubernetes clusters.
For databases, we previously wrote about managed databases solutions for Postgres, MySQL, and Redis, that can be combined with serverless container solutions.
Serverless solutions are not magic, they require some configuration, but you don't need strong system engineering skills to work on it and that's the fundamental difference.
Looking forward to the Serverless future
Looking only at the execution cost of your containers is like only looking at the tip of the iceberg. We believe serverless solutions should also encompass advanced features including:
- seamless continuous deployment
- transparent load-balancing with native TLS
- native secure and private networking with service mesh capabilities
- easy global deployment
That's why built them natively into the Koyeb serverless platform. You can use Koyeb to host web apps and services, Docker containers, APIs, event-driven functions, cron jobs, and more. The platform has built-in Docker container deployment and also provides git-driven continuous deployment!
Discover the serverless experience by signing up today. Also, feel free to join us on the community platform.
And if you want to help build a serverless cloud service provider, we're hiring!
Start for free, pay as you grow
