Kubernetes cluster administration¶
How many cluster¶
The following question will help you to design your Kubernetes environment
- Should you run all application instances on a single cluster?
- Or should you have a separate cluster for each application instance?
- Or should you use a combination of the above?
Different approaches:¶
- Large shared cluster: one cluster hosting all the environments and all the applications
- Cluster per environment: One cluster per environments for all your applications
- Cluster per application: One cluster hosting all the environments for one application
- Small single-user clusters: One cluster per application per environment
Pros and cons of various approaches:¶
Info
We can improve the resilience of the first two solutions with a multi cloud cluster
references
Multi cloud solution¶
Use cases¶
- service spike
- disaster recovery
- active active
Drawback
- Each cloud provider has its own set of API, making multi-cloud solutions hard to architect and to maintain
- IaaS tools such as Terraform have not solved this issue
Kubernetes¶
- Standardized application delivery
- Decoupled from the underlying cloud
- Kubernetes API is a solid foundation for multi-cloud and hybrid cloud
- Example KubeCDN
Topology¶
Stacked etcd¶
- etcd is stacked on top of the cluster
- Simpler to set up than a cluster with external etcd nodes, and simpler to manage for replication
- etcd as well as kube-controller-manager and kube-scheduler communicate only with the kube-apiserver of the hosting node
- This is the default topology in kubeadm
Drawback
- Loosing one control plane host implies to also loose a etcd member thus the cluster data storage (cluster memory) is impacted
External etcd¶
- etcd run in hosts separated from the Kubernetes cluster ones
- kube-controller-manager and kube-scheduler communicate only with the kube-apiserver of the hosting node
- loosing a control plane will not impact the cluster data storage
Drawback
- It is more expensive as you need at least 3 more hosts for the etcd cluster
references
Deployments¶
They are a lot of ways to deploy and maintain a Kubernetes cluster. I will just list some of its to give you a brief overview.
Local solutions¶
Managed solutions¶
- Amazon EKS
- Azure AKS
- DigitalOcean Kubernetes
- Kubermatic
- OpenShift Online
Bare metal¶
- CoreOS
- Docker Enterprise
- OpenShift Container Platform
- Kubespray
Management / deployment tools¶
- kubectl: command line tool lets you control Kubernetes clusters
- Kops: Deploy the infrastructure as well as Kubernetes. Currently support AWS, GCE, OpenStack(beta), VMWare vsphere(alpha)
- kubeadm: Deploy Kubernetes on an existing set of servers
- kubespray: Deploy kubernetes on an existing infrastructure. It comes with contributions to spin the infrastructure on multiple cloud provider.
Cluster resources¶
Even if Kubernetes is built to be highly scalable you have to correctly size the worker nodes to host your workload. To increase the control of the resource usage you can use resource request, limit and quota.
Default resources¶
You can configure default CPU and Memory used by pods. If a pod does not specify the required resources then the default values will be configured at the pod creation.
CPU and memory management¶
To better control the amount of memory and CPU used by a user (multi tenancy) or by an application, you can configure a limitation per namespace. Like a pod, a namespace can be configured to request a certain amount of resources and burst until it reaches a limit.
Number of pods¶
To ensure the available namespace resources will not be spread across too many pods, you can limit the number of pods per namespace.
References
Network Policy¶
Network policy providers are used to control the traffic flow at the IP or port level (OSI layer 3 or 4). In other words you can configure a kind of firewall to authorize traffic flowing between:
- A pod and other pods
- A pod and namespaces
- A pod and particular IPs
Providers¶
- Calico
- Cilium
- Kube-router
- Romana
- Weave net
Upgrade¶
Kubernetes upgrade should be a zero downtime operation, it is made to support a hot upgrade. However it is really important to backup the etcd cluster before an upgrade and to test the upgrade on a test stack before applying it to production.
The cluster upgrade is composed of the following steps :
-
Drain node: Perform the pods eviction operation which consists in moving the pods to other nodes. It also flag the node to prevent new pods to be scheduled on it.
-
Upgrade node: Perform the node upgrade does not matter the technology used.
-
Uncordon node: If you leave the node in the cluster during the maintenance operation, you need to run the uncordon command to tell Kubernetes that it can resume scheduling new pods onto the node.