Operating Kubernetes at scale means juggling many object types—deployments, sidecars, cron jobs, config maps, secrets—and the platform around them: nodes, networking, load balancers, storage, and add-ons such as service meshes and backup tooling.

This post covers one of essential parts of kubernetes ecosystem which is node pools. My team is maintaining gitlab runner workloads on 3 different clusters. We had a case. We noticed that some of our node pools won’t scale down to the minimum node counts we set. Although we knew achieving this condition was very likely impossible, there were pods that never go offline. If this type of pod are disseminated separately to all nodes, nodes would never scale down to their minimum node count.

But the thing got really weird when we found that the amount of node was always at the maximum node counts though the number of workloads were insignificant. These node pools won’t scale down despite low cpu utilizations. This pattern was also pretty consistent during weekends across all clusters.

A colleague rigorously reviewed a large volume of Kubernetes logs. He found that there was a suspicious log showing that these nodes couldn’t be scaled down due to pods have a local storage. It didn’t take him a very long to come up with a hypothesis that revealed the bad actor behind this outrageous stalled kubernetes node autoscaling. He pointed out that all of our runners mounted a volume from node hosts. This was the deployment configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab-runner
  ...
spec:
  ...
    spec:
      ...
      containers:
      - args:
        ...
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: cacerts
          readOnly: true
				...
      volumes:
      - hostPath:
          path: /etc/ssl/certs
        name: cacerts
			...
	...

My colleague immediately proposed to remove the hostPath volume mount. But we didn’t know why this configuration was there, we didn’t know what’s going to happen if we removed the volume. My job was to make sure that removing this configuration won’t produce any unexpected behaviors.

We had identified the likely cause and understood the failure mode. Even with that confidence, we still had to validate the fix without a dedicated test cluster—the equivalent of changing a critical component while the system remained in full use.

This was the kind of condition we had. We didn’t have a test environment for our gitlab runner. Yes we had 3 kubernetes clusters, but all of them served all services pipeline packaging and shipping operations like automated tests, security checks, application builds, bundling assets, and deployments. If one of clusters ran into an issue, these services operations would be impeded.

So how did we test our change?

Test it in production

Why do you test changes on a production cluster? Spawning a test environment for one type of workload in our case is gitlab runner is considered to be expensive. The value offered by this test environment in our case is a little less critical.

We did not run an uncontrolled experiment in production. Kubernetes scopes changes to the objects you modify.

When you look at the kubernetes hierarchy, you’ll see that kubernetes purposefully segregate objects capabilities to their own territories. You change the cpu requests of a particular deployment, kubernetes won’t modify other deployments cpu requests. When you adjust verbs of a serviceaccount it will apply to all workloads that use that serviceaccount. You delete a namespace you delete all workloads within that namespace.

Relying on production for tests usually signals missing isolation elsewhere. Kubernetes still helps because declarative objects can be changed, observed, and rolled back in a bounded way.

Next, the practical work.

The way we tested our change was picking up a safe contained spot within a particular cluster. We ran our test on a node that was dedicated only for our test workloads. We then deployed the unmounted node volume deployment change and ran numbers of pods to trigger node pool autoscaling.

This is the benefit of the sandboxed principle that I’m talking about. kubernetes gives us a capability to isolate our test on a dedicated node to avoid exposing undesired side effects to a place where they don’t belong.

Kubernetes provides a safe way to mark nodes as “unavailable”. I have to put it in double quotes because what i mean by unavailable is setting up certain nodes to not pick up production workloads.

Commonly, we could mark nodes as unschedulable with kubectl drain or kubectl cordon. In our case, we didn’t want to our dedicated node to be unschedulable. We wanted our node to behave like a healthy node that is available to schedule workloads but only for our test workloads.

We can achieve this with taints and tolerations. If node affinity invites pods to certain nodes by node selector labels, taints acts the opposite. Taints won’t let any pods to be scheduled to particular nodes unless pods define their tolerations. Setting up taints prevents our production pods get executed on our test node.

This is the way to set up taints for an existing node

kubectl taint nodes pool-2 dedicated=experimental:NoSchedule

In our case we used gke, we provisioned a node pool with node taints using a gcloud command.

gcloud container node-pools create pool-2 \
  --cluster cluster-name \
  --node-taints dedicated=experimental:NoSchedule

And then this is how to define tolerations for pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab-runner
  ...
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "experimental"
      effect: "NoSchedule"

Right after taints node was ready, we deployed the unmounted node volume deployment which was the gitlab runner jobs manager pod and tested the node pool autoscaling. We made sure that the gitlab runner jobs manager pod was able to distribute ci jobs pods to the tainted node without leaving any error vestiges.

Conclusion

Should you test on production as a default? No—prefer isolated environments when you can.

In our case, narrow isolation on a dedicated node pool made a production validation acceptable. Treat that as a constraint-driven exception, not a general recommendation.

Thanks for reading.