Handling Multiple Kubernetes Clusters Using kubesh

Handling Multiple Kubernetes Clusters Using kubesh

When you are working with multiple Kuberenetes clusters, you have to tell the various tools that you use which cluster to look for. There are some tools and workarounds for this, but I find them cumbersome and/or error prone. So scratched my itch: I introduce you kubesh:

Motivation

I use GKE for work, and for my builderscon project.

This does mean I have two google accounts: One for work, and one for my private needs. And I have multiple clusters in each.

For the longest time (while cumbersome) re-authenticating myself and letting gcloud store Application Default Credentials in ~/.config/gcloud/application_default_credentials.json and have my kubeconfig updated via gcloud container clusters get-credentials from time to time did the job for me.

But over time, I started to face some problematic situations.

One problem was that I started to have incidents where I would be working on one cluster during the morning, and then sometime later, I decide to work on some other project, and I realize that I was working on the cluster from the morning, when I actually wanted to work on another. I haven’t done this yet, but I shudder to think what would happen if I somehow decided delete a Kubernetes resource on an unintended cluster.

There were also cases where I’d be working on two clusters which needed to communicate with each other. For example, maybe you have cluster that runs an organization-wide Prometheus instance, and it wants to monitor other Kubernetes clusters. In this situation, I’d like to be monitoring both clusters at the same time.

Prior Arts

There are other tools that do similar things. One tool that I was recommended was kubectx:

While this makes managing contexts much easier, because it uses kubectl use-context underneath, the effects are global. That means, if you use it in one terminal, and move back to another terminal to use kubectl, you may not be working on the intended cluster anymore.

You could also create multiple kubecfg files, and switch by setting different KUBE_CONFIG variables. This seems like it’s the most widespread way, but I find having to maintain kubeconfig entries unacceptable. Actually, I think I can certainly do it, but would I want to tell my colleagues to learn how to create and manage those files? Hmm…

Scratching My Itch

So now I have kubesh. This basically spawns a new shell, which points kubectl, stern, and helm at their respective wrappers. These wrappers just provide the kubernetes context that you specified when you started kubesh

$ kubesh --context=my-context
$ kubectl get pods # gets data from "my-context"
$ helm list        # lists charts from "my-context"
$ stern pod-name   # tails pod-name from "my-context"

The effects are local to your shell session. On another terminal, you are perfectly safe from its effects, and if you were working on a different cluster, your setup should not be changed.

If you want more consistency, you can open up another new shell, and do the following:

$ kubesh --context=my-other-context

And this will force subsequent calls to kubectl and friends refer to the cluster specified by my-other-context only.

You can even run random kubectl use-context in some other terminal, and it would still work (note: don’t run use-context while kubesh is in effect).

peco integration

If you get tired of writing long context names, just integrate kubesh with peco (disclaimer: I’m the author)

Look into the README for kubesh for how to integrate peco :

Now you can select a context interactively!