A Kubernetes dev environment for Mac

This post will explain how you can hack Kubernetes on a Mac, and easily compile and run the resulting cluster locally. Specifically, I’m talking about working on the Kubernetes codebase directly and not a project that uses Kubernetes.¹ The goal is to be able to run the local-up-cluster script as described in the documentation for running locally.

That documentation mentions that local-up-cluster depends on Linux, and offers Minikube as an alternative for others. Maybe I don’t understand Minikube well enough, but it doesn’t seem like a viable alternative to me. Minikube appears to work off of a Kubernetes build and the local-up-cluster script is used exactly to avoid creating a build – it compiles the various Kubernetes components and runs them directly, only dockerizing things as required/requested.

After checking that, yes, local-up-cluster gets hung-up when run from OS X, I instead created a Docker image with all of the prerequisites for compiling and running Kubernetes. By mounting your local Kubernetes source directory on a container running this image, you can modify Kubernetes and then run the results with only the one extra step of starting the container.²

Running the dev environment

Our setup requires having a working Docker environment and a local copy of the Kubernetes source:

mkdir -p $GOPATH/src/k8s.io/
git clone https://github.com/kubernetes/kubernetes.git $GOPATH/src/k8s.io/kubernetes

I’ve put the source in my GOPATH, but this isn’t necessary.

We then run the alexcharlton/k8s-dev image, binding our local docker socket³ and the Kubernetes directory:

docker run -v /var/run/docker.sock:/var/run/docker.sock \
  -v $GOPATH/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes \
  --name k8s-dev --rm -it alex-charlton/k8s-dev /bin/bash

With this we have a functioning development environment from which we can run local-up-cluster. Before we do, though, let’s prove to ourselves that we can leave our mark on the codebase. We’ll add a log statement at the entrypoint of the API server, affecting the following change on cmd/kube-apiserver/app/options/options.go:

@@ -25,6 +25,7 @@ import (
        kubeletclient "k8s.io/kubernetes/pkg/kubelet/client"
        "k8s.io/kubernetes/pkg/master/ports"

+       "github.com/golang/glog"
        "github.com/spf13/pflag"
)

@@ -55,6 +56,7 @@ func NewAPIServer() *APIServer {
                },
                WebhookTokenAuthnCacheTTL: 2 * time.Minute,
        }
+       glog.Infof("Hello from inside Kubernetes!")
        return &s
}

And now:

root@bcc2570ffab1:/go/src/k8s.io/kubernetes# ./hack/local-up-cluster.sh

...
[Some compiling and other stuff that takes a while]
...

Local Kubernetes cluster is running. Press Ctrl-C to shut it down.

Logs:
  /tmp/kube-apiserver.log
  /tmp/kube-controller-manager.log
  /tmp/kube-proxy.log
  /tmp/kube-scheduler.log
  /tmp/kubelet.log

To start using your cluster, open up another terminal/tab and run:

  export KUBERNETES_PROVIDER=local

  cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/apiserver.crt
  cluster/kubectl.sh config set-credentials myself --username=admin --password=admin
  cluster/kubectl.sh config set-context local --cluster=local --user=myself
  cluster/kubectl.sh config use-context local
  cluster/kubectl.sh

The local-up-cluster script will output something similar to the above. We are invited to interact with our new cluster, via kubectl, and who can resist that? In another terminal, we must first ssh into our container with docker exec -it k8s-dev /bin/bash before we can paste in the above output. After doing so we can peer into our new cluster:

root@14525c848683:/go/src/k8s.io/kubernetes# cluster/kubectl.sh get services
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   10.0.0.1     <none>        443/TCP   39s

And of course, I haven’t forgotten about the log statement:

root@14525c848683:/go/src/k8s.io/kubernetes# head -n 1 /tmp/kube-apiserver.log

I1025 14:29:13.361459   20983 options.go:59] Hello from inside Kubernetes!