Local development with Kubernetes on Mac OS
Recently I started a new project where we decided to use a microservice achitecture running on Kubernetes. Lucky for us, Docker for Mac just shipped with Kubernetes on board, so setting it up was easy. However, figuring out how to route traffic between services inside the cluster, and the Java code we’re working on in our IDE wasn’t easy. In this blog, I’d like to show you how we set it up.
On a Mac, docker runs in a Virtual Machine, and in that machine Kubernetes runs
its nodes, which in turn run the pods. Let’s say we have two services foo
and
bar
which need to talk to each other over HTTP. If we want to be able to work
on the services from an IDE we have to cover three scenario’s:
foo
andbar
run in the cluster- One of them runs in the IDE, the other in the cluster
- Both run in the IDE.
Let’s look at the Kubernetes descriptor for scenario 1. We use a simple
deployment on a port, and a service of type LoadBalancer
to make sure the port
is available from outside the cluster in the browser on our host machine.
The service bar
has the
exact same configuration,
but then on port 8081
. We use the an echo service which is a container that
just echo’s the request headers back as a JSON object.
I’m assuming you have a Kubernetes running. Let’s deploy the two services
kubectl apply -f foo.yaml
kubectl apply -f bar.yaml
It that went well, kubectl get pods
will show something like:
NAME READY STATUS RESTARTS AGE
bar-7b9fd9f6b8-tjdtd 1/1 Running 0 2m
foo-6d5457d5b4-njvrq 1/1 Running 0 2m
Cool! Now let’s see if both work visit http://localhost:8080 and http://localhost:8081 to see the services running. Since you opened it in your browser, it means you can also reach it from your IDE.
Next, let’s verify the services can reach each other inside kubernetes. We can exec into the machine using
kubectl exec -t -i bar-7b9fd9f6b8-tjdtd sh
We want to use curl, which is not installed yet, so we install it using
apk update && apk add curl
After installing those you should be able to curl the services using Kubernete’s DNS using
curl foo.default.svc.cluster.local:8080
curl bar.default.svc.cluster.local:8081
Keep this window open, as we’ll use it in the next step.
That completes scenario one. Now for the harder part. Let’s say we want to work
on the foo
service in our IDE, outside kubernetes. We still need foo
to talk
to bar
which runs inside the cluster, and vice versa. But how do we set that
up?
It turns out that besides deployments, you can also define manual endpoints. The way it works is that Kubernetes creates a service, but sends the traffic to a custom IP you specify. If we use the LAN IP of our host machine, services from inside the cluster can route out of the Kubernetes and reach it. Vice versa, we can still reach the other services via the LoadBalance end points. Let’s see how that works:
First, in a new window delete the foo deployment and service:
kubectl delete service foo
kubectl delete deployment foo
Then, mimic working on Foo in my IDE by simply rinning it with docker (normally you’d start this application in your IDE or whatever development environment you use).
docker run -p 8080:8000 paddycarey/go-echo
Now verify you can reach that service on
localhost:8080. Now go back to the window that is
logged into the bar
pod, and notice that you cannot reach foo
anymore using
curl. That’s because we have no service in the cluster anymore that serves
foo
.
Next, we have to set up the new service and endpoint. For that we need the LAN
ip of our host machine, which on mac you can find using ifconfig
or if you’re
on WiFi you can click the WiFi icon in the menu bar while holding the option
key. Let’s say your IP is 192.168.0.2
. We are goign setup a new service, but
this time, the service will not have the LoadBalance
type. If we would use
that, it would conflict with the host machine that is running the docker image.
Besides that, it’s completely the same. Instead of adding a deployment, we add
an endpoint that tells the service on which IP and port it can find the service:
After applying these configs, we can see the service is up and running using
kubectl get services
:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bar LoadBalancer 10.105.255.191 localhost 8081:31545/TCP 1h
foo ClusterIP 10.111.34.149 <none> 8080/TCP 11s
Now, switch back to the window that has the bar
service open. If you retry
curl foo.default.svc.cluster.local:8080
you’ll notice that it is working
again! Congratulations, you can now send traffic back and forth between servies
in the Kuberenetes cluster, and services outside of it.
Automating
Naturally, you want to automate this, because your LAN IP changes often, and you don’t want to keep double copies of all your descriptors. In the gist, you’ll find a small bash script that grabs your IP address, and create a services from a template. This makes it easy while developing to switch to running a service from your IDE, and switch back by applying your original configuration. You can simply run
bash create_ide_service.sh foo 8080
Which will do the steps we did by hand before. If I want to switch back to the
in-cluster version, I apply the original foo.yaml
, and everything is restored.
By adding *.tmp.ide.yml
to your .gitignore
these temporary templates will
not be added to git.
This blog was written using Docker Version 18.03.0-ce-rc1-mac54 (23022), and Kubernetes v1.9.2.