Recently I started a new project where we decided to use a microservice achitecture running on Kubernetes. Lucky for us, Docker for Mac just shipped with Kubernetes on board, so setting it up was easy. However, figuring out how to route traffic between services inside the cluster, and the Java code we’re working on in our IDE wasn’t easy. In this blog, I’d like to show you how we set it up.
On a Mac, docker runs in a Virtual Machine, and in that machine Kubernetes
runs its nodes, which in turn run the pods. Let’s say we have two services
bar which need to talk to each other over HTTP. If we want to be able
to work on the services from an IDE we have to cover three scenario’s:
barrun in the cluster
- One of them runs in the IDE, the other in the cluster
- Both run in the IDE.
Let’s look at the Kubernetes descriptor for scenario 1. We use a simple
deployment on a port, and a service of type
LoadBalancer to make sure
the port is available from outside the cluster in the browser on our host machine.
bar has the exact same configuration,
but then on port
We use the an echo service which is a container that just echo’s the request
headers back as a JSON object.
I’m assuming you have a Kubernetes running. Let’s deploy the two services
kubectl apply -f foo.yaml kubectl apply -f bar.yaml
It that went well,
kubectl get pods will show something like:
NAME READY STATUS RESTARTS AGE bar-7b9fd9f6b8-tjdtd 1/1 Running 0 2m foo-6d5457d5b4-njvrq 1/1 Running 0 2m
Next, let’s verify the services can reach each other inside kubernetes. We can exec into the machine using
kubectl exec -t -i bar-7b9fd9f6b8-tjdtd sh
We want to use curl, which is not installed yet, so we install it using
apk update && apk add curl
After installing those you should be able to curl the services using Kubernete’s DNS using
curl foo.default.svc.cluster.local:8080 curl bar.default.svc.cluster.local:8081
Keep this window open, as we’ll use it in the next step.
That completes scenario one. Now for the harder part. Let’s say we
want to work on the
foo service in our IDE, outside kubernetes.
We still need
foo to talk to
bar which runs inside the cluster,
and vice versa. But how do we set that up?
It turns out that besides deployments, you can also define manual endpoints. The way it works is that Kubernetes creates a service, but sends the traffic to a custom IP you specify. If we use the LAN IP of our host machine, services from inside the cluster can route out of the Kubernetes and reach it. Vice versa, we can still reach the other services via the LoadBalance end points. Let’s see how that works:
First, in a new window delete the foo deployment and service:
kubectl delete service foo kubectl delete deployment foo
Then, mimic working on Foo in my IDE by simply rinning it with docker (normally you’d start this application in your IDE or whatever development environment you use).
docker run -p 8080:8000 paddycarey/go-echo
Now verify you can reach that service on localhost:8080.
Now go back to the window that is logged into the
bar pod, and notice that
you cannot reach
foo anymore using curl. That’s because we have no service
in the cluster anymore that serves
Next, we have to set up the new service and endpoint. For that we need the LAN ip of
our host machine, which on mac you can find using
ifconfig or if you’re
on WiFi you can click the WiFi icon in the menu bar while holding the option key.
Let’s say your IP is
192.168.0.2. We are goign setup a new service, but
this time, the service will not have the
LoadBalance type. If we would use that, it
would conflict with the host machine that is running the docker image. Besides
that, it’s completely the same. Instead of adding a deployment, we add an endpoint
that tells the service on which IP and port it can find the service:
After applying these configs, we can see the service is up and running
kubectl get services:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bar LoadBalancer 10.105.255.191 localhost 8081:31545/TCP 1h foo ClusterIP 10.111.34.149 <none> 8080/TCP 11s
Now, switch back to the window that has the
bar service open. If you retry
curl foo.default.svc.cluster.local:8080 you’ll notice that it is working again!
Congratulations, you can now send traffic back and forth between servies in
the Kuberenetes cluster, and services outside of it.
Naturally, you want to automate this, because your LAN IP changes often, and you don’t want to keep double copies of all your descriptors. In the gist, you’ll find a small bash script that grabs your IP address, and create a services from a template. This makes it easy while developing to switch to running a service from your IDE, and switch back by applying your original configuration. You can simply run
bash create_ide_service.sh foo 8080
Which will do the steps we did by hand before. If I want to switch back
to the in-cluster version, I apply the original
foo.yaml, and everything
is restored. By adding
*.tmp.ide.yml to your
.gitignore these temporary
templates will not be added to git.
This blog was written using Docker Version 18.03.0-ce-rc1-mac54 (23022), and Kubernetes v1.9.2.