Træfik as Kubernetes Ingress Controller on ARM with automatic TLS certificates
TL;DR: How I configured Træfik with automatic TLS certificates from Let’s Encrypt as an Ingress Controller for my Kubernetes Cluster on a bare metal ARM hardware running in my living room.
Several blog posts have already described how they deployed Træfik as an Ingress Controller for Kubernetes with automatic TLS using Let’s Encrypt. While configuring it myself, I realized it wasn’t so straightforward for my setup. I decided to share with you how I made it work for me, and how you can customize this configuration to meet your needs.
In this blog post I’m not going to explain what an Ingress Controller is, for that please refer to the official documentation. You can also find a comprehensible and exhaustive description of Træfik at the project’s homepage.
An Kubernetes Cluster on ARM
I run a Kubernetes Cluster on a bare metal ARM hardware. My cluster consists of a mix of Orange Pi PCs and Raspberry Pi boards. In this cluster I carry out my experiments on deployment automation with CI/CD, monitoring, networking, storage, etc.
An early version of my Kubernetes cluster on ARM with cardboard.
I have a toy domain name homeservice.click pointing to my home router. I use the Dynu free dynamic DNS service which allows me to resolve my domain name to my home IP address. I don’t run any critical services in that domain, so it’s not a big deal in case my internet connection is down. I should probably be more worried about the possible security issues related to that setup. But I do my best and try to keep all my software up to date and not let any interesting ports be left unguarded. A relatively small price to pay for the enormous fun of having my own “cloud provider”. This setup was mostly guided by the Hypriot blog post “Setup Kubernetes on a Raspberry Pi Cluster easily the official way!” plus some additional tweaks.
Having this setup I wanted to guarantee that when I’m accessing the domain I’m accessing my own machines at home. So I decided to have TLS connections and use Let’s Encrypt to easily get a certificate for my domain. I also knew it would be fun figuring out how to configure it.
Ingress Controller and related manifests
I use Kubernetes 1.6 and RBAC authorization is enabled by default when setting up the cluster using kubeadm. I decided to stick with it to learn how it works. I defined a service account for the Træfik application and allowed this account to perform routing and proxying related operations in the Kubernetes API server. Here’s how my manifests for that look like:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-role
rules:
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs:
- get
- watch
- list
- proxy
- use
- redirect
- apiGroups:
- "extensions"
resources:
- "ingresses"
verbs:
- get
- watch
- list
- proxy
- use
- redirect
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-role
subjects:
- kind: ServiceAccount
name: traefik
namespace: kube-system
I want the Træfik application to manage the Ingress resources across
all namespaces of my Kubernetes cluster. That’s why it’s a ClusterRole
with access to all my services, ingresses, and endpoints resources. I
believe there’s still too much power defined in this ClusterRole
, but
I didn’t have the time yet to figure out the bare minimum
necessary. Since I’m not enabling any changes to be performed by the
Træfik service account, it shouldn’t be a big deal to leave it as it
is now.
I then define a ConfigMap
resource with the Træfik configuration. Here
we’ll need to add some information to enable the TLS certificate
generation.
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-https-cfg
namespace: kube-system
data:
traefik.toml: |
# traefik.toml
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[acme]
email = "youremail@example.com"
storage = "/etc/traefik/acme.json"
entryPoint = "https"
onDemand = true
onHostRule = true
caServer = "https://acme-v01.api.letsencrypt.org/directory"
[[acme.domains]]
main = "example.com"
You should replace youremail@example.com with your own
e-mail. Remember to also update the main = example.com
line. The
port 80 is also enabled here, so Træfik can redirect all the traffic
to the port 443. Another interesting field is the storage =
"/etc/traefik/acme.json"
line, this indicates where Træfik will save
the certificates generated by Let’s Encrypt. It should be a mount in
your Pod
. Where this mount will point is going to depend on your
setup. That’s my Træfik deployment manifest:
apiVersion: v1
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: traefik-ingress-controller-https
namespace: kube-system
labels:
k8s-app: traefik-ingress-controller-https
spec:
replicas: 1
selector:
matchLabels:
k8s-app: traefik-ingress-controller-https
template:
metadata:
labels:
k8s-app: traefik-ingress-controller-https
spec:
serviceAccount: traefik
terminationGracePeriodSeconds: 15
hostNetwork: true
nodeSelector:
kubernetes.io/hostname: red
volumes:
- name: traefik-cache
hostPath:
path: /tmp/traefik
- name: traefik-config
configMap:
name: traefik-https-cfg
containers:
- image: hypriot/rpi-traefik
name: traefik-ingress-controller
resources:
limits:
cpu: 1500m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
- name: admin
containerPort: 8888
volumeMounts:
- mountPath: /etc/traefik
name: traefik-cache
- mountPath: /config-files
name: traefik-config
args:
- --web
- --web.address=:8888
- --kubernetes
- --logLevel=INFO
- --configFile=/config-files/traefik.toml
The deployment manifest has a few things you could consider
customizing for your own use. The first part to customize is the
nodeSelector
field. Because of my setup I assigned the Traefik Pod
to a specific node of my cluster. In my setup it’s required because I
need a fixed internal IP that I can configure in my router to redirect
all the traffic on the ports 80 and 443 to it. The hostNetwork
field
also complies with this requirement. Depending on where and how you
run your Kubernetes cluster you might not have this limitation.
Another interesting part of the deployment manifest to customize is
the “traefik-cache” volume. I use hostPath
type of volume because,
as I mentioned previously, I assigned the Traefik Pod
to a specific
node. This volume can also be an emptyDir
type, but every time your
Pod
restarts, new certificates are going to be requested from Let’s
Encrypt. This might lead you to reach the rate limit in Let’s Encrypt
and stop you from getting certificates for your domain. The best
option is to either use hostPath
or a volume provisioned
specifically for this.
With little modification in those manifests you can also have TLS certificates assigned to your domain. Træfik will automatically manage everything necessary for it to work from there.
Exposing a service under a new subdomain
Traefik is now configured as our Ingress Controller and is watching the Kubernetes API for the creation of new Ingress resources. We can expose a service from the Kubernetes cluster to the internet with a Ingress resource manifest. First, let’s deploy an example application.
$ kubectl run example-app --image=hypriot/rpi-busybox-httpd --replicas=2 --port=80
Our deployment have triggered the creation of 2 Pod
s of our example
application. Now we create a Service that will balance the traffic
between those Pod
s.
$ kubectl expose deployment example-app --port 80
Finally, we can expose the example-app
to the internet under a new
subdomain with an Ingress controller like:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-app-ingress
namespace: default
spec:
rules:
- host: www.homeservice.click
http:
paths:
- path: /
backend:
serviceName: example-app
servicePort: 80
After that, in a few seconds, it will be possible to access our
example-app
through the URL
https://www.homeservice.click. When
accessing the new subdomain for the first time, right after deploying,
the certificate will appear as invalid. The certificate for a new
subdomain takes a few minutes to be issued.
To clean up, you can run:
$ kubectl delete deployment example-app
$ kubectl delete ingress example-app-ingress
Moving on
It might be silly, but since I have configured the HTTPS, I feel more confident to deploy applications to my Kubernetes Cluster. I also had fun figuring out how to make it work. I believe I don’t have a common setup, but it might be of help for you to know how it works and use it as reference for your own manifests.
A small improvement on my OrangePi PC boards organization.
In a future post I plan to dig into more details on how I have configured my Kubernetes Cluster and home network. Maybe it’ll serve as an inspiration for you to also have your own “cloud” at home to deploy your projects. If you’re interested in this, please let me know on Twitter (@rafaelcaricio) or in the comments section below. If you have any questions about anything I wrote here, I’ll be glad to answer.
Post by: Rafael Caricio