Træfik as Kubernetes Ingress Controller on ARM with automatic TLS certificates

TL;DR: How I configured Træfik with automatic TLS certificates from Let’s Encrypt as an Ingress Controller for my Kubernetes Cluster on a bare metal ARM hardware running in my living room.

Several blog posts have already described how they deployed Træfik as an Ingress Controller for Kubernetes with automatic TLS using Let’s Encrypt. While configuring it myself, I realized it wasn’t so straightforward for my setup. I decided to share with you how I made it work for me, and how you can customize this configuration to meet your needs.

In this blog post I’m not going to explain what an Ingress Controller is, for that please refer to the official documentation. You can also find a comprehensible and exhaustive description of Træfik at the project’s homepage.

An Kubernetes Cluster on ARM

I run a Kubernetes Cluster on a bare metal ARM hardware. My cluster consists of a mix of Orange Pi PCs and Raspberry Pi boards. In this cluster I carry out my experiments on deployment automation with CI/CD, monitoring, networking, storage, etc.

An early version of my Kubernetes cluster on ARM with cardboard.

I have a toy domain name homeservice.click pointing to my home router. I use the Dynu free dynamic DNS service which allows me to resolve my domain name to my home IP address. I don’t run any critical services in that domain, so it’s not a big deal in case my internet connection is down. I should probably be more worried about the possible security issues related to that setup. But I do my best and try to keep all my software up to date and not let any interesting ports be left unguarded. A relatively small price to pay for the enormous fun of having my own “cloud provider”. This setup was mostly guided by the Hypriot blog post “Setup Kubernetes on a Raspberry Pi Cluster easily the official way!” plus some additional tweaks.

Having this setup I wanted to guarantee that when I’m accessing the domain I’m accessing my own machines at home. So I decided to have TLS connections and use Let’s Encrypt to easily get a certificate for my domain. I also knew it would be fun figuring out how to configure it.

Ingress Controller and related manifests

I use Kubernetes 1.6 and RBAC authorization is enabled by default when setting up the cluster using kubeadm. I decided to stick with it to learn how it works. I defined a service account for the Træfik application and allowed this account to perform routing and proxying related operations in the Kubernetes API server. Here’s how my manifests for that look like:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-role
rules:
- apiGroups: [""]
  resources: ["services", "endpoints"]
  verbs:
    - get
    - watch
    - list
    - proxy
    - use
    - redirect
- apiGroups:
    - "extensions"
  resources:
    - "ingresses"
  verbs:
    - get
    - watch
    - list
    - proxy
    - use
    - redirect
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-role
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-role
subjects:
- kind: ServiceAccount
  name: traefik
  namespace: kube-system

I want the Træfik application to manage the Ingress resources across all namespaces of my Kubernetes cluster. That’s why it’s a ClusterRole with access to all my services, ingresses, and endpoints resources. I believe there’s still too much power defined in this ClusterRole, but I didn’t have the time yet to figure out the bare minimum necessary. Since I’m not enabling any changes to be performed by the Træfik service account, it shouldn’t be a big deal to leave it as it is now.

I then define a ConfigMap resource with the Træfik configuration. Here we’ll need to add some information to enable the TLS certificate generation.

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-https-cfg
  namespace: kube-system
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.http.redirect]
      entryPoint = "https"
      [entryPoints.https]
      address = ":443"
      [entryPoints.https.tls]
    [acme]
    email = "youremail@example.com"
    storage = "/etc/traefik/acme.json"
    entryPoint = "https"
    onDemand = true
    onHostRule = true
    caServer = "https://acme-v01.api.letsencrypt.org/directory"
    [[acme.domains]]
    main = "example.com"

You should replace youremail@example.com with your own e-mail. Remember to also update the main = example.com line. The port 80 is also enabled here, so Træfik can redirect all the traffic to the port 443. Another interesting field is the storage = "/etc/traefik/acme.json" line, this indicates where Træfik will save the certificates generated by Let’s Encrypt. It should be a mount in your Pod. Where this mount will point is going to depend on your setup. That’s my Træfik deployment manifest:

apiVersion: v1
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller-https
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-controller-https
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-controller-https
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-controller-https
    spec:
      serviceAccount: traefik
      terminationGracePeriodSeconds: 15
      hostNetwork: true
      nodeSelector:
        kubernetes.io/hostname: red
      volumes:
      - name: traefik-cache
        hostPath:
          path: /tmp/traefik
      - name: traefik-config
        configMap:
          name: traefik-https-cfg
      containers:
      - image: hypriot/rpi-traefik
        name: traefik-ingress-controller
        resources:
          limits:
            cpu: 1500m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: admin
          containerPort: 8888
        volumeMounts:
        - mountPath: /etc/traefik
          name: traefik-cache
        - mountPath: /config-files
          name: traefik-config
        args:
        - --web
        - --web.address=:8888
        - --kubernetes
        - --logLevel=INFO
        - --configFile=/config-files/traefik.toml

The deployment manifest has a few things you could consider customizing for your own use. The first part to customize is the nodeSelector field. Because of my setup I assigned the Traefik Pod to a specific node of my cluster. In my setup it’s required because I need a fixed internal IP that I can configure in my router to redirect all the traffic on the ports 80 and 443 to it. The hostNetwork field also complies with this requirement. Depending on where and how you run your Kubernetes cluster you might not have this limitation.

Another interesting part of the deployment manifest to customize is the “traefik-cache” volume. I use hostPath type of volume because, as I mentioned previously, I assigned the Traefik Pod to a specific node. This volume can also be an emptyDir type, but every time your Pod restarts, new certificates are going to be requested from Let’s Encrypt. This might lead you to reach the rate limit in Let’s Encrypt and stop you from getting certificates for your domain. The best option is to either use hostPath or a volume provisioned specifically for this.

With little modification in those manifests you can also have TLS certificates assigned to your domain. Træfik will automatically manage everything necessary for it to work from there.

Exposing a service under a new subdomain

Traefik is now configured as our Ingress Controller and is watching the Kubernetes API for the creation of new Ingress resources. We can expose a service from the Kubernetes cluster to the internet with a Ingress resource manifest. First, let’s deploy an example application.

$ kubectl run example-app --image=hypriot/rpi-busybox-httpd --replicas=2 --port=80

Our deployment have triggered the creation of 2 Pods of our example application. Now we create a Service that will balance the traffic between those Pods.

$ kubectl expose deployment example-app --port 80

Finally, we can expose the example-app to the internet under a new subdomain with an Ingress controller like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-app-ingress
  namespace: default
spec:
  rules:
    - host: www.homeservice.click
      http:
        paths:
          - path: /
            backend:
              serviceName: example-app
              servicePort: 80

After that, in a few seconds, it will be possible to access our example-app through the URL https://www.homeservice.click. When accessing the new subdomain for the first time, right after deploying, the certificate will appear as invalid. The certificate for a new subdomain takes a few minutes to be issued.

To clean up, you can run:

$ kubectl delete deployment example-app
$ kubectl delete ingress example-app-ingress

Moving on

It might be silly, but since I have configured the HTTPS, I feel more confident to deploy applications to my Kubernetes Cluster. I also had fun figuring out how to make it work. I believe I don’t have a common setup, but it might be of help for you to know how it works and use it as reference for your own manifests.

A small improvement on my OrangePi PC boards organization.

In a future post I plan to dig into more details on how I have configured my Kubernetes Cluster and home network. Maybe it’ll serve as an inspiration for you to also have your own “cloud” at home to deploy your projects. If you’re interested in this, please let me know on Twitter (@rafaelcaricio) or in the comments section below. If you have any questions about anything I wrote here, I’ll be glad to answer.