Use of Let's Encrypt wildcard certs in Kubernetes

A wildcard certificate can secure any number of subdomains of a base domain (e.g. *.example.com). This allows to use a single certificate and key pair for a domain and all of its subdomains, which can make HTTPS deployment significantly easier.

Let's Encrypt wildcard certificates support went live in March 2018.

In this blog post you will learn how to setup Kubernetes Ingress controller with Heptio Contour, automate the management and issuance of wildcard TLS certificates with Jetstack Cert-Manager and sync the TLS certs across different namespaces with AppsCode Kubed.

Pre-Requisites

You need to have Kubernetes cluster available, if you don't have it, follow one the docs below to set it up on:

Helm is installed in your Kubernetes cluster.

Note: For the blog post simplicity and some cloud providers limitation I'm not using RBAC enabled cluster, all charts used in the blog post have support for RBAC, which is recommend to have on for production use.

Contour Ingress Controller

Ok, first we need to install Kubernetes Ingress controller.

As there is no Helm chart for Heptio Contour, I wrote the chart and stored it in my Helm repository.
To be able to use it you need to add my Chart repo to Helm:

$ helm repo add rimusz https://helm-charts.rimusz.net
$ helm repo up

Ok, let's install contour chart:

helm install --name contour rimusz/contour  

And to check that ingress controller is running:

$ kubectl --namespace=heptio-contour get pods -l "app=contour"
NAME                       READY     STATUS    RESTARTS   AGE  
contour-5d7f6fc8bd-nv474   2/2       Running   0          20s  

It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status with:

$ kubectl get svc --namespace heptio-contour -w contour
NAME     TYPE          CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE  
contour  LoadBalancer  10.64.12.134   <pending>     80:30321/TCP,443:32400/TCP   34s  
contour  LoadBalancer  10.64.12.134   35.238.152.138   80:30321/TCP,443:32400/TCP   46s  

Now please update your domain DNS record with the External IP.

Nice, we got our Ingress Controller installed.

TLS Cert-Manager

cert-manager high level overview diagram

We are going to install cert-manager from Helm upstream stable repo:

$ helm install --name cert-manager --namespace cert-manager \
    stable/cert-manager --set rbac.create=false

And we check it is running:

$ kubectl get pods -n cert-manager
NAME                                        READY     STATUS    RESTARTS   AGE  
cert-manager-cert-manager-7dddc6d4b-sk5x5   1/1       Running   0          28s  

Awesome.

Generating wildcard certs with Cert-Manager

For this blog post I'm using CloudFlare (sorry I'ma big fan of their services) as DNS01 Challenge Provider, check for other supported ones here.

Now we need to create a secret with CloudFlare API Key, Cert-Manager Issuer with DNS1 Challenge Provider, which will use that secret and the Cert-Manager Certificate which will save the wildcard cert of *.example.com to secret example-com-tls:

$ cat <<EOF | kubectl create -f -  
---
apiVersion: v1  
kind: Secret  
metadata:  
  name: cloudflare-api-key
  namespace: cert-manager
type: Opaque  
data:  
  api-key.txt: <copy here base64 encoded CloudFlare API Key>

---
apiVersion: certmanager.k8s.io/v1alpha1  
kind: Issuer  
metadata:  
  name: letsencrypt-staging
  namespace: cert-manager
spec:  
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging

    # ACME DNS-01 provider configurations
    dns01:
      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:
        - name: cf-dns
          cloudflare:
            email: [email protected]
            # A secretKeyRef to a cloudflare api key
            apiKeySecretRef:
              name: cloudflare-api-key
              key: api-key.txt

---
apiVersion: certmanager.k8s.io/v1alpha1  
kind: Certificate  
metadata:  
  name: example-com
  namespace: cert-manager
spec:  
  secretName: example-com-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: '*.example.com'
  acme:
    config:
    - dns01:
        provider: cf-dns
      domains:
      - '*.example.com'

EOF  

Note: Please update the example template above with your CloudFlare API key, email address and domain name.

If all went successfully you should see your wildcard cert example-com-tls:

$ kubectl -n cert-manager get secret
NAME                                    TYPE                                  DATA      AGE  
cert-manager-cert-manager-token-whctf   kubernetes.io/service-account-token   3         4h  
cloudflare-api-key                      Opaque                                1         17m  
default-token-cfmln                     kubernetes.io/service-account-token   3         4h  
letsencrypt-staging                     Opaque                                1         37m  
example-com-tls                         kubernetes.io/tls                     2         12m  

Yay.

Sync wildcard certs between Kubernetes namespaces

Kubernetes best practices recommend to install app per namespace, but our wildcard cert example-com-tls is stored in the cert-manager namespace, so we need to sync it across namespaces when the cert gets reissued or new app in the new namespace gets added.

For that we are going to use a nice tool from AppsCode Kubed.
Check their other tools, you could find more useful ones for you.

Kubed can be installed via Helm using the chart from AppsCode Charts Repository:

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm install appscode/kubed --name kubed --namespace kube-system \
    --set apiserver.enabled=false \
    --set config.clusterName=my_staging_cluster \
    --set rbac.create=false

Note: Set cluster-name to something meaningful to you, say, prod, prod-us-east, qa, etc. Which also enables kubed's ConfigSyncer feature.

To verify that Kubed has started, run:

$  kubectl --namespace=kube-system get deployments -l "release=kubed, app=kubed"
NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE  
kubed-kubed   1         1         1            1           15s  

Cool.

Ok, now let's create a few namespaces where we are going to sync our wildcard cert to:

$ kubectl create namespace demo1
$ kubectl create namespace demo2

Now we need to put the annotation as per docs to our wildcard secret, so kubed knows what to sync across namespaces:

$ kubectl annotate secret example-com-tls -n cert-manager kubed.appscode.com/sync="app=kubed"
secret "example-com-tls" annotated  

From now on kubed will start syncing secret example-com-tls to only namespaces which have label-selector app=kubed.

Let's annotate namespace demo1:

$ kubectl label namespace demo1 app=kubed

Now we should be able to see the secret in the namespace demo1:

$ kubectl -n demo1 get secret example-com-tls
NAME              TYPE                DATA      AGE  
example-com-tls   kubernetes.io/tls   2         2h  

And if we check namespace demo2 we will not see the example-com-tls there:

$ kubectl -n demo2 get secret example-com-tls
Error from server (NotFound): secrets "example-com-tls" not found  

Putting the annotation on to demo2 namespace will trigger an instant secret sync as well.

Now you can install web apps into demo1 and demo2 namespaces, and apps be able to use the same wildcard cert for their ingress rules.

What's next

Moving forward to production releases and receiving the real TLS certs from Let's Encrypt add new Issuer and Certificate as per example below:

$ cat <<EOF | kubectl create -f -  
---
apiVersion: certmanager.k8s.io/v1alpha1  
kind: Issuer  
metadata:  
  name: letsencrypt-prod
  namespace: cert-manager
spec:  
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]e.com

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod

    # ACME DNS-01 provider configurations
    dns01:
      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:
        - name: cf-dns-prod
          cloudflare:
            email: [email protected]
            # A secretKeyRef to a cloudflare api key
            apiKeySecretRef:
              name: cloudflare-api-key
              key: api-key.txt

---
apiVersion: certmanager.k8s.io/v1alpha1  
kind: Certificate  
metadata:  
  name: example-com-prod
  namespace: cert-manager
spec:  
  secretName: example-com-tls-prod
  issuerRef:
    name: letsencrypt-prod
  commonName: '*.example.com'
  acme:
    config:
    - dns01:
        provider: cf-dns-prod
      domains:
      - '*.example.com'

EOF  

Happy secure web content serving for you :)

Wrap Up

In this blog post we learned how to install Heptio Contour Ingress Controller,
install Jetstack Cert-Manager and set it up to issue wildcard certs from Let's Encrypt, and with the help of AppsCode Kubed sync wildcard certs across labeled namespaces.