Using ExternalDNS on Google Cloud Platform to automate DNS setup

ExternalDNS is a tool that synchronizes exposed Kubernetes Services and Ingresses with DNS providers.

This doc explains how to set up ExternalDNS within a Knative cluster using Google Cloud DNS to automate the process of publishing the Knative domain.

Set up environtment variables

Run the following command to configure the environment variables

export PROJECT_NAME=<your-google-cloud-project-name>

export CUSTOM_DOMAIN=<your-custom-domain-used-in-knative>

export CLUSTER_NAME=<knative-cluster-name>

export CLUSTER_ZONE=<knative-cluster-zone>

Set up Kubernetes Engine cluster with CloudDNS read/write permissions

There are two ways to set up a Kubernetes Engine cluster with CloudDNS read/write permissions.

Cluster with Cloud DNS scope

You can create a GKE cluster with Cloud DNS scope by entering the following command:

gcloud container clusters create $CLUSTER_NAME \
    --zone=$CLUSTER_ZONE \
    --cluster-version=latest \
    --machine-type=n1-standard-4 \
    --enable-autoscaling --min-nodes=1 --max-nodes=10 \
    --enable-autorepair \
    --scopes=service-control,service-management,compute-rw,storage-ro,cloud-platform,logging-write,monitoring-write,pubsub,datastore,"https://www.googleapis.com/auth/ndev.clouddns.readwrite" \
    --num-nodes=3

Note that by using this way, any pod within the cluster will have permissions to read/write CloudDNS.

Cluster with Cloud DNS Admin Service Account credential

Create a GKE cluster without Cloud DNS scope by entering the following command:

gcloud container clusters create $CLUSTER_NAME \
    --zone=$CLUSTER_ZONE \
    --cluster-version=latest \
    --machine-type=n1-standard-4 \
    --enable-autoscaling --min-nodes=1 --max-nodes=10 \
    --enable-autorepair \
    --scopes=service-control,service-management,compute-rw,storage-ro,cloud-platform,logging-write,monitoring-write,pubsub,datastore \
    --num-nodes=3

Create a new service account for Cloud DNS admin role.

# Name of the service account you want to create.
export CLOUD_DNS_SA=cloud-dns-admin

gcloud --project $PROJECT_NAME iam service-accounts \
    create $CLOUD_DNS_SA \
    --display-name "Service Account to support ACME DNS-01 challenge."

Bind the role dns.admin to the newly created service account.

# Fully-qualified service account name also has project-id information.
export CLOUD_DNS_SA=$CLOUD_DNS_SA@$PROJECT_NAME.iam.gserviceaccount.com

gcloud projects add-iam-policy-binding $PROJECT_NAME \
    --member serviceAccount:$CLOUD_DNS_SA \
    --role roles/dns.admin

Download the secret key file for your service account.

gcloud iam service-accounts keys create ~/key.json \
    --iam-account=$CLOUD_DNS_SA

Upload the service account credential to your cluster. This command uses the secret name cloud-dns-key, but you can choose a different name.

kubectl create secret generic cloud-dns-key \
    --from-file=key.json=$HOME/key.json

Delete the local secret

rm ~/key.json

Now your cluster has the credential of your CloudDNS admin service account. And it can be used to access your Cloud DNS. You can enforce the access of the credentail secret within your cluster, so that only the pods that have the permission to get the credential secret can access your Cloud DNS.

Set up Knative

Follow the instruction to install Knative on your cluster.
Configure Knative to use your custom domain.

kubectl edit cm config-domain --namespace knative-serving

This command opens your default text editor and allows you to edit the config map.

apiVersion: v1
data:
  example.com: ""
kind: ConfigMap
[...]

Edit the file to replace example.com with your custom domain (the value of $CUSTOM_DOMAIN) and save your changes. In this example, we use domain external-dns-test.my-org.do for all routes:

apiVersion: v1
data:
  external-dns-test.my-org.do: ""
kind: ConfigMap
[...]

Set up ExternalDNS

This guide uses Google Cloud Platform as an example to show how to set up ExternalDNS. You can find detailed instructions for other cloud providers in the ExternalDNS documentation.

Create a DNS zone for managing DNS records

Skip this step if you already have a zone for managing the DNS records of your custom domain.

A DNS zone which will contain the managed DNS records needs to be created.

Use the following command to create a DNS zone with Google Cloud DNS:

export DNS_ZONE_NAME=<dns-zone-name>

gcloud dns managed-zones create $DNS_ZONE_NAME \
    --dns-name $CUSTOM_DOMAIN \
    --description "Automatically managed zone by kubernetes.io/external-dns"

Make a note of the nameservers that were assigned to your new zone.

gcloud dns record-sets list \
    --zone $DNS_ZONE_NAME \
    --name $CUSTOM_DOMAIN \
    --type NS

You should see output similar to the following assuming your custom domain is external-dns-test.my-org.do:

NAME                             TYPE  TTL    DATA
external-dns-test.my-org.do.  NS    21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.

In this case, the DNS nameservers are ns-cloud-{e1-e4}.googledomains.com. Yours could differ slightly, e.g. {a1-a4}, {b1-b4} etc.

If this zone has the parent zone, you need to add NS records of this zone into the parent zone so that this zone can be found from the parent. Assuming the parent zone is my-org-do and the parent domain is my-org.do, and the parent zone is also hosted at Google Cloud DNS, you can follow these steps to add the NS records of this zone into the parent zone:

gcloud dns record-sets transaction start --zone "my-org-do"
gcloud dns record-sets transaction add ns-cloud-e{1..4}.googledomains.com. \
    --name "external-dns-test.my-org.do." --ttl 300 --type NS --zone "my-org-do"
gcloud dns record-sets transaction execute --zone "my-org-do"

Deploy ExternalDNS

Firstly, choose the manifest of ExternalDNS.

Use below manifest if you set up your cluster with CloudDNS scope.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.opensource.zalan.do/teapot/external-dns:latest
          args:
            - --source=service
            - --domain-filter=$CUSTOM_DOMAIN # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
            - --provider=google
            - --google-project=$PROJECT_NAME # Use this to specify a project different from the one external-dns is running inside
            - --policy=sync # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
            - --registry=txt
            - --txt-owner-id=my-identifier

Or use below manifest if you set up your cluster with CloudDNS service account credential.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["pods,secrets"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      volumes:
        - name: google-cloud-key
          secret:
            secretName: cloud-dns-key
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.opensource.zalan.do/teapot/external-dns:latest
          volumeMounts:
            - name: google-cloud-key
              mountPath: /var/secrets/google
          env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /var/secrets/google/key.json
          args:
            - --source=service
            - --domain-filter=$CUSTOM_DOMAIN # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
            - --provider=google
            - --google-project=$PROJECT_NAME # Use this to specify a project different from the one external-dns is running inside
            - --policy=sync # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
            - --registry=txt
            - --txt-owner-id=my-identifier

Then use the following command to apply the manifest you chose to install ExternalDNS

cat <<EOF | kubectl apply --filename -
<your-chosen-manifest>
EOF

You should see ExternalDNS is installed by running:

kubectl get deployment external-dns

Configuring Knative Gateway service

In order to publish the Knative Gateway service, the annotation external-dns.alpha.kubernetes.io/hostname: '*.$CUSTOM_DOMAIN needs to be added into Knative gateway service:

INGRESSGATEWAY=istio-ingressgateway

kubectl edit svc $INGRESSGATEWAY --namespace istio-system

This command opens your default text editor and allows you to add the annotation to knative-ingressgateway service. After you've added your annotation, your file may look similar to this (assuming your custom domain is external-dns-test.my-org.do):

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: '*.external-dns-test.my-org.do'
    ...

Verify ExternalDNS works

After roughly two minutes, check that a corresponding DNS record for your service was created.

gcloud dns record-sets list     --zone $DNS_ZONE_NAME     --name "*.$CUSTOM_DOMAIN."

You should see output similar to:

NAME                            TYPE  TTL  DATA
*.external-dns-test.my-org.do.  A     300  35.231.248.30
*.external-dns-test.my-org.do.  TXT   300  "heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=service/istio-system/knative-ingressgateway"

Verify domain has been published

You can check if the domain has been published to the Internet be entering the following command:

host test.external-dns-test.my-org.do

You should see the below result after the domain is published:

test.external-dns-test.my-org.do has address 35.231.248.30

Note: The process of publishing the domain to the Internet can take several minutes.

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.