[Kubernetes] Add new DNS server in CoreDNS Configuration

After finished the K8S installation, I had encountered an issue that the pod cannot resolve the domain name. I use these commands to check the DNS issue.

$ systemd-resolve --status
nameservers:
                  addresses:
                  - 210.240.232.1
                  search: []

# use busybox pod to run nslookup
$ kubectl apply -f https://k8s.io/examples/admin/dns/busybox.yaml
$ kubectl exec -it busybox -- nslookup kubernetes.default
$ kubectl exec -it busybox -- nslookup google.com

# check local DNS configuration
$ kubectl exec busybox cat /etc/resolv.conf

For my case, the easy way to solve it is to add additional forwards in here:

$ kubectl edit configmaps coredns -n kube-system
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-08-08T03:45:43Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "198"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 178e9cc4-7167-49a6-bb10-c7f3dda732c2

Add your DNS server address like 8.8.4.4 here:

forward . 8.8.4.4 /etc/resolv.conf

Find your CoreDNS pods and restart them

$ kubectl get pods -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
calico-node-rs7xw                           2/2     Running   17         11d
coredns-5c98db65d4-d4zdk                    1/1     Running   8          11d
coredns-5c98db65d4-nh6hr                    1/1     Running   9          11d
etcd-51-0a50338-01                          1/1     Running   9          11d
kube-apiserver-51-0a50338-01                1/1     Running   9          11d
kube-controller-manager-51-0a50338-01       1/1     Running   8          11d
kube-proxy-pflzg                            1/1     Running   9          11d
kube-scheduler-51-0a50338-01                1/1     Running   8          11d
nvidia-device-plugin-daemonset-1.12-l5vdw   1/1     Running   8          11d

$ kubectl delete pod coredns-5c98db65d4-d4zdk coredns-5c98db65d4-nh6hr -n kube-system

OK, that's done. ( coredns pods will be restarted automatically )

Reference:

https://docs.giantswarm.io/guides/advanced-coredns-configuration/
https://k8smeetup.github.io/docs/tasks/administer-cluster/dns-debugging-resolution/

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/