12 KiB
Running fancy-dns outside the cluster
This is how to run a second resolver on a machine that is not a pod, for
example the controller node, as a backup for the in-cluster one. It uses the
same code and the same cluster objects (DNSZone, ClientGroup, BlockList,
AddressOverride, DomainRule), so an edit made in the console reaches both
resolvers and nobody keeps a second copy of the configuration.
There are two binaries and you probably only want the first:
| Binary | Role | On the controller node |
|---|---|---|
fancy-dns |
The resolver. Answers DNS, reads the cluster objects with a watch, never writes them. | Yes: this is the backup. |
fancy-dns-console |
The web UI. The only thing that writes the objects, and it proxies the rest to a resolver. Not on the DNS path. | Optional. See the console. |
If you meant something else by "the client", say so; the rest of this file assumes the resolver.
How it behaves when the cluster is gone
This is the reason to run it, so know it first.
| Situation | What the resolver does |
|---|---|
| Cluster reachable | Serves the cluster objects live and saves a snapshot of them to <data dir>/kube-snapshot.json after every change. |
| Cluster unreachable at start | Waits 30 seconds, then starts from the saved snapshot and keeps retrying in the background. GET /api/status lists kubernetes: not synced, serving the last saved cluster configuration under components.degraded (and GET /api/healthz, which needs no key, reports degraded). |
| Cluster unreachable at start, never synced before | Starts with the YAML file only: no cluster zones, address overrides, groups or lists. GET /api/status says not synced and no saved snapshot. Start it once while the cluster is up, and keep its data directory. |
| Cluster goes away while running | Keeps serving what it has and reconnects when the API server returns. |
| The kubeconfig is missing or unreadable | Refuses to start. A resolver that cannot build a client at all is a configuration error, not a degraded cluster. Test the kubeconfig (below) before relying on it. |
Because of the snapshot, the backup needs a data directory that survives reboots. That is also where the cache snapshot lives, so it is the same directory you would have anyway.
Plain DNS to the backup does not depend on the console, on Valkey, or on the in-cluster resolver being up.
1. Give it read-only access to the objects
Do not copy the cluster's admin kubeconfig onto a DNS server if you can avoid it:
the resolver only ever needs to read five resource types in one namespace.
deploy/k8s/crds/rbac.yaml already defines that Role (fancy-dns-objects).
Bind a dedicated service account to it and give that account a long-lived token:
# kubectl apply -f this-file.yaml
apiVersion: v1
kind: ServiceAccount
metadata: {name: fancy-dns-backup, namespace: dns}
---
apiVersion: v1
kind: Secret
metadata:
name: fancy-dns-backup-token
namespace: dns
annotations: {kubernetes.io/service-account.name: fancy-dns-backup}
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: fancy-dns-objects-backup, namespace: dns}
subjects: [{kind: ServiceAccount, name: fancy-dns-backup, namespace: dns}]
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: fancy-dns-objects}
Build a kubeconfig from it. Run this where you have admin kubectl, then copy
the file to the node as /etc/fancy-dns/kubeconfig:
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.token}' | base64 -d)
cat > fancy-dns.kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:
- name: cluster
cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}
users:
- name: fancy-dns-backup
user: {token: ${TOKEN}}
contexts:
- name: backup
context: {cluster: cluster, user: fancy-dns-backup, namespace: dns}
current-context: backup
EOF
chmod 600 fancy-dns.kubeconfig
On the controller node itself you can point server: at the local API server
(for example https://127.0.0.1:6443) so the backup does not depend on the
pod network to reach it. Use the address the API server's certificate is valid
for, or the connection fails the TLS check.
Check the credentials before you trust them. This is the check that matters,
because -check below validates only the YAML and not the cluster connection:
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns get clientgroups,dnszones,blocklists
# and the negative: this must be refused
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns delete clientgroups --all --dry-run=server
2. Install and configure the resolver
Build the binary (make build writes dist/fancy-dns and
dist/fancy-dns-console, with the console embedded) or copy it out of the image.
Use the same version as the cluster: an older binary rejects objects that use
fields it does not know, and reports kubernetes: objects rejected in components.degraded.
sudo useradd --system --home /var/lib/fancy-dns --shell /usr/sbin/nologin fancy-dns
sudo install -m 0755 dist/fancy-dns /usr/local/bin/fancy-dns
sudo install -d -o fancy-dns -g fancy-dns /var/lib/fancy-dns
sudo install -d /etc/fancy-dns
sudo install -m 0600 -o fancy-dns -g fancy-dns fancy-dns.kubeconfig /etc/fancy-dns/kubeconfig
Start from deploy/compose/config.yaml (a backup-resolver config) or
deploy/config.example.yaml. The parts that matter here:
cache:
persist:
enabled: true
path: /var/lib/fancy-dns/cache.db # its directory is the data dir: the
# cluster snapshot is saved beside it
api:
enabled: true
listen: ["127.0.0.1:8080"] # loopback only
ui: false
auth:
mode: apikey
api_keys: ["${FANCYDNS_API_KEY}"]
Keep in the YAML only what is genuinely local: listeners, upstreams, cache.
Do not copy the cluster's groups, lists or zones into it. YAML wins every
collision, so a copy would shadow the live object and you would be editing in
the console what the backup ignores. The upstreams and forward_zones are
config, not objects, so they must match the in-cluster ones by hand; compare the
two files after any change.
The service reads its secrets from an environment file:
sudo install -m 0600 /dev/null /etc/fancy-dns/fancy-dns.env
echo "FANCYDNS_API_KEY=$(openssl rand -hex 24)" | sudo tee -a /etc/fancy-dns/fancy-dns.env
An unset ${VAR} is a startup failure on purpose: a missing key is a crash
loop, not an unauthenticated API.
systemd
/etc/systemd/system/fancy-dns.service:
[Unit]
Description=fancy-dns backup resolver
Wants=network-online.target
After=network-online.target
[Service]
User=fancy-dns
Group=fancy-dns
EnvironmentFile=/etc/fancy-dns/fancy-dns.env
ExecStart=/usr/local/bin/fancy-dns \
-config /etc/fancy-dns/config.yaml \
-kubeconfig /etc/fancy-dns/kubeconfig \
-kube-namespace dns
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
TimeoutStopSec=20
LimitNOFILE=65536
# Port 53 without root.
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/fancy-dns
[Install]
WantedBy=multi-user.target
Always pass -kube-namespace. Outside a pod there is no namespace to default
to, and watching all namespaces would need a cluster-wide Role.
-kubeconfig turns the Kubernetes source on by itself. reload is a SIGHUP:
the YAML is re-read and every component is swapped without dropping a query.
Changes to the cluster objects need no reload at all.
Port 53
Check what owns it first: sudo ss -lntup | grep ':53 '. systemd-resolved's
stub listener binds only 127.0.0.53, but a wildcard bind to 0.0.0.0:53
still collides with it. Either bind the node's address in dns.udp/dns.tcp
or set DNSStubListener=no in /etc/systemd/resolved.conf. Open UDP and TCP 53
(and 853/443 if you enabled DoT/DoH) in the node's firewall.
Do not make this resolver the node's only DNS server (/etc/resolv.conf): after
a reboot the node needs name resolution before the resolver is up.
3. Start it and check
sudo systemctl daemon-reload
sudo systemctl enable --now fancy-dns
journalctl -u fancy-dns -n 30 --no-pager # look for: watching Kubernetes for fancy-dns objects
KEY=$(sudo sed -n 's/^FANCYDNS_API_KEY=//p' /etc/fancy-dns/fancy-dns.env)
curl -s -H "X-API-Key: $KEY" http://127.0.0.1:8080/api/status | jq '.components'
dig @127.0.0.1 example.com +short
dig @127.0.0.1 <a name from one of your DNSZones or AddressOverrides> +short
components.degraded should be empty. The two digs tell you it resolves and
that it has the cluster's data. Then simulate the outage you bought this for:
block the API server from the node (or stop the resolver and start it with the
cluster unreachable), and confirm the same names still answer and
components.degraded now names the snapshot.
Finally point clients at it as a second nameserver (DHCP option 6, or the router's secondary). It is a backup, not a load balancer: most stub resolvers only try the second server after the first times out.
The console outside the cluster
Optional. You only need it on the node if you want to edit while the in-cluster console is unavailable, and it can only save while the API server is reachable, since an edit is a write to the cluster. Reads of statistics come from whichever resolver you point it at.
It needs a write Role, which is a separate account from the resolver's.
deploy/k8s/console/console.yaml defines the Role fancy-dns-console; bind a
second service account to it the same way as above (name it
fancy-dns-console-ext, build /etc/fancy-dns/console.kubeconfig).
/etc/fancy-dns/console.yaml (see deploy/console.example.yaml):
blocklist: {enabled: false}
api:
enabled: true
listen: ["127.0.0.1:8081"]
base_path: /api
ui: true
auth:
mode: apikey
api_keys: ["${FANCYDNS_CONSOLE_KEY}"]
trust_loopback: false
/etc/fancy-dns/console.env:
FANCYDNS_CONSOLE_KEY=<a long random key: what you type into the UI>
FANCYDNS_RUNTIME_KEY=<the resolver's FANCYDNS_API_KEY: what the console sends it>
Run it with a unit like the resolver's, minus the port-53 capability:
ExecStart=/usr/local/bin/fancy-dns-console \
-config /etc/fancy-dns/console.yaml \
-runtime-url http://127.0.0.1:8080 \
-kubeconfig /etc/fancy-dns/console.kubeconfig \
-kube-namespace dns
Keep it on loopback and reach it with an SSH tunnel
(ssh -L 8081:127.0.0.1:8081 controller) or an authenticated TLS reverse proxy.
The UI can change DNS for every client you have; do not publish it with an API
key on plain HTTP.
Troubleshooting
| Symptom | Likely cause |
|---|---|
Service exits immediately: no usable Kubernetes credentials |
-kubeconfig points at a file that is missing, unreadable by the fancy-dns user, or malformed. |
Starts, components.degraded says not synced |
The API server is unreachable or the token is rejected (kubectl --kubeconfig … get clientgroups), or the CRDs are not installed. |
…not synced and no saved snapshot and cluster data missing |
First start with the cluster down. Start it once with the cluster up. Also check the data directory is writable. |
kubernetes: objects rejected in components.degraded |
An object failed validation (or this binary is older than the cluster's). The last good version stays in force. Which objects, and why: jq .problems /var/lib/fancy-dns/kube-snapshot.json. |
| Cluster answers differ from the backup's | The upstream/forward-zone config differs (it is YAML, not an object), or the backup's cache holds older answers. POST /api/cache/flush on it. |
bind: address already in use |
Something owns port 53; see Port 53. |
Console: cluster: …forbidden |
It is using the read-only kubeconfig; it needs the write Role. |