293 lines
12 KiB
Markdown
293 lines
12 KiB
Markdown
# Running fancy-dns outside the cluster
|
|
|
|
This is how to run a second resolver on a machine that is not a pod, for
|
|
example the controller node, as a backup for the in-cluster one. It uses the
|
|
same code and the same cluster objects (`DNSZone`, `ClientGroup`, `BlockList`,
|
|
`AddressOverride`, `DomainRule`), so an edit made in the console reaches both
|
|
resolvers and nobody keeps a second copy of the configuration.
|
|
|
|
There are two binaries and you probably only want the first:
|
|
|
|
| Binary | Role | On the controller node |
|
|
|---|---|---|
|
|
| `fancy-dns` | The resolver. Answers DNS, reads the cluster objects with a watch, never writes them. | **Yes**: this is the backup. |
|
|
| `fancy-dns-console` | The web UI. The only thing that writes the objects, and it proxies the rest to a resolver. Not on the DNS path. | Optional. See [the console](#the-console-outside-the-cluster). |
|
|
|
|
If you meant something else by "the client", say so; the rest of this file
|
|
assumes the resolver.
|
|
|
|
## How it behaves when the cluster is gone
|
|
|
|
This is the reason to run it, so know it first.
|
|
|
|
| Situation | What the resolver does |
|
|
|---|---|
|
|
| Cluster reachable | Serves the cluster objects live and saves a snapshot of them to `<data dir>/kube-snapshot.json` after every change. |
|
|
| Cluster unreachable at start | Waits 30 seconds, then starts from the saved snapshot and keeps retrying in the background. `GET /api/status` lists `kubernetes: not synced, serving the last saved cluster configuration` under `components.degraded` (and `GET /api/healthz`, which needs no key, reports `degraded`). |
|
|
| Cluster unreachable at start, **never synced before** | Starts with the YAML file only: no cluster zones, address overrides, groups or lists. `GET /api/status` says `not synced and no saved snapshot`. Start it once while the cluster is up, and keep its data directory. |
|
|
| Cluster goes away while running | Keeps serving what it has and reconnects when the API server returns. |
|
|
| The kubeconfig is missing or unreadable | **Refuses to start.** A resolver that cannot build a client at all is a configuration error, not a degraded cluster. Test the kubeconfig (below) before relying on it. |
|
|
|
|
Because of the snapshot, the backup needs a data directory that survives
|
|
reboots. That is also where the cache snapshot lives, so it is the same
|
|
directory you would have anyway.
|
|
|
|
Plain DNS to the backup does not depend on the console, on Valkey, or on the
|
|
in-cluster resolver being up.
|
|
|
|
## 1. Give it read-only access to the objects
|
|
|
|
Do not copy the cluster's admin kubeconfig onto a DNS server if you can avoid it:
|
|
the resolver only ever needs to read five resource types in one namespace.
|
|
`deploy/k8s/crds/rbac.yaml` already defines that Role (`fancy-dns-objects`).
|
|
Bind a dedicated service account to it and give that account a long-lived token:
|
|
|
|
```yaml
|
|
# kubectl apply -f this-file.yaml
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata: {name: fancy-dns-backup, namespace: dns}
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: fancy-dns-backup-token
|
|
namespace: dns
|
|
annotations: {kubernetes.io/service-account.name: fancy-dns-backup}
|
|
type: kubernetes.io/service-account-token
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata: {name: fancy-dns-objects-backup, namespace: dns}
|
|
subjects: [{kind: ServiceAccount, name: fancy-dns-backup, namespace: dns}]
|
|
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: fancy-dns-objects}
|
|
```
|
|
|
|
Build a kubeconfig from it. Run this where you have admin `kubectl`, then copy
|
|
the file to the node as `/etc/fancy-dns/kubeconfig`:
|
|
|
|
```sh
|
|
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
|
|
CA=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.ca\.crt}')
|
|
TOKEN=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.token}' | base64 -d)
|
|
|
|
cat > fancy-dns.kubeconfig <<EOF
|
|
apiVersion: v1
|
|
kind: Config
|
|
clusters:
|
|
- name: cluster
|
|
cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}
|
|
users:
|
|
- name: fancy-dns-backup
|
|
user: {token: ${TOKEN}}
|
|
contexts:
|
|
- name: backup
|
|
context: {cluster: cluster, user: fancy-dns-backup, namespace: dns}
|
|
current-context: backup
|
|
EOF
|
|
chmod 600 fancy-dns.kubeconfig
|
|
```
|
|
|
|
On the controller node itself you can point `server:` at the local API server
|
|
(for example `https://127.0.0.1:6443`) so the backup does not depend on the
|
|
pod network to reach it. Use the address the API server's certificate is valid
|
|
for, or the connection fails the TLS check.
|
|
|
|
Check the credentials before you trust them. This is the check that matters,
|
|
because `-check` below validates only the YAML and not the cluster connection:
|
|
|
|
```sh
|
|
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns get clientgroups,dnszones,blocklists
|
|
# and the negative: this must be refused
|
|
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns delete clientgroups --all --dry-run=server
|
|
```
|
|
|
|
## 2. Install and configure the resolver
|
|
|
|
Build the binary (`make build` writes `dist/fancy-dns` and
|
|
`dist/fancy-dns-console`, with the console embedded) or copy it out of the image.
|
|
Use the same version as the cluster: an older binary rejects objects that use
|
|
fields it does not know, and reports `kubernetes: objects rejected` in `components.degraded`.
|
|
|
|
```sh
|
|
sudo useradd --system --home /var/lib/fancy-dns --shell /usr/sbin/nologin fancy-dns
|
|
sudo install -m 0755 dist/fancy-dns /usr/local/bin/fancy-dns
|
|
sudo install -d -o fancy-dns -g fancy-dns /var/lib/fancy-dns
|
|
sudo install -d /etc/fancy-dns
|
|
sudo install -m 0600 -o fancy-dns -g fancy-dns fancy-dns.kubeconfig /etc/fancy-dns/kubeconfig
|
|
```
|
|
|
|
Start from `deploy/compose/config.yaml` (a backup-resolver config) or
|
|
`deploy/config.example.yaml`. The parts that matter here:
|
|
|
|
```yaml
|
|
cache:
|
|
persist:
|
|
enabled: true
|
|
path: /var/lib/fancy-dns/cache.db # its directory is the data dir: the
|
|
# cluster snapshot is saved beside it
|
|
api:
|
|
enabled: true
|
|
listen: ["127.0.0.1:8080"] # loopback only
|
|
ui: false
|
|
auth:
|
|
mode: apikey
|
|
api_keys: ["${FANCYDNS_API_KEY}"]
|
|
```
|
|
|
|
Keep in the YAML only what is genuinely local: listeners, upstreams, cache.
|
|
**Do not copy the cluster's groups, lists or zones into it.** YAML wins every
|
|
collision, so a copy would shadow the live object and you would be editing in
|
|
the console what the backup ignores. The upstreams and `forward_zones` are
|
|
config, not objects, so they must match the in-cluster ones by hand; compare the
|
|
two files after any change.
|
|
|
|
The service reads its secrets from an environment file:
|
|
|
|
```sh
|
|
sudo install -m 0600 /dev/null /etc/fancy-dns/fancy-dns.env
|
|
echo "FANCYDNS_API_KEY=$(openssl rand -hex 24)" | sudo tee -a /etc/fancy-dns/fancy-dns.env
|
|
```
|
|
|
|
An unset `${VAR}` is a startup failure on purpose: a missing key is a crash
|
|
loop, not an unauthenticated API.
|
|
|
|
### systemd
|
|
|
|
`/etc/systemd/system/fancy-dns.service`:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=fancy-dns backup resolver
|
|
Wants=network-online.target
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
User=fancy-dns
|
|
Group=fancy-dns
|
|
EnvironmentFile=/etc/fancy-dns/fancy-dns.env
|
|
ExecStart=/usr/local/bin/fancy-dns \
|
|
-config /etc/fancy-dns/config.yaml \
|
|
-kubeconfig /etc/fancy-dns/kubeconfig \
|
|
-kube-namespace dns
|
|
ExecReload=/bin/kill -HUP $MAINPID
|
|
Restart=on-failure
|
|
RestartSec=2
|
|
TimeoutStopSec=20
|
|
LimitNOFILE=65536
|
|
|
|
# Port 53 without root.
|
|
AmbientCapabilities=CAP_NET_BIND_SERVICE
|
|
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
|
|
NoNewPrivileges=yes
|
|
ProtectSystem=strict
|
|
ProtectHome=yes
|
|
PrivateTmp=yes
|
|
ReadWritePaths=/var/lib/fancy-dns
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
Always pass `-kube-namespace`. Outside a pod there is no namespace to default
|
|
to, and watching all namespaces would need a cluster-wide Role.
|
|
|
|
`-kubeconfig` turns the Kubernetes source on by itself. `reload` is a SIGHUP:
|
|
the YAML is re-read and every component is swapped without dropping a query.
|
|
Changes to the cluster objects need no reload at all.
|
|
|
|
### Port 53
|
|
|
|
Check what owns it first: `sudo ss -lntup | grep ':53 '`. `systemd-resolved`'s
|
|
stub listener binds only `127.0.0.53`, but a wildcard bind to `0.0.0.0:53`
|
|
still collides with it. Either bind the node's address in `dns.udp`/`dns.tcp`
|
|
or set `DNSStubListener=no` in `/etc/systemd/resolved.conf`. Open UDP and TCP 53
|
|
(and 853/443 if you enabled DoT/DoH) in the node's firewall.
|
|
|
|
Do not make this resolver the node's only DNS server (`/etc/resolv.conf`): after
|
|
a reboot the node needs name resolution before the resolver is up.
|
|
|
|
## 3. Start it and check
|
|
|
|
```sh
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now fancy-dns
|
|
journalctl -u fancy-dns -n 30 --no-pager # look for: watching Kubernetes for fancy-dns objects
|
|
|
|
KEY=$(sudo sed -n 's/^FANCYDNS_API_KEY=//p' /etc/fancy-dns/fancy-dns.env)
|
|
curl -s -H "X-API-Key: $KEY" http://127.0.0.1:8080/api/status | jq '.components'
|
|
dig @127.0.0.1 example.com +short
|
|
dig @127.0.0.1 <a name from one of your DNSZones or AddressOverrides> +short
|
|
```
|
|
|
|
`components.degraded` should be empty. The two `dig`s tell you it resolves and
|
|
that it has the cluster's data. Then simulate the outage you bought this for:
|
|
block the API server from the node (or stop the resolver and start it with the
|
|
cluster unreachable), and confirm the same names still answer and
|
|
`components.degraded` now names the snapshot.
|
|
|
|
Finally point clients at it as a **second** nameserver (DHCP option 6, or the
|
|
router's secondary). It is a backup, not a load balancer: most stub resolvers
|
|
only try the second server after the first times out.
|
|
|
|
## The console outside the cluster
|
|
|
|
Optional. You only need it on the node if you want to edit while the in-cluster
|
|
console is unavailable, and it can only save while the API server is reachable,
|
|
since an edit is a write to the cluster. Reads of statistics come from whichever
|
|
resolver you point it at.
|
|
|
|
It needs a **write** Role, which is a separate account from the resolver's.
|
|
`deploy/k8s/console/console.yaml` defines the Role `fancy-dns-console`; bind a
|
|
second service account to it the same way as above (name it
|
|
`fancy-dns-console-ext`, build `/etc/fancy-dns/console.kubeconfig`).
|
|
|
|
`/etc/fancy-dns/console.yaml` (see `deploy/console.example.yaml`):
|
|
|
|
```yaml
|
|
blocklist: {enabled: false}
|
|
api:
|
|
enabled: true
|
|
listen: ["127.0.0.1:8081"]
|
|
base_path: /api
|
|
ui: true
|
|
auth:
|
|
mode: apikey
|
|
api_keys: ["${FANCYDNS_CONSOLE_KEY}"]
|
|
trust_loopback: false
|
|
```
|
|
|
|
`/etc/fancy-dns/console.env`:
|
|
|
|
```sh
|
|
FANCYDNS_CONSOLE_KEY=<a long random key: what you type into the UI>
|
|
FANCYDNS_RUNTIME_KEY=<the resolver's FANCYDNS_API_KEY: what the console sends it>
|
|
```
|
|
|
|
Run it with a unit like the resolver's, minus the port-53 capability:
|
|
|
|
```ini
|
|
ExecStart=/usr/local/bin/fancy-dns-console \
|
|
-config /etc/fancy-dns/console.yaml \
|
|
-runtime-url http://127.0.0.1:8080 \
|
|
-kubeconfig /etc/fancy-dns/console.kubeconfig \
|
|
-kube-namespace dns
|
|
```
|
|
|
|
Keep it on loopback and reach it with an SSH tunnel
|
|
(`ssh -L 8081:127.0.0.1:8081 controller`) or an authenticated TLS reverse proxy.
|
|
The UI can change DNS for every client you have; do not publish it with an API
|
|
key on plain HTTP.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Likely cause |
|
|
|---|---|
|
|
| Service exits immediately: `no usable Kubernetes credentials` | `-kubeconfig` points at a file that is missing, unreadable by the `fancy-dns` user, or malformed. |
|
|
| Starts, `components.degraded` says `not synced` | The API server is unreachable or the token is rejected (`kubectl --kubeconfig … get clientgroups`), or the CRDs are not installed. |
|
|
| `…not synced and no saved snapshot` and cluster data missing | First start with the cluster down. Start it once with the cluster up. Also check the data directory is writable. |
|
|
| `kubernetes: objects rejected` in `components.degraded` | An object failed validation (or this binary is older than the cluster's). The last good version stays in force. Which objects, and why: `jq .problems /var/lib/fancy-dns/kube-snapshot.json`. |
|
|
| Cluster answers differ from the backup's | The upstream/forward-zone config differs (it is YAML, not an object), or the backup's cache holds older answers. `POST /api/cache/flush` on it. |
|
|
| `bind: address already in use` | Something owns port 53; see [Port 53](#port-53). |
|
|
| Console: `cluster: …forbidden` | It is using the read-only kubeconfig; it needs the write Role. |
|