# Running fancy-dns outside the cluster

This is how to run a second resolver on a machine that is not a pod, for
example the controller node, as a backup for the in-cluster one. It uses the
same code and the same cluster objects (`DNSZone`, `ClientGroup`, `BlockList`,
`AddressOverride`, `DomainRule`), so an edit made in the console reaches both
resolvers and nobody keeps a second copy of the configuration.

There are two binaries and you probably only want the first:

| Binary | Role | On the controller node |
|---|---|---|
| `fancy-dns` | The resolver. Answers DNS, reads the cluster objects with a watch, never writes them. | **Yes**: this is the backup. |
| `fancy-dns-console` | The web UI. The only thing that writes the objects, and it proxies the rest to a resolver. Not on the DNS path. | Optional. See [the console](#the-console-outside-the-cluster). |

If you meant something else by "the client", say so; the rest of this file
assumes the resolver.

## How it behaves when the cluster is gone

This is the reason to run it, so know it first.

| Situation | What the resolver does |
|---|---|
| Cluster reachable | Serves the cluster objects live and saves a snapshot of them to `<data dir>/kube-snapshot.json` after every change. |
| Cluster unreachable at start | Waits 30 seconds, then starts from the saved snapshot and keeps retrying in the background. `GET /api/status` lists `kubernetes: not synced, serving the last saved cluster configuration` under `components.degraded` (and `GET /api/healthz`, which needs no key, reports `degraded`). |
| Cluster unreachable at start, **never synced before** | Starts with the YAML file only: no cluster zones, address overrides, groups or lists. `GET /api/status` says `not synced and no saved snapshot`. Start it once while the cluster is up, and keep its data directory. |
| Cluster goes away while running | Keeps serving what it has and reconnects when the API server returns. |
| The kubeconfig is missing or unreadable | **Refuses to start.** A resolver that cannot build a client at all is a configuration error, not a degraded cluster. Test the kubeconfig (below) before relying on it. |

Because of the snapshot, the backup needs a data directory that survives
reboots. That is also where the cache snapshot lives, so it is the same
directory you would have anyway.

Plain DNS to the backup does not depend on the console, on Valkey, or on the
in-cluster resolver being up.

## 1. Give it read-only access to the objects

Do not copy the cluster's admin kubeconfig onto a DNS server if you can avoid it:
the resolver only ever needs to read five resource types in one namespace.
`deploy/k8s/crds/rbac.yaml` already defines that Role (`fancy-dns-objects`).
Bind a dedicated service account to it and give that account a long-lived token:

```yaml
# kubectl apply -f this-file.yaml
apiVersion: v1
kind: ServiceAccount
metadata: {name: fancy-dns-backup, namespace: dns}
---
apiVersion: v1
kind: Secret
metadata:
  name: fancy-dns-backup-token
  namespace: dns
  annotations: {kubernetes.io/service-account.name: fancy-dns-backup}
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: fancy-dns-objects-backup, namespace: dns}
subjects: [{kind: ServiceAccount, name: fancy-dns-backup, namespace: dns}]
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: fancy-dns-objects}
```

Build a kubeconfig from it. Run this where you have admin `kubectl`, then copy
the file to the node as `/etc/fancy-dns/kubeconfig`:

```sh
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.token}' | base64 -d)

cat > fancy-dns.kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:
  - name: cluster
    cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}
users:
  - name: fancy-dns-backup
    user: {token: ${TOKEN}}
contexts:
  - name: backup
    context: {cluster: cluster, user: fancy-dns-backup, namespace: dns}
current-context: backup
EOF
chmod 600 fancy-dns.kubeconfig
```

On the controller node itself you can point `server:` at the local API server
(for example `https://127.0.0.1:6443`) so the backup does not depend on the
pod network to reach it. Use the address the API server's certificate is valid
for, or the connection fails the TLS check.

Check the credentials before you trust them. This is the check that matters,
because `-check` below validates only the YAML and not the cluster connection:

```sh
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns get clientgroups,dnszones,blocklists
# and the negative: this must be refused
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns delete clientgroups --all --dry-run=server
```

## 2. Install and configure the resolver

Build the binary (`make build` writes `dist/fancy-dns` and
`dist/fancy-dns-console`, with the console embedded) or copy it out of the image.
Use the same version as the cluster: an older binary rejects objects that use
fields it does not know, and reports `kubernetes: objects rejected` in `components.degraded`.

```sh
sudo useradd --system --home /var/lib/fancy-dns --shell /usr/sbin/nologin fancy-dns
sudo install -m 0755 dist/fancy-dns /usr/local/bin/fancy-dns
sudo install -d -o fancy-dns -g fancy-dns /var/lib/fancy-dns
sudo install -d /etc/fancy-dns
sudo install -m 0600 -o fancy-dns -g fancy-dns fancy-dns.kubeconfig /etc/fancy-dns/kubeconfig
```

Start from `deploy/compose/config.yaml` (a backup-resolver config) or
`deploy/config.example.yaml`. The parts that matter here:

```yaml
cache:
  persist:
    enabled: true
    path: /var/lib/fancy-dns/cache.db   # its directory is the data dir: the
                                        # cluster snapshot is saved beside it
api:
  enabled: true
  listen: ["127.0.0.1:8080"]            # loopback only
  ui: false
  auth:
    mode: apikey
    api_keys: ["${FANCYDNS_API_KEY}"]
```

Keep in the YAML only what is genuinely local: listeners, upstreams, cache.
**Do not copy the cluster's groups, lists or zones into it.** YAML wins every
collision, so a copy would shadow the live object and you would be editing in
the console what the backup ignores. The upstreams and `forward_zones` are
config, not objects, so they must match the in-cluster ones by hand; compare the
two files after any change.

The service reads its secrets from an environment file:

```sh
sudo install -m 0600 /dev/null /etc/fancy-dns/fancy-dns.env
echo "FANCYDNS_API_KEY=$(openssl rand -hex 24)" | sudo tee -a /etc/fancy-dns/fancy-dns.env
```

An unset `${VAR}` is a startup failure on purpose: a missing key is a crash
loop, not an unauthenticated API.

### systemd

`/etc/systemd/system/fancy-dns.service`:

```ini
[Unit]
Description=fancy-dns backup resolver
Wants=network-online.target
After=network-online.target

[Service]
User=fancy-dns
Group=fancy-dns
EnvironmentFile=/etc/fancy-dns/fancy-dns.env
ExecStart=/usr/local/bin/fancy-dns \
    -config /etc/fancy-dns/config.yaml \
    -kubeconfig /etc/fancy-dns/kubeconfig \
    -kube-namespace dns
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
TimeoutStopSec=20
LimitNOFILE=65536

# Port 53 without root.
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/fancy-dns

[Install]
WantedBy=multi-user.target
```

Always pass `-kube-namespace`. Outside a pod there is no namespace to default
to, and watching all namespaces would need a cluster-wide Role.

`-kubeconfig` turns the Kubernetes source on by itself. `reload` is a SIGHUP:
the YAML is re-read and every component is swapped without dropping a query.
Changes to the cluster objects need no reload at all.

### Port 53

Check what owns it first: `sudo ss -lntup | grep ':53 '`. `systemd-resolved`'s
stub listener binds only `127.0.0.53`, but a wildcard bind to `0.0.0.0:53`
still collides with it. Either bind the node's address in `dns.udp`/`dns.tcp`
or set `DNSStubListener=no` in `/etc/systemd/resolved.conf`. Open UDP and TCP 53
(and 853/443 if you enabled DoT/DoH) in the node's firewall.

Do not make this resolver the node's only DNS server (`/etc/resolv.conf`): after
a reboot the node needs name resolution before the resolver is up.

## 3. Start it and check

```sh
sudo systemctl daemon-reload
sudo systemctl enable --now fancy-dns
journalctl -u fancy-dns -n 30 --no-pager      # look for: watching Kubernetes for fancy-dns objects

KEY=$(sudo sed -n 's/^FANCYDNS_API_KEY=//p' /etc/fancy-dns/fancy-dns.env)
curl -s -H "X-API-Key: $KEY" http://127.0.0.1:8080/api/status | jq '.components'
dig @127.0.0.1 example.com +short
dig @127.0.0.1 <a name from one of your DNSZones or AddressOverrides> +short
```

`components.degraded` should be empty. The two `dig`s tell you it resolves and
that it has the cluster's data. Then simulate the outage you bought this for:
block the API server from the node (or stop the resolver and start it with the
cluster unreachable), and confirm the same names still answer and
`components.degraded` now names the snapshot.

Finally point clients at it as a **second** nameserver (DHCP option 6, or the
router's secondary). It is a backup, not a load balancer: most stub resolvers
only try the second server after the first times out.

## The console outside the cluster

Optional. You only need it on the node if you want to edit while the in-cluster
console is unavailable, and it can only save while the API server is reachable,
since an edit is a write to the cluster. Reads of statistics come from whichever
resolver you point it at.

It needs a **write** Role, which is a separate account from the resolver's.
`deploy/k8s/console/console.yaml` defines the Role `fancy-dns-console`; bind a
second service account to it the same way as above (name it
`fancy-dns-console-ext`, build `/etc/fancy-dns/console.kubeconfig`).

`/etc/fancy-dns/console.yaml` (see `deploy/console.example.yaml`):

```yaml
blocklist: {enabled: false}
api:
  enabled: true
  listen: ["127.0.0.1:8081"]
  base_path: /api
  ui: true
  auth:
    mode: apikey
    api_keys: ["${FANCYDNS_CONSOLE_KEY}"]
    trust_loopback: false
```

`/etc/fancy-dns/console.env`:

```sh
FANCYDNS_CONSOLE_KEY=<a long random key: what you type into the UI>
FANCYDNS_RUNTIME_KEY=<the resolver's FANCYDNS_API_KEY: what the console sends it>
```

Run it with a unit like the resolver's, minus the port-53 capability:

```ini
ExecStart=/usr/local/bin/fancy-dns-console \
    -config /etc/fancy-dns/console.yaml \
    -runtime-url http://127.0.0.1:8080 \
    -kubeconfig /etc/fancy-dns/console.kubeconfig \
    -kube-namespace dns
```

Keep it on loopback and reach it with an SSH tunnel
(`ssh -L 8081:127.0.0.1:8081 controller`) or an authenticated TLS reverse proxy.
The UI can change DNS for every client you have; do not publish it with an API
key on plain HTTP.

## Troubleshooting

| Symptom | Likely cause |
|---|---|
| Service exits immediately: `no usable Kubernetes credentials` | `-kubeconfig` points at a file that is missing, unreadable by the `fancy-dns` user, or malformed. |
| Starts, `components.degraded` says `not synced` | The API server is unreachable or the token is rejected (`kubectl --kubeconfig … get clientgroups`), or the CRDs are not installed. |
| `…not synced and no saved snapshot` and cluster data missing | First start with the cluster down. Start it once with the cluster up. Also check the data directory is writable. |
| `kubernetes: objects rejected` in `components.degraded` | An object failed validation (or this binary is older than the cluster's). The last good version stays in force. Which objects, and why: `jq .problems /var/lib/fancy-dns/kube-snapshot.json`. |
| Cluster answers differ from the backup's | The upstream/forward-zone config differs (it is YAML, not an object), or the backup's cache holds older answers. `POST /api/cache/flush` on it. |
| `bind: address already in use` | Something owns port 53; see [Port 53](#port-53). |
| Console: `cluster: …forbidden` | It is using the read-only kubeconfig; it needs the write Role. |