mats committed
1 parent
289a8cc2d8
commit
66f36cfac6
19 files changed
+1905
-817
No files matched your search
@@ -0,0 +1,292 @@
|
||||
# Running fancy-dns outside the cluster
|
||||
|
||||
This is how to run a second resolver on a machine that is not a pod, for
|
||||
example the controller node, as a backup for the in-cluster one. It uses the
|
||||
same code and the same cluster objects (`DNSZone`, `ClientGroup`, `BlockList`,
|
||||
`AddressOverride`, `DomainRule`), so an edit made in the console reaches both
|
||||
resolvers and nobody keeps a second copy of the configuration.
|
||||
|
||||
There are two binaries and you probably only want the first:
|
||||
|
||||
| Binary | Role | On the controller node |
|
||||
|---|---|---|
|
||||
| `fancy-dns` | The resolver. Answers DNS, reads the cluster objects with a watch, never writes them. | **Yes**: this is the backup. |
|
||||
| `fancy-dns-console` | The web UI. The only thing that writes the objects, and it proxies the rest to a resolver. Not on the DNS path. | Optional. See [the console](#the-console-outside-the-cluster). |
|
||||
|
||||
If you meant something else by "the client", say so; the rest of this file
|
||||
assumes the resolver.
|
||||
|
||||
## How it behaves when the cluster is gone
|
||||
|
||||
This is the reason to run it, so know it first.
|
||||
|
||||
| Situation | What the resolver does |
|
||||
|---|---|
|
||||
| Cluster reachable | Serves the cluster objects live and saves a snapshot of them to `<data dir>/kube-snapshot.json` after every change. |
|
||||
| Cluster unreachable at start | Waits 30 seconds, then starts from the saved snapshot and keeps retrying in the background. `GET /api/status` lists `kubernetes: not synced, serving the last saved cluster configuration` under `components.degraded` (and `GET /api/healthz`, which needs no key, reports `degraded`). |
|
||||
| Cluster unreachable at start, **never synced before** | Starts with the YAML file only: no cluster zones, address overrides, groups or lists. `GET /api/status` says `not synced and no saved snapshot`. Start it once while the cluster is up, and keep its data directory. |
|
||||
| Cluster goes away while running | Keeps serving what it has and reconnects when the API server returns. |
|
||||
| The kubeconfig is missing or unreadable | **Refuses to start.** A resolver that cannot build a client at all is a configuration error, not a degraded cluster. Test the kubeconfig (below) before relying on it. |
|
||||
|
||||
Because of the snapshot, the backup needs a data directory that survives
|
||||
reboots. That is also where the cache snapshot lives, so it is the same
|
||||
directory you would have anyway.
|
||||
|
||||
Plain DNS to the backup does not depend on the console, on Valkey, or on the
|
||||
in-cluster resolver being up.
|
||||
|
||||
## 1. Give it read-only access to the objects
|
||||
|
||||
Do not copy the cluster's admin kubeconfig onto a DNS server if you can avoid it:
|
||||
the resolver only ever needs to read five resource types in one namespace.
|
||||
`deploy/k8s/crds/rbac.yaml` already defines that Role (`fancy-dns-objects`).
|
||||
Bind a dedicated service account to it and give that account a long-lived token:
|
||||
|
||||
```yaml
|
||||
# kubectl apply -f this-file.yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata: {name: fancy-dns-backup, namespace: dns}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: fancy-dns-backup-token
|
||||
namespace: dns
|
||||
annotations: {kubernetes.io/service-account.name: fancy-dns-backup}
|
||||
type: kubernetes.io/service-account-token
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata: {name: fancy-dns-objects-backup, namespace: dns}
|
||||
subjects: [{kind: ServiceAccount, name: fancy-dns-backup, namespace: dns}]
|
||||
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: fancy-dns-objects}
|
||||
```
|
||||
|
||||
Build a kubeconfig from it. Run this where you have admin `kubectl`, then copy
|
||||
the file to the node as `/etc/fancy-dns/kubeconfig`:
|
||||
|
||||
```sh
|
||||
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
|
||||
CA=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.ca\.crt}')
|
||||
TOKEN=$(kubectl -n dns get secret fancy-dns-backup-token -o jsonpath='{.data.token}' | base64 -d)
|
||||
|
||||
cat > fancy-dns.kubeconfig <<EOF
|
||||
apiVersion: v1
|
||||
kind: Config
|
||||
clusters:
|
||||
- name: cluster
|
||||
cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}
|
||||
users:
|
||||
- name: fancy-dns-backup
|
||||
user: {token: ${TOKEN}}
|
||||
contexts:
|
||||
- name: backup
|
||||
context: {cluster: cluster, user: fancy-dns-backup, namespace: dns}
|
||||
current-context: backup
|
||||
EOF
|
||||
chmod 600 fancy-dns.kubeconfig
|
||||
```
|
||||
|
||||
On the controller node itself you can point `server:` at the local API server
|
||||
(for example `https://127.0.0.1:6443`) so the backup does not depend on the
|
||||
pod network to reach it. Use the address the API server's certificate is valid
|
||||
for, or the connection fails the TLS check.
|
||||
|
||||
Check the credentials before you trust them. This is the check that matters,
|
||||
because `-check` below validates only the YAML and not the cluster connection:
|
||||
|
||||
```sh
|
||||
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns get clientgroups,dnszones,blocklists
|
||||
# and the negative: this must be refused
|
||||
kubectl --kubeconfig /etc/fancy-dns/kubeconfig -n dns delete clientgroups --all --dry-run=server
|
||||
```
|
||||
|
||||
## 2. Install and configure the resolver
|
||||
|
||||
Build the binary (`make build` writes `dist/fancy-dns` and
|
||||
`dist/fancy-dns-console`, with the console embedded) or copy it out of the image.
|
||||
Use the same version as the cluster: an older binary rejects objects that use
|
||||
fields it does not know, and reports `kubernetes: objects rejected` in `components.degraded`.
|
||||
|
||||
```sh
|
||||
sudo useradd --system --home /var/lib/fancy-dns --shell /usr/sbin/nologin fancy-dns
|
||||
sudo install -m 0755 dist/fancy-dns /usr/local/bin/fancy-dns
|
||||
sudo install -d -o fancy-dns -g fancy-dns /var/lib/fancy-dns
|
||||
sudo install -d /etc/fancy-dns
|
||||
sudo install -m 0600 -o fancy-dns -g fancy-dns fancy-dns.kubeconfig /etc/fancy-dns/kubeconfig
|
||||
```
|
||||
|
||||
Start from `deploy/compose/config.yaml` (a backup-resolver config) or
|
||||
`deploy/config.example.yaml`. The parts that matter here:
|
||||
|
||||
```yaml
|
||||
cache:
|
||||
persist:
|
||||
enabled: true
|
||||
path: /var/lib/fancy-dns/cache.db # its directory is the data dir: the
|
||||
# cluster snapshot is saved beside it
|
||||
api:
|
||||
enabled: true
|
||||
listen: ["127.0.0.1:8080"] # loopback only
|
||||
ui: false
|
||||
auth:
|
||||
mode: apikey
|
||||
api_keys: ["${FANCYDNS_API_KEY}"]
|
||||
```
|
||||
|
||||
Keep in the YAML only what is genuinely local: listeners, upstreams, cache.
|
||||
**Do not copy the cluster's groups, lists or zones into it.** YAML wins every
|
||||
collision, so a copy would shadow the live object and you would be editing in
|
||||
the console what the backup ignores. The upstreams and `forward_zones` are
|
||||
config, not objects, so they must match the in-cluster ones by hand; compare the
|
||||
two files after any change.
|
||||
|
||||
The service reads its secrets from an environment file:
|
||||
|
||||
```sh
|
||||
sudo install -m 0600 /dev/null /etc/fancy-dns/fancy-dns.env
|
||||
echo "FANCYDNS_API_KEY=$(openssl rand -hex 24)" | sudo tee -a /etc/fancy-dns/fancy-dns.env
|
||||
```
|
||||
|
||||
An unset `${VAR}` is a startup failure on purpose: a missing key is a crash
|
||||
loop, not an unauthenticated API.
|
||||
|
||||
### systemd
|
||||
|
||||
`/etc/systemd/system/fancy-dns.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=fancy-dns backup resolver
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
User=fancy-dns
|
||||
Group=fancy-dns
|
||||
EnvironmentFile=/etc/fancy-dns/fancy-dns.env
|
||||
ExecStart=/usr/local/bin/fancy-dns \
|
||||
-config /etc/fancy-dns/config.yaml \
|
||||
-kubeconfig /etc/fancy-dns/kubeconfig \
|
||||
-kube-namespace dns
|
||||
ExecReload=/bin/kill -HUP $MAINPID
|
||||
Restart=on-failure
|
||||
RestartSec=2
|
||||
TimeoutStopSec=20
|
||||
LimitNOFILE=65536
|
||||
|
||||
# Port 53 without root.
|
||||
AmbientCapabilities=CAP_NET_BIND_SERVICE
|
||||
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
PrivateTmp=yes
|
||||
ReadWritePaths=/var/lib/fancy-dns
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Always pass `-kube-namespace`. Outside a pod there is no namespace to default
|
||||
to, and watching all namespaces would need a cluster-wide Role.
|
||||
|
||||
`-kubeconfig` turns the Kubernetes source on by itself. `reload` is a SIGHUP:
|
||||
the YAML is re-read and every component is swapped without dropping a query.
|
||||
Changes to the cluster objects need no reload at all.
|
||||
|
||||
### Port 53
|
||||
|
||||
Check what owns it first: `sudo ss -lntup | grep ':53 '`. `systemd-resolved`'s
|
||||
stub listener binds only `127.0.0.53`, but a wildcard bind to `0.0.0.0:53`
|
||||
still collides with it. Either bind the node's address in `dns.udp`/`dns.tcp`
|
||||
or set `DNSStubListener=no` in `/etc/systemd/resolved.conf`. Open UDP and TCP 53
|
||||
(and 853/443 if you enabled DoT/DoH) in the node's firewall.
|
||||
|
||||
Do not make this resolver the node's only DNS server (`/etc/resolv.conf`): after
|
||||
a reboot the node needs name resolution before the resolver is up.
|
||||
|
||||
## 3. Start it and check
|
||||
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now fancy-dns
|
||||
journalctl -u fancy-dns -n 30 --no-pager # look for: watching Kubernetes for fancy-dns objects
|
||||
|
||||
KEY=$(sudo sed -n 's/^FANCYDNS_API_KEY=//p' /etc/fancy-dns/fancy-dns.env)
|
||||
curl -s -H "X-API-Key: $KEY" http://127.0.0.1:8080/api/status | jq '.components'
|
||||
dig @127.0.0.1 example.com +short
|
||||
dig @127.0.0.1 <a name from one of your DNSZones or AddressOverrides> +short
|
||||
```
|
||||
|
||||
`components.degraded` should be empty. The two `dig`s tell you it resolves and
|
||||
that it has the cluster's data. Then simulate the outage you bought this for:
|
||||
block the API server from the node (or stop the resolver and start it with the
|
||||
cluster unreachable), and confirm the same names still answer and
|
||||
`components.degraded` now names the snapshot.
|
||||
|
||||
Finally point clients at it as a **second** nameserver (DHCP option 6, or the
|
||||
router's secondary). It is a backup, not a load balancer: most stub resolvers
|
||||
only try the second server after the first times out.
|
||||
|
||||
## The console outside the cluster
|
||||
|
||||
Optional. You only need it on the node if you want to edit while the in-cluster
|
||||
console is unavailable, and it can only save while the API server is reachable,
|
||||
since an edit is a write to the cluster. Reads of statistics come from whichever
|
||||
resolver you point it at.
|
||||
|
||||
It needs a **write** Role, which is a separate account from the resolver's.
|
||||
`deploy/k8s/console/console.yaml` defines the Role `fancy-dns-console`; bind a
|
||||
second service account to it the same way as above (name it
|
||||
`fancy-dns-console-ext`, build `/etc/fancy-dns/console.kubeconfig`).
|
||||
|
||||
`/etc/fancy-dns/console.yaml` (see `deploy/console.example.yaml`):
|
||||
|
||||
```yaml
|
||||
blocklist: {enabled: false}
|
||||
api:
|
||||
enabled: true
|
||||
listen: ["127.0.0.1:8081"]
|
||||
base_path: /api
|
||||
ui: true
|
||||
auth:
|
||||
mode: apikey
|
||||
api_keys: ["${FANCYDNS_CONSOLE_KEY}"]
|
||||
trust_loopback: false
|
||||
```
|
||||
|
||||
`/etc/fancy-dns/console.env`:
|
||||
|
||||
```sh
|
||||
FANCYDNS_CONSOLE_KEY=<a long random key: what you type into the UI>
|
||||
FANCYDNS_RUNTIME_KEY=<the resolver's FANCYDNS_API_KEY: what the console sends it>
|
||||
```
|
||||
|
||||
Run it with a unit like the resolver's, minus the port-53 capability:
|
||||
|
||||
```ini
|
||||
ExecStart=/usr/local/bin/fancy-dns-console \
|
||||
-config /etc/fancy-dns/console.yaml \
|
||||
-runtime-url http://127.0.0.1:8080 \
|
||||
-kubeconfig /etc/fancy-dns/console.kubeconfig \
|
||||
-kube-namespace dns
|
||||
```
|
||||
|
||||
Keep it on loopback and reach it with an SSH tunnel
|
||||
(`ssh -L 8081:127.0.0.1:8081 controller`) or an authenticated TLS reverse proxy.
|
||||
The UI can change DNS for every client you have; do not publish it with an API
|
||||
key on plain HTTP.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause |
|
||||
|---|---|
|
||||
| Service exits immediately: `no usable Kubernetes credentials` | `-kubeconfig` points at a file that is missing, unreadable by the `fancy-dns` user, or malformed. |
|
||||
| Starts, `components.degraded` says `not synced` | The API server is unreachable or the token is rejected (`kubectl --kubeconfig … get clientgroups`), or the CRDs are not installed. |
|
||||
| `…not synced and no saved snapshot` and cluster data missing | First start with the cluster down. Start it once with the cluster up. Also check the data directory is writable. |
|
||||
| `kubernetes: objects rejected` in `components.degraded` | An object failed validation (or this binary is older than the cluster's). The last good version stays in force. Which objects, and why: `jq .problems /var/lib/fancy-dns/kube-snapshot.json`. |
|
||||
| Cluster answers differ from the backup's | The upstream/forward-zone config differs (it is YAML, not an object), or the backup's cache holds older answers. `POST /api/cache/flush` on it. |
|
||||
| `bind: address already in use` | Something owns port 53; see [Port 53](#port-53). |
|
||||
| Console: `cluster: …forbidden` | It is using the read-only kubeconfig; it needs the write Role. |
|
||||
Reference in new issue
Block a user