133 lines
7.5 KiB
Markdown
133 lines
7.5 KiB
Markdown
# Status: error triage, spike detection, IP blocking (2026-09-19)
|
|
|
|
Pick-up notes. Nothing below is committed, deployed or applied to the cluster
|
|
unless stated. Greenfield, single user: radical changes are fine.
|
|
|
|
## Access
|
|
|
|
- ClickHouse (otel instance) is now a **LoadBalancer**: `monitoring/clickhouse`
|
|
at `10.10.3.45` (native 9000, HTTP 8123). Set in `k8s/clickhouse.yaml` and
|
|
patched live. `docs/ML.md` still says "keep ClusterIP"; that is superseded.
|
|
- Credentials are in `.env` (gitignored, mode 600). Load with
|
|
`set -a; . ./.env; set +a`. `clickhouse-client` is installed locally.
|
|
- `.env` deliberately does not enable Needle; pass `-needle URL` explicitly.
|
|
|
|
## What exists now
|
|
|
|
| Piece | Where | State |
|
|
| --- | --- | --- |
|
|
| Normalized rows written on ingest | `internal/storage/normalized.go`, `Append` in `clickhouse.go` | Code done, tests pass. **Running collector is still the old build**, so nothing new is written until deployed. |
|
|
| Live schema migration | `EnsureNormalizedSchema` (runs at collector start) | Applied once by hand via backfill; `issue_fingerprint` etc. now exist. |
|
|
| 404/410 are not errors | `isErrorStatus` in `internal/normalization/normalizer.go` | Done. Rows stored before this still count 404s as errors. |
|
|
| Backfill | `cmd/normalized-backfill` | Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need `FINAL` (duplicates until merge). |
|
|
| Spike rule | `internal/features/spike.go` | Excess-error rule, trimmed-mean baseline, `Suppressor` (cooldown, 2x escalation). Tested. |
|
|
| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories; Needle optional and off. |
|
|
| Cause categories | `internal/triage` | Deterministic rules, first match wins. Tested. |
|
|
| Rules scoring | `cmd/triage-eval` | Scores a labelled JSONL; also prepares rows for labelling. |
|
|
| Offender report | `cmd/bad-actors` | Read-only. Ranks IPs from raw nginx spans, marks `[BLOCK]` / `[INTERNAL]`. |
|
|
| Tracker | `internal/honeypot` | Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses `triage.IsProbePath`. |
|
|
| Blocker | `internal/blocker` | Writes ingress-nginx `block-cidrs`. **Dry-run by default.** Tested with a fake client. Wired into `cmd/server`. |
|
|
| RBAC + env | `k8s/deployment.yaml` | Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied. |
|
|
|
|
Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`,
|
|
`stale_reference`, `service_defect`, `capacity_saturation`, `expected_lifecycle`,
|
|
`needs_review`. They are cause-level verdicts, not restatements of status.
|
|
|
|
## Findings worth remembering
|
|
|
|
- **Needle scored 10% on the 40-row golden set** (about chance for 8 classes).
|
|
Reversing the tool order changed its answers wholesale (15%, 34/40 `service_defect`),
|
|
so it picks by position, not content. Confidence is uncorrelated with
|
|
correctness. Decision: **Needle stays dormant.** Intended later use: a gate that
|
|
decides whether a larger model should act on an issue (affects anyone? fixable by
|
|
a smart model?). Do not use it for categorization.
|
|
- **Rules scored 39/40 on the golden set, but that is a fit** (written after seeing
|
|
the rows). A fresh 60-row holdout is in `cmd/needle-sidecar/eval/holdout.jsonl`
|
|
(from 2026-09-18, unseen); it still needs your `expected` labels. Known
|
|
probe-like paths that still land in `stale_reference`: `/ssl/server.key`,
|
|
`/_debugbar/open`, `/debug/pprof`, `/actuator/*`, `/graphql`, `/server-info`,
|
|
`/.bash_profile`, `/settings.py`, `/__env.js`.
|
|
- **The top error sources were noise:** 404 probes and a 401 poller
|
|
(`/api/repos/available`, `/api/controller/instances`, about 700 per hour, counted
|
|
twice: nginx and the orchestrator).
|
|
- **IP data:** 44 public IPs account for 63% of probe requests (12.5 h window).
|
|
**34% show a peer of `127.0.0.1`** because `net.peer.ip` is the TCP peer behind
|
|
the PROXY-protocol hop, so the tracker cannot see them. The ingress has
|
|
`enable-real-ip=true`, `forwarded-for-header=proxy_protocol`,
|
|
`proxy-real-ip-cidr=127.0.0.1/32`, so `block-cidrs` itself works on the real
|
|
client address.
|
|
- **Internal addresses are a hard stop** in both tracker and blocker
|
|
(`honeypot.Untrackable`): loopback, RFC1918, ULA, link-local, multicast,
|
|
unspecified, IPv4-mapped, CGNAT `100.64/10` (Tailscale), `192.0.0.0/24`,
|
|
`198.18/15`, `240/4`. The allowlist only adds exclusions.
|
|
|
|
## Before enabling enforcement
|
|
|
|
1. Deploy the new build (dry-run on). Watch `GET /api/blocker/status` for a day;
|
|
compare against `go run ./cmd/bad-actors -since ... -until ...`.
|
|
2. Put your own public IP(s) in `OTEL_BLOCKER_ALLOW`. A blocked IP gets 403, so
|
|
you would lock yourself out of the UI. Recovery: `kubectl edit cm
|
|
ingress-nginx-controller -n ingress-nginx` (remove entries; managed ones are
|
|
listed in the `otel-debug.k6n.net/blocked` annotation).
|
|
3. Then set `OTEL_BLOCKER_DRY_RUN=false`.
|
|
4. Do not `kubectl apply` the whole `k8s/deployment.yaml` blindly: the dry run
|
|
showed it would create a new `otel-sourcemaps` PVC not present in the cluster.
|
|
|
|
## Next steps (suggested order)
|
|
|
|
1. Deploy the collector so normalized rows stay current; verify with
|
|
`SELECT max(timestamp) FROM otel.normalized_events`.
|
|
2. Label `holdout.jsonl` (fill `expected`), run
|
|
`go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl`, fix
|
|
the rule families that miss. Then re-export a fresh holdout so scoring stays
|
|
honest.
|
|
3. Warm-up guard in `DetectSpikes`: require baseline data before alerting (the
|
|
first window after data begins alerts on everything as NEW).
|
|
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the
|
|
ingress OTel config first; not looked at yet).
|
|
5. Alert delivery: run `DetectSpikes` on a timer in the collector and post
|
|
somewhere (UI panel / ntfy / Slack). Category chips and `same / different /
|
|
noise` + "mute pattern" in the UI, storing labels.
|
|
6. Reduce 401-poller noise: mute list or per-pattern rate limit;
|
|
`client_credential_issue` is the caller's fault, likely a lower severity.
|
|
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the
|
|
logistic regression / MLP from `docs/ML.md`; the rule detector is what they
|
|
must beat.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
set -a; . ./.env; set +a
|
|
|
|
# replay the spike rule over a range (Needle off)
|
|
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
|
|
# add -show-muted to see probes / stale / lifecycle patterns too
|
|
|
|
# who would be blocked
|
|
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z
|
|
|
|
# score the rules against labelled rows
|
|
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl
|
|
|
|
# Needle eval (needs the sidecar running; venv was in scratch and is gone)
|
|
pip install cactus-needle # python 3.12 recommended
|
|
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
|
|
python cmd/needle-sidecar/eval/run_eval.py
|
|
```
|
|
|
|
Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows
|
|
through a `kubectl port-forward` hung and dropped.
|
|
|
|
## Git state
|
|
|
|
Everything is uncommitted. Files changed by this work: `internal/{storage,
|
|
normalization,features,triage,honeypot,blocker}`, `cmd/{spike-alerts,
|
|
triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}`,
|
|
`k8s/{clickhouse,deployment}.yaml`, `.gitignore`.
|
|
|
|
**Not from this work; check before committing:** `AGENTS.md`,
|
|
`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**`
|
|
(including a deleted `GaugeMeter.tsx`). Also tracked `__pycache__/*.pyc` files
|
|
under `cmd/needle-sidecar` show as modified; they should be untracked and
|
|
gitignored.
|