206 lines
12 KiB
Markdown
206 lines
12 KiB
Markdown
# Status: error triage, spike detection, IP blocking (2026-09-19)
|
|
|
|
Pick-up notes. Nothing below is committed, deployed or applied to the cluster
|
|
unless stated. Greenfield, single user: radical changes are fine.
|
|
|
|
## Access
|
|
|
|
- ClickHouse (otel instance) is now a **LoadBalancer**: `monitoring/clickhouse`
|
|
at `10.10.3.45` (native 9000, HTTP 8123). Set in `k8s/clickhouse.yaml` and
|
|
patched live. `docs/ML.md` still says "keep ClusterIP"; that is superseded.
|
|
- Credentials are in `.env` (gitignored, mode 600). Load with
|
|
`set -a; . ./.env; set +a`. `clickhouse-client` is installed locally.
|
|
- `.env` deliberately does not enable Needle; pass `-needle URL` explicitly.
|
|
|
|
## What exists now
|
|
|
|
| Piece | Where | State |
|
|
| --- | --- | --- |
|
|
| Normalized rows written on ingest | `internal/storage/normalized.go`, `Append` in `clickhouse.go` | Code done, tests pass. **Running collector is still the old build**, so nothing new is written until deployed. |
|
|
| Live schema migration | `EnsureNormalizedSchema` (runs at collector start) | Applied once by hand via backfill; `issue_fingerprint` etc. now exist. |
|
|
| 404/410 are not errors | `isErrorStatus` in `internal/normalization/normalizer.go` | Done. Rows stored before this still count 404s as errors. |
|
|
| Backfill | `cmd/normalized-backfill` | Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need `FINAL` (duplicates until merge). |
|
|
| Spike rule | `internal/features/spike.go` | Excess-error rule, trimmed-mean baseline, `Suppressor` (cooldown, 2x escalation). Tested. |
|
|
| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories; Needle optional and off. |
|
|
| Cause categories | `internal/triage` | Deterministic rules, first match wins. Tested. |
|
|
| Rules scoring | `cmd/triage-eval` | Scores a labelled JSONL; also prepares rows for labelling. |
|
|
| Offender report | `cmd/bad-actors` | Read-only. Ranks IPs from raw nginx spans, marks `[BLOCK]` / `[INTERNAL]`. |
|
|
| Tracker | `internal/honeypot` | Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses `triage.IsProbePath`. |
|
|
| Blocker | `internal/blocker` | Writes ingress-nginx `block-cidrs`. **Dry-run by default.** Tested with a fake client. Wired into `cmd/server`. |
|
|
| RBAC + env | `k8s/deployment.yaml` | Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied. |
|
|
|
|
Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`,
|
|
`stale_reference`, `service_defect`, `capacity_saturation`, `expected_lifecycle`,
|
|
`needs_review`. They are cause-level verdicts, not restatements of status.
|
|
|
|
## Findings worth remembering
|
|
|
|
- **Needle scored 10% on the 40-row golden set** (about chance for 8 classes).
|
|
Reversing the tool order changed its answers wholesale (15%, 34/40 `service_defect`),
|
|
so it picks by position, not content. Confidence is uncorrelated with
|
|
correctness. Decision: **Needle stays dormant.** Intended later use: a gate that
|
|
decides whether a larger model should act on an issue (affects anyone? fixable by
|
|
a smart model?). Do not use it for categorization.
|
|
- **Rules scored 39/40 on the golden set, but that is a fit** (written after seeing
|
|
the rows). A fresh 60-row holdout is in `cmd/needle-sidecar/eval/holdout.jsonl`
|
|
(from 2026-09-18, unseen); it still needs your `expected` labels. Known
|
|
probe-like paths that still land in `stale_reference`: `/ssl/server.key`,
|
|
`/_debugbar/open`, `/debug/pprof`, `/actuator/*`, `/graphql`, `/server-info`,
|
|
`/.bash_profile`, `/settings.py`, `/__env.js`.
|
|
- **The top error sources were noise:** 404 probes and a 401 poller
|
|
(`/api/repos/available`, `/api/controller/instances`, about 700 per hour, counted
|
|
twice: nginx and the orchestrator).
|
|
- **IP data:** 44 public IPs account for 63% of probe requests (12.5 h window).
|
|
**34% show a peer of `127.0.0.1`** because `net.peer.ip` is the TCP peer behind
|
|
the PROXY-protocol hop, so the tracker cannot see them. The ingress has
|
|
`enable-real-ip=true`, `forwarded-for-header=proxy_protocol`,
|
|
`proxy-real-ip-cidr=127.0.0.1/32`, so `block-cidrs` itself works on the real
|
|
client address.
|
|
- **Internal addresses are a hard stop** in both tracker and blocker
|
|
(`honeypot.Untrackable`): loopback, RFC1918, ULA, link-local, multicast,
|
|
unspecified, IPv4-mapped, CGNAT `100.64/10` (Tailscale), `192.0.0.0/24`,
|
|
`198.18/15`, `240/4`. The allowlist only adds exclusions.
|
|
|
|
## Before enabling enforcement
|
|
|
|
1. Deploy the new build (dry-run on). Watch `GET /api/blocker/status` for a day;
|
|
compare against `go run ./cmd/bad-actors -since ... -until ...`.
|
|
2. Put your own public IP(s) in `OTEL_BLOCKER_ALLOW`. A blocked IP gets 403, so
|
|
you would lock yourself out of the UI. Recovery: `kubectl edit cm
|
|
ingress-nginx-controller -n ingress-nginx` (remove entries; managed ones are
|
|
listed in the `otel-debug.k6n.net/blocked` annotation).
|
|
3. Then set `OTEL_BLOCKER_DRY_RUN=false`.
|
|
4. Do not `kubectl apply` the whole `k8s/deployment.yaml` blindly: the dry run
|
|
showed it would create a new `otel-sourcemaps` PVC not present in the cluster.
|
|
|
|
## Next steps (suggested order)
|
|
|
|
1. Deploy the collector so normalized rows stay current; verify with
|
|
`SELECT max(timestamp) FROM otel.normalized_events`.
|
|
2. Label `holdout.jsonl` (fill `expected`), run
|
|
`go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl`, fix
|
|
the rule families that miss. Then re-export a fresh holdout so scoring stays
|
|
honest.
|
|
3. Warm-up guard in `DetectSpikes`: require baseline data before alerting (the
|
|
first window after data begins alerts on everything as NEW).
|
|
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the
|
|
ingress OTel config first; not looked at yet).
|
|
5. Alert delivery: run `DetectSpikes` on a timer in the collector and post
|
|
somewhere (UI panel / ntfy / Slack). Category chips and `same / different /
|
|
noise` + "mute pattern" in the UI, storing labels.
|
|
6. Reduce 401-poller noise: mute list or per-pattern rate limit;
|
|
`client_credential_issue` is the caller's fault, likely a lower severity.
|
|
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the
|
|
logistic regression / MLP from `docs/ML.md`; the rule detector is what they
|
|
must beat.
|
|
|
|
## Update 2026-09-20
|
|
|
|
- Warm-up guard: `SpikeConfig.DataSince` / `WarmupWindows` (default 6). No alert
|
|
until that much baseline exists; `spike-alerts` sets it from
|
|
`min(timestamp)` of `normalized_events`. Test: `TestDetectSpikesWarmupGuard`.
|
|
- `holdout.jsonl` labelled (by Claude, review it; ambiguous 404s such as `/mcp`,
|
|
`/graphql`, `/api`, `/api/v2/*` were labelled `stale_reference`). Rules fixed for
|
|
the misses: more probe paths (`_debugbar`, `debug/pprof`, `server-info`,
|
|
`actuator`, `__env`, `.bash_*`, `*.key|pem|py`, `graphql/console`) and all
|
|
1xx/2xx/3xx are `expected_lifecycle`. 60/60, but the set is now tuned
|
|
against, so re-export a fresh holdout before trusting the number.
|
|
`IsProbePath` is shared with the blocker, so these also widen what counts as a probe.
|
|
- Client IP gap (last 6h of nginx spans): public IPs 10.3k spans (2192 IPs),
|
|
`10.10.0.1` 25k (the user's firewall: traffic it NATs to ingress arrives with the firewall as source), loopback 4.1k. The loopback rows include
|
|
real external scanners (CameraScannerV3, MSIE8 UAs, raw-IP hosts), so ~10% of
|
|
spans lose the attacker IP. Ingress has `externalTrafficPolicy: Local`, no
|
|
`use-proxy-protocol`, and `x-real-ip` is never attached. Cause: something on the
|
|
node forwards to ingress from 127.0.0.1; not yet identified.
|
|
|
|
- UA classes: `internal/uaclass` holds the known crawler / AI agent / monitor /
|
|
scan-tool list (first match wins; UA is spoofable, so a class is a positive
|
|
signal only). Stored as `normalized_events.ua_class` (added by
|
|
`EnsureNormalizedSchema` on next start; old rows are `''`), served as
|
|
`uaClass` on issue samples and endpoint stats. Issues UI: add-filter offers
|
|
`client: X` and `hide client: X`. Not yet a triage/spike feature.
|
|
|
|
- Trace context (`internal/tracectx`): one request is several spans in several
|
|
envelopes (nginx outer span on :442 with the real client, nginx inner span on
|
|
:8181 from 127.0.0.1, then the app span), so the ingest-time same-envelope
|
|
lookup in `normalizeTraces` almost never finds the parent (7 of 1154 nginx
|
|
spans measured). `tracectx.Resolve` works on a loaded span set instead and is
|
|
order independent. It yields `ClientIP` (first public peer up the chain),
|
|
`IsDuplicate` (repeats an ancestor's failure: same status code), `IsRootCause`,
|
|
child / upstream-child counts, `ParentMissing`.
|
|
- `bad-actors` resolves loopback hops to the real client (839 hop spans /
|
|
583 probes recovered in a 3h sample).
|
|
- `spike-alerts` loads the spans of every trace with an error, drops duplicate
|
|
hops from the counts and reports distinct clients per pattern; it falls back
|
|
when the trace columns are absent. Historical rows have empty `span_id`, so it
|
|
only helps for data written since the trace columns shipped (~08:09 UTC 2026-09-20).
|
|
- `features.Row` gained `LogicalErrors`, `DuplicateErrors`, `RootCauseErrors`,
|
|
`DistinctClients`, `TopClientShare`, `BotErrorShare` (additive).
|
|
`window-features` reads optional `clientIp/uaClass/isDuplicate/isRootCause/hasUpstream`.
|
|
- Trace view in the UI shows the resolved client (`lib/trace-client.ts`).
|
|
- Not done: the collector still records the hop IP on issue samples at ingest;
|
|
span name `GET /` for orchestrator spans is unchanged (changing it alters every
|
|
orchestrator pattern signature); `HasUpstream` is loaded but not yet a triage
|
|
rule; trace propagation is missing in cart/checkout/profile/scheduling (roots only).
|
|
|
|
- Alerts (`internal/alerts`, env `OTEL_ALERT_*`, secret `otel-alerts`): the server
|
|
runs the spike detector every minute and posts to Telegram (and/or a generic
|
|
webhook). First tick after a start only primes the suppressor. `GET /api/alerts`,
|
|
`POST /api/alerts/test`, `go run ./cmd/alert-test`. Recent alerts are in memory only.
|
|
- Bad actors (`features.MarkBadActors`): an event is marked, never dropped, when its
|
|
client hit >=3 probe paths (`probe-scanner`), walked >=15 not-found routes
|
|
(`enumerator`), or is a declared crawler/agent/scanner getting a 4xx (`bot-4xx`;
|
|
their 5xx stay visible). Marked events are ignored by `DetectSpikes`/alerts and
|
|
counted in `Row.BadActorErrors`. **`Row.Label` now targets the clean future error
|
|
rate** (logical requests, no bad actors); `RawLabel` is the old raw-count label. AI
|
|
crawler rows stay in the data on purpose (see memory: flag, not block).
|
|
Client rules need a resolved client IP, so they only cover data since the trace
|
|
columns shipped; the bot rule works on any row with a UA class.
|
|
- Route ids: `term_<hex>` style segments normalize to `{id}` (per-session websocket
|
|
routes were creating a new pattern each session).
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
set -a; . ./.env; set +a
|
|
|
|
# replay the spike rule over a range (Needle off)
|
|
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
|
|
# add -show-muted to see probes / stale / lifecycle patterns too
|
|
|
|
# who would be blocked
|
|
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z
|
|
|
|
# score the rules against labelled rows
|
|
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl
|
|
|
|
# Needle eval (needs the sidecar running; venv was in scratch and is gone)
|
|
pip install cactus-needle # python 3.12 recommended
|
|
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
|
|
python cmd/needle-sidecar/eval/run_eval.py
|
|
```
|
|
|
|
Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows
|
|
through a `kubectl port-forward` hung and dropped.
|
|
|
|
## Git state
|
|
|
|
Everything is uncommitted. Files changed by this work: `internal/{storage,
|
|
normalization,features,triage,honeypot,blocker}`, `cmd/{spike-alerts,
|
|
triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}`,
|
|
`k8s/{clickhouse,deployment}.yaml`, `.gitignore`.
|
|
|
|
**Not from this work; check before committing:** `AGENTS.md`,
|
|
`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**`
|
|
(including a deleted `GaugeMeter.tsx`). Also tracked `__pycache__/*.pyc` files
|
|
under `cmd/needle-sidecar` show as modified; they should be untracked and
|
|
gitignored.
|
|
|
|
## IP details (MaxMind)
|
|
|
|
The issues view's source IP opens a dialog (`GET /api/ip/{ip}`, authed) with
|
|
country/region/city, timezone, ASN and network. Lookups are local. Enable by
|
|
mounting GeoLite2 files and setting `OTEL_GEOIP_CITY_DB` and/or
|
|
`OTEL_GEOIP_ASN_DB` (the Dockerfile copies `db/*.mmdb` into the image and sets both); without them the dialog says so. Private/loopback
|
|
addresses are labelled instead of looked up.
|