# Status: error triage, spike detection, IP blocking (2026-09-19)

Pick-up notes. Nothing below is committed, deployed or applied to the cluster
unless stated. Greenfield, single user: radical changes are fine.

## Access

- ClickHouse (otel instance) is now a **LoadBalancer**: `monitoring/clickhouse`
  at `10.10.3.45` (native 9000, HTTP 8123). Set in `k8s/clickhouse.yaml` and
  patched live. `docs/ML.md` still says "keep ClusterIP"; that is superseded.
- Credentials are in `.env` (gitignored, mode 600). Load with
  `set -a; . ./.env; set +a`. `clickhouse-client` is installed locally.
- `.env` deliberately does not enable Needle; pass `-needle URL` explicitly.

## What exists now

| Piece | Where | State |
| --- | --- | --- |
| Normalized rows written on ingest | `internal/storage/normalized.go`, `Append` in `clickhouse.go` | Code done, tests pass. **Running collector is still the old build**, so nothing new is written until deployed. |
| Live schema migration | `EnsureNormalizedSchema` (runs at collector start) | Applied once by hand via backfill; `issue_fingerprint` etc. now exist. |
| 404/410 are not errors | `isErrorStatus` in `internal/normalization/normalizer.go` | Done. Rows stored before this still count 404s as errors. |
| Backfill | `cmd/normalized-backfill` | Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need `FINAL` (duplicates until merge). |
| Spike rule | `internal/features/spike.go` | Excess-error rule, trimmed-mean baseline, `Suppressor` (cooldown, 2x escalation). Tested. |
| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories; Needle optional and off. |
| Cause categories | `internal/triage` | Deterministic rules, first match wins. Tested. |
| Rules scoring | `cmd/triage-eval` | Scores a labelled JSONL; also prepares rows for labelling. |
| Offender report | `cmd/bad-actors` | Read-only. Ranks IPs from raw nginx spans, marks `[BLOCK]` / `[INTERNAL]`. |
| Tracker | `internal/honeypot` | Now needs distinct paths (default 3), never tracks internal/reserved IPs, uses `triage.IsProbePath`. |
| Blocker | `internal/blocker` | Writes ingress-nginx `block-cidrs`. **Dry-run by default.** Tested with a fake client. Wired into `cmd/server`. |
| RBAC + env | `k8s/deployment.yaml` | Role limited to the one ConfigMap; blocker env set to enabled + dry-run. Not applied. |

Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`,
`stale_reference`, `service_defect`, `capacity_saturation`, `expected_lifecycle`,
`needs_review`. They are cause-level verdicts, not restatements of status.

## Findings worth remembering

- **Needle scored 10% on the 40-row golden set** (about chance for 8 classes).
  Reversing the tool order changed its answers wholesale (15%, 34/40 `service_defect`),
  so it picks by position, not content. Confidence is uncorrelated with
  correctness. Decision: **Needle stays dormant.** Intended later use: a gate that
  decides whether a larger model should act on an issue (affects anyone? fixable by
  a smart model?). Do not use it for categorization.
- **Rules scored 39/40 on the golden set, but that is a fit** (written after seeing
  the rows). A fresh 60-row holdout is in `cmd/needle-sidecar/eval/holdout.jsonl`
  (from 2026-09-18, unseen); it still needs your `expected` labels. Known
  probe-like paths that still land in `stale_reference`: `/ssl/server.key`,
  `/_debugbar/open`, `/debug/pprof`, `/actuator/*`, `/graphql`, `/server-info`,
  `/.bash_profile`, `/settings.py`, `/__env.js`.
- **The top error sources were noise:** 404 probes and a 401 poller
  (`/api/repos/available`, `/api/controller/instances`, about 700 per hour, counted
  twice: nginx and the orchestrator).
- **IP data:** 44 public IPs account for 63% of probe requests (12.5 h window).
  **34% show a peer of `127.0.0.1`** because `net.peer.ip` is the TCP peer behind
  the PROXY-protocol hop, so the tracker cannot see them. The ingress has
  `enable-real-ip=true`, `forwarded-for-header=proxy_protocol`,
  `proxy-real-ip-cidr=127.0.0.1/32`, so `block-cidrs` itself works on the real
  client address.
- **Internal addresses are a hard stop** in both tracker and blocker
  (`honeypot.Untrackable`): loopback, RFC1918, ULA, link-local, multicast,
  unspecified, IPv4-mapped, CGNAT `100.64/10` (Tailscale), `192.0.0.0/24`,
  `198.18/15`, `240/4`. The allowlist only adds exclusions.

## Before enabling enforcement

1. Deploy the new build (dry-run on). Watch `GET /api/blocker/status` for a day;
   compare against `go run ./cmd/bad-actors -since ... -until ...`.
2. Put your own public IP(s) in `OTEL_BLOCKER_ALLOW`. A blocked IP gets 403, so
   you would lock yourself out of the UI. Recovery: `kubectl edit cm
   ingress-nginx-controller -n ingress-nginx` (remove entries; managed ones are
   listed in the `otel-debug.k6n.net/blocked` annotation).
3. Then set `OTEL_BLOCKER_DRY_RUN=false`.
4. Do not `kubectl apply` the whole `k8s/deployment.yaml` blindly: the dry run
   showed it would create a new `otel-sourcemaps` PVC not present in the cluster.

## Next steps (suggested order)

1. Deploy the collector so normalized rows stay current; verify with
   `SELECT max(timestamp) FROM otel.normalized_events`.
2. Label `holdout.jsonl` (fill `expected`), run
   `go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl`, fix
   the rule families that miss. Then re-export a fresh holdout so scoring stays
   honest.
3. Warm-up guard in `DetectSpikes`: require baseline data before alerting (the
   first window after data begins alerts on everything as NEW).
4. Real client IP in nginx spans, to close the 127.0.0.1 gap (inspect the
   ingress OTel config first; not looked at yet).
5. Alert delivery: run `DetectSpikes` on a timer in the collector and post
   somewhere (UI panel / ntfy / Slack). Category chips and `same / different /
   noise` + "mute pattern" in the UI, storing labels.
6. Reduce 401-poller noise: mute list or per-pattern rate limit;
   `client_credential_issue` is the caller's fault, likely a lower severity.
7. After 2+ weeks of data: time-of-day baseline, rollout proximity, then the
   logistic regression / MLP from `docs/ML.md`; the rule detector is what they
   must beat.

## Update 2026-09-20

- Warm-up guard: `SpikeConfig.DataSince` / `WarmupWindows` (default 6). No alert
  until that much baseline exists; `spike-alerts` sets it from
  `min(timestamp)` of `normalized_events`. Test: `TestDetectSpikesWarmupGuard`.
- `holdout.jsonl` labelled (by Claude, review it; ambiguous 404s such as `/mcp`,
  `/graphql`, `/api`, `/api/v2/*` were labelled `stale_reference`). Rules fixed for
  the misses: more probe paths (`_debugbar`, `debug/pprof`, `server-info`,
  `actuator`, `__env`, `.bash_*`, `*.key|pem|py`, `graphql/console`) and all
  1xx/2xx/3xx are `expected_lifecycle`. 60/60, but the set is now tuned
  against, so re-export a fresh holdout before trusting the number.
  `IsProbePath` is shared with the blocker, so these also widen what counts as a probe.
- Client IP gap (last 6h of nginx spans): public IPs 10.3k spans (2192 IPs),
  `10.10.0.1` 25k (the user's firewall: traffic it NATs to ingress arrives with the firewall as source), loopback 4.1k. The loopback rows include
  real external scanners (CameraScannerV3, MSIE8 UAs, raw-IP hosts), so ~10% of
  spans lose the attacker IP. Ingress has `externalTrafficPolicy: Local`, no
  `use-proxy-protocol`, and `x-real-ip` is never attached. Cause: something on the
  node forwards to ingress from 127.0.0.1; not yet identified.

- UA classes: `internal/uaclass` holds the known crawler / AI agent / monitor /
  scan-tool list (first match wins; UA is spoofable, so a class is a positive
  signal only). Stored as `normalized_events.ua_class` (added by
  `EnsureNormalizedSchema` on next start; old rows are `''`), served as
  `uaClass` on issue samples and endpoint stats. Issues UI: add-filter offers
  `client: X` and `hide client: X`. Not yet a triage/spike feature.

- Trace context (`internal/tracectx`): one request is several spans in several
  envelopes (nginx outer span on :442 with the real client, nginx inner span on
  :8181 from 127.0.0.1, then the app span), so the ingest-time same-envelope
  lookup in `normalizeTraces` almost never finds the parent (7 of 1154 nginx
  spans measured). `tracectx.Resolve` works on a loaded span set instead and is
  order independent. It yields `ClientIP` (first public peer up the chain),
  `IsDuplicate` (repeats an ancestor's failure: same status code), `IsRootCause`,
  child / upstream-child counts, `ParentMissing`.
  - `bad-actors` resolves loopback hops to the real client (839 hop spans /
    583 probes recovered in a 3h sample).
  - `spike-alerts` loads the spans of every trace with an error, drops duplicate
    hops from the counts and reports distinct clients per pattern; it falls back
    when the trace columns are absent. Historical rows have empty `span_id`, so it
    only helps for data written since the trace columns shipped (~08:09 UTC 2026-09-20).
  - `features.Row` gained `LogicalErrors`, `DuplicateErrors`, `RootCauseErrors`,
    `DistinctClients`, `TopClientShare`, `BotErrorShare` (additive).
    `window-features` reads optional `clientIp/uaClass/isDuplicate/isRootCause/hasUpstream`.
  - Trace view in the UI shows the resolved client (`lib/trace-client.ts`).
  - Not done: the collector still records the hop IP on issue samples at ingest;
    span name `GET /` for orchestrator spans is unchanged (changing it alters every
    orchestrator pattern signature); `HasUpstream` is loaded but not yet a triage
    rule; trace propagation is missing in cart/checkout/profile/scheduling (roots only).

## Commands

```bash
set -a; . ./.env; set +a

# replay the spike rule over a range (Needle off)
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
# add -show-muted to see probes / stale / lifecycle patterns too

# who would be blocked
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z

# score the rules against labelled rows
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl

# Needle eval (needs the sidecar running; venv was in scratch and is gone)
pip install cactus-needle   # python 3.12 recommended
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
python cmd/needle-sidecar/eval/run_eval.py
```

Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows
through a `kubectl port-forward` hung and dropped.

## Git state

Everything is uncommitted. Files changed by this work: `internal/{storage,
normalization,features,triage,honeypot,blocker}`, `cmd/{spike-alerts,
triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}`,
`k8s/{clickhouse,deployment}.yaml`, `.gitignore`.

**Not from this work; check before committing:** `AGENTS.md`,
`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**`
(including a deleted `GaugeMeter.tsx`). Also tracked `__pycache__/*.pyc` files
under `cmd/needle-sidecar` show as modified; they should be untracked and
gitignored.

## IP details (MaxMind)

The issues view's source IP opens a dialog (`GET /api/ip/{ip}`, authed) with
country/region/city, timezone, ASN and network. Lookups are local. Enable by
mounting GeoLite2 files and setting `OTEL_GEOIP_CITY_DB` and/or
`OTEL_GEOIP_ASN_DB` (the Dockerfile copies `db/*.mmdb` into the image and sets both); without them the dialog says so. Private/loopback
addresses are labelled instead of looked up.