mats/go-otel
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove Needle; stop paging on nginx 4xx

Browse Source 
Needle scored ~chance on the golden set and never gated alerts, so drop the
sidecar, proxy handlers, UI buttons, k8s container and build step. The labelled
eval sets move to cmd/triage-eval/data.

nginx 4xx (registry 401 challenges, /oauth2/auth, scanner 405s) is now dropped
in FilterActionable instead of only being discounted, so it cannot ride along
with a real 5xx. nginx 5xx still alerts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
matsandClaude Sonnet 5.5 committed 2026-11-01 11:40:53 +01:00
1 parent e046bd86d1
commit 51a8412ae1
22 files changed
+29 -1162
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/ML.md
+1 -1
View File
@@ -1,7 +1,7 @@
# Error detection model
> Latest status, decisions and next steps: [STATUS.md](STATUS.md). Where they
> differ (ClickHouse is now a LoadBalancer; Needle is dormant), STATUS.md wins.
> differ (ClickHouse is now a LoadBalancer; Needle is removed), STATUS.md wins.
## Current handoff (2026-09-18)
docs/STATUS.md
+13 -22
View File
@@ -10,7 +10,6 @@ unless stated. Greenfield, single user: radical changes are fine.
patched live. `docs/ML.md` still says "keep ClusterIP"; that is superseded.
- Credentials are in `.env` (gitignored, mode 600). Load with
`set -a; . ./.env; set +a`. `clickhouse-client` is installed locally.
- `.env` deliberately does not enable Needle; pass `-needle URL` explicitly.
## What exists now
@@ -21,7 +20,7 @@ unless stated. Greenfield, single user: radical changes are fine.
| 404/410 are not errors | `isErrorStatus` in `internal/normalization/normalizer.go` | Done. Rows stored before this still count 404s as errors. |
| Backfill | `cmd/normalized-backfill` | Ran 2026-09-19 12:00 to ~21:00 UTC. Queries need `FINAL` (duplicates until merge). |
| Spike rule | `internal/features/spike.go` | Excess-error rule, trimmed-mean baseline, `Suppressor` (cooldown, 2x escalation). Tested. |
| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories; Needle optional and off. |
| Replay CLI | `cmd/spike-alerts` | Replays a range; mutes non-actionable categories. |
| Cause categories | `internal/triage` | Deterministic rules, first match wins. Tested. |
| Rules scoring | `cmd/triage-eval` | Scores a labelled JSONL; also prepares rows for labelling. |
| Offender report | `cmd/bad-actors` | Read-only. Ranks IPs from raw nginx spans, marks `[BLOCK]` / `[INTERNAL]`. |
@@ -35,14 +34,14 @@ Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`
## Findings worth remembering
- **Needle scored 10% on the 40-row golden set** (about chance for 8 classes).
Reversing the tool order changed its answers wholesale (15%, 34/40 `service_defect`),
so it picks by position, not content. Confidence is uncorrelated with
correctness. Decision: **Needle stays dormant.** Intended later use: a gate that
decides whether a larger model should act on an issue (affects anyone? fixable by
a smart model?). Do not use it for categorization.
- **Needle was removed (2026-11-01).** It scored 10% on the 40-row golden set (about
chance for 8 classes), picked by tool position rather than content, and its
confidence was uncorrelated with correctness. Sidecar, proxy handlers, UI and k8s
container are gone; the labelled sets live in `cmd/triage-eval/data/`.
- **Edge 4xx never pages.** nginx 4xx (registry 401 challenges, `/oauth2/auth`,
scanner 405s) is dropped in `FilterActionable`; nginx 5xx still alerts.
- **Rules scored 39/40 on the golden set, but that is a fit** (written after seeing
the rows). A fresh 60-row holdout is in `cmd/needle-sidecar/eval/holdout.jsonl`
the rows). A fresh 60-row holdout is in `cmd/triage-eval/data/holdout.jsonl`
(from 2026-09-18, unseen); it still needs your `expected` labels. Known
probe-like paths that still land in `stale_reference`: `/ssl/server.key`,
`/_debugbar/open`, `/debug/pprof`, `/actuator/*`, `/graphql`, `/server-info`,
@@ -78,7 +77,7 @@ Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`
1. Deploy the collector so normalized rows stay current; verify with
`SELECT max(timestamp) FROM otel.normalized_events`.
2. Label `holdout.jsonl` (fill `expected`), run
`go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl`, fix
`go run ./cmd/triage-eval -input cmd/triage-eval/data/holdout.jsonl`, fix
the rule families that miss. Then re-export a fresh holdout so scoring stays
honest.
3. Warm-up guard in `DetectSpikes`: require baseline data before alerting (the
@@ -164,7 +163,7 @@ Categories: `scanner_probe`, `dependency_unreachable`, `client_credential_issue`
```bash
set -a; . ./.env; set +a
# replay the spike rule over a range (Needle off)
# replay the spike rule over a range
go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00Z -baseline-windows 12
# add -show-muted to see probes / stale / lifecycle patterns too
@@ -172,12 +171,7 @@ go run ./cmd/spike-alerts -since 2026-09-19T12:00:00Z -until 2026-09-19T20:30:00
go run ./cmd/bad-actors -since 2026-09-19T08:00:00Z -until 2026-09-19T20:30:00Z
# score the rules against labelled rows
go run ./cmd/triage-eval -input cmd/needle-sidecar/eval/holdout.jsonl
# Needle eval (needs the sidecar running; venv was in scratch and is gone)
pip install cactus-needle # python 3.12 recommended
NEEDLE_PORT=5005 NEEDLE_HOST=127.0.0.1 python cmd/needle-sidecar/server.py
python cmd/needle-sidecar/eval/run_eval.py
go run ./cmd/triage-eval -input cmd/triage-eval/data/holdout.jsonl
```
Backfill in small windows only (10 to 30 minutes over the LAN); one-hour windows
@@ -187,14 +181,11 @@ through a `kubectl port-forward` hung and dropped.
Everything is uncommitted. Files changed by this work: `internal/{storage,
normalization,features,triage,honeypot,blocker}`, `cmd/{spike-alerts,
triage-eval,bad-actors,normalized-backfill,server,needle-sidecar}`,
triage-eval,bad-actors,normalized-backfill,server}`,
`k8s/{clickhouse,deployment}.yaml`, `.gitignore`.
**Not from this work; check before committing:** `AGENTS.md`,
`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**`
(including a deleted `GaugeMeter.tsx`). Also tracked `__pycache__/*.pyc` files
under `cmd/needle-sidecar` show as modified; they should be untracked and
gitignored.
`internal/collector/{collector_test,issue_extract}.go`, `otel-ui/src/**` (including a deleted `GaugeMeter.tsx`).
## IP details (MaxMind)