Adds internal/laya, a client and safety-gated decision engine for the
Laya decision API (https://laya.k6n.net). The API's OpenAPI document
leaves every response schema empty, so the wire format was pinned by
probing the live service: questions are typed (choice/score/noul) and
carry "criteria" as an object for a choice and an array for a score.
An alert is rendered into a compact factual state and asked four typed
questions: which action (restart/ignore/notify/escalate), severity and
how urgent. The state always includes the pods' last events - restart
count, readiness, last termination reason and exit code, log tail, and
any overlapping rollout - because a 503 burst from a pod being replaced
is otherwise indistinguishable from an outage.
The model proposes, the policy disposes. Every gate records why it
changed an answer: an executable-action allowlist, a probability and
lead-over-runner-up threshold, no restart while pods are crash-looping
(the kubelet is already doing it) or while a rollout is replacing every
pod (the 503s are the rollout's own), never ignoring a new error or a
high-priority spike without a human's mute, an hourly restart budget
per deployment, and a plain notification whenever the model is
unreachable. The margin, not the raw probability, is the useful signal:
label probabilities are diffuse and a correct answer rarely exceeds 0.5.
topology.PodLog now carries the pod's lifecycle facts alongside the log
tail. PodsNotReady/CrashLooping helpers build on them.
Verified end to end against the live API: a crash loop is withheld as
"already crash-looping", a rollout's 503 burst is held back at a 2%
margin, and crawler 404s are ignored at 13%. LAYA_URL=... go test
./internal/laya -run TestLiveModel replays it.
Standalone cmd that runs the same pipeline as the webhook notifications
(alerts.LoadErrorEvents -> alerts.DetectAlerts -> cooldown suppressor) on
a schedule and renders what is firing as a row of severity-colored cards -
built for an ultra-wide, low display. A pattern shows PAGED when the scan
would have notified and ongoing while suppressed; disappearing spikes are
reported as recovered. -once prints one snapshot for cron/pipe use.
Colors follow the otel-ui design-system tokens (dark and light).
- POST /v1/csp collects browser CSP reports in both wire formats (legacy
application/csp-report and Reporting API application/reports+json), with
CORS limited to the operator's domains (any subdomain of k6n.net,
tornberg.me, knatofs.se) and preflight handling.
- Reports are stored in a dedicated typed ClickHouse table (csp_reports,
90-day TTL) with a JSONL file fallback (OTEL_CSP_PATH).
- New authenticated view endpoints /api/csp/reports and /api/csp/summary
plus a 'CSP reports' screen in the UI (tab 12) with time window, site and
directive filters and report drill-down.
- Documented in instrumentation.md; unit tests for parsing/handlers/file
store and an env-gated ClickHouse integration test.
Verified end to end against users.k6n.net: device grant approved in a
browser, refresh token stored 0600, access token carries aud
[dev-console, https://otel.k6n.net].
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AuthClient dev-console + AuthServer deviceFlow via k8s/auth-dev.yaml; otel
server accepts its own URL as an extra audience (OTEL_OIDC_AUDIENCE). Vite
dev-only device-flow sign-in, scripts/dev-auth.mjs helper, AGENTS.md notes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Normalize radii, drop shadows/blur and font-black, and document the
one-view-at-a-time plan in AGENTS.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Restore IssuesVisualizer redesign reverted by merge 67ce419 and re-apply
Needle triage/test additions. Migrate all remaining screens off the MD3
palette, raw Tailwind colors, gradients and Material Symbols. Remove the
legacy theme block and fonts, drop the duplicate tab strip, rebuild
TelemetryFilters on FilterBar, and document status in AGENTS.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add static/otel-rum.js: a framework-free browser script that captures
uncaught errors, unhandled promise rejections and failed resource loads,
translating them into OTLP log records POSTed to the existing /v1/logs
endpoint (source: frontend attribute) so they flow through the same
history/search/issue pipeline as backend telemetry.
- Add correlation-id support: traceHeaders() (manual) and same-origin
fetch auto-instrumentation inject a W3C traceparent header, so a
frontend error and the backend spans it triggered share a trace id and
can be found together via GET /api/traces/{traceId}.
- Add CORS handling (withCORS in cmd/server/main.go) on the OTLP HTTP
endpoints, required for any browser-origin POST.
- Add an optional shared ingest key (OTEL_INGEST_KEY, via '?key=' or
X-Otel-Ingest-Key) and a per-IP rate limit (20 rps/burst 50) on OTLP
HTTP ingestion, to scope/protect a public endpoint fed by a browser
script. Verified end-to-end (CORS preflight, cross-origin POST, ingest
key enforcement, rate limiting) against a running server.
- Serve static/otel-rum.js and agents_example.md as static HTTP routes so
both the script and the agent instructions doc are fetchable directly
from the base URL.
- Ignore the separately-cloned otel-ui/ frontend repo (has its own .git;
not meant to be nested into this repo's history).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add internal/issues.Tracker: fingerprints (kind, service, title) into
deduplicated Issues with count/firstSeen/lastSeen/affected services and
bounded sample occurrences, evicting the least-recently-seen issue once
a cap is reached.
- Add internal/collector/issue_extract.go to pull issues out of ingested
OTLP data: spans with an error status (preferring exception.type/message
from an "exception" event, falling back to the span status message), and
logs at ERROR/FATAL severity (by severity number or text).
- Wire extraction into Collector.ConsumeTraces/ConsumeLogs and expose
GET /api/issues and GET /api/issues/{id}.
- Fix a pre-existing data race in the collector test's mockStore (unsynced
access to entries while addToHistory appends asynchronously), found via
`go test -race` while adding the new issue-tracking tests.
- Update docs/PLAN.md, README.md and agents_example.md for the new API.
The UI "Issues" tab itself lives in the separate frontend project that
builds this server's dist/ bundle; this commit only covers the backend.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Fix /api/history/{type} to search the full persistent store instead of
only the bounded in-memory ring buffer, so filtered history isn't
silently truncated.
- Add structured type/since/until filters to /api/history/search and
/api/history/wait, alongside the existing free-text query.
- Add GET /api/traces/{traceId} to fetch all entries for a trace in one
call instead of relying on substring search.
- Rate-limit /api/history/wait per client IP.
- Add docs/PLAN.md tracking the phased improvement plan; update README
and agents_example.md for the new/changed endpoints.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
developer
mats