more otel

README.md

@@ -103,8 +103,26 @@ gRPC services for traces, metrics, and logs using standard OTLP protobuf definit
 - `GET /api/traces/{traceId}` - Retrieve all stored entries (spans and
   correlated logs) referencing a given trace id.
 
-All endpoints above search the full persistent store (when `OTELLOG_PATH`
-storage is enabled), not just the in-memory ring buffer.
+All endpoints above search the full persistent store, not just the
+in-memory ring buffer.
+
+### Stats API Endpoints (Port 8080, requires ClickHouse)
+
+Aggregate time-series over the stored telemetry, computed server-side from
+an hourly rollup table. Available only when `OTEL_CLICKHOUSE_ADDR` is
+configured (see Configuration below); otherwise they respond `501` with an
+explanatory JSON body and the UI hides its "Trends" tab.
+
+- `GET /api/stats/errors?hours=72&service=` - Total/error entry counts per
+  hour (per day when `hours > 336`). Response:
+  `{"granularity":"hour","points":[{"bucket":"2026-09-17 14:00:00","total":42,"errors":3}]}`.
+- `GET /api/stats/releases?hours=72&service=` - Per-release error-count
+  series: `{"series":{"v1.3":[{bucket,total,errors}...],"v1.4":[...]}}`
+  for release-over-release comparison. Entries need a `service.release`
+  attribute to appear here.
+- `GET /api/stats/services` - Distinct services seen in the rollup.
+
+All accept `hours` (1..2160, default 72) and an optional `service` filter.
 
 ### Issues API Endpoints (Port 8080)
 
@@ -115,6 +133,20 @@ Errors seen in traces (spans with an error status) and logs (severity
   Supports `limit`.
 - `GET /api/issues/{id}` - Fetch a single issue, including up to 5 recent
   sample occurrences (timestamp, trace/span id, message).
+- `GET /api/patterns` - List deterministic groups of related exact issues.
+  Pattern grouping is explainable and does not merge issue IDs.
+- `GET /api/issues/{id}/similar` - Return bounded, deterministic related-issue
+  suggestions with a score and matching reasons.
+- `POST /api/issues/{id}/similar` - Record a review label with
+  `{"issueId":"...","decision":"same|different"}` for future training data.
+- `GET /api/rollouts` - List read-only Kubernetes Deployment rollout state.
+- `GET /api/rollouts/{id}` - Fetch one normalized rollout.
+- `GET /api/rollouts/{id}/impact` - Return optional Prometheus error-rate points
+  around the rollout.
+- `POST /api/sourcemaps` - Upload a JavaScript source map. Public multipart upload;
+  the release build script sends the source-map file contents directly.
+  multipart fields are `release`, `source` (the generated bundle URL/path),
+  and `file`. Uploaded maps are used server-side to resolve frontend stacks.
 
 ### Web Interface
 
@@ -127,6 +159,9 @@ Errors seen in traces (spans with an error status) and logs (severity
 - `GET /agents_example.md` - Serves this repo's agent instructions doc
   directly, so an AI agent can fetch it from the base URL before making any
   API calls.
+- `GET /instrumentation.md` - Serves the frontend/backend instrumentation
+  guide, including React error boundaries and source-map deployment.
+- `GET /static/instrumentation.md` - Static/public alias for the same guide.
 
 ## Usage
 
@@ -175,7 +210,12 @@ through the same history/search/issue pipeline as backend telemetry, tagged
 ```html
 <script src="https://otel.k6n.net/static/otel-rum.js"></script>
 <script>
-  var rum = OtelRUM.init({ serviceName: "my-web-app" });
+  var rum = OtelRUM.init({
+    serviceName: "my-web-app",
+    release: "2026.09.18-abc123",
+    // Optional; defaults to https://otel.k6n.net.
+    baseUrl: "https://otel.example.com",
+  });
 
   // Inject a W3C traceparent header into any HTTP client so a backend span
   // (if the service is OTel-instrumented) shares this request's trace id -
@@ -188,23 +228,128 @@ through the same history/search/issue pipeline as backend telemetry, tagged
 </script>
 ```
 
+React render failures should be reported from an error boundary as well. The
+project UI uses this pattern in `otel-ui/src/components/ViewErrorBoundary.tsx`:
+
+```tsx
+class AppBoundary extends React.Component<{ children: React.ReactNode }> {
+  componentDidCatch(error: Error, info: React.ErrorInfo) {
+    OtelRUM.last?.captureException(error, {
+      "exception.type": "ReactErrorBoundary",
+      "react.component_stack": info.componentStack,
+    });
+  }
+
+  render() { return this.props.children; }
+}
+```
+
+The boundary should also render a fallback UI; otherwise a broken component
+can leave the whole application blank while still being logged.
+
 `fetch` is auto-instrumented by default (same-origin requests only) to
 inject `traceparent` and record request/response breadcrumbs; this can be
 disabled via `injectTraceHeaders: false` / `autoInstrumentFetch: false`.
-Out of scope for v1: source map resolution (stack traces stay minified),
-session replay, PII scrubbing - don't route sensitive data through
-breadcrumbs or attributes.
+The log ingest endpoint supports unauthenticated cross-origin POSTs; no OIDC
+session or `Authorization` header is needed.
+Source maps are kept server-side. When a captured frontend error has a
+matching `service.release` and generated bundle path, the Issues view displays
+the resolved original source frame while retaining the generated frame.
+
+#### Source-map deploy workflow
+
+Build the UI with a stable release identifier and keep the generated map next
+to the bundle. Do not publish the map publicly:
+
+```sh
+export RELEASE="2026.09.18-$(git rev-parse --short HEAD)"
+npm --prefix otel-ui run build
+
+# Upload after the new UI is deployed. Use the generated URL/path exactly as
+# it appears in browser stack traces.
+curl -fS -X POST https://otel.example.com/api/sourcemaps \
+  -F "release=$RELEASE" \
+  -F "source=/assets/index-ABC123.js" \
+  -F "file=@otel-ui/dist/assets/index-ABC123.js.map"
+```
+
+For this project, `otel-ui/src/main.tsx` sends `VITE_APP_RELEASE` as the RUM
+`service.release` value; set it during the build, for example:
+
+```sh
+VITE_APP_RELEASE="$RELEASE" npm --prefix otel-ui run build
+```
+
+The repository also includes `scripts/upload-sourcemap.sh`; after deploying,
+run it against the emitted bundle, for example:
+
+```sh
+RELEASE="$RELEASE" OTEL_URL="https://otel.example.com" \
+  ./scripts/upload-sourcemap.sh otel-ui/dist/assets/index-ABC123.js
+```
+
+The receiver stores uploaded maps under `OTEL_SOURCEMAP_DIR` (default
+`./sourcemaps`, `/sourcemaps` in the Kubernetes deployment). Upload maps for
+each release before investigating its errors. The API never returns map
+contents or original source files.
+
+`build.sh` passes the release into the Docker UI build. If `OTEL_URL` is set,
+it also builds the matching UI locally and uploads all generated JavaScript
+maps. The Dockerfile removes `.map` files from the
+final image by default; set `PUBLIC_SOURCEMAPS=true` for a development or
+staging image where Chrome DevTools should fetch the original sources:
+
+```sh
+PUBLIC_SOURCEMAPS=true ./build.sh
+```
+
+Public maps can expose application source code, so keep this disabled for
+production unless that exposure is intentional. Credentials are never passed
+to Docker or stored in the image.
 
 ## Configuration
 
 - `OTELLOG_PATH` - path to the persistent telemetry log file (default
-  `otel-agent.log`).
+  `otel-agent.log`). Only used when ClickHouse is not configured.
+- `OTEL_SOURCEMAP_DIR` - directory for uploaded release source maps (default
+  `./sourcemaps`; mount persistent storage there in production).
+- `OTEL_KUBERNETES_ENABLED` - set to `false` to disable rollout observation;
+  otherwise it activates in-cluster or `KUBECONFIG` observation when available.
+- `OTEL_KUBERNETES_NAMESPACE` - optional namespace restriction; empty watches
+  all namespaces permitted by the service account.
+- `OTEL_KUBERNETES_LABEL_SELECTOR` - optional Deployment label selector.
+- `OTEL_PROMETHEUS_URL` - optional Prometheus base URL for rollout impact.
+- `OTEL_PROMETHEUS_ERROR_QUERY` - optional `fmt.Sprintf`-style PromQL template;
+  `%q` placeholders receive service and release in that order.
+- `OTEL_CLICKHOUSE_ADDR` - optional `host:port` (native protocol, usually
+  `9000`). When set, telemetry is stored in ClickHouse instead of the JSONL
+  file: history search stays fast and unbounded, and the aggregate stats
+  endpoints (`/api/stats/*`) and the UI "Trends" tab are enabled. The
+  schema (`otel` database, raw `entries` table + materialized hourly
+  rollup) is created automatically on first connect with a 90-day TTL; if
+  ClickHouse is unreachable at startup the server falls back to the file
+  store.
+- `OTEL_CLICKHOUSE_DB` / `OTEL_CLICKHOUSE_USER` /
+  `OTEL_CLICKHOUSE_PASSWORD` - optional ClickHouse connection settings
+  (defaults: `otel` / `default` / empty).
 - `OTEL_INGEST_KEY` - optional shared key required (via `?key=` or the
   `X-Otel-Ingest-Key` header) on `/v1/traces`, `/v1/metrics` and `/v1/logs`
   over HTTP. Unset by default (no key required). This is meant to
   scope/rate-limit a public ingest endpoint fed by a browser script, not to
   provide real authentication - pair it with the built-in per-IP rate
   limiting on those endpoints if the endpoint is public.
+- `OTEL_AGENT_TOKEN` - optional service token for non-interactive debugging
+  agents. Send it as `X-Otel-Agent-Token` or `Authorization: Bearer ...` when
+  calling protected `/api/history`, `/api/issues`, `/api/traces`, and stats
+  endpoints. Keep it in a Kubernetes Secret and never expose it to browsers.
+
+A single-node ClickHouse instance for Kubernetes is provided in
+`k8s/clickhouse.yaml` (StatefulSet + Service + PVC, deployed in the
+`monitoring` namespace); the receiver Deployment in `k8s/deployment.yaml`
+already points at it. The UI's Trends tab (and live issue updates) appear
+automatically once the stats endpoints answer; on file-store deployments the
+tab hides itself. SPA routes (/issues, /traces, /logs, /metrics, /trends,
+/setup) are served client-side by wouter and survive the OIDC login redirect.
 
 Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.