# Go OTel Debugger

A debug receiver for OpenTelemetry telemetry data (traces, metrics, and logs) that provides real-time visualization and historical data access. It accepts OTLP data via HTTP and gRPC protocols, stores a configurable history of received telemetry, and broadcasts updates to connected WebSocket clients for live debugging.

## Features

- **OTLP Support**: Receives traces, metrics, and logs via both HTTP (JSON/Protobuf) and gRPC (Protobuf) protocols on standard OTLP ports.
- **Real-time Broadcasting**: Uses WebSocket connections to broadcast incoming telemetry data to all connected clients instantly.
- **Historical Data**: Maintains a configurable ring buffer of recent telemetry data, accessible via REST API endpoints.
- **Web Interface**: Includes a built-in HTML interface for visualizing telemetry data in real-time with counters and clear buttons.
- **Docker Support**: Containerized for easy deployment.
- **Kubernetes Ready**: Includes deployment manifests for Kubernetes clusters.

## Architecture

The application consists of several key components:

1. **HTTP/gRPC Servers**: Handle incoming OTLP requests on ports 8080 (HTTP) and 4317 (gRPC).
2. **Collector**: Processes and stores telemetry data in a ring buffer for history. Implements OpenTelemetry consumer interfaces.
3. **WebSocket Server**: Manages client connections and broadcasts new telemetry data as JSON messages.
4. **Web Interface**: Static HTML page served on the root path that connects to the WebSocket for real-time updates.
5. **History API**: REST endpoints to retrieve stored telemetry data by type or all combined.

Data flow:
```
OTLP Client → HTTP/gRPC Server → Collector → WebSocket Broadcast → Connected Clients
                                      ↓
                                 History Storage → REST API → Clients
```

## Getting Started

### Prerequisites

- Go 1.25+ (for local development)
- Docker (optional, for containerized deployment)

### Running Locally

1. Clone the repository:
   ```sh
   git clone https://git.k6n.net/go-otel.git
   cd go-otel
   ```

2. Run the server:
   ```sh
   go run ./cmd/server
   ```

The server will start on ports 8080 (HTTP/WebSocket) and 4317 (gRPC).

### Running with Docker

1. Build the Docker image:
   ```sh
   docker build -t go-otel .
   ```

2. Run the container:
   ```sh
   docker run -p 8080:8080 -p 4317:4317 go-otel
   ```

### Kubernetes Deployment

Apply the included Kubernetes manifests:
```sh
kubectl apply -f k8s/
```

This deploys the application with a Service and Ingress for external access.

## API Endpoints

### OTLP HTTP Endpoints (Port 8080)

- `POST /v1/traces` - Receive OTLP traces
- `POST /v1/metrics` - Receive OTLP metrics  
- `POST /v1/logs` - Receive OTLP logs

Supported Content-Types: `application/json`, `application/x-protobuf`

### OTLP gRPC Endpoints (Port 4317)

gRPC services for traces, metrics, and logs using standard OTLP protobuf definitions.

### WebSocket Endpoint (Port 8080)

- `GET /ws` - WebSocket connection for real-time telemetry streaming

### History API Endpoints (Port 8080)

- `GET /api/history` - Retrieve all historical telemetry data
- `GET /api/history/traces` - Retrieve only trace history
- `GET /api/history/metrics` - Retrieve only metrics history
- `GET /api/history/logs` - Retrieve only logs history
- `GET /api/history/search` - Search history. Supports `q` (free-text
  substring), `type`, `since`/`until` (unix nanoseconds), and `limit`.
- `GET /api/history/wait` - Block until a matching entry arrives. Same
  filters as `search`, plus `timeout` (seconds, default 30). Rate-limited
  per client IP.
- `GET /api/traces/{traceId}` - Retrieve all stored entries (spans and
  correlated logs) referencing a given trace id. With ClickHouse this is an
  indexed lookup (`entries.trace_ids`) and each payload is trimmed to that
  trace's own spans/log records.
- `GET /api/traces/summaries?ids=a,b,c` - Per-trace rollups (name, start/end,
  duration, span count, error count, services; max 100 ids) without reading
  span payloads. Maintained on insert by the `trace_summaries` view; empty
  with the file store.

All endpoints above search the full persistent store, not just the
in-memory ring buffer.

### Stats API Endpoints (Port 8080, requires ClickHouse)

Aggregate time-series over the stored telemetry, computed server-side from
an hourly rollup table. Available only when `OTEL_CLICKHOUSE_ADDR` is
configured (see Configuration below); otherwise they respond `501` with an
explanatory JSON body and the UI hides its "Trends" tab.

- `GET /api/stats/errors?hours=72&service=` - Total/error entry counts per
  hour (per day when `hours > 336`). Response:
  `{"granularity":"hour","points":[{"bucket":"2026-09-17 14:00:00","total":42,"errors":3}]}`.
- `GET /api/stats/releases?hours=72&service=` - Per-release error-count
  series: `{"series":{"v1.3":[{bucket,total,errors}...],"v1.4":[...]}}`
  for release-over-release comparison. Entries need a `service.release`
  attribute to appear here.
- `GET /api/stats/services` - Distinct services seen in the rollup.
- `GET /api/stats/endpoints?hours=72&host=&service=&limit=100` - HTTP span
  traffic grouped by normalized service, host, method, path, route, status,
  user-agent, and safe browser request metadata. `spanCount` exposes nested
  proxy/instrumentation duplication while `traceCount` approximates logical
  requests. Origin and referrer are returned as host-only values; query
  strings are stripped from paths.

All accept `hours` (1..2160, default 72) and an optional `service` filter.

### Issues API Endpoints (Port 8080)

Errors seen in traces (spans with an error status) and logs (severity
`ERROR`/`FATAL`) are deduplicated into "issues" as they're ingested.

- `GET /api/issues` - List deduplicated issues, most recently seen first.
  Supports `limit`.
- `GET /api/issues/{id}` - Fetch a single issue, including up to 5 recent
  sample occurrences (timestamp, trace/span id, message).
- `GET /api/issues/{id}/sources` - Return every distinct source IP recorded
  for the issue, with occurrence count and first/last seen timestamps. Requires
  the ClickHouse occurrence store; returns `501` with file/in-memory storage.
- `GET /api/issues/{id}/traces` - Every trace the issue occurred in, most
  recently hit first, with each trace's summary and its occurrences, in one
  request (`limit`, `before` = `nextBefore` from the previous page).
  ClickHouse keeps *all* occurrences (`issue_occurrences`, 90-day TTL);
  without it the response is built from the 5 in-memory samples and
  `complete` is `false`.
- `GET /api/patterns` - List deterministic groups of related exact issues.
  Pattern grouping is explainable and does not merge issue IDs.
- `GET /api/issues/{id}/similar` - Return bounded, deterministic related-issue
  suggestions with a score and matching reasons.
- `POST /api/issues/{id}/similar` - Record a review label with
  `{"issueId":"...","decision":"same|different"}` for future training data.
- `GET /api/rollouts` - List read-only Kubernetes Deployment rollout state.
- `GET /api/rollouts/{id}` - Fetch one normalized rollout.
- `GET /api/rollouts/{id}/impact` - Return optional Prometheus error-rate points
  around the rollout.
- `POST /api/sourcemaps` - Upload a JavaScript source map. Requires the shared
  `OTEL_SOURCEMAP_KEY` in an `X-Sourcemap-Key` header (uploads are refused while
  it is unset). Multipart upload;
  multipart fields are `release`, `source` (the generated bundle URL/path),
  and `file`. Uploaded maps are used server-side to resolve frontend stacks.

### Web Interface

- `GET /` - Serves the HTML visualization interface

### Static Routes

- `GET /static/otel-rum.js` - The frontend error collector script (see
  below).
- `GET /agents_example.md` - Serves this repo's agent instructions doc
  directly, so an AI agent can fetch it from the base URL before making any
  API calls.
- `GET /instrumentation.md` - Serves the frontend/backend instrumentation
  guide, including React error boundaries and source-map deployment.
- `GET /static/instrumentation.md` - Static/public alias for the same guide.

## Usage

### Sending Telemetry Data

Send OTLP data to the appropriate endpoints. The data will be stored in history and broadcast to all connected WebSocket clients.

#### HTTP Example (JSON)

```sh
curl -X POST http://localhost:8080/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"scope":{"name":"test-scope"},"spans":[{"traceId":"0102030405060708090a0b0c0d0e0f10","spanId":"0102030405060708","name":"test-span","startTimeUnixNano":"1640995200000000000","endTimeUnixNano":"1640995260000000000","attributes":[{"key":"test.key","value":{"stringValue":"test-value"}}]}]}]}]}'
```

#### gRPC Example

Use any OTLP gRPC client library to send data to `localhost:4317`.

### Real-time Visualization

1. Open `http://localhost:8080` in your browser
2. The interface will automatically connect via WebSocket
3. Send telemetry data to see it appear in real-time
4. Use the "Clear" buttons to reset individual data sections
5. View message counters for each telemetry type

### Accessing Historical Data

Use the history API endpoints to retrieve previously received data:

```sh
curl http://localhost:8080/api/history/traces
```

This returns JSON arrays of historical entries with timestamps.

### Frontend Error Collector (`static/otel-rum.js`)

A framework-free browser script that captures uncaught errors, unhandled
promise rejections and failed resource loads (img/script/link), and ships
them as OTLP log records to the existing `/v1/logs` endpoint - so they flow
through the same history/search/issue pipeline as backend telemetry, tagged
`source: frontend`.

```html
<script src="https://otel.k6n.net/static/otel-rum.js"></script>
<script>
  var rum = OtelRUM.init({
    serviceName: "my-web-app",
    release: "2026.09.18-abc123",
    // Optional; defaults to https://otel.k6n.net.
    baseUrl: "https://otel.example.com",
  });

  // Inject a W3C traceparent header into any HTTP client so a backend span
  // (if the service is OTel-instrumented) shares this request's trace id -
  // then GET /api/traces/{traceId} shows the frontend error and backend
  // spans for the same request together.
  fetch("/api/orders", { headers: rum.traceHeaders() });

  // Manual error reporting.
  try { risky(); } catch (err) { rum.captureException(err); }
</script>
```

React render failures should be reported from an error boundary as well. The
project UI uses this pattern in `otel-ui/src/components/ViewErrorBoundary.tsx`:

```tsx
class AppBoundary extends React.Component<{ children: React.ReactNode }> {
  componentDidCatch(error: Error, info: React.ErrorInfo) {
    OtelRUM.last?.captureException(error, {
      "exception.type": "ReactErrorBoundary",
      "react.component_stack": info.componentStack,
    });
  }

  render() { return this.props.children; }
}
```

The boundary should also render a fallback UI; otherwise a broken component
can leave the whole application blank while still being logged.

`fetch` is auto-instrumented by default (same-origin requests only) to
inject `traceparent` and record request/response breadcrumbs; this can be
disabled via `injectTraceHeaders: false` / `autoInstrumentFetch: false`.
The log ingest endpoint supports unauthenticated cross-origin POSTs; no OIDC
session or `Authorization` header is needed.
Source maps are kept server-side. When a captured frontend error has a
matching `service.release` and generated bundle path, the Issues view displays
the resolved original source frame while retaining the generated frame.

#### Source-map deploy workflow

Build the UI with a stable release identifier and keep the generated map next
to the bundle. Do not publish the map publicly:

```sh
export RELEASE="2026.09.18-$(git rev-parse --short HEAD)"
npm --prefix otel-ui run build

# Upload after the new UI is deployed. Use the generated URL/path exactly as
# it appears in browser stack traces.
curl -fS -X POST https://otel.example.com/api/sourcemaps \
  -H "X-Sourcemap-Key: $OTEL_SOURCEMAP_KEY" \
  -F "release=$RELEASE" \
  -F "source=/assets/index-ABC123.js" \
  -F "file=@otel-ui/dist/assets/index-ABC123.js.map"
```

For this project, `otel-ui/src/main.tsx` sends `VITE_APP_RELEASE` as the RUM
`service.release` value; set it during the build, for example:

```sh
VITE_APP_RELEASE="$RELEASE" npm --prefix otel-ui run build
```

The repository also includes `scripts/upload-sourcemap.sh`; after deploying,
run it against the emitted bundle, for example:

```sh
RELEASE="$RELEASE" OTEL_URL="https://otel.example.com" \
  ./scripts/upload-sourcemap.sh otel-ui/dist/assets/index-ABC123.js
```

The receiver stores uploaded maps under `OTEL_SOURCEMAP_DIR` (default
`./sourcemaps`, `/sourcemaps` in the Kubernetes deployment). Maps are reloaded on start, so keep the directory on persistent storage. See
`instrumentation.md` (*Source maps → Requirements checklist*) for everything a
project needs for stacks to resolve. `GET /api/sourcemaps` lists loaded maps.
The API never returns map files themselves.

`build.sh` passes the release into the Docker UI build. If `OTEL_URL` is set,
it also builds the matching UI locally and uploads all generated JavaScript
maps. The Dockerfile removes `.map` files from the
final image by default; set `PUBLIC_SOURCEMAPS=true` for a development or
staging image where Chrome DevTools should fetch the original sources:

```sh
PUBLIC_SOURCEMAPS=true ./build.sh
```

Public maps can expose application source code, so keep this disabled for
production unless that exposure is intentional. Credentials are never passed
to Docker or stored in the image.

## Configuration

- `OTELLOG_PATH` - path to the persistent telemetry log file (default
  `otel-agent.log`). Only used when ClickHouse is not configured.
- `OTEL_SOURCEMAP_KEY` - shared key required to upload source maps.
- `OTEL_ISSUE_STATUS_PATH` - file that keeps resolved/ignored issue decisions across restarts when ClickHouse is not used (default: next to `OTELLOG_PATH`).
- `OTEL_TODOS_PATH` - file for todos filed via `otel-todo.js` when ClickHouse is not used (default: next to `OTELLOG_PATH`).
- `OTEL_SOURCEMAP_CACHE_MB` - size budget (on-disk MB, default 64) for maps kept
  parsed in memory; maps are indexed at start and parsed lazily on first use.
- `OTEL_SOURCEMAP_DIR` - directory for uploaded release source maps (default
  `./sourcemaps`; mount persistent storage there in production).
- `OTEL_IP_INVESTIGATION_NMAP` - set to `true` to enable the authenticated,
  rate-limited active service probe. It is off by default because Nmap sends
  active network traffic to the investigated public IP.
- `ABUSE_KEY` - optional AbuseIPDB API key. When set, the authenticated IP
  detail view can check a public address and submit a report without exposing
  the key to the browser. Keep it in a secret, never in the frontend build.
- `OTEL_PPROF_ADDR` - address for the Go pprof server (default
  `127.0.0.1:6060`). It serves `/debug/pprof/` separately from the public UI
  port; use a port-forward or set this explicitly while diagnosing memory or
  CPU usage.
- `OTEL_KUBERNETES_ENABLED` - set to `false` to disable rollout observation;
  otherwise it activates in-cluster or `KUBECONFIG` observation when available.
- `OTEL_KUBERNETES_NAMESPACE` - optional namespace restriction; empty watches
  all namespaces permitted by the service account.
- `OTEL_KUBERNETES_LABEL_SELECTOR` - optional Deployment label selector.
- `OTEL_PROMETHEUS_URL` - optional Prometheus base URL for rollout impact.
- `OTEL_PROMETHEUS_ERROR_QUERY` - optional `fmt.Sprintf`-style PromQL template;
  `%q` placeholders receive service and release in that order.
- `OTEL_CLICKHOUSE_ADDR` - optional `host:port` (native protocol, usually
  `9000`). When set, telemetry is stored in ClickHouse instead of the JSONL
  file: history search stays fast and unbounded, and the aggregate stats
  endpoints (`/api/stats/*`) and the UI "Trends" tab are enabled. The
  schema (`otel` database, raw `entries` table + materialized hourly
  rollup) is created automatically on first connect with a 90-day TTL; if
  ClickHouse is unreachable at startup the server falls back to the file
  store.
- `OTEL_CLICKHOUSE_DB` / `OTEL_CLICKHOUSE_USER` /
  `OTEL_CLICKHOUSE_PASSWORD` - optional ClickHouse connection settings
  (defaults: `otel` / `default` / empty).

### Issues display

The standalone `issues-tui` command is a terminal dashboard for currently
firing alert spikes. It reads normalized error events directly from
ClickHouse; it does not replace the web UI's persisted `/api/issues` view.
When a spike stops firing, its card stays on the display marked resolved
until something fires again.

Build and run it on the display computer (or use `go run`):

```sh
go build -o issues-tui ./cmd/issues-tui
OTEL_CLICKHOUSE_ADDR=clickhouse.example.net:9000 \
OTEL_CLICKHOUSE_DB=otel \
OTEL_CLICKHOUSE_USER=default \
OTEL_CLICKHOUSE_PASSWORD='…' \
OTEL_ALERT_BASE_URL=https://otel.example.net \
./issues-tui
```

The display computer must be able to reach ClickHouse's native TCP port. For a
Kubernetes-only ClickHouse service, use a private network path or tunnel; the
in-cluster service name is not reachable from an ordinary workstation. The
dashboard refreshes using the `OTEL_ALERT_*` interval, window, baseline,
cooldown and lag settings. Press `r` to scan immediately and `q` to quit.
Use `-once` for a one-shot, non-interactive output suitable for a process
supervisor health check.

- `OTEL_INGEST_KEY` - optional shared key required (via `?key=` or the
  `X-Otel-Ingest-Key` header) on `/v1/traces`, `/v1/metrics` and `/v1/logs`
  over HTTP. Unset by default (no key required). This is meant to
  scope/rate-limit a public ingest endpoint fed by a browser script, not to
  provide real authentication - pair it with the built-in per-IP rate
  limiting on those endpoints if the endpoint is public.
- `OTEL_AGENT_TOKEN` - optional service token for non-interactive debugging
  agents. Send it as `X-Otel-Agent-Token` or `Authorization: Bearer ...` when
  calling protected `/api/history`, `/api/issues`, `/api/traces`, and stats
  endpoints. Keep it in a Kubernetes Secret and never expose it to browsers.

A single-node ClickHouse instance for Kubernetes is provided in
`k8s/clickhouse.yaml` (StatefulSet + Service + PVC, deployed in the
`monitoring` namespace); the receiver Deployment in `k8s/deployment.yaml`
already points at it. The UI's Trends tab (and live issue updates) appear
automatically once the stats endpoints answer; on file-store deployments the
tab hides itself. SPA routes (/issues, /traces, /logs, /metrics, /trends,
/setup) are served client-side by wouter and survive the OIDC login redirect.
Selection lives in the URL too, so every step can be reloaded, stepped with
back/forward, and shared (each detail view has a link chip that copies its
permalink):

| Path | Shows |
| --- | --- |
| `/issues/<issueId>` | one issue with its traces |
| `/issues/<issueId>/trace/<traceId>` | the same, with a trace selected |
| `/patterns/<patternId>[/<issueId>]` | a pattern card open, optionally reviewing one member |
| `/traces/<traceId>[/<spanId>]` | a trace, optionally with a span selected |
| `/rum/session/<sessionId>` | a RUM session |
| `/rum/errors[/<errorId>]` | JS errors, optionally one error expanded (`errorId` is a hash of type + message) |

Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.

## Development

To modify the application:

1. Edit source files in `cmd/server/` and `internal/`
2. Run locally with `go run ./cmd/server`
3. For UI changes, edit `index.html` directly

The application uses standard Go modules and OpenTelemetry collector libraries for data processing.