Go OTel Debugger

A debug receiver for OpenTelemetry telemetry data (traces, metrics, and logs) that provides real-time visualization and historical data access. It accepts OTLP data via HTTP and gRPC protocols, stores a configurable history of received telemetry, and broadcasts updates to connected WebSocket clients for live debugging.

Features

• OTLP Support: Receives traces, metrics, and logs via both HTTP (JSON/Protobuf) and gRPC (Protobuf) protocols on standard OTLP ports.
• Real-time Broadcasting: Uses WebSocket connections to broadcast incoming telemetry data to all connected clients instantly.
• Historical Data: Maintains a configurable ring buffer of recent telemetry data, accessible via REST API endpoints.
• Web Interface: Includes a built-in HTML interface for visualizing telemetry data in real-time with counters and clear buttons.
• Docker Support: Containerized for easy deployment.
• Kubernetes Ready: Includes deployment manifests for Kubernetes clusters.

Architecture

The application consists of several key components:

1. HTTP/gRPC Servers: Handle incoming OTLP requests on ports 8080 (HTTP) and 4317 (gRPC).
2. Collector: Processes and stores telemetry data in a ring buffer for history. Implements OpenTelemetry consumer interfaces.
3. WebSocket Server: Manages client connections and broadcasts new telemetry data as JSON messages.
4. Web Interface: Static HTML page served on the root path that connects to the WebSocket for real-time updates.
5. History API: REST endpoints to retrieve stored telemetry data by type or all combined.

Data flow:

OTLP Client → HTTP/gRPC Server → Collector → WebSocket Broadcast → Connected Clients
                                      ↓
                                 History Storage → REST API → Clients

Getting Started

Prerequisites

• Go 1.25+ (for local development)
• Docker (optional, for containerized deployment)

Running Locally

1. Clone the repository:

   git clone https://git.k6n.net/go-otel.git
   cd go-otel
   

2. Run the server:

   go run ./cmd/server
   

The server will start on ports 8080 (HTTP/WebSocket) and 4317 (gRPC).

Running with Docker

1. Build the Docker image:

   docker build -t go-otel .
   

2. Run the container:

   docker run -p 8080:8080 -p 4317:4317 go-otel
   

Kubernetes Deployment

Apply the included Kubernetes manifests:

kubectl apply -f k8s/

This deploys the application with a Service and Ingress for external access.

API Endpoints

OTLP HTTP Endpoints (Port 8080)

• POST /v1/traces - Receive OTLP traces
• POST /v1/metrics - Receive OTLP metrics
• POST /v1/logs - Receive OTLP logs

Supported Content-Types: application/json, application/x-protobuf

OTLP gRPC Endpoints (Port 4317)

gRPC services for traces, metrics, and logs using standard OTLP protobuf definitions.

WebSocket Endpoint (Port 8080)

• GET /ws - WebSocket connection for real-time telemetry streaming

History API Endpoints (Port 8080)

• GET /api/history - Retrieve all historical telemetry data
• GET /api/history/traces - Retrieve only trace history
• GET /api/history/metrics - Retrieve only metrics history
• GET /api/history/logs - Retrieve only logs history
• GET /api/history/search - Search history. Supports q (free-text substring), type, since/until (unix nanoseconds), and limit.
• GET /api/history/wait - Block until a matching entry arrives. Same filters as search, plus timeout (seconds, default 30). Rate-limited per client IP.
• GET /api/traces/{traceId} - Retrieve all stored entries (spans and correlated logs) referencing a given trace id. With ClickHouse this is an indexed lookup (entries.trace_ids) and each payload is trimmed to that trace's own spans/log records.
• GET /api/traces/summaries?ids=a,b,c - Per-trace rollups (name, start/end, duration, span count, error count, services; max 100 ids) without reading span payloads. Maintained on insert by the trace_summaries view; empty with the file store.

All endpoints above search the full persistent store, not just the in-memory ring buffer.

Stats API Endpoints (Port 8080, requires ClickHouse)

Aggregate time-series over the stored telemetry, computed server-side from an hourly rollup table. Available only when OTEL_CLICKHOUSE_ADDR is configured (see Configuration below); otherwise they respond 501 with an explanatory JSON body and the UI hides its "Trends" tab.

• GET /api/stats/errors?hours=72&service= - Total/error entry counts per hour (per day when hours > 336). Response: {"granularity":"hour","points":[{"bucket":"2026-09-17 14:00:00","total":42,"errors":3}]}.
• GET /api/stats/releases?hours=72&service= - Per-release error-count series: {"series":{"v1.3":[{bucket,total,errors}...],"v1.4":[...]}} for release-over-release comparison. Entries need a service.release attribute to appear here.
• GET /api/stats/services - Distinct services seen in the rollup.
• GET /api/stats/endpoints?hours=72&host=&service=&limit=100 - HTTP span traffic grouped by normalized service, host, method, path, route, status, user-agent, and safe browser request metadata. spanCount exposes nested proxy/instrumentation duplication while traceCount approximates logical requests. Origin and referrer are returned as host-only values; query strings are stripped from paths.

All accept hours (1..2160, default 72) and an optional service filter.

Issues API Endpoints (Port 8080)

Errors seen in traces (spans with an error status) and logs (severity ERROR/FATAL) are deduplicated into "issues" as they're ingested.

• GET /api/issues - List deduplicated issues, most recently seen first. Supports limit.
• GET /api/issues/{id} - Fetch a single issue, including up to 5 recent sample occurrences (timestamp, trace/span id, message).
• GET /api/issues/{id}/sources - Return every distinct source IP recorded for the issue, with occurrence count and first/last seen timestamps. Requires the ClickHouse occurrence store; returns 501 with file/in-memory storage.
• GET /api/issues/{id}/traces - Every trace the issue occurred in, most recently hit first, with each trace's summary and its occurrences, in one request (limit, before = nextBefore from the previous page). ClickHouse keeps all occurrences (issue_occurrences, 90-day TTL); without it the response is built from the 5 in-memory samples and complete is false.
• GET /api/patterns - List deterministic groups of related exact issues. Pattern grouping is explainable and does not merge issue IDs.
• GET /api/issues/{id}/similar - Return bounded, deterministic related-issue suggestions with a score and matching reasons.
• POST /api/issues/{id}/similar - Record a review label with {"issueId":"...","decision":"same|different"} for future training data.
• GET /api/rollouts - List read-only Kubernetes Deployment rollout state.
• GET /api/rollouts/{id} - Fetch one normalized rollout.
• GET /api/rollouts/{id}/impact - Return optional Prometheus error-rate points around the rollout.
• POST /api/sourcemaps - Upload a JavaScript source map. Requires the shared OTEL_SOURCEMAP_KEY in an X-Sourcemap-Key header (uploads are refused while it is unset). Multipart upload; multipart fields are release, source (the generated bundle URL/path), and file. Uploaded maps are used server-side to resolve frontend stacks.

Web Interface

• GET / - Serves the HTML visualization interface

Static Routes

• GET /static/otel-rum.js - The frontend error collector script (see below).
• GET /agents_example.md - Serves this repo's agent instructions doc directly, so an AI agent can fetch it from the base URL before making any API calls.
• GET /instrumentation.md - Serves the frontend/backend instrumentation guide, including React error boundaries and source-map deployment.
• GET /static/instrumentation.md - Static/public alias for the same guide.

Usage

Sending Telemetry Data

Send OTLP data to the appropriate endpoints. The data will be stored in history and broadcast to all connected WebSocket clients.

HTTP Example (JSON)

curl -X POST http://localhost:8080/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"scope":{"name":"test-scope"},"spans":[{"traceId":"0102030405060708090a0b0c0d0e0f10","spanId":"0102030405060708","name":"test-span","startTimeUnixNano":"1640995200000000000","endTimeUnixNano":"1640995260000000000","attributes":[{"key":"test.key","value":{"stringValue":"test-value"}}]}]}]}]}'

gRPC Example

Use any OTLP gRPC client library to send data to localhost:4317.

Real-time Visualization

1. Open http://localhost:8080 in your browser
2. The interface will automatically connect via WebSocket
3. Send telemetry data to see it appear in real-time
4. Use the "Clear" buttons to reset individual data sections
5. View message counters for each telemetry type

Accessing Historical Data

Use the history API endpoints to retrieve previously received data:

curl http://localhost:8080/api/history/traces

This returns JSON arrays of historical entries with timestamps.

Frontend Error Collector (static/otel-rum.js)

A framework-free browser script that captures uncaught errors, unhandled promise rejections and failed resource loads (img/script/link), and ships them as OTLP log records to the existing /v1/logs endpoint - so they flow through the same history/search/issue pipeline as backend telemetry, tagged source: frontend.

<script src="https://otel.k6n.net/static/otel-rum.js"></script>
<script>
  var rum = OtelRUM.init({
    serviceName: "my-web-app",
    release: "2026.09.18-abc123",
    // Optional; defaults to https://otel.k6n.net.
    baseUrl: "https://otel.example.com",
  });

  // Inject a W3C traceparent header into any HTTP client so a backend span
  // (if the service is OTel-instrumented) shares this request's trace id -
  // then GET /api/traces/{traceId} shows the frontend error and backend
  // spans for the same request together.
  fetch("/api/orders", { headers: rum.traceHeaders() });

  // Manual error reporting.
  try { risky(); } catch (err) { rum.captureException(err); }
</script>

React render failures should be reported from an error boundary as well. The project UI uses this pattern in otel-ui/src/components/ViewErrorBoundary.tsx:

class AppBoundary extends React.Component<{ children: React.ReactNode }> {
  componentDidCatch(error: Error, info: React.ErrorInfo) {
    OtelRUM.last?.captureException(error, {
      "exception.type": "ReactErrorBoundary",
      "react.component_stack": info.componentStack,
    });
  }

  render() { return this.props.children; }
}

The boundary should also render a fallback UI; otherwise a broken component can leave the whole application blank while still being logged.

fetch is auto-instrumented by default (same-origin requests only) to inject traceparent and record request/response breadcrumbs; this can be disabled via injectTraceHeaders: false / autoInstrumentFetch: false. The log ingest endpoint supports unauthenticated cross-origin POSTs; no OIDC session or Authorization header is needed. Source maps are kept server-side. When a captured frontend error has a matching service.release and generated bundle path, the Issues view displays the resolved original source frame while retaining the generated frame.

Source-map deploy workflow

Build the UI with a stable release identifier and keep the generated map next to the bundle. Do not publish the map publicly:

export RELEASE="2026.09.18-$(git rev-parse --short HEAD)"
npm --prefix otel-ui run build

# Upload after the new UI is deployed. Use the generated URL/path exactly as
# it appears in browser stack traces.
curl -fS -X POST https://otel.example.com/api/sourcemaps \
  -H "X-Sourcemap-Key: $OTEL_SOURCEMAP_KEY" \
  -F "release=$RELEASE" \
  -F "source=/assets/index-ABC123.js" \
  -F "file=@otel-ui/dist/assets/index-ABC123.js.map"

For this project, otel-ui/src/main.tsx sends VITE_APP_RELEASE as the RUM service.release value; set it during the build, for example:

VITE_APP_RELEASE="$RELEASE" npm --prefix otel-ui run build

The repository also includes scripts/upload-sourcemap.sh; after deploying, run it against the emitted bundle, for example:

RELEASE="$RELEASE" OTEL_URL="https://otel.example.com" \
  ./scripts/upload-sourcemap.sh otel-ui/dist/assets/index-ABC123.js

The receiver stores uploaded maps under OTEL_SOURCEMAP_DIR (default ./sourcemaps, /sourcemaps in the Kubernetes deployment). Maps are reloaded on start, so keep the directory on persistent storage. See instrumentation.md (Source maps → Requirements checklist) for everything a project needs for stacks to resolve. GET /api/sourcemaps lists loaded maps. The API never returns map files themselves.

build.sh passes the release into the Docker UI build. If OTEL_URL is set, it also builds the matching UI locally and uploads all generated JavaScript maps. The Dockerfile removes .map files from the final image by default; set PUBLIC_SOURCEMAPS=true for a development or staging image where Chrome DevTools should fetch the original sources:

PUBLIC_SOURCEMAPS=true ./build.sh

Public maps can expose application source code, so keep this disabled for production unless that exposure is intentional. Credentials are never passed to Docker or stored in the image.

Configuration

• OTELLOG_PATH - path to the persistent telemetry log file (default otel-agent.log). Only used when ClickHouse is not configured.
• OTEL_SOURCEMAP_KEY - shared key required to upload source maps.
• OTEL_ISSUE_STATUS_PATH - file that keeps resolved/ignored issue decisions across restarts when ClickHouse is not used (default: next to OTELLOG_PATH).
• OTEL_TODOS_PATH - file for todos filed via otel-todo.js when ClickHouse is not used (default: next to OTELLOG_PATH).
• OTEL_SOURCEMAP_CACHE_MB - size budget (on-disk MB, default 64) for maps kept parsed in memory; maps are indexed at start and parsed lazily on first use.
• OTEL_SOURCEMAP_DIR - directory for uploaded release source maps (default ./sourcemaps; mount persistent storage there in production).
• OTEL_IP_INVESTIGATION_NMAP - set to true to enable the authenticated, rate-limited active service probe. It is off by default because Nmap sends active network traffic to the investigated public IP.
• ABUSE_KEY - optional AbuseIPDB API key. When set, the authenticated IP detail view can check a public address and submit a report without exposing the key to the browser. Keep it in a secret, never in the frontend build.
• OTEL_PPROF_ADDR - address for the Go pprof server (default 127.0.0.1:6060). It serves /debug/pprof/ separately from the public UI port; use a port-forward or set this explicitly while diagnosing memory or CPU usage.
• OTEL_KUBERNETES_ENABLED - set to false to disable rollout observation; otherwise it activates in-cluster or KUBECONFIG observation when available.
• OTEL_KUBERNETES_NAMESPACE - optional namespace restriction; empty watches all namespaces permitted by the service account.
• OTEL_KUBERNETES_LABEL_SELECTOR - optional Deployment label selector.
• OTEL_PROMETHEUS_URL - optional Prometheus base URL for rollout impact.
• OTEL_PROMETHEUS_ERROR_QUERY - optional fmt.Sprintf-style PromQL template; %q placeholders receive service and release in that order.
• OTEL_CLICKHOUSE_ADDR - optional host:port (native protocol, usually 9000). When set, telemetry is stored in ClickHouse instead of the JSONL file: history search stays fast and unbounded, and the aggregate stats endpoints (/api/stats/*) and the UI "Trends" tab are enabled. The schema (otel database, raw entries table + materialized hourly rollup) is created automatically on first connect with a 90-day TTL; if ClickHouse is unreachable at startup the server falls back to the file store.
• OTEL_CLICKHOUSE_DB / OTEL_CLICKHOUSE_USER / OTEL_CLICKHOUSE_PASSWORD - optional ClickHouse connection settings (defaults: otel / default / empty).
• OTEL_INGEST_KEY - optional shared key required (via ?key= or the X-Otel-Ingest-Key header) on /v1/traces, /v1/metrics and /v1/logs over HTTP. Unset by default (no key required). This is meant to scope/rate-limit a public ingest endpoint fed by a browser script, not to provide real authentication - pair it with the built-in per-IP rate limiting on those endpoints if the endpoint is public.
• OTEL_AGENT_TOKEN - optional service token for non-interactive debugging agents. Send it as X-Otel-Agent-Token or Authorization: Bearer ... when calling protected /api/history, /api/issues, /api/traces, and stats endpoints. Keep it in a Kubernetes Secret and never expose it to browsers.

A single-node ClickHouse instance for Kubernetes is provided in k8s/clickhouse.yaml (StatefulSet + Service + PVC, deployed in the monitoring namespace); the receiver Deployment in k8s/deployment.yaml already points at it. The UI's Trends tab (and live issue updates) appear automatically once the stats endpoints answer; on file-store deployments the tab hides itself. SPA routes (/issues, /traces, /logs, /metrics, /trends, /setup) are served client-side by wouter and survive the OIDC login redirect. Selection lives in the URL too, so every step can be reloaded, stepped with back/forward, and shared (each detail view has a link chip that copies its permalink):

Path	Shows
/issues/<issueId>	one issue with its traces
/issues/<issueId>/trace/<traceId>	the same, with a trace selected
/patterns/<patternId>[/<issueId>]	a pattern card open, optionally reviewing one member
/traces/<traceId>[/<spanId>]	a trace, optionally with a span selected
/rum/session/<sessionId>	a RUM session
/rum/errors[/<errorId>]	JS errors, optionally one error expanded (errorId is a hash of type + message)

Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.

Development

To modify the application:

1. Edit source files in cmd/server/ and internal/
2. Run locally with go run ./cmd/server
3. For UI changes, edit index.html directly

The application uses standard Go modules and OpenTelemetry collector libraries for data processing.