go-otel/README.md

Go OTel Debugger

A debug receiver for OpenTelemetry telemetry data (traces, metrics, and logs) that provides real-time visualization and historical data access. It accepts OTLP data via HTTP and gRPC protocols, stores a configurable history of received telemetry, and broadcasts updates to connected WebSocket clients for live debugging.

Features

• OTLP Support: Receives traces, metrics, and logs via both HTTP (JSON/Protobuf) and gRPC (Protobuf) protocols on standard OTLP ports.
• Real-time Broadcasting: Uses WebSocket connections to broadcast incoming telemetry data to all connected clients instantly.
• Historical Data: Maintains a configurable ring buffer of recent telemetry data, accessible via REST API endpoints.
• Web Interface: Includes a built-in HTML interface for visualizing telemetry data in real-time with counters and clear buttons.
• Docker Support: Containerized for easy deployment.
• Kubernetes Ready: Includes deployment manifests for Kubernetes clusters.

Architecture

The application consists of several key components:

1. HTTP/gRPC Servers: Handle incoming OTLP requests on ports 8080 (HTTP) and 4317 (gRPC).
2. Collector: Processes and stores telemetry data in a ring buffer for history. Implements OpenTelemetry consumer interfaces.
3. WebSocket Server: Manages client connections and broadcasts new telemetry data as JSON messages.
4. Web Interface: Static HTML page served on the root path that connects to the WebSocket for real-time updates.
5. History API: REST endpoints to retrieve stored telemetry data by type or all combined.

Data flow:

OTLP Client → HTTP/gRPC Server → Collector → WebSocket Broadcast → Connected Clients
                                      ↓
                                 History Storage → REST API → Clients

Getting Started

Prerequisites

• Go 1.25+ (for local development)
• Docker (optional, for containerized deployment)

Running Locally

1. Clone the repository:

   git clone https://git.k6n.net/go-otel.git
   cd go-otel
   

2. Run the server:

   go run ./cmd/server
   

The server will start on ports 8080 (HTTP/WebSocket) and 4317 (gRPC).

Running with Docker

1. Build the Docker image:

   docker build -t go-otel .
   

2. Run the container:

   docker run -p 8080:8080 -p 4317:4317 go-otel
   

Kubernetes Deployment

Apply the included Kubernetes manifests:

kubectl apply -f k8s/

This deploys the application with a Service and Ingress for external access.

API Endpoints

OTLP HTTP Endpoints (Port 8080)

• POST /v1/traces - Receive OTLP traces
• POST /v1/metrics - Receive OTLP metrics
• POST /v1/logs - Receive OTLP logs

Supported Content-Types: application/json, application/x-protobuf

OTLP gRPC Endpoints (Port 4317)

gRPC services for traces, metrics, and logs using standard OTLP protobuf definitions.

WebSocket Endpoint (Port 8080)

• GET /ws - WebSocket connection for real-time telemetry streaming

History API Endpoints (Port 8080)

• GET /api/history - Retrieve all historical telemetry data
• GET /api/history/traces - Retrieve only trace history
• GET /api/history/metrics - Retrieve only metrics history
• GET /api/history/logs - Retrieve only logs history
• GET /api/history/search - Search history. Supports q (free-text substring), type, since/until (unix nanoseconds), and limit.
• GET /api/history/wait - Block until a matching entry arrives. Same filters as search, plus timeout (seconds, default 30). Rate-limited per client IP.
• GET /api/traces/{traceId} - Retrieve all stored entries (spans and correlated logs) referencing a given trace id.

All endpoints above search the full persistent store (when OTELLOG_PATH storage is enabled), not just the in-memory ring buffer.

Issues API Endpoints (Port 8080)

Errors seen in traces (spans with an error status) and logs (severity ERROR/FATAL) are deduplicated into "issues" as they're ingested.

• GET /api/issues - List deduplicated issues, most recently seen first. Supports limit.
• GET /api/issues/{id} - Fetch a single issue, including up to 5 recent sample occurrences (timestamp, trace/span id, message).

Web Interface

• GET / - Serves the HTML visualization interface

Static Routes

• GET /static/otel-rum.js - The frontend error collector script (see below).
• GET /agents_example.md - Serves this repo's agent instructions doc directly, so an AI agent can fetch it from the base URL before making any API calls.

Usage

Sending Telemetry Data

Send OTLP data to the appropriate endpoints. The data will be stored in history and broadcast to all connected WebSocket clients.

HTTP Example (JSON)

curl -X POST http://localhost:8080/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"scope":{"name":"test-scope"},"spans":[{"traceId":"0102030405060708090a0b0c0d0e0f10","spanId":"0102030405060708","name":"test-span","startTimeUnixNano":"1640995200000000000","endTimeUnixNano":"1640995260000000000","attributes":[{"key":"test.key","value":{"stringValue":"test-value"}}]}]}]}]}'

gRPC Example

Use any OTLP gRPC client library to send data to localhost:4317.

Real-time Visualization

1. Open http://localhost:8080 in your browser
2. The interface will automatically connect via WebSocket
3. Send telemetry data to see it appear in real-time
4. Use the "Clear" buttons to reset individual data sections
5. View message counters for each telemetry type

Accessing Historical Data

Use the history API endpoints to retrieve previously received data:

curl http://localhost:8080/api/history/traces

This returns JSON arrays of historical entries with timestamps.

Frontend Error Collector (static/otel-rum.js)

A framework-free browser script that captures uncaught errors, unhandled promise rejections and failed resource loads (img/script/link), and ships them as OTLP log records to the existing /v1/logs endpoint - so they flow through the same history/search/issue pipeline as backend telemetry, tagged source: frontend.

<script src="https://otel.k6n.net/static/otel-rum.js"></script>
<script>
  var rum = OtelRUM.init({ serviceName: "my-web-app" });

  // Inject a W3C traceparent header into any HTTP client so a backend span
  // (if the service is OTel-instrumented) shares this request's trace id -
  // then GET /api/traces/{traceId} shows the frontend error and backend
  // spans for the same request together.
  fetch("/api/orders", { headers: rum.traceHeaders() });

  // Manual error reporting.
  try { risky(); } catch (err) { rum.captureException(err); }
</script>

fetch is auto-instrumented by default (same-origin requests only) to inject traceparent and record request/response breadcrumbs; this can be disabled via injectTraceHeaders: false / autoInstrumentFetch: false. Out of scope for v1: source map resolution (stack traces stay minified), session replay, PII scrubbing - don't route sensitive data through breadcrumbs or attributes.

Configuration

• OTELLOG_PATH - path to the persistent telemetry log file (default otel-agent.log).
• OTEL_INGEST_KEY - optional shared key required (via ?key= or the X-Otel-Ingest-Key header) on /v1/traces, /v1/metrics and /v1/logs over HTTP. Unset by default (no key required). This is meant to scope/rate-limit a public ingest endpoint fed by a browser script, not to provide real authentication - pair it with the built-in per-IP rate limiting on those endpoints if the endpoint is public.

Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.

Development

To modify the application:

1. Edit source files in cmd/server/ and internal/
2. Run locally with go run ./cmd/server
3. For UI changes, edit index.html directly

The application uses standard Go modules and OpenTelemetry collector libraries for data processing.