go-otel/README.md

# Go OTel Debugger

A debug receiver for OpenTelemetry telemetry data (traces, metrics, and logs) that provides real-time visualization and historical data access. It accepts OTLP data via HTTP and gRPC protocols, stores a configurable history of received telemetry, and broadcasts updates to connected WebSocket clients for live debugging.

## Features

- **OTLP Support**: Receives traces, metrics, and logs via both HTTP (JSON/Protobuf) and gRPC (Protobuf) protocols on standard OTLP ports.
- **Real-time Broadcasting**: Uses WebSocket connections to broadcast incoming telemetry data to all connected clients instantly.
- **Historical Data**: Maintains a configurable ring buffer of recent telemetry data, accessible via REST API endpoints.
- **Web Interface**: Includes a built-in HTML interface for visualizing telemetry data in real-time with counters and clear buttons.
- **Docker Support**: Containerized for easy deployment.
- **Kubernetes Ready**: Includes deployment manifests for Kubernetes clusters.

## Architecture

The application consists of several key components:

1. **HTTP/gRPC Servers**: Handle incoming OTLP requests on ports 8080 (HTTP) and 4317 (gRPC).
2. **Collector**: Processes and stores telemetry data in a ring buffer for history. Implements OpenTelemetry consumer interfaces.
3. **WebSocket Server**: Manages client connections and broadcasts new telemetry data as JSON messages.
4. **Web Interface**: Static HTML page served on the root path that connects to the WebSocket for real-time updates.
5. **History API**: REST endpoints to retrieve stored telemetry data by type or all combined.

Data flow:
```
OTLP Client → HTTP/gRPC Server → Collector → WebSocket Broadcast → Connected Clients
                                      ↓
                                 History Storage → REST API → Clients
```

## Getting Started

### Prerequisites

- Go 1.25+ (for local development)
- Docker (optional, for containerized deployment)

### Running Locally

1. Clone the repository:
   ```sh
   git clone https://git.k6n.net/go-otel.git
   cd go-otel
   ```

2. Run the server:
   ```sh
   go run ./cmd/server
   ```

The server will start on ports 8080 (HTTP/WebSocket) and 4317 (gRPC).

### Running with Docker

1. Build the Docker image:
   ```sh
   docker build -t go-otel .
   ```

2. Run the container:
   ```sh
   docker run -p 8080:8080 -p 4317:4317 go-otel
   ```

### Kubernetes Deployment

Apply the included Kubernetes manifests:
```sh
kubectl apply -f k8s/
```

This deploys the application with a Service and Ingress for external access.

## API Endpoints

### OTLP HTTP Endpoints (Port 8080)

- `POST /v1/traces` - Receive OTLP traces
- `POST /v1/metrics` - Receive OTLP metrics  
- `POST /v1/logs` - Receive OTLP logs

Supported Content-Types: `application/json`, `application/x-protobuf`

### OTLP gRPC Endpoints (Port 4317)

gRPC services for traces, metrics, and logs using standard OTLP protobuf definitions.

### WebSocket Endpoint (Port 8080)

- `GET /ws` - WebSocket connection for real-time telemetry streaming

### History API Endpoints (Port 8080)

- `GET /api/history` - Retrieve all historical telemetry data
- `GET /api/history/traces` - Retrieve only trace history
- `GET /api/history/metrics` - Retrieve only metrics history
- `GET /api/history/logs` - Retrieve only logs history
- `GET /api/history/search` - Search history. Supports `q` (free-text
  substring), `type`, `since`/`until` (unix nanoseconds), and `limit`.
- `GET /api/history/wait` - Block until a matching entry arrives. Same
  filters as `search`, plus `timeout` (seconds, default 30). Rate-limited
  per client IP.
- `GET /api/traces/{traceId}` - Retrieve all stored entries (spans and
  correlated logs) referencing a given trace id.

All endpoints above search the full persistent store (when `OTELLOG_PATH`
storage is enabled), not just the in-memory ring buffer.

### Issues API Endpoints (Port 8080)

Errors seen in traces (spans with an error status) and logs (severity
`ERROR`/`FATAL`) are deduplicated into "issues" as they're ingested.

- `GET /api/issues` - List deduplicated issues, most recently seen first.
  Supports `limit`.
- `GET /api/issues/{id}` - Fetch a single issue, including up to 5 recent
  sample occurrences (timestamp, trace/span id, message).

### Web Interface

- `GET /` - Serves the HTML visualization interface

### Static Routes

- `GET /static/otel-rum.js` - The frontend error collector script (see
  below).
- `GET /agents_example.md` - Serves this repo's agent instructions doc
  directly, so an AI agent can fetch it from the base URL before making any
  API calls.

## Usage

### Sending Telemetry Data

Send OTLP data to the appropriate endpoints. The data will be stored in history and broadcast to all connected WebSocket clients.

#### HTTP Example (JSON)

```sh
curl -X POST http://localhost:8080/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"scope":{"name":"test-scope"},"spans":[{"traceId":"0102030405060708090a0b0c0d0e0f10","spanId":"0102030405060708","name":"test-span","startTimeUnixNano":"1640995200000000000","endTimeUnixNano":"1640995260000000000","attributes":[{"key":"test.key","value":{"stringValue":"test-value"}}]}]}]}]}'
```

#### gRPC Example

Use any OTLP gRPC client library to send data to `localhost:4317`.

### Real-time Visualization

1. Open `http://localhost:8080` in your browser
2. The interface will automatically connect via WebSocket
3. Send telemetry data to see it appear in real-time
4. Use the "Clear" buttons to reset individual data sections
5. View message counters for each telemetry type

### Accessing Historical Data

Use the history API endpoints to retrieve previously received data:

```sh
curl http://localhost:8080/api/history/traces
```

This returns JSON arrays of historical entries with timestamps.

### Frontend Error Collector (`static/otel-rum.js`)

A framework-free browser script that captures uncaught errors, unhandled
promise rejections and failed resource loads (img/script/link), and ships
them as OTLP log records to the existing `/v1/logs` endpoint - so they flow
through the same history/search/issue pipeline as backend telemetry, tagged
`source: frontend`.

```html
<script src="https://otel.k6n.net/static/otel-rum.js"></script>
<script>
  var rum = OtelRUM.init({ serviceName: "my-web-app" });

  // Inject a W3C traceparent header into any HTTP client so a backend span
  // (if the service is OTel-instrumented) shares this request's trace id -
  // then GET /api/traces/{traceId} shows the frontend error and backend
  // spans for the same request together.
  fetch("/api/orders", { headers: rum.traceHeaders() });

  // Manual error reporting.
  try { risky(); } catch (err) { rum.captureException(err); }
</script>
```

`fetch` is auto-instrumented by default (same-origin requests only) to
inject `traceparent` and record request/response breadcrumbs; this can be
disabled via `injectTraceHeaders: false` / `autoInstrumentFetch: false`.
Out of scope for v1: source map resolution (stack traces stay minified),
session replay, PII scrubbing - don't route sensitive data through
breadcrumbs or attributes.

## Configuration

- `OTELLOG_PATH` - path to the persistent telemetry log file (default
  `otel-agent.log`).
- `OTEL_INGEST_KEY` - optional shared key required (via `?key=` or the
  `X-Otel-Ingest-Key` header) on `/v1/traces`, `/v1/metrics` and `/v1/logs`
  over HTTP. Unset by default (no key required). This is meant to
  scope/rate-limit a public ingest endpoint fed by a browser script, not to
  provide real authentication - pair it with the built-in per-IP rate
  limiting on those endpoints if the endpoint is public.

Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.

## Development

To modify the application:

1. Edit source files in `cmd/server/` and `internal/`
2. Run locally with `go run ./cmd/server`
3. For UI changes, edit `index.html` directly

The application uses standard Go modules and OpenTelemetry collector libraries for data processing.