445 lines
20 KiB
Markdown
445 lines
20 KiB
Markdown
# Go OTel Debugger
|
|
|
|
A debug receiver for OpenTelemetry telemetry data (traces, metrics, and logs) that provides real-time visualization and historical data access. It accepts OTLP data via HTTP and gRPC protocols, stores a configurable history of received telemetry, and broadcasts updates to connected WebSocket clients for live debugging.
|
|
|
|
## Features
|
|
|
|
- **OTLP Support**: Receives traces, metrics, and logs via both HTTP (JSON/Protobuf) and gRPC (Protobuf) protocols on standard OTLP ports.
|
|
- **Real-time Broadcasting**: Uses WebSocket connections to broadcast incoming telemetry data to all connected clients instantly.
|
|
- **Historical Data**: Maintains a configurable ring buffer of recent telemetry data, accessible via REST API endpoints.
|
|
- **Web Interface**: Includes a built-in HTML interface for visualizing telemetry data in real-time with counters and clear buttons.
|
|
- **Docker Support**: Containerized for easy deployment.
|
|
- **Kubernetes Ready**: Includes deployment manifests for Kubernetes clusters.
|
|
|
|
## Architecture
|
|
|
|
The application consists of several key components:
|
|
|
|
1. **HTTP/gRPC Servers**: Handle incoming OTLP requests on ports 8080 (HTTP) and 4317 (gRPC).
|
|
2. **Collector**: Processes and stores telemetry data in a ring buffer for history. Implements OpenTelemetry consumer interfaces.
|
|
3. **WebSocket Server**: Manages client connections and broadcasts new telemetry data as JSON messages.
|
|
4. **Web Interface**: Static HTML page served on the root path that connects to the WebSocket for real-time updates.
|
|
5. **History API**: REST endpoints to retrieve stored telemetry data by type or all combined.
|
|
|
|
Data flow:
|
|
```
|
|
OTLP Client → HTTP/gRPC Server → Collector → WebSocket Broadcast → Connected Clients
|
|
↓
|
|
History Storage → REST API → Clients
|
|
```
|
|
|
|
## Getting Started
|
|
|
|
### Prerequisites
|
|
|
|
- Go 1.25+ (for local development)
|
|
- Docker (optional, for containerized deployment)
|
|
|
|
### Running Locally
|
|
|
|
1. Clone the repository:
|
|
```sh
|
|
git clone https://git.k6n.net/go-otel.git
|
|
cd go-otel
|
|
```
|
|
|
|
2. Run the server:
|
|
```sh
|
|
go run ./cmd/server
|
|
```
|
|
|
|
The server will start on ports 8080 (HTTP/WebSocket) and 4317 (gRPC).
|
|
|
|
### Running with Docker
|
|
|
|
1. Build the Docker image:
|
|
```sh
|
|
docker build -t go-otel .
|
|
```
|
|
|
|
2. Run the container:
|
|
```sh
|
|
docker run -p 8080:8080 -p 4317:4317 go-otel
|
|
```
|
|
|
|
### Kubernetes Deployment
|
|
|
|
Apply the included Kubernetes manifests:
|
|
```sh
|
|
kubectl apply -f k8s/
|
|
```
|
|
|
|
This deploys the application with a Service and Ingress for external access.
|
|
|
|
## API Endpoints
|
|
|
|
### OTLP HTTP Endpoints (Port 8080)
|
|
|
|
- `POST /v1/traces` - Receive OTLP traces
|
|
- `POST /v1/metrics` - Receive OTLP metrics
|
|
- `POST /v1/logs` - Receive OTLP logs
|
|
|
|
Supported Content-Types: `application/json`, `application/x-protobuf`
|
|
|
|
### OTLP gRPC Endpoints (Port 4317)
|
|
|
|
gRPC services for traces, metrics, and logs using standard OTLP protobuf definitions.
|
|
|
|
### WebSocket Endpoint (Port 8080)
|
|
|
|
- `GET /ws` - WebSocket connection for real-time telemetry streaming
|
|
|
|
### History API Endpoints (Port 8080)
|
|
|
|
- `GET /api/history` - Retrieve all historical telemetry data
|
|
- `GET /api/history/traces` - Retrieve only trace history
|
|
- `GET /api/history/metrics` - Retrieve only metrics history
|
|
- `GET /api/history/logs` - Retrieve only logs history
|
|
- `GET /api/history/search` - Search history. Supports `q` (free-text
|
|
substring), `type`, `since`/`until` (unix nanoseconds), and `limit`.
|
|
- `GET /api/history/wait` - Block until a matching entry arrives. Same
|
|
filters as `search`, plus `timeout` (seconds, default 30). Rate-limited
|
|
per client IP.
|
|
- `GET /api/traces/{traceId}` - Retrieve all stored entries (spans and
|
|
correlated logs) referencing a given trace id. With ClickHouse this is an
|
|
indexed lookup (`entries.trace_ids`) and each payload is trimmed to that
|
|
trace's own spans/log records.
|
|
- `GET /api/traces/summaries?ids=a,b,c` - Per-trace rollups (name, start/end,
|
|
duration, span count, error count, services; max 100 ids) without reading
|
|
span payloads. Maintained on insert by the `trace_summaries` view; empty
|
|
with the file store.
|
|
|
|
All endpoints above search the full persistent store, not just the
|
|
in-memory ring buffer.
|
|
|
|
### Stats API Endpoints (Port 8080, requires ClickHouse)
|
|
|
|
Aggregate time-series over the stored telemetry, computed server-side from
|
|
an hourly rollup table. Available only when `OTEL_CLICKHOUSE_ADDR` is
|
|
configured (see Configuration below); otherwise they respond `501` with an
|
|
explanatory JSON body and the UI hides its "Trends" tab.
|
|
|
|
- `GET /api/stats/errors?hours=72&service=` - Total/error entry counts per
|
|
hour (per day when `hours > 336`). Response:
|
|
`{"granularity":"hour","points":[{"bucket":"2026-09-17 14:00:00","total":42,"errors":3}]}`.
|
|
- `GET /api/stats/releases?hours=72&service=` - Per-release error-count
|
|
series: `{"series":{"v1.3":[{bucket,total,errors}...],"v1.4":[...]}}`
|
|
for release-over-release comparison. Entries need a `service.release`
|
|
attribute to appear here.
|
|
- `GET /api/stats/services` - Distinct services seen in the rollup.
|
|
- `GET /api/stats/endpoints?hours=72&host=&service=&limit=100` - HTTP span
|
|
traffic grouped by normalized service, host, method, path, route, status,
|
|
user-agent, and safe browser request metadata. `spanCount` exposes nested
|
|
proxy/instrumentation duplication while `traceCount` approximates logical
|
|
requests. Origin and referrer are returned as host-only values; query
|
|
strings are stripped from paths.
|
|
|
|
All accept `hours` (1..2160, default 72) and an optional `service` filter.
|
|
|
|
### Issues API Endpoints (Port 8080)
|
|
|
|
Errors seen in traces (spans with an error status) and logs (severity
|
|
`ERROR`/`FATAL`) are deduplicated into "issues" as they're ingested.
|
|
|
|
- `GET /api/issues` - List deduplicated issues, most recently seen first.
|
|
Supports `limit`.
|
|
- `GET /api/issues/{id}` - Fetch a single issue, including up to 5 recent
|
|
sample occurrences (timestamp, trace/span id, message).
|
|
- `GET /api/issues/{id}/sources` - Return every distinct source IP recorded
|
|
for the issue, with occurrence count and first/last seen timestamps. Requires
|
|
the ClickHouse occurrence store; returns `501` with file/in-memory storage.
|
|
- `GET /api/issues/{id}/traces` - Every trace the issue occurred in, most
|
|
recently hit first, with each trace's summary and its occurrences, in one
|
|
request (`limit`, `before` = `nextBefore` from the previous page).
|
|
ClickHouse keeps *all* occurrences (`issue_occurrences`, 90-day TTL);
|
|
without it the response is built from the 5 in-memory samples and
|
|
`complete` is `false`.
|
|
- `GET /api/patterns` - List deterministic groups of related exact issues.
|
|
Pattern grouping is explainable and does not merge issue IDs.
|
|
- `GET /api/issues/{id}/similar` - Return bounded, deterministic related-issue
|
|
suggestions with a score and matching reasons.
|
|
- `POST /api/issues/{id}/similar` - Record a review label with
|
|
`{"issueId":"...","decision":"same|different"}` for future training data.
|
|
- `GET /api/rollouts` - List read-only Kubernetes Deployment rollout state.
|
|
- `GET /api/rollouts/{id}` - Fetch one normalized rollout.
|
|
- `GET /api/rollouts/{id}/impact` - Return optional Prometheus error-rate points
|
|
around the rollout.
|
|
- `POST /api/sourcemaps` - Upload a JavaScript source map. Requires the shared
|
|
`OTEL_SOURCEMAP_KEY` in an `X-Sourcemap-Key` header (uploads are refused while
|
|
it is unset). Multipart upload;
|
|
multipart fields are `release`, `source` (the generated bundle URL/path),
|
|
and `file`. Uploaded maps are used server-side to resolve frontend stacks.
|
|
|
|
### Web Interface
|
|
|
|
- `GET /` - Serves the HTML visualization interface
|
|
|
|
### Static Routes
|
|
|
|
- `GET /static/otel-rum.js` - The frontend error collector script (see
|
|
below).
|
|
- `GET /agents_example.md` - Serves this repo's agent instructions doc
|
|
directly, so an AI agent can fetch it from the base URL before making any
|
|
API calls.
|
|
- `GET /instrumentation.md` - Serves the frontend/backend instrumentation
|
|
guide, including React error boundaries and source-map deployment.
|
|
- `GET /static/instrumentation.md` - Static/public alias for the same guide.
|
|
|
|
## Usage
|
|
|
|
### Sending Telemetry Data
|
|
|
|
Send OTLP data to the appropriate endpoints. The data will be stored in history and broadcast to all connected WebSocket clients.
|
|
|
|
#### HTTP Example (JSON)
|
|
|
|
```sh
|
|
curl -X POST http://localhost:8080/v1/traces \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"scope":{"name":"test-scope"},"spans":[{"traceId":"0102030405060708090a0b0c0d0e0f10","spanId":"0102030405060708","name":"test-span","startTimeUnixNano":"1640995200000000000","endTimeUnixNano":"1640995260000000000","attributes":[{"key":"test.key","value":{"stringValue":"test-value"}}]}]}]}]}'
|
|
```
|
|
|
|
#### gRPC Example
|
|
|
|
Use any OTLP gRPC client library to send data to `localhost:4317`.
|
|
|
|
### Real-time Visualization
|
|
|
|
1. Open `http://localhost:8080` in your browser
|
|
2. The interface will automatically connect via WebSocket
|
|
3. Send telemetry data to see it appear in real-time
|
|
4. Use the "Clear" buttons to reset individual data sections
|
|
5. View message counters for each telemetry type
|
|
|
|
### Accessing Historical Data
|
|
|
|
Use the history API endpoints to retrieve previously received data:
|
|
|
|
```sh
|
|
curl http://localhost:8080/api/history/traces
|
|
```
|
|
|
|
This returns JSON arrays of historical entries with timestamps.
|
|
|
|
### Frontend Error Collector (`static/otel-rum.js`)
|
|
|
|
A framework-free browser script that captures uncaught errors, unhandled
|
|
promise rejections and failed resource loads (img/script/link), and ships
|
|
them as OTLP log records to the existing `/v1/logs` endpoint - so they flow
|
|
through the same history/search/issue pipeline as backend telemetry, tagged
|
|
`source: frontend`.
|
|
|
|
```html
|
|
<script src="https://otel.k6n.net/static/otel-rum.js"></script>
|
|
<script>
|
|
var rum = OtelRUM.init({
|
|
serviceName: "my-web-app",
|
|
release: "2026.09.18-abc123",
|
|
// Optional; defaults to https://otel.k6n.net.
|
|
baseUrl: "https://otel.example.com",
|
|
});
|
|
|
|
// Inject a W3C traceparent header into any HTTP client so a backend span
|
|
// (if the service is OTel-instrumented) shares this request's trace id -
|
|
// then GET /api/traces/{traceId} shows the frontend error and backend
|
|
// spans for the same request together.
|
|
fetch("/api/orders", { headers: rum.traceHeaders() });
|
|
|
|
// Manual error reporting.
|
|
try { risky(); } catch (err) { rum.captureException(err); }
|
|
</script>
|
|
```
|
|
|
|
React render failures should be reported from an error boundary as well. The
|
|
project UI uses this pattern in `otel-ui/src/components/ViewErrorBoundary.tsx`:
|
|
|
|
```tsx
|
|
class AppBoundary extends React.Component<{ children: React.ReactNode }> {
|
|
componentDidCatch(error: Error, info: React.ErrorInfo) {
|
|
OtelRUM.last?.captureException(error, {
|
|
"exception.type": "ReactErrorBoundary",
|
|
"react.component_stack": info.componentStack,
|
|
});
|
|
}
|
|
|
|
render() { return this.props.children; }
|
|
}
|
|
```
|
|
|
|
The boundary should also render a fallback UI; otherwise a broken component
|
|
can leave the whole application blank while still being logged.
|
|
|
|
`fetch` is auto-instrumented by default (same-origin requests only) to
|
|
inject `traceparent` and record request/response breadcrumbs; this can be
|
|
disabled via `injectTraceHeaders: false` / `autoInstrumentFetch: false`.
|
|
The log ingest endpoint supports unauthenticated cross-origin POSTs; no OIDC
|
|
session or `Authorization` header is needed.
|
|
Source maps are kept server-side. When a captured frontend error has a
|
|
matching `service.release` and generated bundle path, the Issues view displays
|
|
the resolved original source frame while retaining the generated frame.
|
|
|
|
#### Source-map deploy workflow
|
|
|
|
Build the UI with a stable release identifier and keep the generated map next
|
|
to the bundle. Do not publish the map publicly:
|
|
|
|
```sh
|
|
export RELEASE="2026.09.18-$(git rev-parse --short HEAD)"
|
|
npm --prefix otel-ui run build
|
|
|
|
# Upload after the new UI is deployed. Use the generated URL/path exactly as
|
|
# it appears in browser stack traces.
|
|
curl -fS -X POST https://otel.example.com/api/sourcemaps \
|
|
-H "X-Sourcemap-Key: $OTEL_SOURCEMAP_KEY" \
|
|
-F "release=$RELEASE" \
|
|
-F "source=/assets/index-ABC123.js" \
|
|
-F "file=@otel-ui/dist/assets/index-ABC123.js.map"
|
|
```
|
|
|
|
For this project, `otel-ui/src/main.tsx` sends `VITE_APP_RELEASE` as the RUM
|
|
`service.release` value; set it during the build, for example:
|
|
|
|
```sh
|
|
VITE_APP_RELEASE="$RELEASE" npm --prefix otel-ui run build
|
|
```
|
|
|
|
The repository also includes `scripts/upload-sourcemap.sh`; after deploying,
|
|
run it against the emitted bundle, for example:
|
|
|
|
```sh
|
|
RELEASE="$RELEASE" OTEL_URL="https://otel.example.com" \
|
|
./scripts/upload-sourcemap.sh otel-ui/dist/assets/index-ABC123.js
|
|
```
|
|
|
|
The receiver stores uploaded maps under `OTEL_SOURCEMAP_DIR` (default
|
|
`./sourcemaps`, `/sourcemaps` in the Kubernetes deployment). Maps are reloaded on start, so keep the directory on persistent storage. See
|
|
`instrumentation.md` (*Source maps → Requirements checklist*) for everything a
|
|
project needs for stacks to resolve. `GET /api/sourcemaps` lists loaded maps.
|
|
The API never returns map files themselves.
|
|
|
|
`build.sh` passes the release into the Docker UI build. If `OTEL_URL` is set,
|
|
it also builds the matching UI locally and uploads all generated JavaScript
|
|
maps. The Dockerfile removes `.map` files from the
|
|
final image by default; set `PUBLIC_SOURCEMAPS=true` for a development or
|
|
staging image where Chrome DevTools should fetch the original sources:
|
|
|
|
```sh
|
|
PUBLIC_SOURCEMAPS=true ./build.sh
|
|
```
|
|
|
|
Public maps can expose application source code, so keep this disabled for
|
|
production unless that exposure is intentional. Credentials are never passed
|
|
to Docker or stored in the image.
|
|
|
|
## Configuration
|
|
|
|
- `OTELLOG_PATH` - path to the persistent telemetry log file (default
|
|
`otel-agent.log`). Only used when ClickHouse is not configured.
|
|
- `OTEL_SOURCEMAP_KEY` - shared key required to upload source maps.
|
|
- `OTEL_ISSUE_STATUS_PATH` - file that keeps resolved/ignored issue decisions across restarts when ClickHouse is not used (default: next to `OTELLOG_PATH`).
|
|
- `OTEL_TODOS_PATH` - file for todos filed via `otel-todo.js` when ClickHouse is not used (default: next to `OTELLOG_PATH`).
|
|
- `OTEL_SOURCEMAP_CACHE_MB` - size budget (on-disk MB, default 64) for maps kept
|
|
parsed in memory; maps are indexed at start and parsed lazily on first use.
|
|
- `OTEL_SOURCEMAP_DIR` - directory for uploaded release source maps (default
|
|
`./sourcemaps`; mount persistent storage there in production).
|
|
- `OTEL_IP_INVESTIGATION_NMAP` - set to `true` to enable the authenticated,
|
|
rate-limited active service probe. It is off by default because Nmap sends
|
|
active network traffic to the investigated public IP.
|
|
- `ABUSE_KEY` - optional AbuseIPDB API key. When set, the authenticated IP
|
|
detail view can check a public address and submit a report without exposing
|
|
the key to the browser. Keep it in a secret, never in the frontend build.
|
|
- `OTEL_PPROF_ADDR` - address for the Go pprof server (default
|
|
`127.0.0.1:6060`). It serves `/debug/pprof/` separately from the public UI
|
|
port; use a port-forward or set this explicitly while diagnosing memory or
|
|
CPU usage.
|
|
- `OTEL_KUBERNETES_ENABLED` - set to `false` to disable rollout observation;
|
|
otherwise it activates in-cluster or `KUBECONFIG` observation when available.
|
|
- `OTEL_KUBERNETES_NAMESPACE` - optional namespace restriction; empty watches
|
|
all namespaces permitted by the service account.
|
|
- `OTEL_KUBERNETES_LABEL_SELECTOR` - optional Deployment label selector.
|
|
- `OTEL_PROMETHEUS_URL` - optional Prometheus base URL for rollout impact.
|
|
- `OTEL_PROMETHEUS_ERROR_QUERY` - optional `fmt.Sprintf`-style PromQL template;
|
|
`%q` placeholders receive service and release in that order.
|
|
- `OTEL_CLICKHOUSE_ADDR` - optional `host:port` (native protocol, usually
|
|
`9000`). When set, telemetry is stored in ClickHouse instead of the JSONL
|
|
file: history search stays fast and unbounded, and the aggregate stats
|
|
endpoints (`/api/stats/*`) and the UI "Trends" tab are enabled. The
|
|
schema (`otel` database, raw `entries` table + materialized hourly
|
|
rollup) is created automatically on first connect with a 90-day TTL; if
|
|
ClickHouse is unreachable at startup the server falls back to the file
|
|
store.
|
|
- `OTEL_CLICKHOUSE_DB` / `OTEL_CLICKHOUSE_USER` /
|
|
`OTEL_CLICKHOUSE_PASSWORD` - optional ClickHouse connection settings
|
|
(defaults: `otel` / `default` / empty).
|
|
|
|
### Issues display
|
|
|
|
The standalone `issues-tui` command is a terminal dashboard for currently
|
|
firing alert spikes. It reads normalized error events directly from
|
|
ClickHouse; it does not replace the web UI's persisted `/api/issues` view.
|
|
When a spike stops firing, its card stays on the display marked resolved
|
|
until something fires again.
|
|
|
|
Build and run it on the display computer (or use `go run`):
|
|
|
|
```sh
|
|
go build -o issues-tui ./cmd/issues-tui
|
|
OTEL_CLICKHOUSE_ADDR=clickhouse.example.net:9000 \
|
|
OTEL_CLICKHOUSE_DB=otel \
|
|
OTEL_CLICKHOUSE_USER=default \
|
|
OTEL_CLICKHOUSE_PASSWORD='…' \
|
|
OTEL_ALERT_BASE_URL=https://otel.example.net \
|
|
./issues-tui
|
|
```
|
|
|
|
The display computer must be able to reach ClickHouse's native TCP port. For a
|
|
Kubernetes-only ClickHouse service, use a private network path or tunnel; the
|
|
in-cluster service name is not reachable from an ordinary workstation. The
|
|
dashboard refreshes using the `OTEL_ALERT_*` interval, window, baseline,
|
|
cooldown and lag settings. Press `r` to scan immediately and `q` to quit.
|
|
Use `-once` for a one-shot, non-interactive output suitable for a process
|
|
supervisor health check.
|
|
|
|
- `OTEL_INGEST_KEY` - optional shared key required (via `?key=` or the
|
|
`X-Otel-Ingest-Key` header) on `/v1/traces`, `/v1/metrics` and `/v1/logs`
|
|
over HTTP. Unset by default (no key required). This is meant to
|
|
scope/rate-limit a public ingest endpoint fed by a browser script, not to
|
|
provide real authentication - pair it with the built-in per-IP rate
|
|
limiting on those endpoints if the endpoint is public.
|
|
- `OTEL_AGENT_TOKEN` - optional service token for non-interactive debugging
|
|
agents. Send it as `X-Otel-Agent-Token` or `Authorization: Bearer ...` when
|
|
calling protected `/api/history`, `/api/issues`, `/api/traces`, and stats
|
|
endpoints. Keep it in a Kubernetes Secret and never expose it to browsers.
|
|
|
|
A single-node ClickHouse instance for Kubernetes is provided in
|
|
`k8s/clickhouse.yaml` (StatefulSet + Service + PVC, deployed in the
|
|
`monitoring` namespace); the receiver Deployment in `k8s/deployment.yaml`
|
|
already points at it. The UI's Trends tab (and live issue updates) appear
|
|
automatically once the stats endpoints answer; on file-store deployments the
|
|
tab hides itself. SPA routes (/issues, /traces, /logs, /metrics, /trends,
|
|
/setup) are served client-side by wouter and survive the OIDC login redirect.
|
|
Selection lives in the URL too, so every step can be reloaded, stepped with
|
|
back/forward, and shared (each detail view has a link chip that copies its
|
|
permalink):
|
|
|
|
| Path | Shows |
|
|
| --- | --- |
|
|
| `/issues/<issueId>` | one issue with its traces |
|
|
| `/issues/<issueId>/trace/<traceId>` | the same, with a trace selected |
|
|
| `/patterns/<patternId>[/<issueId>]` | a pattern card open, optionally reviewing one member |
|
|
| `/traces/<traceId>[/<spanId>]` | a trace, optionally with a span selected |
|
|
| `/rum/session/<sessionId>` | a RUM session |
|
|
| `/rum/errors[/<errorId>]` | JS errors, optionally one error expanded (`errorId` is a hash of type + message) |
|
|
|
|
Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.
|
|
|
|
## Development
|
|
|
|
To modify the application:
|
|
|
|
1. Edit source files in `cmd/server/` and `internal/`
|
|
2. Run locally with `go run ./cmd/server`
|
|
3. For UI changes, edit `index.html` directly
|
|
|
|
The application uses standard Go modules and OpenTelemetry collector libraries for data processing.
|