365 lines
15 KiB
Markdown
365 lines
15 KiB
Markdown
# Go OTel Debugger
|
|
|
|
A debug receiver for OpenTelemetry telemetry data (traces, metrics, and logs) that provides real-time visualization and historical data access. It accepts OTLP data via HTTP and gRPC protocols, stores a configurable history of received telemetry, and broadcasts updates to connected WebSocket clients for live debugging.
|
|
|
|
## Features
|
|
|
|
- **OTLP Support**: Receives traces, metrics, and logs via both HTTP (JSON/Protobuf) and gRPC (Protobuf) protocols on standard OTLP ports.
|
|
- **Real-time Broadcasting**: Uses WebSocket connections to broadcast incoming telemetry data to all connected clients instantly.
|
|
- **Historical Data**: Maintains a configurable ring buffer of recent telemetry data, accessible via REST API endpoints.
|
|
- **Web Interface**: Includes a built-in HTML interface for visualizing telemetry data in real-time with counters and clear buttons.
|
|
- **Docker Support**: Containerized for easy deployment.
|
|
- **Kubernetes Ready**: Includes deployment manifests for Kubernetes clusters.
|
|
|
|
## Architecture
|
|
|
|
The application consists of several key components:
|
|
|
|
1. **HTTP/gRPC Servers**: Handle incoming OTLP requests on ports 8080 (HTTP) and 4317 (gRPC).
|
|
2. **Collector**: Processes and stores telemetry data in a ring buffer for history. Implements OpenTelemetry consumer interfaces.
|
|
3. **WebSocket Server**: Manages client connections and broadcasts new telemetry data as JSON messages.
|
|
4. **Web Interface**: Static HTML page served on the root path that connects to the WebSocket for real-time updates.
|
|
5. **History API**: REST endpoints to retrieve stored telemetry data by type or all combined.
|
|
|
|
Data flow:
|
|
```
|
|
OTLP Client → HTTP/gRPC Server → Collector → WebSocket Broadcast → Connected Clients
|
|
↓
|
|
History Storage → REST API → Clients
|
|
```
|
|
|
|
## Getting Started
|
|
|
|
### Prerequisites
|
|
|
|
- Go 1.25+ (for local development)
|
|
- Docker (optional, for containerized deployment)
|
|
|
|
### Running Locally
|
|
|
|
1. Clone the repository:
|
|
```sh
|
|
git clone https://git.k6n.net/go-otel.git
|
|
cd go-otel
|
|
```
|
|
|
|
2. Run the server:
|
|
```sh
|
|
go run ./cmd/server
|
|
```
|
|
|
|
The server will start on ports 8080 (HTTP/WebSocket) and 4317 (gRPC).
|
|
|
|
### Running with Docker
|
|
|
|
1. Build the Docker image:
|
|
```sh
|
|
docker build -t go-otel .
|
|
```
|
|
|
|
2. Run the container:
|
|
```sh
|
|
docker run -p 8080:8080 -p 4317:4317 go-otel
|
|
```
|
|
|
|
### Kubernetes Deployment
|
|
|
|
Apply the included Kubernetes manifests:
|
|
```sh
|
|
kubectl apply -f k8s/
|
|
```
|
|
|
|
This deploys the application with a Service and Ingress for external access.
|
|
|
|
## API Endpoints
|
|
|
|
### OTLP HTTP Endpoints (Port 8080)
|
|
|
|
- `POST /v1/traces` - Receive OTLP traces
|
|
- `POST /v1/metrics` - Receive OTLP metrics
|
|
- `POST /v1/logs` - Receive OTLP logs
|
|
|
|
Supported Content-Types: `application/json`, `application/x-protobuf`
|
|
|
|
### OTLP gRPC Endpoints (Port 4317)
|
|
|
|
gRPC services for traces, metrics, and logs using standard OTLP protobuf definitions.
|
|
|
|
### WebSocket Endpoint (Port 8080)
|
|
|
|
- `GET /ws` - WebSocket connection for real-time telemetry streaming
|
|
|
|
### History API Endpoints (Port 8080)
|
|
|
|
- `GET /api/history` - Retrieve all historical telemetry data
|
|
- `GET /api/history/traces` - Retrieve only trace history
|
|
- `GET /api/history/metrics` - Retrieve only metrics history
|
|
- `GET /api/history/logs` - Retrieve only logs history
|
|
- `GET /api/history/search` - Search history. Supports `q` (free-text
|
|
substring), `type`, `since`/`until` (unix nanoseconds), and `limit`.
|
|
- `GET /api/history/wait` - Block until a matching entry arrives. Same
|
|
filters as `search`, plus `timeout` (seconds, default 30). Rate-limited
|
|
per client IP.
|
|
- `GET /api/traces/{traceId}` - Retrieve all stored entries (spans and
|
|
correlated logs) referencing a given trace id.
|
|
|
|
All endpoints above search the full persistent store, not just the
|
|
in-memory ring buffer.
|
|
|
|
### Stats API Endpoints (Port 8080, requires ClickHouse)
|
|
|
|
Aggregate time-series over the stored telemetry, computed server-side from
|
|
an hourly rollup table. Available only when `OTEL_CLICKHOUSE_ADDR` is
|
|
configured (see Configuration below); otherwise they respond `501` with an
|
|
explanatory JSON body and the UI hides its "Trends" tab.
|
|
|
|
- `GET /api/stats/errors?hours=72&service=` - Total/error entry counts per
|
|
hour (per day when `hours > 336`). Response:
|
|
`{"granularity":"hour","points":[{"bucket":"2026-09-17 14:00:00","total":42,"errors":3}]}`.
|
|
- `GET /api/stats/releases?hours=72&service=` - Per-release error-count
|
|
series: `{"series":{"v1.3":[{bucket,total,errors}...],"v1.4":[...]}}`
|
|
for release-over-release comparison. Entries need a `service.release`
|
|
attribute to appear here.
|
|
- `GET /api/stats/services` - Distinct services seen in the rollup.
|
|
|
|
All accept `hours` (1..2160, default 72) and an optional `service` filter.
|
|
|
|
### Issues API Endpoints (Port 8080)
|
|
|
|
Errors seen in traces (spans with an error status) and logs (severity
|
|
`ERROR`/`FATAL`) are deduplicated into "issues" as they're ingested.
|
|
|
|
- `GET /api/issues` - List deduplicated issues, most recently seen first.
|
|
Supports `limit`.
|
|
- `GET /api/issues/{id}` - Fetch a single issue, including up to 5 recent
|
|
sample occurrences (timestamp, trace/span id, message).
|
|
- `GET /api/patterns` - List deterministic groups of related exact issues.
|
|
Pattern grouping is explainable and does not merge issue IDs.
|
|
- `GET /api/issues/{id}/similar` - Return bounded, deterministic related-issue
|
|
suggestions with a score and matching reasons.
|
|
- `POST /api/issues/{id}/similar` - Record a review label with
|
|
`{"issueId":"...","decision":"same|different"}` for future training data.
|
|
- `GET /api/rollouts` - List read-only Kubernetes Deployment rollout state.
|
|
- `GET /api/rollouts/{id}` - Fetch one normalized rollout.
|
|
- `GET /api/rollouts/{id}/impact` - Return optional Prometheus error-rate points
|
|
around the rollout.
|
|
- `POST /api/sourcemaps` - Upload a JavaScript source map. Public multipart upload;
|
|
the release build script sends the source-map file contents directly.
|
|
multipart fields are `release`, `source` (the generated bundle URL/path),
|
|
and `file`. Uploaded maps are used server-side to resolve frontend stacks.
|
|
|
|
### Web Interface
|
|
|
|
- `GET /` - Serves the HTML visualization interface
|
|
|
|
### Static Routes
|
|
|
|
- `GET /static/otel-rum.js` - The frontend error collector script (see
|
|
below).
|
|
- `GET /agents_example.md` - Serves this repo's agent instructions doc
|
|
directly, so an AI agent can fetch it from the base URL before making any
|
|
API calls.
|
|
- `GET /instrumentation.md` - Serves the frontend/backend instrumentation
|
|
guide, including React error boundaries and source-map deployment.
|
|
- `GET /static/instrumentation.md` - Static/public alias for the same guide.
|
|
|
|
## Usage
|
|
|
|
### Sending Telemetry Data
|
|
|
|
Send OTLP data to the appropriate endpoints. The data will be stored in history and broadcast to all connected WebSocket clients.
|
|
|
|
#### HTTP Example (JSON)
|
|
|
|
```sh
|
|
curl -X POST http://localhost:8080/v1/traces \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeSpans":[{"scope":{"name":"test-scope"},"spans":[{"traceId":"0102030405060708090a0b0c0d0e0f10","spanId":"0102030405060708","name":"test-span","startTimeUnixNano":"1640995200000000000","endTimeUnixNano":"1640995260000000000","attributes":[{"key":"test.key","value":{"stringValue":"test-value"}}]}]}]}]}'
|
|
```
|
|
|
|
#### gRPC Example
|
|
|
|
Use any OTLP gRPC client library to send data to `localhost:4317`.
|
|
|
|
### Real-time Visualization
|
|
|
|
1. Open `http://localhost:8080` in your browser
|
|
2. The interface will automatically connect via WebSocket
|
|
3. Send telemetry data to see it appear in real-time
|
|
4. Use the "Clear" buttons to reset individual data sections
|
|
5. View message counters for each telemetry type
|
|
|
|
### Accessing Historical Data
|
|
|
|
Use the history API endpoints to retrieve previously received data:
|
|
|
|
```sh
|
|
curl http://localhost:8080/api/history/traces
|
|
```
|
|
|
|
This returns JSON arrays of historical entries with timestamps.
|
|
|
|
### Frontend Error Collector (`static/otel-rum.js`)
|
|
|
|
A framework-free browser script that captures uncaught errors, unhandled
|
|
promise rejections and failed resource loads (img/script/link), and ships
|
|
them as OTLP log records to the existing `/v1/logs` endpoint - so they flow
|
|
through the same history/search/issue pipeline as backend telemetry, tagged
|
|
`source: frontend`.
|
|
|
|
```html
|
|
<script src="https://otel.k6n.net/static/otel-rum.js"></script>
|
|
<script>
|
|
var rum = OtelRUM.init({
|
|
serviceName: "my-web-app",
|
|
release: "2026.09.18-abc123",
|
|
// Optional; defaults to https://otel.k6n.net.
|
|
baseUrl: "https://otel.example.com",
|
|
});
|
|
|
|
// Inject a W3C traceparent header into any HTTP client so a backend span
|
|
// (if the service is OTel-instrumented) shares this request's trace id -
|
|
// then GET /api/traces/{traceId} shows the frontend error and backend
|
|
// spans for the same request together.
|
|
fetch("/api/orders", { headers: rum.traceHeaders() });
|
|
|
|
// Manual error reporting.
|
|
try { risky(); } catch (err) { rum.captureException(err); }
|
|
</script>
|
|
```
|
|
|
|
React render failures should be reported from an error boundary as well. The
|
|
project UI uses this pattern in `otel-ui/src/components/ViewErrorBoundary.tsx`:
|
|
|
|
```tsx
|
|
class AppBoundary extends React.Component<{ children: React.ReactNode }> {
|
|
componentDidCatch(error: Error, info: React.ErrorInfo) {
|
|
OtelRUM.last?.captureException(error, {
|
|
"exception.type": "ReactErrorBoundary",
|
|
"react.component_stack": info.componentStack,
|
|
});
|
|
}
|
|
|
|
render() { return this.props.children; }
|
|
}
|
|
```
|
|
|
|
The boundary should also render a fallback UI; otherwise a broken component
|
|
can leave the whole application blank while still being logged.
|
|
|
|
`fetch` is auto-instrumented by default (same-origin requests only) to
|
|
inject `traceparent` and record request/response breadcrumbs; this can be
|
|
disabled via `injectTraceHeaders: false` / `autoInstrumentFetch: false`.
|
|
The log ingest endpoint supports unauthenticated cross-origin POSTs; no OIDC
|
|
session or `Authorization` header is needed.
|
|
Source maps are kept server-side. When a captured frontend error has a
|
|
matching `service.release` and generated bundle path, the Issues view displays
|
|
the resolved original source frame while retaining the generated frame.
|
|
|
|
#### Source-map deploy workflow
|
|
|
|
Build the UI with a stable release identifier and keep the generated map next
|
|
to the bundle. Do not publish the map publicly:
|
|
|
|
```sh
|
|
export RELEASE="2026.09.18-$(git rev-parse --short HEAD)"
|
|
npm --prefix otel-ui run build
|
|
|
|
# Upload after the new UI is deployed. Use the generated URL/path exactly as
|
|
# it appears in browser stack traces.
|
|
curl -fS -X POST https://otel.example.com/api/sourcemaps \
|
|
-F "release=$RELEASE" \
|
|
-F "source=/assets/index-ABC123.js" \
|
|
-F "file=@otel-ui/dist/assets/index-ABC123.js.map"
|
|
```
|
|
|
|
For this project, `otel-ui/src/main.tsx` sends `VITE_APP_RELEASE` as the RUM
|
|
`service.release` value; set it during the build, for example:
|
|
|
|
```sh
|
|
VITE_APP_RELEASE="$RELEASE" npm --prefix otel-ui run build
|
|
```
|
|
|
|
The repository also includes `scripts/upload-sourcemap.sh`; after deploying,
|
|
run it against the emitted bundle, for example:
|
|
|
|
```sh
|
|
RELEASE="$RELEASE" OTEL_URL="https://otel.example.com" \
|
|
./scripts/upload-sourcemap.sh otel-ui/dist/assets/index-ABC123.js
|
|
```
|
|
|
|
The receiver stores uploaded maps under `OTEL_SOURCEMAP_DIR` (default
|
|
`./sourcemaps`, `/sourcemaps` in the Kubernetes deployment). Upload maps for
|
|
each release before investigating its errors. The API never returns map
|
|
contents or original source files.
|
|
|
|
`build.sh` passes the release into the Docker UI build. If `OTEL_URL` is set,
|
|
it also builds the matching UI locally and uploads all generated JavaScript
|
|
maps. The Dockerfile removes `.map` files from the
|
|
final image by default; set `PUBLIC_SOURCEMAPS=true` for a development or
|
|
staging image where Chrome DevTools should fetch the original sources:
|
|
|
|
```sh
|
|
PUBLIC_SOURCEMAPS=true ./build.sh
|
|
```
|
|
|
|
Public maps can expose application source code, so keep this disabled for
|
|
production unless that exposure is intentional. Credentials are never passed
|
|
to Docker or stored in the image.
|
|
|
|
## Configuration
|
|
|
|
- `OTELLOG_PATH` - path to the persistent telemetry log file (default
|
|
`otel-agent.log`). Only used when ClickHouse is not configured.
|
|
- `OTEL_SOURCEMAP_DIR` - directory for uploaded release source maps (default
|
|
`./sourcemaps`; mount persistent storage there in production).
|
|
- `OTEL_KUBERNETES_ENABLED` - set to `false` to disable rollout observation;
|
|
otherwise it activates in-cluster or `KUBECONFIG` observation when available.
|
|
- `OTEL_KUBERNETES_NAMESPACE` - optional namespace restriction; empty watches
|
|
all namespaces permitted by the service account.
|
|
- `OTEL_KUBERNETES_LABEL_SELECTOR` - optional Deployment label selector.
|
|
- `OTEL_PROMETHEUS_URL` - optional Prometheus base URL for rollout impact.
|
|
- `OTEL_PROMETHEUS_ERROR_QUERY` - optional `fmt.Sprintf`-style PromQL template;
|
|
`%q` placeholders receive service and release in that order.
|
|
- `OTEL_CLICKHOUSE_ADDR` - optional `host:port` (native protocol, usually
|
|
`9000`). When set, telemetry is stored in ClickHouse instead of the JSONL
|
|
file: history search stays fast and unbounded, and the aggregate stats
|
|
endpoints (`/api/stats/*`) and the UI "Trends" tab are enabled. The
|
|
schema (`otel` database, raw `entries` table + materialized hourly
|
|
rollup) is created automatically on first connect with a 90-day TTL; if
|
|
ClickHouse is unreachable at startup the server falls back to the file
|
|
store.
|
|
- `OTEL_CLICKHOUSE_DB` / `OTEL_CLICKHOUSE_USER` /
|
|
`OTEL_CLICKHOUSE_PASSWORD` - optional ClickHouse connection settings
|
|
(defaults: `otel` / `default` / empty).
|
|
- `OTEL_INGEST_KEY` - optional shared key required (via `?key=` or the
|
|
`X-Otel-Ingest-Key` header) on `/v1/traces`, `/v1/metrics` and `/v1/logs`
|
|
over HTTP. Unset by default (no key required). This is meant to
|
|
scope/rate-limit a public ingest endpoint fed by a browser script, not to
|
|
provide real authentication - pair it with the built-in per-IP rate
|
|
limiting on those endpoints if the endpoint is public.
|
|
- `OTEL_AGENT_TOKEN` - optional service token for non-interactive debugging
|
|
agents. Send it as `X-Otel-Agent-Token` or `Authorization: Bearer ...` when
|
|
calling protected `/api/history`, `/api/issues`, `/api/traces`, and stats
|
|
endpoints. Keep it in a Kubernetes Secret and never expose it to browsers.
|
|
|
|
A single-node ClickHouse instance for Kubernetes is provided in
|
|
`k8s/clickhouse.yaml` (StatefulSet + Service + PVC, deployed in the
|
|
`monitoring` namespace); the receiver Deployment in `k8s/deployment.yaml`
|
|
already points at it. The UI's Trends tab (and live issue updates) appear
|
|
automatically once the stats endpoints answer; on file-store deployments the
|
|
tab hides itself. SPA routes (/issues, /traces, /logs, /metrics, /trends,
|
|
/setup) are served client-side by wouter and survive the OIDC login redirect.
|
|
|
|
Currently, the history buffer size is hardcoded to 100 entries. This can be made configurable in future versions.
|
|
|
|
## Development
|
|
|
|
To modify the application:
|
|
|
|
1. Edit source files in `cmd/server/` and `internal/`
|
|
2. Run locally with `go run ./cmd/server`
|
|
3. For UI changes, edit `index.html` directly
|
|
|
|
The application uses standard Go modules and OpenTelemetry collector libraries for data processing.
|