go-otel/instrumentation.md

# Instrumentation guide

This document describes the recommended instrumentation contract for this
receiver. It is available at `/instrumentation.md` so frontend and backend
agents can read it before integrating telemetry.

## Shared identity

Use the same values across all signals from one deployed application:

- `service.name`: stable application name, for example `checkout-web`.
- `service.release`: immutable deploy identifier, for example
  `2026.09.18-a1b2c3`.
- `service.environment`: `production`, `staging`, or `development`.

The release must exactly match the source-map upload release.

## Browser frontend

Load the RUM script and initialize it once, before the application starts:

```html
<script src="https://otel.example.com/static/otel-rum.js"></script>
<script>
  const rum = OtelRUM.init({
    serviceName: "checkout-web",
    release: "2026.09.18-a1b2c3",
    environment: "production",
    baseUrl: "https://otel.example.com"
  });
</script>
```

To avoid parser-blocking downloads, use a deferred script with declarative
initialization instead of the two scripts above:

```html
<script defer src="https://otel.example.com/static/otel-rum.js"
  data-service-name="checkout-web"
  data-release="2026.09.18-a1b2c3"
  data-environment="production"
  data-base-url="https://otel.example.com"></script>
```

Declarative initialization runs at `DOMContentLoaded`. Use the earlier,
explicit initialization when capturing errors and requests during application
startup matters more than avoiding parser blocking. Do not initialize twice.

The collector captures uncaught errors, unhandled promise rejections, failed
HTTP requests, breadcrumbs, Web Vitals, and optional trace context. Correlate
frontend requests with backend spans when possible:

```js
fetch("/api/orders", { headers: rum.traceHeaders() });
```

For manually handled errors:

```js
try {
  riskyOperation();
} catch (error) {
  rum.captureException(error, { "feature.name": "checkout" });
}
```

Do not put secrets, tokens, or sensitive customer data in breadcrumbs or
custom attributes.

### Cookieless diagnostics and consent

Neither RUM nor Todo reads or writes cookies, sessionStorage, or localStorage.
Configuration and consent are held only in memory. Integrate with the
application's existing consent manager rather than adding a tracking cookie
or another banner:

```js
const rum = OtelRUM.init({ serviceName: "checkout-web" });
rum.setConsent("granted");
rum.setConsent("denied");
```

For a previously accepted choice, supply `consent: "granted"` at initialization
or `data-consent="granted"` on the script. Omitted consent means `"denied"`;
explicit configuration overrides the attribute. `rum.getConsent()` returns
the current choice. The application is responsible for persisting that choice
and applying it on the next page load.

The SDK separates error diagnostics from optional performance collection:

| Policy | Error payload | Performance collection |
| --- | --- | --- |
| Default `counts` | Generic event, severity, recognised error type, HTTP status, release/environment | Off without consent |
| Configured `minimal` | Counts plus application-sanitized diagnostic fields | Still off without consent |
| `consented` | Detailed errors, stacks, context, breadcrumbs and page metadata | On unless disabled |
| Explicit, host-allowlisted `development` | Detailed diagnostics for an already-approved development site | On unless disabled |

`rum.getPrivacyMode()` reports the configured active policy. An individual
minimal event falls back to counts if its sanitizer does not approve it.
Counts/minimal payloads exclude automatic URLs, user agents, viewport/connection
details, session ids, DOM selectors, breadcrumbs, context and custom attributes.
Per-request trace-header instrumentation still runs without persistent visitor
identity, but counts/minimal log records do not include trace ids. Neither
policy is a guarantee of anonymity or a consent exemption.

### Minimal actual-error diagnostics

Only configure this policy after separately assessing the diagnostic purpose,
ePrivacy position and GDPR legal basis; it is not automatically enabled when
tracking consent is rejected. Provide an application-owned sanitizer that maps
known errors to useful, reviewed diagnostics:

```js
const safeDiagnostics = new Map([
  ["checkout.price-missing", {
    message: "Checkout price is missing",
    code: "CHECKOUT_PRICE_MISSING"
  }]
]);

const rum = OtelRUM.init({
  serviceName: "checkout-web",
  minimalDiagnostics: {
    route: "/checkout/:step",
    sanitize: event => safeDiagnostics.get(event.message) || null
  }
});

rum.captureMessage("checkout.price-missing");
```

The sanitizer receives bounded `message`, optional `type` and `stacktrace`.
It may return `{message, code?, stacktrace?}`; only those fields and the
configured route template are added. Return `null` for unknown/free-text
events. Missing, throwing or invalid sanitizers fall back to counts without
sending the original message or stack. Output lengths are bounded, but **the
SDK trusts sanitizer output: it does not automatically scrub returned text**.
Only return reviewed messages and necessary scrubbed stack frames; never
forward `event` unchanged. Exclude credentials, query strings, fragments,
user-specific paths, user content and sensitive data. The route must be an
application-controlled template, not `location.pathname`.

On withdrawal, detailed queued records are dropped and an independently
configured minimal policy continues. Without that policy, counts continue.
Do not use this to relabel consent-required processing after refusal. Even
counts require a deployment-specific assessment. See `docs/EU-RUM-PRIVACY.md`.

### Development sites with existing access and acceptance

Development mode is explicit and requires an **exact hostname allowlist**:

```js
const rum = OtelRUM.init({
  serviceName: "checkout-web",
  environment: "development",
  devMode: accessState.loggedIn && accessState.diagnosticsAccepted,
  devModeHosts: ["localhost", "checkout.dev.example.com"]
});
```

Here `accessState` is provided by the host application's existing login and
acceptance flow. Do not initialize development mode before that flow completes.
Login alone does not establish consent or another legal basis. The SDK cannot
verify the login, notice, acceptance or purpose; the application must do so.
The allowlist is an accidental-production safeguard, not an authentication
boundary. Wildcards/subdomains do not match automatically; an environment
label never enables this mode. Ports are not part of the hostname match.

For declarative setup after the access gate, use `data-dev-mode="true"` and
`data-dev-mode-hosts="localhost,checkout.dev.example.com"`. Explicit
`devMode: false` overrides the attribute. Manual `captureMessage(message,
"INFO")`, exceptions, context and breadcrumbs are available for development
improvement logs. Do not include passwords, tokens or sensitive user content.

Development mode does not manufacture a consent grant: `getConsent()` still
returns the actual configured choice. Call `rum.setConsent("denied")` on logout
or withdrawal; it disables development mode even if consent was already denied,
discards queued data and returns to minimal/counts. Re-enabling requires a new
page initialization after the access gate. Todo remains consent-controlled
separately, and is still disabled by default.

### Performance permission and withdrawal

Web Vitals is not loaded until consent is granted or allowed development mode
is active. `captureWebVitals: false` disables it in all modes. If a separate
assessment establishes permission for cookieless performance collection, set
`captureWebVitalsWithoutConsent: true` or `data-vitals-without-consent="true"`.
Explicit `false` overrides the attribute; this option is not a legal exemption.
In that mode only values, deltas and numeric attribution are sent without
consent. A random, in-memory page-view id deduplicates snapshots within one
load; metric ids, selectors and other text attribution are omitted.

Detailed diagnostics use an in-memory-only session id, renewed on reload or
re-grant. Every consent change discards queued records, breadcrumbs, context
and pending vitals; data already sent cannot be recalled. Reapply identifying
context only after permission. Withdrawal suppresses later vitals callbacks
and reports unless cookieless performance was independently enabled. Observers
already installed inside the vendored Web Vitals library cannot be unloaded by
this SDK; reload the page after withdrawal if complete observer teardown is
required. If consent is revoked before its download completes, no metric
subscriptions are started.

Telemetry uses `fetch` with `credentials: "omit"`, `referrerPolicy: "no-referrer"`
and `keepalive: true` in every mode; no cookie-bearing `sendBeacon` fallback is used. If fetch
is unavailable or fails, telemetry is dropped. Browser-managed network headers
and client IP addresses remain visible to the receiving infrastructure, so
this is client-side data minimization, not a guarantee of legal anonymization
or a determination that consent is unnecessary. Review server logging,
retention, enrichment and applicable requirements separately.

### Building the browser scripts

The maintained TypeScript sources are `rum/src/otel-rum.ts` and
`rum/src/otel-todo.ts`; public options and APIs are defined in
`rum/src/types.ts`. Do not edit the generated `static/otel-rum.js` or
`static/otel-todo.js` directly.

Run `make build-rum` to install locked dependencies, typecheck with strict
TypeScript, and generate separate minified IIFE bundles at the existing
`/static/otel-rum.js` and `/static/otel-todo.js` URLs. Todo stays separate so
it is never downloaded unless enabled. Run `make test-rum` for the same build
plus regression tests against the generated JavaScript. After installing
dependencies, `npm --prefix rum test` also runs the build and tests.

`make build` builds the browser scripts and Go server. Docker builds and tests
the browser scripts in an isolated Node stage automatically, so `build.sh`
also uses fresh bundles. Generated JavaScript remains checked in for direct
`go run ./cmd/server` development; rebuild it whenever TypeScript changes.

External source maps are generated locally but are not downloaded by browsers
automatically. Docker omits them by default; `PUBLIC_SOURCEMAPS=true` retains
them, as for the UI. TypeScript and esbuild are build-time dependencies only.

## React error boundaries

Rendering errors are not reliably caught by global `window.onerror` handlers.
Report them from an error boundary and render a recovery screen:

```tsx
import React from "react";

export class AppErrorBoundary extends React.Component<
  { children: React.ReactNode },
  { error: Error | null }
> {
  state = { error: null as Error | null };

  static getDerivedStateFromError(error: Error) {
    return { error };
  }

  componentDidCatch(error: Error, info: React.ErrorInfo) {
    window.OtelRUM?.last?.captureException(error, {
      "exception.type": "ReactErrorBoundary",
      "react.component_stack": info.componentStack,
    });
  }

  render() {
    if (this.state.error) {
      return <button onClick={() => this.setState({ error: null })}>Retry</button>;
    }
    return this.props.children;
  }
}
```

Use the same pattern in your application, including a user-visible fallback
for individual views and the application root.

## Todo widget

`otel-todo.js` adds a small floating symbol and a keyboard shortcut
(default `Alt+Shift+T`) that opens a corner dialog for filing a todo/issue
from the page, with an optional component picker. User-submitted text, selected
components and explicitly picked element references are sent as requested;
URLs, user agents and session metadata are attached only with consent. A note
can itself contain personal information, so it is not guaranteed anonymous.

Todo is **disabled by default** in `otel-rum.js`. Opt in with
`OtelRUM.init({ enableTodo: true })` or `data-enable-todo="true"` on the RUM
script. It reuses the service name, base URL and ingest key, and loads from
the same directory as the RUM script.

When enabled, both loading and initialization wait until after `window.load`,
then run during `requestIdleCallback` with no forced timeout. If idle callbacks
are unavailable, loading waits three seconds after `load`. On a continuously
busy page, idle loading may be delayed indefinitely. Error capture and request
instrumentation do not wait for Todo.

Explicit `enableTodo: false` overrides the opt-in attribute. The legacy
`excludeTodo: true` and `data-exclude-todo="true"` remain supported and veto
opt-in. Passing `excludeTodo: false` no longer enables Todo.

Alternatively, load it standalone (this bypasses RUM's idle scheduling):

```html
<script src="https://otel.example.com/static/otel-todo.js"
        data-service-name="checkout-web"
        data-ingest-key="..."
        data-components="Header,Cart,Payment"></script>
```

Components are also discovered from `[data-component="..."]` elements. Other
options (on the RUM or todo script, or `OtelRUM.init({ todo: {...} })`): `data-shortcut`, `data-hide-button="true"`, `data-base-url`.

When loaded by RUM, Todo reads the current RUM consent at submission time,
including after withdrawal. Standalone Todo defaults to denied consent; use
`OtelTodo.instance.setConsent("granted" | "denied")` from the consent manager,
or initialize with `consent` / `data-consent`. Submissions always omit browser
credentials and referrers, and no browser storage is used.

- `POST /v1/todos` (ingest key, CORS) body `{project, text, component, path, hash, url, ua, session}`
- `GET /api/todos?project=<service>&limit=100` (authenticated; omit `project` for all), newest first

Todos go to ClickHouse when configured, else to the JSONL file at `OTEL_TODOS_PATH`
(default `otel-todos.jsonl`).

## CSP violation reports

Browsers report Content-Security-Policy violations straight to the collector.
Point the policy's `report-uri` (or a Reporting API `report-to` endpoint) at
`/v1/csp`; both the legacy `application/csp-report` object and the Reporting
API's `application/reports+json` batch are accepted:

```html
policy: default-src 'self'; report-uri https://otel.example.com/v1/csp?key=...
```

Delivery is allowed cross-origin from `*.k6n.net`, `*.tornberg.me` and
`*.knatofs.se` (the collector answers CORS preflights and echoes the origin
for those domains only). Like the vitals beacon, the optional ingest key
travels as `?key=` on the reporting URL, since browsers cannot attach headers
to violation reports. Storage follows the configured store: a dedicated typed
ClickHouse table (`csp_reports`, 90-day TTL), else the JSONL file at
`OTEL_CSP_PATH` (default `otel-csp-reports.jsonl`).

- `POST /v1/csp` (ingest key as `?key=`, CORS) — browser-reported, send nothing by hand
- `GET /api/csp/reports?site=&directive=&q=&hours=&limit=` (authenticated), newest first
- `GET /api/csp/summary?hours=24` (authenticated) — totals plus top directives, blocked hosts and sites

## Backend instrumentation

Instrument backend services with an OpenTelemetry SDK or auto-instrumentation
and export OTLP to this receiver:

- OTLP/gRPC: `otel.example.com:4317`
- OTLP/HTTP traces: `https://otel.example.com/v1/traces`
- OTLP/HTTP metrics: `https://otel.example.com/v1/metrics`
- OTLP/HTTP logs: `https://otel.example.com/v1/logs`

Set the resource attributes on every provider/exporter:

```text
service.name=checkout-api
service.release=2026.09.18-a1b2c3
service.environment=production
```

Create spans around inbound requests, database calls, external HTTP calls,
queue operations, and important business operations. Record HTTP status,
route, method, and exception details. Mark failed spans with an error status.
The receiver derives backend issues from error spans and ERROR/FATAL logs.

If HTTP ingest authentication is enabled, provide `OTEL_INGEST_KEY` as the
`X-Otel-Ingest-Key` header or `key` query parameter. Keep that key server-side;
browser clients should not receive it.

For non-interactive agents that need to query protected debug endpoints, use a
separate `OTEL_AGENT_TOKEN` service token. Send it as
`X-Otel-Agent-Token: $OTEL_AGENT_TOKEN`; do not put it in browser code or
telemetry payloads.

## Source maps

Build the frontend with the same immutable release used by RUM. The command
and output directory depend on your build tool; the example below assumes a
standard `dist/` output:

```sh
export RELEASE="2026.09.18-a1b2c3"
VITE_APP_RELEASE="$RELEASE" npm run build
```

After deployment, upload the contents of each generated JavaScript map to the
public multipart endpoint. No repository checkout or upload script is needed.
`source` must match the generated bundle URL/path captured in the browser
stack, while `file=@...` reads the local source-map file contents:

```sh
curl --fail-with-body --silent --show-error \
  -H "X-Sourcemap-Key: $OTEL_SOURCEMAP_KEY" \
  -X POST "https://otel.example.com/api/sourcemaps" \
  -F "release=$RELEASE" \
  -F "source=/assets/app-HASH.js" \
  -F "file=@dist/assets/app-HASH.js.map;type=application/json"
```

For example, if the deployed JavaScript is available at
`https://app.example.com/assets/main-abc123.js`, upload its matching map with
`source=/assets/main-abc123.js`. The source-map filename on the build machine
does not need to match the URL, but the `source` field must.

Maps are parsed, written to `OTEL_SOURCEMAP_DIR`, and reloaded on receiver
start. Issue responses include a resolved stack when `service.release` and the
generated bundle path match. Uploaded maps are never returned, but when a map
embeds `sourcesContent` the RUM "JS errors" view shows a few lines of original
source around each mapped frame to signed-in users.

### Requirements checklist

A stack only resolves when every one of these holds. When it does not, check
them in this order:

1. **The build emits maps, with sources embedded.** Enable source maps in the
   bundler (Vite `build.sourcemap: true`, webpack `devtool: "source-map"`).
   Keep `sourcesContent` (the default) or the code view has nothing to show.
2. **The browser reports a release.** Pass the release to `OtelRUM.init({ release })`
   (or `data-release` on the script tag), injected at build time. An event
   with no release, or `dev`, can never match an upload.
3. **The uploaded release is byte-for-byte the same string.** Use one value for
   both the bundle build and the upload (a git short SHA works).
4. **`source` matches the path in the browser stack.** Everything before
   `/assets/` and any `?query` is ignored, so `source=/assets/main-abc123.js`
   matches `https://app.example.com/assets/main-abc123.js`. Content-hashed
   names must match exactly, and chunks each need their own upload.
5. **The map belongs to the deployed bundle.** Upload the maps from the very
   build that produced the deployed files. A second local rebuild can differ
   (dependencies, Node version) and silently mismatch. In Docker, copy the maps
   out of the build stage (`docker build --target <stage>`, then `docker cp`)
   instead of rebuilding. `build.sh` does this.
6. **Every chunk is uploaded.** Loop over all `*.js.map` files and fail the
   deploy when none are found, since an empty loop otherwise succeeds silently.
7. **The receiver keeps its maps.** `OTEL_SOURCEMAP_DIR` must be on persistent
   storage (a PVC in Kubernetes, not `emptyDir`). Maps written to an ephemeral
   directory vanish on restart.
8. **Maps are not published with the site.** Delete `*.map` from the served
   image unless public maps are intended (see below).

### Verifying a deploy

List what the receiver has loaded (requires a signed-in session or token):

```sh
curl -H "Authorization: Bearer $TOKEN" https://otel.example.com/api/sourcemaps
# [{"release":"2026.09.18-a1b2c3","source":"/assets/main-abc123.js"}, ...]
```

Then trigger a real uncaught error from application code, for example a
button that runs `setTimeout(() => { throw new Error("source-map check") })`
(this repository's RUM "JS errors" view has a *Generate test error* button),
and confirm the card shows **Source-mapped** with the original file and line.

Frame status in the stack explorer explains a miss: `no-release` (event has no
release), `no-map` (nothing uploaded for that release and file), or `unmapped`
(a map exists but has no mapping for that position).

The Docker build removes `.map` files from production images by default. For
development or staging, public maps can be enabled for Chrome DevTools:

```sh
PUBLIC_SOURCEMAPS=true ./build.sh
```

Public maps expose application source code and should not be enabled in
production unless that is intentional.

## Verification checklist

1. Trigger a controlled frontend exception.
2. Trigger a controlled backend error span or ERROR log.
3. Confirm both appear under Issues as separate frontend/backend issues.
4. Confirm the issue sample contains the expected release.
5. Confirm a mapped frontend stack shows original file and line information
   (see *Requirements checklist* under Source maps if it does not).
6. Confirm trace IDs connect frontend requests to backend traces where headers
   were propagated.