go-otel/agents_example.md

AI Agent Debugging with OTEL Collector

This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics). It is also served as-is at {Base URL}/agents_example.md, so an agent can fetch it directly before making any API calls.

Base Configuration

• Base URL: https://otel.k6n.net
• Authentication: set the service token in OTEL_AGENT_TOKEN and send it as X-Otel-Agent-Token: $OTEL_AGENT_TOKEN on every protected debug request.
• Endpoints:
  • GET /api/history/search?q={query}&type={type}&since={ns}&until={ns}&limit={limit}: Search all historical telemetry data.
    • Default Limit: 50 (max 100 recommended for context).
    • Ordering: Latest entries first.
    • Match Scope: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.), combinable with structured filters.
    • type: optional, one of traces, metrics, logs.
    • since / until: optional, unix nanosecond timestamps to bound the time range.
  • GET /api/history/wait?q={query}&type={type}&timeout={seconds}: Wait for a specific signal to arrive.
    • Blocking: This call blocks until a match is found or the timeout (default 30s) is reached.
    • Ideal for Automation: Use this after triggering an action to wait for the exact result.
    • Rate limited: capped per client IP; back off if you receive 429.
  • GET /api/traces/{traceId}: Fetch every stored entry (spans and correlated logs) referencing a specific trace id in one call. Prefer this over substring-searching for a trace id.
  • GET /api/issues: List deduplicated error issues (from error-status spans and error/fatal-severity logs), most recently seen first. Supported filters are limit (default 100, max 200), cursor, service, kind, status, q (title substring), sourceIp, route, since, and until (the time bounds are unix nanoseconds). Results remain a JSON array for compatibility; when another page exists, use the X-Next-Cursor response header or the Link: ...; rel=next header in the next request. The cursor is opaque and must be reused unchanged. Use this to triage what's currently broken before diving into raw search.
  • GET /api/issues/{id}: Fetch a single issue's detail, including sample occurrences with trace ids you can feed into /api/traces/{traceId}. PATCH the same path with {"status":"resolved"|"ignored"|"open"} to update triage state (a resolved issue that fires again is auto-reopened).
  • GET /api/stats/errors?hours=72&service=: Hourly (daily when hours > 336) total/error entry counts, ClickHouse-backed. Responds 501 when the server runs on file storage. Use for "when did errors spike?" and trend questions.
  • GET /api/stats/releases?hours=72&service=: Per-release error-count series ({"series":{"v1.3":[...],"v1.4":[...]}}) to compare error rates before/after a deploy. Only entries carrying a service.release attribute appear.
  • GET /api/stats/services: Distinct services present in the stats rollup.
  • GET /api/history: Fetch recent history of all telemetry (searches the full persistent store, not just recent in-memory entries).
  • GET /api/history/traces: Fetch recent traces.
  • GET /api/history/logs: Fetch recent logs.
  • GET /api/history/metrics: Fetch recent metrics.

Agent Instructions

1. Searching for Context

When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are latest first, you will always see the most recent events related to your search.

Example Request:

# Search for recent "ERROR" occurrences (default limit 50, latest first)
curl "https://otel.k6n.net/api/history/search?q=error" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"

2. Waiting for a Signal

Use the wait endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.

Example Request:

# Wait up to 60s for a specific traceId to show up in the collector
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"

3. Identifying Trace Chains

If you find a log entry with a traceId, use the dedicated trace endpoint to fetch all spans and logs associated with that specific trace and understand the full request flow.

Example Request:

# Fetch all activity related to a specific trace
curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"

3. Fetching Recent Telemetry

To see the most recent activity without specific keywords, use the search endpoint with an empty query.

Example Request:

curl "https://otel.k6n.net/api/history/search?limit=20" \
  -H "X-Otel-Agent-Token: $OTEL_AGENT_TOKEN"

4. Data Format

The returned data is a JSON array of HistoryEntry objects:

[
  {
    "type": "logs",
    "timestamp": 1712132978000000000,
    "data": { ... OTEL Log Records ... }
  },
  {
    "type": "traces",
    "timestamp": 1712132979000000000,
    "data": { ... OTEL Resource Spans ... }
  }
]

Best Practices for AI Agents

• Iterative Search: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
• Limit results: Always use the limit parameter to manage context window size (default is 100).
• Correlate Signals: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.