go-otel/agents_example.md

AI Agent Debugging with OTEL Collector

This document provides instructions for AI agents on how to utilize the OTEL Collector's debugging features to search and fetch telemetry data (traces, logs, and metrics).

Base Configuration

• Base URL: https://otel.k6n.net
• Endpoints:
  • GET /api/history/search?q={query}&type={type}&since={ns}&until={ns}&limit={limit}: Search all historical telemetry data.
    • Default Limit: 50 (max 100 recommended for context).
    • Ordering: Latest entries first.
    • Match Scope: Case-insensitive substring match on raw JSON (covers service name, span body, attributes, etc.), combinable with structured filters.
    • type: optional, one of traces, metrics, logs.
    • since / until: optional, unix nanosecond timestamps to bound the time range.
  • GET /api/history/wait?q={query}&type={type}&timeout={seconds}: Wait for a specific signal to arrive.
    • Blocking: This call blocks until a match is found or the timeout (default 30s) is reached.
    • Ideal for Automation: Use this after triggering an action to wait for the exact result.
    • Rate limited: capped per client IP; back off if you receive 429.
  • GET /api/traces/{traceId}: Fetch every stored entry (spans and correlated logs) referencing a specific trace id in one call. Prefer this over substring-searching for a trace id.
  • GET /api/history: Fetch recent history of all telemetry (searches the full persistent store, not just recent in-memory entries).
  • GET /api/history/traces: Fetch recent traces.
  • GET /api/history/logs: Fetch recent logs.
  • GET /api/history/metrics: Fetch recent metrics.

Agent Instructions

1. Searching for Context

When debugging an issue, start by searching for relevant keywords in the logs and traces. This helps locate specific errors or trace IDs associated with a failure. Since results are latest first, you will always see the most recent events related to your search.

Example Request:

# Search for recent "ERROR" occurrences (default limit 50, latest first)
curl "https://otel.k6n.net/api/history/search?q=error"

2. Waiting for a Signal

Use the wait endpoint to synchronize your debugging flow. This is perfect for verifying that a specific event occurred after you triggered an action.

Example Request:

# Wait up to 60s for a specific traceId to show up in the collector
curl "https://otel.k6n.net/api/history/wait?q=5682138e4a9b4629852899d45e542456&timeout=60"

3. Identifying Trace Chains

If you find a log entry with a traceId, use the dedicated trace endpoint to fetch all spans and logs associated with that specific trace and understand the full request flow.

Example Request:

# Fetch all activity related to a specific trace
curl "https://otel.k6n.net/api/traces/5682138e4a9b4629852899d45e542456"

3. Fetching Recent Telemetry

To see the most recent activity without specific keywords, use the search endpoint with an empty query.

Example Request:

curl "https://otel.k6n.net/api/history/search?limit=20"

4. Data Format

The returned data is a JSON array of HistoryEntry objects:

[
  {
    "type": "logs",
    "timestamp": 1712132978000000000,
    "data": { ... OTEL Log Records ... }
  },
  {
    "type": "traces",
    "timestamp": 1712132979000000000,
    "data": { ... OTEL Resource Spans ... }
  }
]

Best Practices for AI Agents

• Iterative Search: Start with a broad search (e.g., service name) and narrow down using specific attributes or error messages found in the results.
• Limit results: Always use the limit parameter to manage context window size (default is 100).
• Correlate Signals: Look for timestamps that coincide across logs and metrics to identify resource constraints or timing issues.