Observability

Pipeline Overview #

The end-to-end signal flow from agent execution to Grafana visualization:

LLM Agents (OTLP HTTP :4318 / gRPC :4317)
       │
       ▼
┌─────────────────────────────────────────────────────────┐
│  OpenTelemetry Collector  (otel/opentelemetry-collector-contrib:0.100.0)  │
│                                                         │
│  receivers:  otlp (grpc :4317, http :4318)              │
│  processors: memory_limiter → attributes/llm_meta       │
│              → batch                                     │
│  exporters:  otlp/tempo (traces) · prometheus (metrics) │
└───────────┬────────────────────────┬────────────────────┘
            │ traces (OTLP gRPC)     │ metrics (Prometheus scrape :8889)
            ▼                        ▼
   ┌─────────────────┐     ┌───────────────────────┐
   │  Grafana Tempo  │     │  Prometheus 2.52       │
   │  2.4.1          │     │  (amtp-prometheus)     │
   │  HTTP :3200     │     │  :9091 (host)          │
   │  gRPC :9095     │     │  15 d retention        │
   └────────┬────────┘     └──────────┬────────────┘
            │ TraceQL               │ PromQL
            └──────────┬────────────┘
                       ▼
              ┌─────────────────┐
              │  Grafana 10.4.2 │
              │  :3000 (host)   │
              │  Datasources:   │
              │    Tempo · Prom │
              │  Dashboard:     │
              │    LLM Token    │
              │    Accounting   │
              └─────────────────┘

OpenTelemetry Collector #

The Collector is the single ingestion point for all AMTP telemetry. Configuration: infra/observability/otel-collector-config.yaml.

Receivers #

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins: ["http://*", "https://*"]

Processors #

Pipeline execution order: memory_limiter → attributes/llm_meta → batch.

Exporters #

Extensions #

Grafana Tempo 2.4.1 #

Distributed trace backend. Receives spans from the OTel Collector via OTLP gRPC on its internal port 4317. Exposes a Grafana datasource on HTTP 3200 for TraceQL queries. Configuration: infra/observability/tempo.yaml.

Metrics Generator #

Tempo’s built-in metrics_generator derives span metrics and service graphs from ingested traces, then remote-writes them to Prometheus:

metrics_generator:
  storage:
    remote_write:
      - url: http://amtp-prometheus:9090/api/v1/write
        send_exemplars: true
  processor:
    service_graphs:
      dimensions: [service.name, span.kind]
    span_metrics:
      dimensions:
        - service.name
        - span.name
        - span.kind
        - status.code
        - llm.model
        - llm.provider
        - gen_ai.system
      enable_target_info: true

The span-metric dimensions include llm.model, llm.provider, and gen_ai.system so Grafana can break down P95 latency and throughput by model and provider without requiring a separate labeling step.

Storage & Retention #

• Backend: local filesystem (/var/tempo/blocks) backed by Docker volume tempo_data.
• Retention: block_retention: 72h — traces older than 72 hours are compacted and dropped.
• WAL: /var/tempo/wal for durability across ingester restarts.
• Bloom filter: false-positive rate 0.01; block encoding zstd.
• Max traces per user: 100,000. Ingestion rate limit: 15 MB/s, burst 20 MB/s.

Prometheus 2.52 (amtp-prometheus) #

Metrics store. The service is named amtp-prometheus (not the default prometheus) to avoid collisions with other Prometheus instances on the amtp_net network. Host port ${OBS_PROMETHEUS_PORT:-9091} maps to container port 9090. Configuration: infra/observability/prometheus.yml.

Scrape Targets #

scrape_configs:
  # LLM token metrics emitted by agents via OTEL collector
  - job_name: otel-collector
    static_configs:
      - targets: ['otel-collector:8889']

  # Tempo span metrics (service graphs, span durations) via metrics_generator
  - job_name: tempo
    static_configs:
      - targets: ['tempo:3200']
    metrics_path: /metrics

Runtime Flags #

• --storage.tsdb.retention.time=15d — metrics retained for 15 days.
• --web.enable-remote-write-receiver — accepts remote-write from Tempo’s metrics generator.
• --web.enable-lifecycle — exposes /-/reload and /-/quit endpoints.
• External label: cluster=zt-amtp added to all scraped metrics.

Grafana 10.4.2 #

Visualization layer. Depends on both Tempo and Prometheus being healthy before starting. Provisioned via Git-backed YAML files mounted from infra/observability/grafana/provisioning/.

Provisioned Datasources #

LLM Token Accounting Dashboard #

UID: llm-token-accounting. Auto-provisioned from infra/observability/grafana/provisioning/dashboards/llm-token-accounting.json. Refresh: 30 s. Default time range: last 3 hours.

Generic OAuth / OIDC SSO #

Grafana supports any OAuth 2.0 / OIDC provider via the GF_AUTH_GENERIC_OAUTH_* environment variable conventions (mapped to the [auth.generic_oauth] ini section). SSO is disabled by default (GF_AUTH_GENERIC_OAUTH_ENABLED=false). Role attribute path defaults to: grant Admin if the user is in group grafana-admins, otherwise Viewer.

SMTP for email alerts is likewise disabled by default (GF_SMTP_ENABLED=false). All configurable values are documented in .env.example § Observability stack.

Alerting #

Alerting is provisioned from infra/observability/grafana/provisioning/alerting/alerting.yaml. The provisioning file defines contact points, notification policies, mute timings, and alert rules in a single YAML document.

Contact Points #

Notification Policies #

• Default receiver: slack-llm-alerts. Group by: alertname, service_name, cluster.
• Group wait 30 s → interval 5 m → repeat 4 h.
• Sub-route for severity=critical: additionally routes to email-llm-alerts with group wait 0 s, interval 1 m, repeat 1 h, continue: true so Slack still fires.

Mute Timings #

A maintenance-window mute timing suppresses all notifications on Saturday and Sunday 00:00–23:59 (server timezone).

Alert Rules #

Pipeline Verification (apps/otel-verify/) #

apps/otel-verify/verify_trace.py is a self-contained Python script that emits one structured LLM trace plus LLM token counters through the real OTel Collector to verify the full observability pipeline end-to-end. It is used by .github/workflows/observability-ci-cd.yml as the post-deploy acceptance gate.

What it emits #

• Trace: a root span llm.completion with two child spans (llm.inference, llm.response.parse). Span attributes match the LLM instrumentation contract below (128 prompt / 64 completion / 192 total tokens; model gpt-4o; provider openai).
• Metrics: counters prompt_tokens, completion_tokens, total_tokens with attributes llm.model, llm.provider, gen_ai.system.

Trace ID extraction #

verify_trace.py prints the phrase “Trace ID” on multiple lines — once with the 32-hex value and once in the user-hint line “search by Trace ID above”. A naïve awk '{print $NF}' produces a multi-line string that breaks the subsequent curl URL. The canonical extraction in CI uses a strict regex:

TRACE_ID=$(
  python apps/otel-verify/verify_trace.py \
    | grep -oE '[0-9a-f]{32}' \
    | head -n 1
)

Tempo polling #

After emitting the trace, CI polls GET /api/traces/{TRACE_ID} on Tempo in a retry loop (20 attempts × 3 s = 60 s budget):

for i in $(seq 1 20); do
  STATUS=$(curl -s -o /dev/null -w '%{http_code}' \
    "http://localhost:${OBS_TEMPO_HTTP_PORT}/api/traces/${TRACE_ID}" || true)
  if [ "$STATUS" = "200" ]; then
    echo "Trace ${TRACE_ID} confirmed in Tempo (attempt $i)"
    exit 0
  fi
  echo "Waiting for Tempo to ingest trace... ($i/20, status=$STATUS)"
  sleep 3
done
exit 1

Dependencies #

opentelemetry-api==1.24.0
opentelemetry-sdk==1.24.0
opentelemetry-exporter-otlp-proto-grpc==1.24.0
opentelemetry-exporter-otlp-proto-http==1.24.0

LLM Instrumentation Contract #

All LLM agent spans must carry the following attributes to be correctly bucketed by the LLM Token Accounting dashboard and the Tempo span-metric dimensions.

Required span attributes #

Required metric counters #

Emitted as OTLP counter metrics (no llm_ prefix in the instrument name — the Collector’s Prometheus exporter adds the llm namespace and the _total suffix):

Required metric attributes: llm.model, llm.provider, gen_ai.system.

GitHub MCP instrumentation #

The amtp-github-mcp service emits its own OpenTelemetry spans and metrics via the same OTLP HTTP endpoint (http://otel-collector:4318). OTEL_SERVICE_NAME=amtp-github-mcp is set in the container environment so traces and metrics are attributed separately from the LLM agents. Key span names: repo.tree, repo.read_file, github.api, valkey.get, valkey.set. Key metric histograms: mcp.tool.duration, mcp.github_api.duration. These use MCP-specific attributes (mcp.tool, mcp.cache_hit, mcp.exclusion_reason) and are not subject to the LLM instrumentation contract above.

Host Ports & OBS_* Overrides #

Every observability host port is parameterized via an OBS_* environment variable so that port conflicts on a shared dev host can be resolved without code changes — set the relevant variable in GitHub Actions Settings → Environments → dev → Variables.

Network Dependency (amtp_net) #

Two remediation paths (apply one):

1. Workflow-level sequencing (recommended). Add needs: healthcheck to the observability job in .github/workflows/ci-cd.yml. This makes the base-stack bring-up a precondition, visible in the workflow graph.
2. Bootstrap-level creation. Add the following idempotent command to infra/bootstrap.sh so the network pre-exists any CI run:

   docker network create amtp_net --label amtp.owner=bootstrap 2>/dev/null || true

Until one path is applied, on a pristine host manually run either docker network create amtp_net or docker compose -f docker-compose.yml up -d --wait healthcheck before triggering the observability workflow.

See also: Observability CI/CD workflow and Dev-host bootstrap runbook.