Grafana LGTM

Grafana LGTM is the "all signals in one container" distribution — Grafana + Loki (logs) + Tempo (traces) + Prometheus (metrics) + an OTel Collector, bundled into grafana/otel-lgtm. (LGTM stands for Loki, Grafana, Tempo, Mimir; the pinned image, 0.34.0, runs Prometheus 3.14 for metrics.) It covers all three OTel signals from a single image with no account, which is why it's the recommended stack for local development and quickstarts.

Signals: traces + metrics + logs. Deployment: local (single container) or you assemble the pieces in production. Cost: OSS.

One-command quickstart

A ready-to-use compose file ships with the plugin:

cd ~/.hermes/plugins/hermes_otel
docker compose -f docker-compose/lgtm/docker-compose.yaml up -d

Wait ~30s for everything to come up, then open Grafana at http://localhost:3000 (admin / admin). Three datasources are pre-provisioned: Tempo, Prometheus and Loki.

Point the plugin at it — minimal config.yaml:

backends:
  - type: lgtm
    endpoint: http://localhost:4318/v1/traces

capture_logs: true   # optional — ship Python logs to Loki with trace-id correlation
log_level: INFO

Give endpoint as the full /v1/traces URL: the lgtm type posts traces to exactly that URL and derives /v1/metrics and /v1/logs from it, so a bare http://localhost:4318 would not work. There is no OTEL_LGTM_ENDPOINT; in env-var mode use OTEL_TEMPO_ENDPOINT (traces only) or a backends: entry.

Restart hermes-agent and you'll see all three signals flow in.

Why a dedicated type: lgtm

On the export side type: lgtm behaves like type: otlp — both send OTLP/HTTP to a collector on 4318. The dedicated type exists for three reasons:

1. Self-documenting config. type: lgtm tells the next reader exactly what stack this entry targets, instead of a bare type: otlp.
2. Startup logs say "LGTM". The fan-out status line reads ✓ LGTM at http://localhost:4318/v1/traces instead of ✓ OTLP at ....
3. The dashboard can read it. The entry gets the Tempo / Prometheus / Loki dashboard adapter; a type: otlp entry has none.

Metrics and logs are both on by default. Override with metrics: false or logs: false per backend if needed. OTLP/gRPC (protocol: grpc) is available only on a type: otlp entry; point one at port 4317 if you want gRPC.

caution

Don't use type: tempo for this stack. That type is for standalone Tempo (traces only) and will refuse to fan out logs or metrics even when pointed at the LGTM collector's endpoint. Use type: lgtm (or type: otlp) to unlock all three signals.

Ports

Port	Service	Purpose
3000	Grafana	UI (admin / admin)
3100	Loki	OTLP/HTTP log writes from the internal collector
3200	Tempo	Tempo's HTTP query API (Grafana talks here)
4317	Collector	OTLP gRPC receiver
4318	Collector	OTLP HTTP receiver — the plugin exports here
9090	Prometheus	Prometheus UI + OTLP write endpoint

Port 3000 collides with the self-hosted Langfuse and Latitude stacks. Port 4318 collides with the Jaeger v1 stack. Bring the LGTM container up on its own or remap the conflicts. docker-compose/all.sh up with no names starts only the light set (Phoenix, Jaeger, OpenObserve), and nothing in the repo starts every stack at once.

Spanmetrics — RED metrics for free

The bundled collector config at docker-compose/lgtm/otelcol-config.yaml enables the spanmetrics connector. The collector derives call counters and duration histograms (traces_spanmetrics_calls_total, traces_spanmetrics_latency_bucket) from every incoming trace — you get rate/errors/duration metrics in Prometheus without the plugin having to emit them from Python.

Dimensions are picked from attributes the plugin actually emits:

• llm.provider — split RED by model provider
• llm.model_name — split by specific model
• openinference.span.kind — split by LLM / TOOL / AGENT
• hermes.session.kind — split by session vs cron
• hermes.tool.outcome — error-rate panels

Add more in the YAML under connectors.spanmetrics.dimensions if you want to slice on additional attributes the plugin sets. Histogram buckets default to [5ms, 25ms, 100ms, 250ms, 1s, 2.5s, 5s, 10s, 30s, 60s] — tuned for LLM workloads.

What you'll see in Grafana

Traces (Tempo): Explore → Tempo. Search by service.name=hermes-agent or span name (agent, tool.*, api.*). Every span carries the OpenInference + GenAI attribute set and the plugin's hermes.* extensions.

Metrics (Prometheus): Explore → Prometheus. Query any hermes_* metric (hermes_session_count_total, hermes_token_usage_total, hermes_tool_duration_milliseconds_bucket, ...) or any traces_spanmetrics_* derived metric. With metric_name_compat: claude_code the session, token, cost and active-time counters arrive as claude_code_session_count_total, claude_code_token_usage_tokens_total, claude_code_cost_usage_USD_total and claude_code_active_time_seconds_total instead (metrics reference).

Logs (Loki): Explore → Loki. Query {service_name="hermes-agent"}. Every logger.info(...) call from hermes or the plugin lands here with the active span's trace_id / span_id automatically attached — clicking a trace_id in a log line jumps straight into the Tempo trace. See OTel logs for the full story.

Grafana dashboards shipped with the stack

The compose file mounts docker-compose/lgtm/grafana-dashboards/ into Grafana as a provisioned folder named hermes-otel (grafana-provisioning-dashboards.yaml; Grafana rescans it every 10 s, so a JSON file dropped there appears without a restart). It ships with:

Dashboard	Source	Needs
Agents: Hermes and Claude Code	this repo, generated by scripts/grafana/agents_dashboard.py; the two agents on one set of axes, read from the four shared claude_code.* metrics and service_name	the same switch; credits and sits beside the community dashboard below
Claude Code Metrics (Prometheus)	grafana.com 25255 by Rock Martel-Langlois, MIT (licence file beside the JSON), bound to the stack's prometheus datasource	metric_name_compat: claude_code (or both) in the plugin config, so Hermes exports claude_code.* names; metric_name_compat_session_id: true for the per-session panels. Claude Code itself pointed at the same collector shows up on the same panels

The walkthrough, with what each panel shows for Hermes, is Claude Code dashboards. The stack also starts Prometheus with --enable-feature=created-timestamp-zero-ingestion (PROMETHEUS_EXTRA_ARGS in the compose file): a zero sample at every cumulative series' OTLP start timestamp, so a counter a one-shot hermes -z run exported once still has an increase() for PromQL panels (the first-sample gap the bundled dashboard's adapter works around in code, #296). To add your own, drop a dashboard JSON into the folder with "datasource": {"type": "prometheus", "uid": "prometheus"} (the Tempo and Loki datasources are tempo and loki).

Dashboard

The bundled dashboard's LGTM adapter (and the observability skill with --source lgtm) reads all three signals back from the stack's own components: traces from Tempo (query_port, default 3200), metrics from Prometheus and logs from Loki. With the compose stack nothing needs configuring; when the pieces live elsewhere, name them:

  - type: lgtm
    endpoint: https://lgtm.example/v1/traces
    query_port: 443                        # Tempo's query API
    prometheus_url: https://prometheus.example
    metrics_match: '{__name__=~"hermes_.*"}'   # optional: keep a shared Prometheus's instrument list short
    loki_url: "off"                        # or https://loki.example; off when there is no Loki yet
    search_fetch: 500                      # optional: traces fetched per page (Tempo's limit is not newest-first)

Tempo's search API returns the first limit matches it finds, in block order, not the newest ones (Tempo 3.0.3: limit=3 over four traces skipped a newer trace in about half the calls). The adapter therefore asks for search_fetch traces per page and keeps the newest limit itself, so keyset paging stays complete; raise it for windows holding more than 500 traces. A native TraceQL query in the search bar replaces the structured predicates (the page marks them ignored) and the minimum duration is applied to the returned rows, because Tempo ignores minDuration next to a q.

Prometheus renames OTLP instruments (hermes.token.usage → hermes_token_usage_total, histograms grow _sum/_count); counters are read as raw cumulative samples over a range vector and the increases are taken by the adapter, gauges as the last sample per bucket. PromQL's increase() never counts a series' first sample, so it would read a one-shot hermes -z run as zero; the raw-sample read counts that first sample when the series started inside the window (#296). Logs use LogQL label-filter stages on the OTLP attributes Loki keeps (trace_id, hermes_session_id, scope_name, severity_number) plus |= for text; the logger list is a count_over_time by scope_name.

Multi-backend config

Fan out to LGTM alongside other backends:

backends:
  - type: lgtm
    endpoint: http://localhost:4318/v1/traces

  - type: phoenix
    endpoint: http://localhost:6006/v1/traces

capture_logs: true

With the above, traces go to both LGTM and Phoenix; metrics and logs go only to LGTM (Phoenix ingests traces only, so the plugin skips it for both automatically).

Production usage

The grafana/otel-lgtm image is for local development and quickstarts. For production, Grafana recommends deploying the components individually:

• Tempo as its own service (or use Grafana Cloud Traces)
• Mimir or plain Prometheus
• Loki as its own service (or use Grafana Cloud Logs)
• A properly-sized OTel Collector in front

Once you've split them, point the plugin at your production collector and set type: otlp (or keep type: lgtm for clarity — the semantics are identical).

Troubleshooting

"Container is unhealthy"

The bundled healthcheck probes /tmp/ready — the image writes that file when every sub-service is up. If the container stays (health: starting) forever, check the logs: docker logs hermes-otel-lgtm. Usually a port bind conflict.

"Grafana is up but the Tempo datasource says 'connection refused'"

The image needs ~30s for Tempo to finish starting — Grafana is up well before Tempo. Wait for docker ps to show (healthy) and retry.

"I see [hermes-otel] ✓ LGTM at ... but no traces appear"

From inside a Docker-networked hermes-agent, localhost:4318 means the agent's own container, not your host. Use host.docker.internal:4318 on Docker Desktop (macOS/Windows) or 172.17.0.1:4318 on Linux to reach the LGTM container running on the host. Alternatively, put both containers on the same docker network.

"Span-metrics histograms look coarse"

The spanmetrics connector's buckets are tuned for LLM latency (5 ms to 60 s). For very fast tools everything may fall into the lowest two buckets. Edit connectors.spanmetrics.histogram.explicit.buckets in otelcol-config.yaml. (This only affects the collector-derived traces_spanmetrics_* metrics, not the plugin's own histograms.)

See also

• OTel logs — the log pipeline that makes trace-id correlation work.
• Multi-backend fan-out — adding LGTM alongside other backends.
• Grafana Tempo — the traces-only standalone option (different type, different use case).
• docker-compose/lgtm/README.md — the on-disk README with the same setup walkthrough.