Skip to main content

Uptrace

Uptrace is an open-source OpenTelemetry backend that stores traces, metrics, and logs in ClickHouse and uses Postgres for metadata (users, projects, alerts). It has a polished UI, PromQL support on metrics, and per-project DSN ingestion tokens so you can multi-tenant one deployment across several apps.

Signals: traces + metrics + logs. Deployment: local (docker compose) or self-hosted. Cost: OSS (premium features require a license key).

One-command quickstart​

The plugin ships a trimmed compose that boots just what hermes-otel needs (ClickHouse + Postgres + Redis + Uptrace — no collector, no Grafana, no Mailpit):

cd ~/.hermes/plugins/hermes_otel
docker compose -f docker-compose/uptrace/docker-compose.yaml up -d

Wait ~20s for ClickHouse to finish applying migrations, then open the UI at http://localhost:14318 and log in with the seeded admin:

  • Email: admin@uptrace.local
  • Password: admin

Point the plugin at it — minimal config.yaml:

backends:
  - type: uptrace
    endpoint: http://localhost:14318/v1/traces
    dsn: http://project1_secret@localhost:14318?grpc=14317

capture_logs: true

The dsn value is the ingestion DSN for the seeded project (hermes-agent under org hermes-otel, token project1_secret). The plugin sends it as the uptrace-dsn header on every OTLP request; Uptrace uses it to route data to the right project.

Why a dedicated type: uptrace​

Uptrace authenticates with a custom uptrace-dsn header that type: otlp can't synthesise for you. Declaring type: uptrace asks the plugin to:

  1. Read dsn: (or dsn_env: → env var → UPTRACE_DSN / OTEL_UPTRACE_DSN).
  2. Set the uptrace-dsn request header automatically.
  3. Display ✓ Uptrace at <endpoint> in startup logs.
  4. Enable all three signals (supports_metrics=True, supports_logs=True).

Keep secrets out of config.yaml — use dsn_env and set the variable in your shell:

backends:
  - type: uptrace
    endpoint: http://localhost:14318/v1/traces
    dsn_env: UPTRACE_DSN
export UPTRACE_DSN='http://project1_secret@localhost:14318?grpc=14317'

Ports​

PortServicePurpose
5432PostgresUptrace metadata store
8123ClickHouseHTTP query API
9000ClickHouseNative protocol
14317UptraceOTLP/gRPC receiver
14318UptraceUI + OTLP/HTTP receiver — the plugin exports here

Port 14318 was picked specifically to avoid the standard 4318 used by LGTM / Jaeger / SigNoz, so Uptrace can run alongside them. The only likely conflict is port 5432 with a host-installed PostgreSQL — stop it or remap in docker-compose/uptrace/docker-compose.yaml.

What you'll see in the UI​

Traces: Navigation → Traces. Every attribute the plugin emits is searchable (OpenInference + GenAI + hermes.*). Group by llm.model_name, openinference.span.kind, hermes.session.kind, etc. Click a trace to see the full waterfall with the same attributes Phoenix and Grafana show.

Metrics: Navigation → Metrics. Query with PromQL over any hermes_* metric the plugin emits:

  • hermes_session_count_total — sessions started
  • hermes_token_usage_total — prompt / completion tokens by provider / model
  • hermes_tool_duration_bucket — tool execution histogram

Logs: Navigation → Logs. When capture_logs: true is on, every logger.info(...) from hermes or the plugin lands here. A line logged during a turn carries that turn's trace_id and span_id, hermes.session_id and hermes.log.attribution; lines the plugin cannot attribute to a turn (between turns, or from another thread while several sessions are active) arrive without trace ids. Click a log record's trace_id to jump into the corresponding trace. See OTel logs for the attribution rules.

Dashboard​

The bundled dashboard's Uptrace adapter (and the observability skill with --source uptrace) reads traces, metrics and logs back through Uptrace 2's /internal/v1 API. That API is authenticated with a user token, not the project token inside the DSN, so the backend entry needs one more key:

  - type: uptrace
    endpoint: http://localhost:14318/v1/traces
    dsn_env: UPTRACE_DSN
    user_token_env: UPTRACE_USER_TOKEN   # Settings → API tokens, or the seeded user_tokens entry
    # query_port: 443                    # when the API sits behind an HTTPS ingress
    # project_id: 1

Uptrace 2.1 and 2.0 spell that API differently (2.1 has one route per signal, /spans/{project}, /logs/{project}, /traces/{project}/{id}, with time_start/time_end and typed attribute keys such as hermes_session_id::str; 2.0 keeps everything under /tracing/{project}/… with time_gte/time_lt). The adapter asks the server once per process which one it speaks and remembers the answer; the dashboard's status shows it as api_dialect. The 2.1 spelling was taken from Uptrace's own UI requests and checked against 2.1.0-beta.5 (the version docker-compose/uptrace/docker-compose.yaml pins); the 2.0 spelling is the one recorded against 2.0.2 when the adapter was written.

Metrics go through MQL (metric=<name>&alias=$m plus $m group by model, sum($m), avg($m), max($m), count($m)); a counter only supports its per-interval sum, so the other aggregates fall back to it. Logs are the span store's log:* systems: --level narrows the systems, --trace, --session, --logger and --event become where clauses on _trace_id, hermes_session_id, otel_library_name and event_name (--events-only is where event_name exists), and --text uses Uptrace's search. The logger list is the otel_library_name attribute's values with counts, scoped to the agent's service so Uptrace's own log lines stay out of it.

Timestamps on this API are milliseconds and attribute keys are flattened with underscores; the adapter strips 2.1's type suffixes and maps the keys back to the documented dotted names.

Rotating the DSN​

The seeded token project1_secret is fine for local dev; change it before exposing the stack anywhere:

  1. Edit docker-compose/uptrace/uptrace.yml and change project_tokens[0].token.
  2. Restart the stack — seed_data.update: true in the config applies changes on boot.
  3. Update the dsn (or UPTRACE_DSN env var) in your hermes-otel config to match.

Or, do it from the UI: Settings → Projects → Edit → rotate token, then update the plugin config.

Multi-backend fan-out​

Uptrace plays nicely alongside other backends because its ports don't collide:

backends:
  - type: uptrace
    endpoint: http://localhost:14318/v1/traces
    dsn_env: UPTRACE_DSN

  - type: phoenix
    endpoint: http://localhost:6006/v1/traces

capture_logs: true

Traces fan out to both; metrics and logs go only to Uptrace (Phoenix ingests traces only — the plugin skips it automatically).

Troubleshooting​

"Uptrace is up but I see unauthorized in the agent logs"

The uptrace-dsn header is missing or its token doesn't match the seed config. Confirm the DSN matches project_tokens[0].token in docker-compose/uptrace/uptrace.yml.

"I see ✓ Uptrace at ... but no traces appear"

Uptrace batches writes to ClickHouse — first traces may take 10-30s to show up. If they still don't appear, check docker logs hermes-otel-uptrace for ClickHouse connection errors.

"ClickHouse container is unhealthy"

Usually a port bind conflict on 8123 or 9000. Stop any other ClickHouse running on the host (a SigNoz stack, for example) before bringing Uptrace up.

"Postgres container is unhealthy"

Host-installed PostgreSQL is almost certainly bound to 5432 already. Either stop it or remap the compose port: ports: ["5433:5432"] and adjust your firewall rules.

Production usage​

The shipped compose uses seed_data for local bootstrap — fine for dev, not for prod. For a production deployment you should:

  • Disable seed_data after initial bootstrap (or set it to a non-admin user / restricted org).
  • Change service.secret to a real secret (used for cryptographic operations).
  • Point ch_cluster.shards[].replicas[].addr at a real ClickHouse cluster (replicated or distributed), not the single-node container.
  • Put a reverse proxy (nginx, Caddy) in front and enable TLS.
  • Replace the seeded users / tokens with real ones via the UI.

See Uptrace's production docs for the full checklist.

See also​