Skip to main content

Multi-backend fan-out

hermes-otel can send the same span tree to several backends in parallel. Every backend entry gets its own BatchSpanProcessor (independent worker thread, independent queue), so a slow or unreachable collector can't block the agent or starve the others.

Why fan out?​

  • Side-by-side evaluation — compare Phoenix and Langfuse on the same real traffic.
  • Operational vs. LLM-specific UIs — ops team watches Jaeger / Grafana Tempo; product team watches Langfuse.
  • Local + cloud — mirror everything to a local Phoenix for development and a Langfuse Cloud for retention.
  • Tenant / project isolation — same traces, different ingest paths with different auth headers.

Configuration​

Multi-backend is YAML-only. When backends: is set and non-empty, single-backend env-var detection is skipped entirely — the YAML takes full control.

# ~/.hermes/hermes_otel.yaml
backends:
  - type: phoenix
    endpoint: http://localhost:6006/v1/traces

  - type: langfuse
    public_key_env: LANGFUSE_PUBLIC_KEY
    secret_key_env: LANGFUSE_SECRET_KEY
    base_url: https://cloud.langfuse.com

  - type: signoz
    endpoint: http://localhost:4328/v1/traces
    ingestion_key_env: OTEL_SIGNOZ_INGESTION_KEY

  - type: jaeger
    endpoint: http://localhost:4318/v1/traces

  - type: tempo
    endpoint: http://localhost:4358/v1/traces

  - type: otlp
    name: honeycomb
    endpoint: https://api.honeycomb.io/v1/traces
    headers:
      x-honeycomb-team: ${HONEYCOMB_API_KEY}

The startup log prints one line per backend: its name (else the type's display name), the traces URL, and which signals it gets when metrics are off:

[hermes-otel] ✓ Phoenix at http://localhost:6006/v1/traces (traces only)
[hermes-otel] ✓ Langfuse at https://cloud.langfuse.com/api/public/otel/v1/traces (traces only)
[hermes-otel] ✓ SigNoz at http://localhost:4328/v1/traces
[hermes-otel] ✓ Jaeger at http://localhost:4318/v1/traces (traces only)
[hermes-otel] ✓ Tempo at http://localhost:4358/v1/traces (traces only)
[hermes-otel] ✓ honeycomb at https://api.honeycomb.io/v1/traces

An entry that cannot be resolved (a missing key, an unknown type) is logged and skipped; the others still start.

Isolation guarantees​

Each backend runs in its own worker thread with its own bounded queue:

  • A single slow collector backs up only its own queue — the others keep draining.
  • If one backend is completely unreachable, the plugin logs the failed POST attempts and drops the oldest spans in that queue when the buffer fills. The others continue.
  • At init the plugin opens one TCP connection to each backend endpoint on a background thread and logs ✓ <name> reachable at host:port (3 ms) or ⚠ <name> unreachable at host:port (Connection refused), so a dead backend is said so at startup rather than 30 s later inside the export worker (#481). It runs once, sends no bytes and never gates an export; a proxy that accepts the connection passes even when the service behind it is down. hermes otel status probes on demand and prints the same verdict per backend. probe_backends: false turns the startup probe off.
  • No backend's latency is on Hermes' hot path. span.end() is still a queue push.

Per-backend signal overrides​

By default, every backend receives trace exports. Metrics and logs follow the type: a type whose backend takes no OTLP metrics (Phoenix, Langfuse, Jaeger, Zipkin, Tempo and the other traces-only types) gets no metrics exporter, and logs go only to types that store them. The signal table lists every type's defaults. Use traces: true|false, metrics: true|false, or logs: true|false to override a signal per backend:

backends:
  - type: otlp
    name: collector
    endpoint: http://collector:4318/v1/traces
    metrics: true
    logs: true

  # Dashboard/query-only Tempo entry: visible to the OTel Traces tab, but it
  # does not receive duplicate span exports because traces are disabled here.
  - type: tempo
    name: tempo-query
    endpoint: http://tempo:3200
    query_port: 3200
    traces: false
    metrics: false
    logs: false

trace: false is accepted as a friendly alias for traces: false; use the plural spelling in new configs to match metrics and logs.

The dashboard's Settings tab shows each entry as a card that opens the backend's web UI. Add ui_url: https://grafana.example.com to an entry when the UI is not on the OTLP endpoint's origin (Grafana for lgtm / tempo, a Jaeger or SigNoz container on its own port); see the dashboard docs.

Forcing a signal on or off:

backends:
  - type: jaeger
    endpoint: http://localhost:4318/v1/traces
    metrics: true   # Force-attempt metrics export (will probably 404; useful for testing collectors-that-happen-to-be-jaeger)

  - type: otlp
    name: traces-only-sink
    endpoint: http://sink:4318/v1/traces
    metrics: false  # Explicit opt-out

Secrets​

Prefer env-var references over inline values. Two forms:

  1. *_env: keys — for first-class backend types:

    - type: langfuse
      public_key_env: LANGFUSE_PUBLIC_KEY
      secret_key_env: LANGFUSE_SECRET_KEY

  2. ${VAR} interpolation — inside headers::

    - type: otlp
      endpoint: https://api.honeycomb.io/v1/traces
      headers:
        x-honeycomb-team: ${HONEYCOMB_API_KEY}

The config file is plain text: even when it is not in git (the plugin repo ignores config.yaml), any editor plugin, pair-programming session or backup system can read it. Keep secrets in env vars. For a credential that expires while Hermes runs, use a headers_helper instead.

LangSmith and fan-out​

LangSmith is an ordinary type: langsmith entry (its OTLP ingest at /otel/v1/traces), so it fans out beside every other backend. Before 1.26 it was a separate run-tree client that LANGSMITH_TRACING=true selected instead of the list; see LangSmith for the migration.

Debugging fan-out​

Enable debug logging:

export HERMES_OTEL_DEBUG=true

Each backend's batches show up in ~/.hermes/plugins/hermes_otel/debug.log as export <backend>: N span(s) -> SUCCESS|FAILURE, with the SDK's reason for any failure on an [sdk] … line. See Debug logging.