Skip to main content

Limitations

Things to be aware of. Some are upstream constraints; some are plugin-specific trade-offs.

Full content capture is the default and large​

content_capture: full (the default since 1.11) writes the complete prompt (system prompt and every message) and the complete response onto every api.* span, once per attribute convention (gen_ai.input.messages and input.value; gen_ai.output.messages and output.value). A turn with several tool rounds repeats the whole prefix on each of its api.* spans, so storage grows with the square of the conversation length. The export batch shrinks to 64 spans per POST automatically in this mode. Set content_capture: preview to keep only the 1,200-character previews, or off to record no content. Backends cap attribute or span sizes differently and some drop oversized spans silently; check your backend's limits before relying on very long prompts.

Hermes itself caps the sanitised copy of the request it hands to plugins (8,000 characters per string, 1,000 past HERMES_PLUGIN_PAYLOAD_MAX_CHARS). The plugin reads the raw request_messages list instead, so captured messages are whole; a ...[truncated N chars] marker inside captured content means a payload only the sanitised copy carried.

Langfuse auth requires both keys​

The plugin builds Langfuse's Basic Auth header from the public and the secret key, so it needs both. A backends: entry with only one is skipped with a warning. In env-var mode (no backends: list) Langfuse is passed over without a warning and the next backend in priority order is tried.

This is deliberate: Langfuse would answer 401 anyway, and a skipped entry is easier to spot than failed exports.

gRPC only for generic otlp entries​

OTLP over HTTP with protobuf bodies is the default for every backend. protocol: grpc (since 1.32.0, #469) is accepted only on a generic type: otlp entry; any other type with protocol: grpc is skipped with a warning. headers_helper works only over HTTP. See gRPC and TLS.

A session is a group of traces, not one trace​

Each user turn is its own trace, rooted at an agent / cron span; a conversation of ten turns is ten traces sharing session.id / gen_ai.conversation.id (and hermes.turn.number on the root). This is deliberate: the OTel SDK exports a span only when it ends, and a gateway session can stay open for hours, so a session-long root would appear late or never and would break tail sampling and trace timeouts. Phoenix, Langfuse, LangSmith and Weave all group by that attribute. on_session_finalize records the session's size (hermes.session.turns, hermes.session.duration) and on_session_reset links a replacing session to the old one; the dashboard's Sessions view groups turns the same way. The turn counter is process-local: a gateway restarted mid-session numbers the next turn 1 again. A resumed session (hermes -r) continues hermes.turn.number from the user prompts in the conversation Hermes passes, so after a context compaction has folded earlier prompts into a summary, a resumed process can number its turn lower than the true count (a turn in the same process never goes backwards).

Single session tracked in memory​

The SpanTracker's parent-stack state is in-memory. If Hermes restarts mid-session, any currently-open spans are lost — not exported, not resumable.

The orphan-span sweep is the partial fix for this: when a new turn fires a hook after a crash, the sweep finalizes stale roots with hermes.turn.final_status=timed_out. But:

  • Buffered-but-not-yet-exported spans in the BatchSpanProcessor queue of the old process are lost (hard crash: atexit doesn't run).
  • Spans that were opened but never ended (and not yet swept) from the old process are just gone — the new process can't resurrect them.

Live with this. Production tracing stacks have the same trade-off; the alternative is persisting span state to disk, which has its own failure modes.

Delegated child agents (delegate_task) are modeled as subagent.* spans, and the child's own root span rejoins the parent trace so a multi-agent run is one connected tree. This relies on the delegation span being reachable from the child's on_session_start — which works because Hermes runs delegated children in the same process (on background threads), where the plugin's in-memory sub-agent registry holds the live span.

If a future Hermes version runs delegated children in a separate process, that registry won't hold the live span. The plugin then degrades gracefully: the child root attaches an OTel span link to the delegation span's SpanContext (when available) and is tagged with hermes.subagent.parent_session_id for correlation — but the child becomes its own trace rather than nesting in the parent. True cross-process trace-context propagation (injecting the parent context into the child process) is out of scope for now; open an issue if you need it.

Delegation spans outlive their turn; timeout details are not reported​

Hermes runs a top-level delegate_task in the background and fires subagent_stop after the parent's turn has ended, so a subagent.* span ends after the turn root it nests under. Trace views draw it past the root's end; that is the delegation's real duration, not a clock problem. A delegation whose stop never arrives is closed after subagent_span_ttl_ms (24 hours by default) with hermes.subagent.swept=true.

When a child times out, Hermes records timeout_phase (before or after the first LLM call) and last_event_age on the delegate tool's result, but the subagent_stop hook does not carry them, so the span has hermes.subagent.status=timeout without them.

A kanban run's trace has two caveats​

One run of a card is one trace (the worker joins the dispatcher's kanban.claim through TRACEPARENT), with two edges:

  • The environment window. The plugin sets TRACEPARENT in the dispatcher's process environment between kanban_task_claimed and on_kanban_worker_spawned (inside Hermes's dispatch lock, a few milliseconds). A dispatcher inside a gateway shares that environment with its chats, so a terminal command that a concurrent turn spawns in that window inherits the variable; an OpenTelemetry-instrumented program it runs would join the kanban run's trace.
  • Late arrivals. A worker run can outlast a backend's trace-assembly window (Tempo, Jaeger), and a one-shot hermes kanban dispatch pass forgets the claim when it exits, so kanban.worker_exit from a later pass is a trace of its own (correlated by hermes.kanban.task_id / run_id). A worker killed with kill -9 never exports its root, so its spans sit in the run's trace under a parent that never arrives.

API error capture depends on the api_request_error hook​

Failed provider API requests (rate limits, timeouts, 5xx, network errors) are captured through the api_request_error hook: the api.{model} span ends ERROR with a recorded exception and retry metadata. On older Hermes builds that don't fire that hook, a failed request's span is instead finalized by the orphan sweep on the next turn and ends OK — so on those builds failures remain invisible until you upgrade Hermes.

When every attempt of a turn fails, Hermes skips on_session_end, so the root is closed later (finalize, the sweep, or the session's next turn) as failed; in a gateway that can be up to root_span_ttl_ms after the failure if the session gets no next message (#467).

A hard crash before the error hook fires (or before the next sweep) can still lose the in-flight span, the same buffered-span trade-off described above.

Note the deliberate asymmetry: only API-level failures map to ERROR. Tool timeout/blocked outcomes stay OK so they don't inflate error rates.

Sampling is head-based only​

ParentBased(TraceIdRatioBased(rate)) makes the sampling decision at the root. There's no tail-based sampler that boosts on error or keeps all traces above a duration threshold.

Why: tail-based sampling requires buffering spans until the trace is complete, which roughly doubles memory use and adds a full trace of latency to every export.

The standard answer is to run an OpenTelemetry Collector in the middle with a tail_sampling processor. The Collector can fan-out to all your backends and apply tail-based policy centrally.

Weave is trace-ingest only​

W&B Weave's documented OTLP endpoint is /otel/v1/traces, authenticated with the wandb-api-key header and routed by wandb.entity / wandb.project Resource attributes. The dedicated type: weave backend therefore disables OTLP metrics and logs by default.

If you need hermes.* metrics or OTel logs next to Weave traces, fan out to Weave plus a metrics/logs-capable backend such as SigNoz, LGTM, or OpenObserve. The bundled dashboard also does not query Weave; use Weave's UI for W&B traces and point the dashboard at a local backend.

Because wandb.entity and wandb.project are Resource attributes on the shared TracerProvider, one Hermes process can route to one Weave project at a time. Multiple configured Weave entries must agree on those values.

Logs are attributed exactly or not at all​

A log line gets a turn's trace and span ids from one of three exact sources: a span current on the logging thread, Hermes' own per-thread session tag on the record, or the single session with a turn in flight. Hermes sets its session tag on the agent's turn thread only (about a quarter of agent.* lines in a measured gateway log, almost none of tools.*, gateway.* or platform-adapter lines), so on a gateway serving several users at once, lines from tool execution and housekeeping threads stay unattributed rather than being pinned to the wrong user. hermes.log.attribution on each record says which source applied. Lines logged between turns never carry a trace id. See OTel logs.

Log redaction covers secrets, not personal data​

Exported log records and live-store rows are redacted with Hermes' agent.redact (credential shapes, auth headers, URL userinfo, vault values) or the plugin's smaller built-in set outside Hermes. Neither recognises names, e-mail addresses or free-text personal data; use a Collector processor or content_capture for those. Masked values keep a head and tail by design, so a leaked key can still be identified.

Debug log has no rotation​

With HERMES_OTEL_DEBUG=true the plugin appends to $HERMES_HOME/plugins/hermes_otel/debug.log (~/.hermes by default) with no rotation and no size cap.

Deliberate: the debug log is meant for troubleshooting, not routine operation. If you keep it on, rotate the file with logrotate or delete it now and then.

Metrics labels are curated​

The plugin deliberately does not put user IDs, session IDs, tool arg values, or similar high-cardinality values on metrics. Labels are restricted to low-cardinality values (model, provider, platform, profile, tool name, outcome, choice, error class) so a Prometheus-style time-series database does not fill up with series. Open-ended labels fold into other after metrics_label_limit distinct values (see Metrics).

If you need high-cardinality breakdowns, use traces (where every attribute is fine), not metrics.

Host metrics are a sampler, not an accountant​

host_metrics reads the Hermes process tree and the host on a fixed interval. Consequences:

  • Per-tool GPU numbers are coincident load. hermes.tool.gpu.utilization.* is the whole host's GPU busy ratio while the tool ran. On an inference host that is mostly other work (the model is idle during a tool call), so read it as context, not as the tool's own consumption. CPU, by contrast, is the process tree and is attributable.
  • Short tools omit the attributes. A tool that starts and finishes between two samples has no window to average; lower host_metrics_interval_ms if you need sub-second tools covered.
  • External inference servers are excluded. Only children of the Hermes process count toward process.cpu.utilization; a model server in its own container is host load (system.cpu.utilization, hw.gpu.*), by design.
  • No GPU on macOS / without an SDK. The GPU series are simply absent; the CPU series still work.

Cost is Hermes's estimate, and only when Hermes has a price​

The plugin keeps no price table. Each api.* span's cost comes from Hermes's own estimate_usage_cost, the function behind state.db's estimated_cost_usd and /usage, so it matches what Hermes shows and inherits its limits:

  • Mostly estimates. hermes.cost.status says estimated for a price-table or models-API figure (OpenRouter's is the models API price times the tokens, not the invoice) and actual only when a provider profile reports the charge.
  • No price, no number. An unknown model (unknown) or a subscription-included route (included) gets the status but no hermes.cost.usage and no metric point, never $0. A turn mixing priced and unpriced calls is partial on the root with no total.
  • Auxiliary calls are priced on their own spans. Calls Hermes makes on the agent's behalf (title generation, compression, MoA, vision, …) arrive through post_auxiliary_call on Hermes builds that fire it. They are priced the same way and counted as hermes.turn.auxiliary.* on the root, beside the turn total rather than inside it. On older builds they are not seen at all.
  • Older Hermes builds without agent.usage_pricing get no cost attributes at all.

Phoenix and Langfuse can also price spans server-side from the token counts; SigNoz can with a derived metric.

Python ≥ 3.9​

The plugin uses from __future__ import annotations and some 3.9-only stdlib features. 3.8 is EOL and not supported.