Dashboard tab
Installing the plugin adds an OTel tab to the Hermes web dashboard (hermes dashboard). It has two data sources:
| Source | What it shows | Needs |
|---|---|---|
| Live (in-process store) | The most recent turns, spans, metrics and logs the plugin recorded on this machine, within seconds of happening | nothing: on by default |
| Backend | Historical trace search and trace detail queried from one of your configured backends | a backend whose type has a dashboard adapter |
The plugin's own dashboard code lives in hermes_otel/dashboard/ (a FastAPI router mounted at /api/plugins/hermes_otel/ and a pre-built bundle registered as the tab). It is installed with the plugin; there is nothing extra to enable.
The live store
The gateway process writes every finished span, every metric point and (when logs.capture or logs.events.enabled is on) every log record at or above logs.live_min_level (default INFO, separate from the OTLP logs.level) and every event into a small SQLite file, $HERMES_HOME/hermes_otel_live.db. Rows are buffered and committed in one transaction by a background thread every 250 ms or 64 rows, so a hook never waits on the disk; the dashboard process, which is separate, reads the same file and sees a row within that interval. A commit that fails (the file locked by another process, a full disk) does not lose the batch: the rows are held, up to 5,000 of them, and committed on the next attempt, with one line in the plugin debug log. The file is created owner-only (0600, and an older 0644 file is tightened on open; the -wal / -shm sidecars inherit the mode) because it holds full prompts, responses and tool I/O. The store is a bounded buffer of recent activity, not a record:
| Key | Default | Meaning |
|---|---|---|
dashboard_live | true | Write to the live store at all. Set false to disable the Live source. |
dashboard_live_max_spans | 1000 | Rows kept per kind (spans, metrics, logs); the oldest are dropped. Host samples (process.cpu.*, system.cpu.*, hw.*, recorded several times a second when host_metrics is on) have their own cap of 200 rows so they can never evict the turn's token and cost points. |
dashboard_live_retention_hours | 168 | Rows older than this are dropped as well. 0 keeps rows until the row cap evicts them. |
HERMES_OTEL_LIVE_DB (env) | $HERMES_HOME/hermes_otel_live.db | Where the file lives. Both processes must resolve the same path. |
The caps are enforced every 64 writes by the gateway and once more whenever either process opens the file, with the values of that home's config, so a store fed only by one-shot hermes -z runs (fewer than 64 rows each) stays bounded too. With the default cap, 1,000 spans is roughly thirty turns of ten tool calls each; raise dashboard_live_max_spans for a busy gateway, or configure a backend for history.
Every key also has a HERMES_OTEL_* environment variable, see Environment variables. For history beyond the buffer, configure a backend.
Metrics in the live store carry the same names as the OTLP instruments (hermes.token.usage, gen_ai.client.token.usage, …), so a name in the Metrics tab's explorer means the same thing for the Live source and for a backend; a Prometheus-style backend shows them as hermes_token_usage.
The same store is what the bundled skill's terminal tool reads, so an agent in the chat can list turns, draw a trace tree or total up cost without the dashboard: see The observability skill.
Rows carry indexed columns (trace id, session id, span name, status, start and end time, log level, logger, metric name) so the tab filters, groups and buckets in SQLite and the browser receives one page of results. The file's schema is versioned (PRAGMA user_version): a file written by a plugin release before 1.9 (v1, a JSON blob only) is recreated on first open; a file from 1.9 to 1.18 (v2) has the same layout as the current v3 and is kept and stamped in place, so a dashboard on the current release reads a file an older gateway is still writing; a file stamped by a newer release than the one reading it is opened as is and never re-stamped. A file that cannot be opened at all (corrupt, wrong permissions) is reported by GET /live/status as live: false with the error, not as an empty store.
A trace whose root span has not finished yet (a turn in flight) is listed as partial: the row describes its earliest stored span, the totals are the api.* subtotal so far, and partial: true is on the row; it becomes a normal row when the root lands. A turn is status=ok only when none of its spans failed. The model filter matches the model attribute of a span (a model name quoted inside a prompt is not a match), and % / _ in the name and text filters are literal.
Tabs
- Live: the newest turns of the last hour (up to 50) from the same
/live/tracesquery the Traces tab uses, with cost, tokens, turns, spans and errors for that window, an activity sparkline built from the rows, and one card per turn. Tokens and cost are counted once per turn (the root span's totals); a total that was not recorded reads as unknown, never as0or$0. A turn that delegated shows its own figures plus the delegations' on its card (+12.4k tok · +$0.03 · 2 sub-agents), and the window's tokens and cost include both (cost per agent). A turn whose root span has not finished yet is marked in progress and its detail refreshes until the root lands. Opening a card shows the trace detail and puts?trace=<id>in the URL. - Traces: one search bar for every source: status, kind, tool, model, session id, minimum duration, free text, trace id and lookback, plus the backend's native query and service as advanced fields. Every field is a URL key (
?tab=traces&status=error&kind=tool&tool=terminal&model=…&session=…&mindur=250&text=…&q=…&service=…&roots=0&lookback=24). The selected source's/statusentry says which fields its adapter honours (see the table below); a field the adapter ignores is marked ignored on the bar, one applied after the fetch is marked after fetch, and the result line lists the fields the server reported as ignored. Results are keyset-paged, newest first, for both sources: Newest / Newer / Older with the cursor stack inbefore=, so a page never shifts while new turns arrive and a refresh lands on the same page. The newest page follows new turns (Pause stops it); an older page is a fixed window. One card for both sources: root span, kind, model, span count, duration, tokens, cost, time, with the input/output previews and tool badge when the source has them. The Turns view lists one card per turn; the Sessions view groups turns by session id with per-session turns, spans, errors, tokens, cost and tool calls: from the live store's/live/sessions(the bar keeps only the lookback, since the session query takes no other field; rows refresh every 10 s; "show more" pages the list), or grouped client-side from the backend's current page of results, in which case every row carries a this page only badge because it is not a whole-session aggregate. A session link from a trace header or a log line opens that one session. Backend cards show the whole-trace span count when the adapter knows it. Opening a span shows a summary built for its kind: a turn's user message, final response as markdown, and its tool calls with outcomes and commands; an LLM or API call's prompt as a conversation with a badge per role (system prompts collapsed), tool calls as cards and the response as markdown; a tool's command, arguments and result as key/value rows with the output text in full; skills, approvals, sub-agents and cron runs with their own facts. Every rich value has a structured / raw switch (JSON, message lists and markdown; one setting for the page, remembered per browser), and the full attribute table below formats counts, durations, booleans, lists and links by key, with long content collapsed behind its size. The status filter applies to the root span on every backend (the live store'serrormeans any span of the turn errored, so itsstatus=errorcan list a turn whose tool failed while the turn succeeded). The kind and tool filters widen the search to every span. On a backend, a kind makes each card the matched span (atool.terminalcard with that tool's input, output and duration); a tool filter alone (or the kind other, which has no name prefix) keeps the card the trace's root, with the matched span as the card's attributes, the reason the turn matched. On the live store it stays the turn's card. - Trace detail (both sources): a header with the requested model (and the served model when it differs), tokens in/out/reasoning/cache-read (from the root span, else the sum of the
api.*spans only), cost or "no pricing data", tools used, outcome, a session link and a copy button for the trace id, an in progress badge while the root is unfinished, a truncated badge when the backend returned only the first spans, a by agent table when the turn delegated (below), plus an "open in …" link to the trace in the backend's own UI, where the adapter can build one (see the table below). Under it, Spans (the waterfall under a time axis, collapsed by default with Expand all; every row opens with Enter or Space to a summary for its kind: prompt and response as a conversation for LLM spans, command, arguments and result for tools, the decision for approvals, a generic summary for anything else, with every attribute grouped by prefix below), Logs (the log lines that carry this trace id, asked for the trace's own window ± 5 minutes) and Raw (the spans as JSON, with a copy button). - Metrics: for the chosen source and range, tiles (tokens, cost, model calls, tool calls, cache-read share), tokens and cost over time, tokens by type, calls by model, tool durations, approvals, CPU and GPU utilisation when recorded, and an explorer that charts any instrument the source holds, grouped by any attribute, with sum/count/avg/max/last; it runs when you press Query, and the range, instrument, grouping and aggregation are URL keys (
?tab=metrics&range=24&inst=hermes.token.usage&group=model&agg=avg). The curated panels find each instrument through the catalogue's canonicalotlp_name, so a backend that spellshermes.token.usageashermes_token_usagestill lights them up. Buckets are computed server-side; the browser never sees raw points; a bucket without data is a gap in the line, not a zero; hovering a chart shows the bucket's values. A missing cost series reads "no pricing data", never$0; the "Tool calls" tile is one point per call on the Live source and is labelled as duration samples on a backend. - Settings: every plugin setting with its effective value, where it came from (
envfor aHERMES_OTEL_*variable,filefor the config file,default) and a one-line description, grouped by topic and searchable, with values changed from the default marked. Invalid values in the file or the environment are shown as ignored rather than silently dropped. Backends appear as cards. The backend's name opens its web UI in a new window: fromui_urlon the entry when set, otherwise derived fromendpointwhere the UI shares the OTLP origin (the tooltip says which; a port is never guessed, so a Jaeger or SigNoz entry on:4318needsui_urlorquery_port). The signal pills separate what the backend type accepts from what the entry exports:on(accepted and exported),off(accepted, switched off in the entry),n/a(the type does not accept the signal) andforced(exported although the type does not accept it). Below them: whether this dashboard can query the backend and for which signals, the metric temporality its reader uses and which rule set it (entry, type preset, top-level, SDK default), query-only keys such asquery_portorproject_name, per-backend headers, a link to the type's docs page, and how each credential is supplied (inline,${VAR}reference,<field>_env, or a fallback variable), never the credential itself. A Raw YAML view shows the config file as written and an Effective config rendering of every setting with a source comment per key, ready to paste intohermes_otel.yaml; an Environment view lists every variable the plugin reads, set or not. Credential values are masked until "show secrets…" is confirmed (the choice is never stored); a find box applies to the Raw YAML and Environment views too. The tab is read-only: edit the file, then reload. Values are resolved by the dashboard process; a gateway started before an edit keeps its old values until it restarts; the header shows the short fingerprint of the resolved configuration, the value the plugin logs when it builds its pipeline, so a match says the gateway runs what the tab shows; flatcapture_logs/log_*spellings in the file are marked as deprecated next to the unknown keys (thelogs:block is the preferred form), and the config file's origin is judged against the profile's own home. - Logs: server-side search over the chosen source with level, logger (picked from the loggers seen), session id, trace id, text, lookback, an events only toggle and an event name box that keep just the plugin's structured events (the name is a badge on the line, and clicking a badge filters to that event; see the log events reference); relative or absolute times; a trace id opens that trace in the Traces tab. Above the list a severity summary counts the page by level, with a sparkline of lines per time bucket (errors and warnings in colour). Every line expands in place: the record's time, logger and severity number, links to its session and trace, its span id, a ±30 s around this line link that reloads the tab as a context window around that instant, copy JSON for the whole record, the body as it was logged, and the attributes grouped by prefix (Event, Hermes, GenAI, Exception, Code location, Other) with a stacktrace as a block. follow turns the page the other way up (oldest first, newest at the bottom) and keeps the end in view as lines arrive; wrap toggles long lines; copy link copies the page's URL, which carries the source and every filter. A session card in the Traces tab links to that session's logs, and a trace's Logs sub-tab shows the same expandable rows with an "open in Logs tab" link. Pages are keyset-paged, newest first: the next page is "the N rows older than the oldest row shown" (
before_ns), so lines arriving while you read never shift an older page and every backend answers it with its native time bound. The newest page follows new lines every few seconds; an older page is a fixed window until you go back to the newest. Page size is 100–1000. Every filter, the page size and the cursor are in the URL (?tab=logs&source=<backend>&level=30&logger=…&session=…&trace=…&text=…&lookback=24&events=1&event=hermes.tool.call¢er=<unix ns>&win=30&size=500&before=<unix ns>; a pasted link'ssourcewins over the browser's remembered one), so a refresh, the back button or a pasted link lands on the same page with the same filters. A log line carries a trace id and session id when the plugin can attribute it exactly: a span current on the logging thread, Hermes' own session tag on the record, or the one session with a turn in flight;hermes.log.attributionon the record says which. With several sessions active at once an unattributable line stays unattributed rather than guessed (details). Rows use the OTel severity spelling (WARN,FATAL); the level filter's values are the Python scale the API takes (20= INFO and up,30= WARN and up,40= ERROR), shown next to the OTel severity number they correspond to. follow stops following when you scroll up and offers "jump to newest". The whole cursor stack is inbefore=(comma-separated), so a refresh on page 3 is still page 3.
The tab keeps its state in the URL, so a refresh, the back button or a pasted link lands on the same view. tab, source, trace and session are shared by every tab; the other keys belong to one tab (Traces: the search fields, view, before; Logs: its filters, size, before; Metrics: range, inst, group, agg) and leave the URL when you switch tabs. Pages stay mounted while hidden, so filters, results and an open card survive a tab switch; polling pauses on the hidden tabs and on a hidden browser tab. A page that hits a rendering error shows the error with a "try again" button instead of a blank tab.
Successful MCP keepalive pings are hidden by default in every list (a switch shows them). Absolute times are shown as YYYY-MM-DD HH:MM:SS in the browser's timezone, named once in the tab's header.
Cost and tokens per agent
A turn's root span carries only the parent agent's own calls; each subagent.* span carries its child's whole subtree (hermes.subagent.usage.*, hermes.subagent.api_call_count, hermes.subagent.cost.*, see sub-agents). The tab therefore reports a delegating turn as the root's figures plus its top-level delegations, never the root plus every span. A delegation another delegation made (a nested orchestrator's children) is already inside its parent's row.
- Trace detail (every source, from the trace's spans): the header's tokens and cost read (main), and a by agent table lists
main(the parent's model, tokens, API calls and cost), one row per top-level delegation (its role, the child's model, tokens, calls, cost; a failed or stalled child in red), and the whole run. The run's cost is an amount only when every row has one, by the same rule as a turn root. - Live and Traces cards, Live totals, Sessions (live store): the row carries the delegations as
subagents, the card adds+tokens · +cost · N sub-agents, the Live tab's tokens and cost tiles include them, and a session row shows+tokens sub-agents. A backend's search results are root spans only, so backend cards show the turn's own figures; open the turn for the table. - No split for a turn still in flight or a run whose root has no totals of its own (a kanban run rooted at
kanban.claim): those totals are already the sum of everyapi.*span, the children's included.
Choosing the source
Every tab has a source selector: Live (in-process) plus one entry per configured backend that can serve what the tab shows. Entries whose type has no adapter, or that cannot serve the tab's signal (metrics, logs), are not options (the host's select cannot disable one); a note next to the selector counts them and names the reason on hover. The choice is remembered per browser and a pasted link's source wins. A failed request shows the server's detail and what kind of failure it was (not found, credentials, configuration, backend down, dashboard unreachable) with a hint. The API takes the same choice as a backend=<name or type> query parameter on /status, /traces/search, /traces/{id}, /metrics/* and /logs/*.
Every backend route answers a failure with the same body, {"detail": "<message>", "kind": "<kind>"}, and a status that follows the kind: request (400: an unknown backend name, an invalid metric name), config (503: no adapter for the type, a missing credential or URL, an entry that cannot be set up such as project_id: abc), auth (503: the backend rejected the key), not_found (404: only on /traces/{id}, when the backend has no such trace) and backend (502: unreachable, a timeout, a 5xx, a non-JSON answer, or an answer the adapter could not read). A backend is contacted once per request: nothing is retried after a network failure.
Without a selection (or a parameter), query_backend: <name or type> in hermes_otel.yaml chooses the default backend, else the first configured one with an adapter that sets up cleanly. The config file is parsed once per change (its path, modification time and size) and the adapter for each entry is kept for as long as the file is unchanged, so an edit is picked up on the next request and the adapters' own caches (Phoenix's project id, OpenObserve's schema, Uptrace's API dialect) hold between requests.
A dashboard that serves several profiles (?profile=<name>) reads each profile's own config and live store: the tab appends the page's profile to every plugin request (the host only does that for its own route families), so the Live source, the backend list and the Settings tab all follow the profile of the page.
Which backends can the tab query?
| Backend type | Trace search and detail | Metrics | Logs | Notes |
|---|---|---|---|---|
phoenix | yes | no | no | GraphQL on the Phoenix port; honours project_name and never substitutes another project. Roots-only is rootSpansOnly on Phoenix builds that still take it and the parent_id is None filter predicate on Phoenix 20+, which dropped the argument (the schema is introspected once per query URL); a widened search shows one card per trace: named after its newest matching span for a kind, the trace's root (one more query for the page's roots) for a tool filter alone, like the other adapters |
openobserve | yes | yes | yes | SQL over the traces, metrics and logs streams; needs user and password. Every trace links to the stream's trace-details page in OpenObserve's UI (ui_url overrides the origin). Counters arrive cumulative and are shown as increases per bucket; events filter on the event_name column (an empty answer, not an error, until the stream has ingested its first event: OpenObserve rejects a query on a column it has never seen), span status from the status_code column (1 ok, 2 error; verified on the traces stream 2026-10-09), and flattened attribute names (hermes_log_attribution) come back dotted |
langfuse | yes | no | no | Public API (limit capped at 100 per page); the card's model, token totals and session come from the list item's metadata.attributes (the root span's attributes); the kind filter widens over /api/public/observations by type (tool. → TOOL, api./llm. → GENERATION, agent → AGENT) and TOOL observations get the exporter's tool. prefix back; the trace's timestamp sits within a few tens of ms of the root span's start; a synthetic root marked synthetic: true holds the observations together. Langfuse v4 (its default events_only mode has no trace endpoints, #246) is read through /api/public/v2/observations instead: the AGENT root observations are the turns, the kind filter lists the typed observations one card per trace, session and an error level filter server-side, and each page's token totals, cost, observation count and model come from one /api/public/v2/metrics query; the detail is the trace's observations (no input, output or per-span usage on v4, the trace totals go on the root). The API is probed once per query URL (query_api: v3 / v4 pins it) |
signoz | yes | yes | yes | Query-builder API (/api/v4/query_range) for all three signals; needs api_key. Counters are read as the latest cumulative value per exporting process and step (service.instance.id joins the grouping as a series key) and the increases are taken by the adapter, because SigNoz's own increase over-counts one-shot turns that share a step, and the query opens on the step grid (SigNoz stamps each step's value at the step's start, aligned to the epoch, so a window opening mid-step would otherwise miss its first samples); the list's numeric span attributes (token counts, turn number) are requested as float64 and the row's start time comes from the list entry; logs filter on trace_id, hermes.session_id, the logger (scope_name), severity and body text; events on the event.name attribute |
uptrace | yes | yes | yes | Uptrace 2.x /internal/v1 API in both of its spellings (2.1's one route per signal, 2.0's /tracing/ routes; probed once per process); needs a user token (user_token_env), not the DSN's project token. Roots are filtered on the server on 2.1 (_parent_id = ""), so pages are exact; on 2.0 they are kept client-side from a wider fetch. Metrics via MQL ($m, sum($m), avg($m)…) with group by; a counter's cumulative value is read per exporting process (service_instance_id joins the grouping as a series key) so turns that share one of Uptrace's intervals do not add their running totals together; logs are the span store's log:* systems, filtered on _trace_id, _parent_id (the span a record was written in, for the trace detail's per-span logs), hermes_session_id, otel_library_name, event_name, level and text. A widened search (kind or tool filter) is built from span rows: several spans of one trace share a page, so the adapter fetches four rows per requested trace, dedupes, and reports has_more when the fetch came back full |
lgtm | yes | yes | yes | Tempo for traces, the stack's Prometheus (prometheus_url, default :9090) for metrics and Loki (loki_url, default :3100, off to disable) for logs. Counters are read as raw cumulative samples (a range-vector query, not increase(), so a one-shot run's single sample counts and nothing between two evaluation instants is lost); trace ids come back padded to 32 hex digits (Tempo drops leading zeros, Loki and the live store do not); ui_url pointing at a Grafana (http://host:3000/explore) gives every trace an "open in" link into Explore on the Tempo datasource (grafana_datasource_uid, default tempo); logs use LogQL label-filter stages on trace_id, hermes_session_id, scope_name, severity_number, event_name and body text, and the underscored labels come back as dotted attribute names. Tempo's search stops at limit matches in block order rather than the newest ones, so each page asks for search_fetch traces (default 500) and keeps the newest limit after sorting; a native TraceQL query replaces every structured predicate (the page marks them ignored) while the minimum duration is still applied to the rows, because Tempo ignores minDuration next to a q |
tempo | yes | optional | optional | Traces from Tempo; add prometheus_url / loki_url to a Tempo entry to get the same metrics and logs as lgtm, and ui_url for the Grafana link |
jaeger | yes | no | no | Jaeger stores traces only. Every search names a service: the entry's service_name, else the plugin's resource_attributes.service.name, else hermes-agent. Roots are kept client-side; widened by the kind filter, each card is the first matching span of its trace (by a tool filter alone, the trace's root carrying that span's attributes). Jaeger v1 (/api/traces) and Jaeger v2 (/api/v3/..., OTLP-JSON, the only API v2 serves) are both spoken: the API is probed once per query URL, classic first because Jaeger 1.x's all-in-one also exposes an /api/v3 gateway with different (snake_case) parameters (query_api: v1 / v3 pins it); on v3 the tag map goes JSON-encoded in query.attributes, the bounds are RFC 3339 and an empty answer is v2's 404 "No traces found" |
zipkin | yes | no | no | Zipkin stores traces only, through the zipkin-otel image (stock Zipkin has no OTLP endpoint). API v2 on the same port as ingest and the UI: /api/v2/traces with serviceName (the entry's service_name, else the plugin's resource_attributes.service.name, else hermes-agent), annotationQuery (key=value and key2=value2) for model, session, tool and the error status (otel.status_code=ERROR), spanName exact, minDuration in µs and an endTs + lookback window in ms; /api/v2/trace/{id} for the detail. Every tag is a string in Zipkin, so numeric and boolean values are coerced back except on id-like keys; span events come back from annotations ("name":{...}); roots are kept client-side (the span with no parentId) and, because annotationQuery matches any span of a trace, roots-only re-checks the root. Each trace links to /zipkin/traces/{id} |
greptime | yes | yes | yes | GreptimeDB on its one HTTP port. Traces through the Jaeger-compatible API under /v1/jaeger (query_path): the Jaeger v1 grammar and typed tags, so the Jaeger adapter does the work; every search names a service, roots are kept client-side, and because limit=N keeps N arbitrary traces rather than the newest the adapter asks for search_fetch (default 500) and cuts the page itself. Metrics through SQL on the per-metric tables (greptime_timestamp, greptime_value, one TAG column per label): raw samples with a five-minute look-behind, counters turned into per-bucket increases with first-sample counting (GreptimeDB's PromQL route answers a range-vector instant query with an empty body); the metric list from PromQL's label/__name__/values. Logs through SQL on opentelemetry_logs (logs_table): trace, span, logger, severity, text, session and event filters, the last two via json_get_string(log_attributes, '["hermes.session_id"]'). No "open in" link (the embedded dashboard has no documented deep link to a trace; ui_url offers one) |
clickstack | yes | yes | yes | ClickStack / HyperDX, read straight from ClickHouse over HTTP (query_port, default 8123; Basic auth from user / password, the all-in-one image's api / api from the host). Traces are SQL on otel_traces: every search-bar field is a server predicate, roots-only included (ParentSpanId = ''), status = ok, free text on the input and output values and a raw SQL predicate ANDed into the WHERE; a widened search cards the earliest matching span of each trace (by a tool filter alone, the trace's root, read with one more query, carrying that span's attributes); every Map(String, String) value is a string, coerced back except on id-like keys. Metrics keep their dotted OTel names (hermes.token.usage), histograms are listed as <name>.sum / <name>.count; counters and those columns are read as raw cumulative samples with the five-minute look-behind and first-sample counting, gauges stay raw; group_by takes either spelling of a key. Logs are SQL on otel_logs (session and event filters on LogAttributes); the Collector lower-cases SeverityText, the adapter upper-cases it again. Each trace links to HyperDX's search page when ui_url is set |
victoria | yes | optional | optional | The VictoriaMetrics family, one entry. Traces through VictoriaTraces' Jaeger-compatible API under /select/jaeger (query_path): the Jaeger v1 grammar, every tag typed string and the OTLP status an error tag ("true" / "false"), so values are coerced back and error="false" is not an error; whether limit keeps the newest traces is undocumented, so the adapter asks for search_fetch (default 500) and cuts the page. Metrics through the shared Prometheus reader on VictoriaMetrics (prometheus_url, default the metrics_endpoint origin; the bundled stack sets -opentelemetry.usePrometheusNaming so the names are the Prometheus spelling the reader expects). Logs through LogsQL on VictoriaLogs (logs_url, default the logs_endpoint origin): _time:[…), trace_id, span_id, hermes.session_id, scope.name, severity_number:>=, event.name, _msg:~ text, then sort by (_time desc); loggers from stats by (scope.name) count(). No "open in" link (vmui's TraceID mode has no documented deep link; ui_url offers one) |
aspire | yes | no | yes | The standalone Aspire Dashboard's read API on its UI port (query_port, default 18888; x-api-key from api_key when the API is key-protected). GET /api/telemetry/traces?resource=<instance>…&limit=<search_fetch> returns OTLP JSON for the matching traces (a resource is one Hermes process, so the adapter lists /api/telemetry/resources and names every instance of the service; the bare name is ambiguous once two have reported), and since the API has no time window and no attribute, status or name filters, the adapter applies the window, every structured filter (status = ok and free text included), roots-only and the keyset cursor to the fetched rows; the detail is GET /api/telemetry/traces/{traceId}, passed through. Logs from GET /api/telemetry/logs with resource, limit, traceId and search on the server and span, logger, severity, session and event filters on the rows; severities arrive as Information / Warning / Critical and are mapped from severityNumber. No metrics route exists, so the source reports metrics: false. Each trace links to the dashboard's /traces/detail/{traceId} page |
gigapipe | yes | yes | yes | Gigapipe (formerly qryn) through the lgtm adapter with every read API defaulting to the one port, plus four detours verified on v5.5.3: /api/search with a TraceQL q returns one row per trace but an empty select() list, so each card's model, tokens and session come from /api/traces/{id} (the root span, or the first matching span on a widened search, carded as the trace's root when no kind is set); that detail is pre-1.0 OTLP JSON (resourceSpans → instrumentationLibrarySpans, every attribute a stringValue, no kind, the service on each span's serviceName) and is reshaped and coerced; a range-vector instant query answers an empty result, so counters are read with query_range at the 15 s rollup step and turned into increases here; Loki streams are labels only, with a level label and no scope label, so the LogQL stages use trace_id, span_id, hermes_session_id, event_name, level=~… and ` |
agenta | yes | no | no | Agenta's POST /api/spans/query with the same project ApiKey the exporter uses (so service is not a filter: the key is the project). A TracingQuery with windowing (oldest / newest from the window and the cursor, order: descending) and filtering conditions: parent_id is null for roots-only, span_name is / startswith, attributes with a dotted key (hermes.session_id, llm.model_name or gen_ai.request.model, raw k=v pairs) with eq, the tool filter as span_name is tool.<name>, status_code is / is_not STATUS_CODE_ERROR, free text as contains on input.value, output.value, ag.data.outputs and ag.unsupported.data.inputs, one query per key merged client-side (the API drops nested or groups; current Agenta moves those keys, #630); the grammar has no duration field, so the minimum duration is applied to the fetched rows; span counts come from one more query over the page's trace ids. The detail flattens Agenta's nested attribute objects to dotted keys, drops its ag.data.* / ag.meta.* copies of the inputs and outputs and keeps ag.metrics.* / ag.type.*; resource attributes are not stored, so the resource is the entry's service_name. Ingest is asynchronous (a few seconds). No "open in" link unless ui_url names the project's observability page |
honeycomb, weave, langsmith, parseable, elastic, openlit, mlflow, opik, laminar, langwatch, latitude, otlp | no | no | no | No adapter yet: shown as unavailable in the selector; use the Live source, or the backend's own UI |
The logs column, including the event filters and the expanded record view, was checked against live stacks on 2026-10-04: grafana/otel-lgtm 0.34 (Loki), OpenObserve, SigNoz v0.119 and Uptrace 2.1.0-beta.5, each fed by a real Hermes turn. The Uptrace 2.0 spelling keeps the request and row shapes recorded against 2.0.2 and was not re-run.
Which search-bar fields reach each backend
Every adapter declares what it does with each field of the search bar, and /status publishes the declaration (available[].filters): server means the field is part of the query the backend runs, client that it is applied to the rows the backend returned (so a page can come back short), none that it is ignored. A search answers with applied_filters and ignored_filters so the tab can say which fields did nothing.
| Field | phoenix | langfuse | jaeger | zipkin | greptime | clickstack | victoria | aspire | gigapipe | agenta | lgtm / tempo | signoz | uptrace | openobserve |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| service | none (a project is the service) | none | server (defaults to the agent's service) | server (defaults to the agent's service) | server (defaults to the agent's service) | server | server (defaults to the agent's service) | server (resource=) | server | none (the key is the project) | server | server | server | server |
| kind (a span-name prefix) | client (a substring pre-filter, the prefix checked on the rows) | client | client (operation= is exact) | client (spanName= is exact) | client (operation= is exact) | server (startsWith) | client (operation= is exact) | client | server | server (startswith) | server | server | server (like "x%") | server |
| model / session / tool | server | session only | server (tags) | server (annotationQuery) | server (tags) | server (SpanAttributes[k] =) | server (tags) | client | server | server (attributes eq) | server | server | server | server |
| min duration | server | client (walks up to five pages) | server | server (µs) | server | server | server | client | server (TraceQL duration >=) | client (no duration field) | server (duration >= in the TraceQL; client-side on the rows when a native query is set) | server | server | server |
| status = error | server | none | server (error=true) | server (otel.status_code=ERROR) | server (error=true) | server | server (error=true) | client | server | server (is) | server | server | server | server |
| status = ok | server | none | none (no negative tag search) | none (no negative tag search) | none (no negative tag search) | server | none (no negative tag search) | client | server | server (is_not) | server | server | server | server |
| free text | server (in input.value) | none | none | none | none | server (input and output values) | none | client (input and output values) | server (input and output) | server (contains, one query per target) | server (input and output) | server (input.value contains) | server (input_value contains) | server (every text column the stream has) |
| native query | server (filterCondition) | server (k=v query parameters; page and limit stay the route's) | server (k=v tags) | server (k=v pairs joined with and) | server (k=v tags) | server (SQL predicate, ANDed) | server (k=v tags) | none (no query grammar) | server (TraceQL) | server (k=v attribute pairs) | server (TraceQL, replaces the predicates; the other fields are reported as ignored, minimum duration is kept client-side) | server (k=v items) | server (UQL, appended) | server (SQL, ANDed) |
| roots only | server | none (the list is per trace) | client (the root span itself must match) | client (the root span itself must match) | client (the root span itself must match) | server (ParentSpanId = '') | client (the root span itself must match) | client (the root span itself must match) | server (nestedSetParent < 0) | server (parent_id is null) | server (nestedSetParent < 0) | server | server on 2.1 (_parent_id = ""), client on 2.0 (from each row's parentId) | server |
Implicit scoping, worth knowing when a filter seems to match nothing: a Jaeger, Zipkin, GreptimeDB or Victoria search always names a service (above); Uptrace log queries are pinned to the same service name so Uptrace's own lines stay out of the Logs tab (service_name: off on the entry removes the pin), and its user token may also come from UPTRACE_USER_TOKEN; Loki filters are label-filter stages that match the labels and structured metadata the collector promoted (trace_id, hermes_session_id, scope_name, severity_number, event_name, as the grafana/otel-lgtm image does), and every Loki window is clamped to the last 30 days of the request (Loki's default max_query_length); the Phoenix project list is read once per adapter and holds up to 200 projects.
Pages: trace search is keyset-paged like logs. The response carries has_more and next_before_ns (the start of the oldest trace shown); pass it back as before_ns for the next, older page, on every source. Tempo answers a search with the first limit matches it finds, not the newest, so the Tempo and LGTM adapters over-fetch (search_fetch, default 500 traces per page) and cut the page here; a window holding more traces than that can still lose one between pages. A trace detail carries span_count and truncated: true when the backend's span cap (500 on Phoenix, SigNoz and OpenObserve, 2,000 on Agenta) was hit.
The whole tab was driven against every backend below on 2026-10-09 (Hermes at 46d7718a), each from the repo's compose file on this machine, with scripts/dashboard_verify: a fresh four-turn batch per backend, every answer compared with the Live source for the same turns, then the real tab in headless Chromium. Versions and results: grafana/otel-lgtm 0.34.0 (Tempo 3.0.3) PASS 51 as lgtm and as tempo with and without Loki; OpenObserve latest PASS 51; SigNoz v0.119.0 PASS 51; Uptrace 2.1.0-beta.5 PASS 51; Jaeger all-in-one:latest and Jaeger 2.21.0 (v3 API) PASS 38 each; Phoenix 20.20.0 PASS 39; Langfuse 3.225.11 PASS 35 and Langfuse 4.56.0 (events-only) PASS 36, the skipped checks being the signals a traces-only backend does not serve. What each run found and fixed is on the per-backend issues (#293–#299, #245, #246) and in the milestone's closing note (#302).
Zipkin joined on 2026-10-11: ghcr.io/openzipkin-contrib/zipkin-otel 0.3.0 (Zipkin 3.6.0) PASS 38 / SKIP 10 as zipkin from a fresh four-turn batch, the skips being the metrics and logs checks Zipkin does not serve. GreptimeDB joined the same day: greptime/greptimedb v1.2.1 as greptime from a fresh four-turn batch, every trace, metric and log check passing once the trace search over-fetched (the only skips are status=ok, which the Jaeger grammar cannot express, and the "open in" checks, since GreptimeDB's dashboard has no documented trace deep link). ClickStack (clickhouse/clickstack-all-in-one 2.40.0, ClickHouse 26.8) as clickstack the same day: PASS 51 from a fresh four-turn batch, no skips. The VictoriaMetrics family (VictoriaTraces v0.12.0, VictoriaLogs v1.53.0, VictoriaMetrics v1.153.0) as victoria the same day: PASS 46 / SKIP 5 from a fresh four-turn batch (the skips: no UI deep link, status=ok). The Aspire Dashboard (mcr.microsoft.com/dotnet/aspire-dashboard 13.6.1) as aspire the same day: PASS 48 / SKIP 3 from a fresh four-turn batch (the skips are the metrics checks; the read API has no metrics route). Gigapipe (ghcr.io/metrico/gigapipe v5.5.3) as gigapipe the same day: PASS 46 / SKIP 5 from a fresh four-turn batch (the skips: no UI deep link, and the loggers list, since Gigapipe keeps no scope label). Agenta (agenta OSS, api v0.122.x) as agenta the same day: PASS 35 / SKIP 13 from a fresh four-turn batch (the skips: no UI deep link, and the metrics and logs checks Agenta does not serve); its one adapter fix was free text, which the API drops when sent as a nested or group, so the adapter sends two flat queries.
API
All routes are under /api/plugins/hermes_otel/. The streaming views poll the cursor endpoints; the query endpoints do the filtering server-side.
| Route | Purpose |
|---|---|
GET /live/status | Whether the store can be read (live, with a reason and the file path when not: the module failed to import, the store could not be created, or the file could not be opened), how full it is, and write_error / pending while a commit is being retried |
GET /live/spans, /live/metrics, /live/logs (since, limit) | Raw rows after a cursor, for streaming |
GET /live/traces (lookback_hours or start_s/end_s, session, status, name, kind, text, trace_id, model, tool, min_duration_ms, limit, before_ns, offset) | One row per trace, newest start first, with totals counted once: {traces, total, has_more, next_before_ns}. Pages are keyset-paged like logs: pass next_before_ns (the start of the oldest row shown) back as before_ns for the next (older) page, so a turn landing meanwhile never shifts it; offset still works. total ignores the cursor. A row carries partial: true while its root span is missing, and subagents: {count, tokens, calls, cost} when the turn delegated (else null). kind accepts every kind the tab shows, including other. A window with start_s after end_s is a 422 |
GET /live/traces/{trace_id} | The trace's spans and its summary row (partial included) |
GET /live/sessions (lookback_hours, limit) | One row per session: turns, spans, errors, tokens, cost, tool calls, subagentTokens (the turns' delegations, on top of tokens), newest first; has_more when more sessions than limit exist in the window |
GET /live/metrics/names | Every instrument in the store with its point count (the whole buffer, not a window) |
| window defaults | lookback_hours defaults to 1 on the trace, metric and log routes and to 24 on /live/sessions and the backend /loggers route (/live/loggers takes no window and counts the whole store); start_s / end_s (unix seconds, 0 is the epoch) replace it, and start_s after end_s is a 422 |
GET /live/metrics/query (name, group_by, agg, lookback_hours or start_s/end_s, bucket_s) | Time buckets for one instrument, optionally split by an attribute. More than 5,000 buckets (window ÷ bucket_s) is a 422 |
GET /live/logs/search (trace_id, span_id, session, min_level on the Python scale: 10 debug, 20 info, 30 warn, 40 error, logger, text, event_name, events_only, lookback_hours or an absolute start_s/end_s window, limit, before_ns) | One page of filtered log lines, newest first: {logs, next_before_ns, has_more}. Pass next_before_ns back as before_ns for the next (older) page. Rows carry the OTel severity_number next to the level |
GET /live/loggers | Logger names with counts |
GET /settings (reveal) | Every setting with value, default, source and description; the config file raw and as an effective YAML; the environment variables the plugin reads. Credentials are masked unless reveal=true |
GET /status (backend) | The chosen or default backend, and every configured one with its capabilities (available[].supported/metrics/logs) |
GET /traces/search (backend, q, service, lookback_hours or start_s/end_s, roots_only, status, min_duration_ms, free_text, name_prefix, name_regex, model, session, tool, limit, before_ns) | Backend trace search, newest first: {traces, has_more, next_before_ns, applied_filters, ignored_filters}. model/session/tool become attribute equalities, name_prefix is the kind filter; each adapter declares what it honours (table above) |
GET /traces/{trace_id} (backend) | Backend trace detail as OTLP JSON plus span_count, truncated and ui_url; 404 {"kind": "not_found"} when the backend has no such trace |
GET /metrics/names, /metrics/query (backend, lookback_hours or start_s/end_s, same parameters as the live ones) | Metrics from a backend that serves them, in the live endpoints' shape; every name row carries otlp_name, the plugin's dotted instrument name behind the backend's spelling (hermes_token_usage_total → hermes.token.usage). At most 5,000 buckets per query (422 otherwise); name and group_by are validated before they reach a query |
GET /logs/search, /loggers (backend, start_s/end_s or lookback_hours, same parameters as the live ones, plus span_id) | Logs from a backend that serves them, in the live record shape |
Troubleshooting
The Live tab says "Live store unavailable". The dashboard could not import the plugin's live_store_registry module, the store could not be created, or the file could not be opened (corrupt, or not readable by the dashboard's user); GET /live/status names the reason and the path. An empty Live tab with live: true and zero rows means nothing is being written: dashboard_live is false, or the gateway and the dashboard resolve different HERMES_HOME values and therefore different files (compare path with the gateway's home).
Backend source says "Not configured". No entry under backends: has a type with an adapter. The message lists the types that do.
Numbers differ between Live and Backend for the same turn. They should not since 1.8.8. Open an issue with the trace id and both screenshots.
Developing the tab
Front-end sources are in dashboard-ui/src (TypeScript, built with esbuild into hermes_otel/dashboard/dist/, which is committed unminified so a PR diff of the bundle stays readable). npm run build rebuilds the bundle and CI fails if it is stale; npm run check runs the type checker, ESLint and Prettier; npm test runs the vitest suite: the pure helpers plus one rendered smoke test per page against a shimmed host SDK (test/sdk-shim.ts). make ci-dashboard runs all of it. The tab must not rely on Tailwind classes from the host dashboard: every class it uses is either in dashboard-ui/host-classes.txt (regenerated from the targeted Hermes release with scripts/host_css_classes.py) or defined in dist/style.css, and a unit test enforces that. Python routes are tested with a FastAPI test client in tests/unit/test_dashboard_api.py.