Skip to main content

Backend quirks: what each one does differently, and what hermes-otel does about it

OTLP fixes the wire format. It does not fix the URL a backend listens on, the header it wants your key in, which of the three signals it stores, whether it wants cumulative or delta counters, how it adds up tokens over a trace, how long an attribute may be, or which API gives the data back. Every backend in this directory answers those questions differently, and the plugin carries one answer per type: so that a backends: entry is one line of intent rather than a page of vendor notes.

This file is the catalog of those differences. Part 1 is one matrix per category. Part 2 lists each backend's quirks with the exact treatment the plugin gives it, and the issue where that was settled. Part 3 is the short list of what the plugin normalizes. Every statement comes from hermes_otel/backend_specs.py, hermes_otel/backends.py, a dashboard adapter under hermes_otel/dashboard/backends/, a folder README here, a page under website/docs/backends/, or a closed issue; nothing here is a guess. Where a figure is quoted it carries the date and version it was measured against.

Companion pages: the overview says which backend to pick, each backend's own page has the setup, and in the repo docker-compose/README.md is the local-stack manual. This page is also docker-compose/QUIRKS.md there.

Part 1: the matrices​

Legend: std means the standard OTLP/HTTP paths /v1/traces, /v1/metrics, /v1/logs. "the plugin completes it" means you can give the base URL (or the vendor prefix, or the full traces URL) and the resolver produces the full per-signal URLs. "derived" means /v1/metrics and /v1/logs are produced from the traces URL by swapping the last path segment (pipeline_env.derive_signal_endpoint).

A. Where OTLP lands​

Backendtype:Ingest pathPort notesWhat the plugin does
Phoenixphoenixstd, traces onlyingest and UI share :6006; no redirect from the root, the URL must end in /v1/tracesendpoint used as given
Langfuselangfusevendor prefix /api/public/otel/v1/tracesingest and UI share :3000completes root URL, /api/public/otel, or full URL; a root URL used to be posted as-is and died with 405
LangSmithlangsmithOTLP/HTTP /otel/v1/traces on api.smith.langchain.com (region: eu → eu.api.smith.langchain.com)cloud onlyapi_key → x-api-key; project → Langsmith-Project; traces only
SigNozsignozstdcollector :4328 here (upstream 4318); UI/API :3301derived
Jaeger v1 / v2jaegerstdv1 ingest :4318, UI :16686; v2 ingest :4368, UI :16696endpoint used as given
Agentaagentavendor prefix /api/otlp/v1/traces, traces only, protobuf only (JSON → 400), 10 MB capone front door (nginx :3060 here, Traefik :80 upstream) for UI, API and ingestcompletes origin / /api / /api/otlp to the full URL; cloud host from region when unset
Gigapipegigapipestd, one port for ingest, the Tempo / Prometheus / Loki APIs and the UI (:3110 here, 3100 upstream)3100 collides with LGTM's Loki → remapped; ClickHouse unpublishedcompletes base URL, derived; no default host
Aspire Dashboardaspirestd on the OTLP/HTTP port (:4418 here, 18890 in-container)UI + read API on :18888, OTLP/gRPC 18889 and MCP 18891 unpublishedcompletes base URL, derived; no default host
VictoriaTraces / Logs / Metricsvictoriathree hosts: /insert/opentelemetry/v1/traces on :10428, /opentelemetry/v1/metrics on :8428, /insert/opentelemetry/v1/logs on :9428upstream ports, nothing else here uses themendpoint + metrics_endpoint + logs_endpoint, each an origin or full URL; the type introduced per-signal URLs on the exporter side; a signal without its store is off
ClickStackclickstackstd on the bundled Collector :4408 (4318 in-container)UI :3050 (8080), ClickHouse :8124 (8123) published for readscompletes base URL, derived; no default host
GreptimeDBgreptimevendor prefix /v1/otlp/v1/traces (+ /metrics, /logs)one HTTP port for ingest, SQL, PromQL, Jaeger API and UI (:4398 here, upstream 4000)completes origin or /v1/otlp to the traces URL, derived; per-signal headers: the trace pipeline header never reaches metrics or logs
Zipkinzipkinstd, traces onlyingest, API v2 and UI share :9411, on the zipkin-otel image only (stock Zipkin has no OTLP)completes the origin to /v1/traces; no default host
Tempotempostdingest :4358 here, API :3210, Grafana :3020endpoint used as given
Grafana LGTMlgtmstdcollector :4318 (gRPC :4317), Grafana :3000, Loki :3100, Tempo :3200, Prometheus :9090derived; lgtm is a self-documenting alias of otlp
Uptraceuptracestdingest and UI share :14318; gRPC :14317derived
OpenObserveopenobserveorg-prefixed /api/<org>/v1/tracesingest and UI share :5080; gRPC :5081derived (the org stays in the path); a wrong path 404s silently
Parseable Cloud / Enterpriseparseablestd, at the ingestor URLquery/UI endpoint may refuse ingest in distributed setupsderived, one dataset header per signal
Parseable OSSotlp via collectorstd on the bundled collector :4348UI :8010collector re-encodes to OTLP/JSON because OSS refuses protobuf
Honeycombhoneycombstd on api.honeycomb.io or api.eu1.honeycomb.ioSaaSbase chosen by region: us|eu, derived
W&B Weaveweavevendor path /otel/v1/traces (SaaS) or /traces/otel/v1/traces (dedicated / self-managed)SaaScompletes from base_url
Elasticelasticstd, one endpoint for all signalsEDOT Collector :14319 here; ES :19201; Kibana :15602strips any /v1/<signal> and re-appends /v1/traces, derived; no default host
OpenLITopenlitstd on the collector :4338 (4318 in-container)UI :3010 does not proxy /v1/* on 2.1.0completes base URL, derived; no default host
MLflowmlflowstd, traces only, on the tracking server :5001 (5000 in-container)one port for UI and ingestcompletes base URL
Comet Opikopikvendor prefix /api/v1/private/otel/v1/tracesingest and UI share :5173 (nginx)completes base, …/api, or prefix form; cloud default host
Laminarlaminarstd on the app-server :8100 (upstream 8000)UI :5667 does not ingest; gRPC :8101completes base URL; cloud default api.lmnr.ai
LangWatchlangwatchvendor prefix /api/otel/v1/tracesingest, API and UI share :5560completes base or prefix form, derived; cloud default host
Latitudelatitudestd, traces only, on the ingest service :3002web :3000, API :3001, Mailpit :8025completes base URL; cloud default ingest.latitude.so
telemetry.devotlpstd on ingest.telemetry.devSaaSgeneric; derived
Langtraceotlpcustom /api/trace (no /v1/)ingest and UI share :3040generic, URL used verbatim; derivation does not apply
Sigirootlpstd :4378 (gRPC :4377)SQL API :9999, no UIgeneric; derived
OpenTelemetry Collector (collector/)otlpstd :4448 (upstream 4318); health :13133the destination is the Collector's COLLECTOR_TARGET (debug, otlphttp, azuremonitor, aws, googlecloud)generic; derived; one hop removed (the plugin only sees the Collector's 200)
otel-desktop-viewerotlpstd :4428 (upstream 4318)UI + JSON-RPC :3070 (upstream 8000)generic; derived
otel-tuiotlpstd :4438 (upstream 4318)no UI port: a TTY (docker attach)generic; derived
Maple Localotlpstd :4388ingest and UI share :4388generic; derived

Transport is OTLP/HTTP with protobuf bodies (opentelemetry-exporter-otlp-proto-http) for every type. Only a generic type: otlp entry can switch to OTLP/gRPC (protocol: grpc, #469); the named types keep their HTTP URL rules. Parseable OSS is the one backend that refuses protobuf, hence its collector.

B. Who you are: authentication​

BackendHeader and schemeRequired?Where the local key comes from
Phoenixnone locally; Arize cloud takes api_key: <key> as an extra headernonone
LangfuseAuthorization: Basic base64(public:secret) plus x-langfuse-ingestion-version: 4yes, both keyspre-seeded by headless init (lf_pk_hermes_dev / lf_sk_hermes_dev)
LangSmithx-api-key (+ Langsmith-Project)yescloud account
SigNozsignoz-ingestion-key (cloud); nothing self-hostedcloud onlycloud UI
AgentaAuthorization: ApiKey <project key> (Bearer accepted too)yes; a missing key skips the entryagenta/mint-api-key.sh (SuperTokens sign-up, header session, POST /api/keys) or the project settings
Gigapipenone by default; Authorization: Basic from user + password for GIGAPIPE_LOGIN / GIGAPIPE_PASSWORD deployments; X-Scope-OrgID is not readnonone
Aspire Dashboardnone in anonymous mode; x-otlp-api-key from api_key when DASHBOARD__OTLP__AUTHMODE=ApiKey (the read API wants the same value as x-api-key)nonone
VictoriaTraces / Logs / Metricsnone; Authorization: Basic from user + password for -httpAuth.* deployments; logs carry VL-Stream-Fields: service.namenonone
ClickStackbare authorization: <team ingestion key> (a Bearer prefix is accepted too)yes: 401 without, "does not match expected scheme or token" with a wrong one; a missing key skips the entryclickstack/mint-api-key.sh (registers the first account, reads the key from the all-in-one's MongoDB), or Team Settings → API Keys
GreptimeDBAuthorization: Basic base64(user:password) only when the server runs a user provider; omitted when either is missingno locallynone; X-Greptime-DB-Name picks the database (default public)
Jaeger, Zipkin, Tempo, LGTMnone (Grafana Cloud: Authorization: Basic <token> via headers)nonone
Uptraceuptrace-dsn: http://<project_token>@host:14318?grpc=14317yesseeded in uptrace.yml (project_tokens[0].token)
OpenObserveAuthorization: Basic base64(email:password)yesseeded root user (root@example.com / Complexpass#123)
Parseable (type)X-API-KeyyesCloud / Enterprise UI. OSS has no API keys: any X-API-Key is a 401, so the collector adds Basic auth instead
Honeycombx-honeycomb-teamyescloud UI; keys are region-specific (a US key is a 401 on EU)
W&B Weavewandb-api-keyyesW&B account
ElasticAuthorization: ApiKey <key> (not Bearer); omitted when no key resolvescloud yes, local EDOT nocloud UI; needs the apm app's event:write privilege
OpenLITAuthorization: Bearer <key>; omitted when noneno (OTLP_REQUIRE_API_KEY=true makes it mandatory)UI; without a key data lands in the INIT_DB_* defaults
MLflowAuthorization: Bearer <tracking token>; omitted when nonenonone locally
Comet OpikAuthorization: <key> (bare, no Bearer) plus Comet-Workspacecloud yes, self-hosted nocloud UI
LaminarAuthorization: Bearer <project key>yes; a missing key skips the entry at startuplaminar/mint-api-key.sh (signs in, then POST /api/cli/api-key)
LangWatchAuthorization: Bearer sk-lw-…yes; missing key skips the entryUI onboarding wizard once, langwatch/mint-api-key.py (Playwright), or select "apiKey" from mydb."Project" in Postgres
LatitudeAuthorization: Bearer <key>yes; missing key skips the entryUI only (encrypted at rest, cannot be read from Postgres)
telemetry.devAuthorization: Bearer td_live_… via headersyescloud UI
Langtracex-api-key via headersyesUI, or the NextAuth curl recipe in its README
Sigiro, Maplenonenonone
otel-desktop-viewernonenonone
OpenTelemetry Collectornone on the plugin side; the target's exporter signs (Entra via connection string, SigV4, OAuth) from the container's environmentnothe cloud account
otel-tuinone by default; Authorization: Bearer when AUTH_TOKEN is setnothe env var you set

Five different spellings of "here is my key" on the Authorization header alone: Basic (Langfuse, OpenObserve), Bearer (OpenLIT, MLflow, Laminar, LangWatch, Latitude, telemetry.dev), ApiKey (Elastic), the bare key (Opik), and vendor headers for the rest. The per-type resolver builds the right one from api_key / api_key_env / the vendor's own env var, and the global headers: block is merged under the resolver's headers so it can never clobber the auth it built. A headers_helper on the entry (a command that prints the headers, re-run on a 401/403, #483) is merged on top of both, and an entry with one no longer needs the static key that would otherwise be required.

Env-var mode has one more rule: a vendor's own credential variable (LANGSMITH_API_KEY, LANGFUSE_PUBLIC_KEY, WANDB_API_KEY, HONEYCOMB_API_KEY, ELASTIC_API_KEY, OPENLIT_API_KEY, MLFLOW_TRACKING_TOKEN, OPIK_API_KEY, LMNR_PROJECT_API_KEY, LANGWATCH_API_KEY, LATITUDE_API_KEY, AGENTA_API_KEY, HYPERDX_API_KEY) never switches export on by itself, because those are often set for other tools and Hermes loads $HERMES_HOME/.env into the process. One plugin-namespaced OTEL_* variable is the opt-in (#259).

C. Where it files your data: routing​

BackendRouting mechanismPlugin field
Phoenixproject = resource attribute openinference.project.nametop-level project_name
Langfuseproject = the key pairnone
SigNoznonenone
GreptimeDBdatabase (X-Greptime-DB-Name, default public), then service.namedb / db_env
ClickStackthe key = the team; ServiceName columnapi_key
Agentathe key = the project; ag.session.id from session.id; resource attributes droppedapi_key
Gigapipeservice.name (Tempo rootServiceName, Prometheus job, Loki service_name label)resource_attributes (service.name) or service_name_from_profile
Aspire Dashboarda resource = service.name + service.instance.id (one per Hermes process); the API's resource= takes the bare service name only while one instance exists, so the adapter names every instanceresource_attributes (service.name) or service_name_from_profile
VictoriaTraces / Logs / Metricsservice.name (VictoriaTraces and VictoriaLogs stream field; a VictoriaMetrics label)resource_attributes (service.name) or service_name_from_profile; headers: {VL-Stream-Fields: …} for the log stream key
Jaeger / Zipkin / Tempo / LGTMservice.nameresource_attributes (service.name) or service_name_from_profile
Uptraceproject = the DSNdsn
OpenObserveorg in the URL path, stream in the stream-name header (default default; a bad name is a 400)stream_name
Parseableone dataset per signal: X-P-Stream: hermes-{traces,metrics,logs} plus X-P-Log-Source: otel-{traces,metrics,logs}traces_dataset, metrics_dataset, logs_dataset
Honeycombx-honeycomb-dataset is honoured only by Classic keys (required there for every signal); Environments keys route traces by service.name and metrics to the Metrics datasetdataset (leave unset on a modern key)
W&B Weaveresource attributes wandb.entity and wandb.project; they live on the one shared Resource, so one Weave project per processentity, project
Elasticresource attributes data_stream.dataset / data_stream.namespace; the collector appends .otel, so dataset: hermes_otel lands in traces-hermes_otel.otel-<namespace>. Components allow [a-z0-9_.] only, no -; an invalid value skips the entry; two elastic entries with different values conflictdataset, namespace
OpenLITorg / project / environment = the keyapi_key
MLflowmandatory x-mlflow-experiment-id (400 without it; default "0", the built-in Default experiment); optional X-MLFLOW-WORKSPACEexperiment_id, workspace
Comet OpikprojectName header (auto-created, default "Default Project"); Comet-Workspace on cloudproject, workspace
Laminarproject = the keyapi_key
LangWatchX-Project-Id, needed only with a service keyproject
Latitudemandatory X-Latitude-Project: <slug>; spans without it are rejected, so a missing slug skips the entryproject
Langtraceproject = the key; one ClickHouse table per project idheaders

Two backends route by resource attribute instead of header (Weave, Elastic). Because the plugin has one TracerProvider and therefore one Resource, those attributes reach every configured backend, and two entries of the same type with different values are a configuration error at init.

D. Which signals it stores, and what happens to the others​

"Stored" is what landed from a real Hermes turn, not what a vendor's docs claim (verified dates are on each folder README). The failure mode matters because the plugin's debug log only sees the HTTP status.

BackendTracesMetricsLogsFailure mode for the missing onesPlugin default
Phoenixyesnono405 on /v1/metrics and /v1/logs (#160)traces-only spec: no metrics or logs exporter is created
Langfuseyesnonono such routestraces-only spec
LangSmithyesnonothe OTLP ingest serves traces onlymetrics and logs off
SigNozyesyesyesbefore first-run setup the collector runs a nop pipeline and resets the connectionall on
Jaegeryesnononot served (a forced metrics: true is expected to 404)traces-only spec
Agentayesnonoonly /api/otlp/v1/traces is mounted (404 elsewhere); ingest is asynchronous via a Redis stream, 200 = acceptedtraces-only spec
Gigapipeyesyes (cumulative only)yesdelta points rejected one by one in partial_success; /v1/logs answers 204all on
Aspire Dashboardyesyes (Gauge, Sum, Histogram; exponential histograms rejected, "is not supported")yes—all on
VictoriaTraces / Logs / Metricsyesyes (its own store)yes (its own store)a signal whose store is not named has nowhere to go; an explicit metrics: true / logs: true without the store skips the entry with a hinton per named store
ClickStackyesyesyesnothing listens on 4317/4318 until the first team exists (the Collector gets its receivers over OpAMP); then 401 without the keyall on
GreptimeDByesyes (no exponential histograms)yestraces without X-Greptime-Pipeline-Name: greptime_trace_v1 → 400 "Pipeline is required"; logs with it → 400 "Unsupported pipeline for logs" (verified 2026-10-11)all on; the pipeline header goes on traces only (metrics_headers / logs_headers), and an empty per-signal header set now means "none", not "the trace headers"
Zipkinyesnono/v1/metrics is 404; /v1/logs answers 202 and keeps a record only as an annotation on a span it can name through an event.name attribute, which the plugin's records lack, so nothing is stored (verified 2026-10-11)traces-only spec
Tempoyesnononot servedtraces-only spec (even behind a collector that would accept them; set metrics: true explicitly)
Grafana LGTMyesyesyes—all on
Uptraceyesyesyes—all on
OpenObserveyesyesyes—all on
ParseableyesyesyesOSS: 400 Protobuf ingestion is not supported without the collectorall on
Honeycombyesyesyes—all on
W&B WeaveyesnonoW&B documents the endpoint for traces onlytraces-only spec
Elasticyesyesyessilent: exporter says SUCCESS, the collector's bulk indexer fails if the cluster is redall on, delta preset
OpenLITyesyesyes—all on
MLflowyesnono404 on /v1/metrics and /v1/logstraces-only spec
Comet Opikyesnono404 under the vendor prefixtraces-only spec
Laminaryesnoyes/v1/metrics answers 200 and stores nothing (a placeholder handler)metrics=False, logs=True spec: metrics off, logs on
LangWatchyesyesyes—all on
Latitudeyesnonothe ingest service has only /v1/traces and health routestraces-only spec
telemetry.devyesyesyes—generic otlp: all on
Langtracepartialnononot servedgeneric otlp: set metrics: false, logs: false on the entry
Sigiro, Mapleyesyesyes—generic otlp: all on
OpenTelemetry Collectoryesyesyesper target: awscloudwatchlogs is alpha; the Collector's 200 says nothing about the targetgeneric otlp: all on
otel-desktop-viewer, otel-tuiyesyesyes— (otel-tui: no exponential histograms, which the plugin does not emit)generic otlp: all on

Three failure modes, one of them invisible: a 404/405 you can see in the debug log, a connection reset you can see, and Laminar's 200-with-nothing and Elastic's red-cluster drop that you cannot. The per-type defaults (metrics / logs on each type's BackendSpec in backend_specs.py) exist so a user never has to discover the invisible ones. The generic otlp type cannot know, so it defaults every signal on and you turn them off per entry.

hermes -z (one-shot) disables Python logging process-wide to keep its stderr clean; the plugin lifts that threshold to the capture level while its handler is installed, so one-shot runs export all three signals (logs only since 1.24.0).

E. Metrics: temporality, histograms, names, staleness​

BackendTemporality it wantsPlugin presetExponential histogramsHow the stored name looks
Grafana LGTM (Mimir), Tempo-side Prometheuscumulative; delta is dropped unless otlp-deltatocumulative is oncumulative (SDK default)Prometheus 3.8+ native histograms; Mimir only with native-histogram ingestion onhermes_token_usage_total, histograms grow _sum / _count / _bucket, units appended (_seconds, _milliseconds)
OpenObservecumulative works; delta undocumentedcumulativeundocumentedone stream per metric, _total dropped (hermes_token_usage), histograms split into _sum / _count / _bucket streams
SigNozrecommends delta; exponential histograms delta-only and self-hosted-onlydeltaself-hosted onlyhermes_token_usage_total, .sum / .count / .bucket views
Uptraceprefers delta, converts cumulativedeltarecommendeddotted names kept; MQL $m
Elasticdelta required: ES does not handle cumulative histograms and converts cumulative counters lossilydeltaacceptedES data streams
Honeycombeithercumulativenot verified—
OpenLITcumulative stored as-is (verified)cumulativestored (exponential histogram type)ClickHouse otel_metrics_* tables
Gigapipecumulative only (delta rejected per point)cumulativeconverted to _bucket / _sum / _countPrometheus spelling per the OTLP-to-Prometheus spec (hermes_token_usage_total, target_info, otel_scope_*); PromQL served from the 15 s rollup
Aspire Dashboardeither, stored as sentcumulativerejecteddotted OTel names, in the UI only (no metrics read route)
VictoriaMetricscumulative natively; delta stored as-is since v1.132cumulativeconverted to vmrange histogramsa server flag: with -opentelemetry.usePrometheusNaming (the bundled stack) hermes_token_usage_total / …_seconds_sum; without it the dotted OTel names "as is"; every resource attribute promoted to a label by default
ClickStackeither, stored as sent by the ClickHouse exportercumulativestored (otel_metrics_exponential_histogram)dotted OTel names kept (hermes.token.usage); one table per type (otel_metrics_sum / _gauge / _histogram with Count + Sum), TimeUnix in seconds, Attributes map with the original keys
GreptimeDBeither, stored as sent (no conversion)cumulativerejected ("not supported yet")one table per metric in Prometheus spelling (hermes_token_usage_total, …_seconds_sum / _count / _bucket), greptime_timestamp ms + greptime_value, one TAG column per label
OpenTelemetry Collectorper target: Application Insights wants delta, so azuremonitor runs cumulative_to_delta; AWS and Google Cloud take cumulativecumulative—per target
Parseable, generic otlp, Sigiro, Maple, otel-desktop-viewer, otel-tuiwhatever the collector / store doescumulativeSigiro has an exp_histogram table—
Datadog, New Relic, Logfire (generic otlp)delta required (Datadog rejects cumulative sums; Logfire dashboards stay empty; New Relic prefers delta)none yet (#232): set metrics_temporality: delta on the entryDatadog and New Relic accept them—
LangWatchstored (73 metric points from one turn, 2026-10)cumulative——

Presets are the temporality field of each type's BackendSpec in backend_specs.py (signoz, uptrace, elastic → delta, #233). An explicit metrics_temporality on the entry wins; the top-level key sets every backend. metrics_histogram: exponential is global, so use it only when every configured backend accepts base-2 histograms.

Two cross-backend facts about short-lived runs: a Prometheus-family store treats a series as stale five minutes after its last sample, so a one-shot turn's single cumulative point disappears from "now" queries (widen the range); and Datadog / New Relic treat the first cumulative point as a baseline, so the same turn counts as zero there but in full as delta. Long running gateways are unaffected either way.

F. Semantics: what the UI understands, and how it adds up a trace​

BackendAttribute namespace it readsTrace-level token totalPrices calls itself?Session / thread key
PhoenixOpenInference (llm.token_count.*, input.value, openinference.span.kind)reads the root's own roll-upyes, from token countssession.id
Langfusegen_ai.* (including legacy gen_ai.system)reads the root; zero-fills usage on non-GENERATION observationsyessession.id (automatic grouping)
LangSmithGenAI (gen_ai.*) through its OTLP ingestnot verified since the OTLP switch (#415)—not verified
SigNozgen_ai.* in its LLM views; otherwise raw tagsn/a (span table)via a derived metricany attribute
AgentaOpenInference, OpenLLMetry and OTel GenAI adapters server-side: ag.type.span (agent / chat / tool), ag.metrics.tokenscumulative down the tree: the root's total equalled the sum of the api.* spans (no double count; root_usage kept)yes, for priced models (ag.metrics.costs)ag.session.id ← session.id
Gigapipenone; raw tags, TraceQLn/anoany attribute (TraceQL)
Aspire DashboardGenAI visualizer for chat-completion spans (gen_ai.input.messages / gen_ai.output.messages, tool definitions since 13.1); otherwise raw attributesn/anoany attribute in the UI
VictoriaTracesnone; attributes stored with span_attr: / resource_attr: / scope_attr: prefixes, searchable in vmuin/anoany attribute (Jaeger tags)
ClickStacknone; every attribute searchable in HyperDX's Lucene-style search (SpanAttributes.gen_ai.usage.input_tokens:>1000)n/anoany attribute
GreptimeDBnone; attributes become typed columns ("span_attributes.gen_ai.usage.input_tokens" Int64)n/anoany column; SQL
Jaeger, Zipkin, Tempo, LGTMnone; raw tags (Zipkin: every value a string)n/anoany attribute (Tempo: TraceQL; Zipkin: annotationQuery)
Uptrace, OpenObserveall attributes searchable, keys flattened with underscoresn/anohermes_session_id
Honeycombcolumns auto-created from whatever arrives (gen_ai.*, llm.*)n/anoany column
W&B Weavegen_ai.*; wandb.thread_id, wandb.is_turn=true on the rootper call—gen_ai.conversation.id / wandb.thread_id
Elasticraw fields; Kibana dashboard uses hermes.token.usage, hermes.model.usage, tool durationsn/anoany field
OpenLITgen_ai.request.model, gen_ai.usage.*, gen_ai.tool.*n/anohermes.session_id on logs
MLflowtranslators for GenAI semconv, OpenInference and OpenLLMetrycomputes itself (mlflow.trace.tokenUsage), reads the rootyes (mlflow.trace.cost, priced models only)mlflow.trace.session = Hermes session id
Comet OpikGenAI semconv and OpenInference; gen_ai.request.model, gen_ai.provider.namesums every span → root roll-up dropped (root_usage: false, 1.24.0)yes (total_estimated_cost, null for unknown models; before 1.24.0 the doubling also doubled the cost)thread_id = Hermes session
Laminargen_ai.usage.*; span types LLM (api.*, llm.*) and TOOLreads the rootyes (0 for unknown models)—
LangWatchgen_ai.*sums every span → root roll-up dropped (root_usage: false, 1.24.0)—thread_id = Hermes session
LatitudeGenAI semconv, explicitlyper span columns (cost_*)yessession column
telemetry.devgen_ai.* only; OpenInference attributes ignored——gen_ai.conversation.id on the root
Langtraceits own gen_ai.*-based schema———
MapleAI trace index (model, tokens, cost, tool); pickup unverified———

Three things the plugin does here:

  1. It emits both conventions on every span: OTel GenAI gen_ai.* and OpenInference llm.*, so Phoenix and Langfuse and OpenLIT each find the spelling they index on without a mapping adapter.
  2. It puts the turn's token total on the agent / cron root and on each api.* span. Backends that total a trace by summing every span (Opik, LangWatch) would then show twice the tokens, so for those two types the exporter bound to that entry rebuilds the root without gen_ai.usage.* / llm.token_count.* (root_usage: false, the root_usage field of their BackendSpec, #327, since 1.24.0; on 1.23 and earlier the totals double). Phoenix, Langfuse, MLflow and Laminar read the root's own numbers and keep it. Only that backend's copy changes; every other exporter and the live store still get the roll-up.
  3. It sets session.id, gen_ai.conversation.id and the legacy hermes.session_id, plus wandb.thread_id for Weave, so each backend's notion of "conversation" lines up with a Hermes session.

G. Attribute limits​

BackendLimitTreatment
Tempotruncates attribute values at 2048 bytes by default; cut gen_ai.input.messages under content_capture: fulltempo/tempo.yaml sets distributor.max_attribute_bytes: 131072
Agentaattributes stored as nested objects per namespace (gen_ai.usage.total_tokens → {gen_ai: {usage: {total_tokens}}}) plus Agenta's ag.*; resource attributes dropped; 10 MB batch capthe adapter flattens to dotted keys and drops the ag.data / ag.meta copies
Gigapipeevery attribute value a string on read (/api/traces/{id}); log attributes become Loki labels (dots → underscores), the scope name is dropped; 64 MiB request capthe adapter coerces values back; logger filter and loggers list unsupported
Aspire Dashboard128 attributes per item (DASHBOARD__TELEMETRYLIMITS__MAXATTRIBUTECOUNT), no value-length cap by default; 100 000 traces and logs, 50 000 metric points per dimension, oldest evicteddocumented
VictoriaTraces / Logsevery Jaeger tag typed string ("23571", "false"), the OTLP status an error tag; log field values strings; 1000 fields per line, 64 MiB per trace request, -search.maxTraces 1000; VictoriaMetrics 40 labels per series, 4096-char valuesthe adapter coerces tag values back and reads the status from error
ClickStackSpanAttributes, ResourceAttributes, LogAttributes and metric Attributes are Map(String, String): every value a stringthe dashboard adapter coerces numbers and booleans back, except on id-like keys
GreptimeDBevery attribute key becomes a column of opentelemetry_traces (schema grows per new key; Int64 may widen to Float64); the trace table is partitioned 16 ways on the first trace_id characterdocumented; quote the dotted column names in SQL
Zipkinevery attribute becomes a string tag (ints, bools and doubles unquoted, arrays comma-joined without brackets, kvlists as JSON); span links are dropped; INTERNAL spans lose their kindthe dashboard adapter coerces numeric and boolean strings back, except on id-like keys
Langtracekeeps string and int attributes only; drops bool and double; reads only resourceSpans[0] of a batchdocumented; evaluation-only backend (#224)
Langfusedrops the exporter's tool. prefix on TOOL observationsthe dashboard adapter restores it
OpenObserve, Uptrace, Lokidots in attribute names become underscores on storage (llm.model_name → llm_model_name), irreversiblythe adapters carry a KNOWN_ATTRIBUTES table to map them back (#158); collisions resolve to the name with more dots
Uptrace 2.1attribute keys carry a type suffix on the read side (hermes_session_id::str)stripped by the adapter
LangWatch, Latituderequest caps of 10 MiB and 32 MiBthe span batch is auto-lowered to 64 when content_capture: full, because a 512-span POST of full prompts exceeds receiver limits (#199)
New Relic (generic otlp)payloads over 1 MB rejected; attribute values at most 4095 chars; at most 128 span attributes and 64 resource attributesuse content_capture: preview or a smaller max_export_batch_size (#227)
All (plugin side)log attribute values are capped by max_attribute_length (default 4096, applied after redaction)configurable

H. Reading it back: query API, auth, version splits​

The dashboard tab and the observability skill read fifteen types back through an adapter. Each one speaks a different API and, in four cases, two versions of it.

BackendRead APIRead-side auth vs write-sideVersion splitHow the adapter finds a turn's root
PhoenixGraphQL only, POST /graphqloptional Bearer (write: none)Phoenix 20 removed rootSpansOnly / orphanSpanAsRootSpan; the adapter introspects the schema once per URL and sends parent_id is None insteadparent_id is None; the project is never substituted (#159)
Langfusev3: /api/public/traces, /api/public/traces/{id}, /api/public/observations; v4: /api/public/v2/observations plus /api/public/v2/metricssame Basic pairv3 vs v4: v4 events_only removed /api/public/traces, /observations, /sessions (404 naming events_only); probed once per URL, query_api: v3|v4 pins (#246); cloud retires v3 routes 2026-11-16v3: list is per trace; detail root = observation with no parent, else a synthetic root (#159). v4: AGENT observations with isRootObservation
SigNozPOST /api/v4/query_range (query builder) for all three signals; GET /api/v1/traces/{id}always required, even on localhost: sent as both SIGNOZ-API-KEY (PAT) and Authorization: Bearer (session JWT; the OSS build has no PAT endpoint, so the key is a 30-minute JWT)—parentSpanID = ""
Jaegerv1: /api/traces; v2: /api/v3/... (OTLP-JSON, the only API v2 serves)optional Bearer (write: none)v1 vs v2 (#245): probed once per URL, classic first because Jaeger 1.x also exposes an /api/v3 gateway with snake_case params; query_api: v1|v3 pins. v2 answers an empty search with 404 "No traces found"client-side: the span with no CHILD_OF reference; every search must name a service
AgentaPOST /api/spans/query (TracingQuery: windowing oldest/newest/limit/order, filtering trees of {field, key, value, operator}; parent_id is null = roots, span_name is/startswith, status_code is/is_not, attributes + dotted key with eq/gt/contains; no duration field; nested or groups dropped)the same ApiKey as the write sidev0.122 verifiedparent_id is null server-side
Gigapipethe three Grafana APIs on one port: Tempo /api/search (TraceQL q, start/end in seconds; select() returns no attributes → cards from /api/traces/{id}, which is pre-1.0 OTLP JSON with instrumentationLibrarySpans and string values), Prometheus /api/v1/* (a range-vector instant query answers empty → counters via query_range at step 15), Loki /loki/api/v1/* (labels only, level label, no scope label)none (Basic when configured)5.x; v5.5.3 verifiednestedSetParent < 0 works server-side
Aspire DashboardGET /api/telemetry/{resources,traces,spans,logs} and /traces/{id} → OTLP JSON with totalCount / returnedCount; filters on 13.6.1: resource, limit, offset, and on logs traceId + search; no time window, no metrics route; ?resource=<unknown> → 404x-api-key when the API is key-protected (anonymous mode: none)13.6+ (the API is new)client-side: the span with no parentSpanId in the fetched trace
VictoriaTraces / Logs / Metricstraces: Jaeger v1 JSON under /select/jaeger (service required, tags on plain keys, operation exact, minDuration with unit, start/end µs; limit ordering undocumented → over-fetch 500); metrics: the Prometheus API on VictoriaMetrics, range-vector instant queries included; logs: POST /select/logsql/query → NDJSON (_msg, _time RFC 3339 ns, trace_id, span_id, severity_number, scope.name, dotted attribute fields), stats by (…) for loggers; no Loki query APIoptional Basic (same as write)VictoriaTraces pre-1.0client-side: the span with no CHILD_OF reference
ClickStackClickHouse HTTP (POST /?database=…&default_format=JSON, output_format_json_quote_64bit_integers=0): SQL on otel_traces, otel_logs, otel_metrics_*; ClickHouse answers a bad query with 400/404 and a text body, so every error is a backend error, never "not found"ClickHouse Basic auth (api / api on the all-in-one; default is localhost-only), separate from the write-side key—ParentSpanId = '' server-side
GreptimeDBtraces: Jaeger v1 JSON under /v1/jaeger (service required, start/end µs, tags JSON, operation exact, minDuration with unit; limit keeps arbitrary traces, so the adapter over-fetches 500 and cuts the page); metrics: SQL raw samples per metric table (PromQL query_range works, a range-vector instant query answers an empty body); logs: SQL on opentelemetry_logs with json_get_string(log_attributes, '["key"]') bracket pathsoptional Basic (same as write)—client-side: the span with no CHILD_OF reference
ZipkinAPI v2: /api/v2/traces (serviceName, spanName exact, annotationQuery as k=v and k2=v2, endTs + lookback in ms, minDuration in µs), /api/v2/trace/{id}optional Bearer for a proxy (write: none)—client-side: the span with no parentId; annotationQuery matches any span of a trace, so roots-only re-checks the root
Tempo / LGTMTempo /api/search (TraceQL) + /api/traces/{id}; Prometheus /api/v1/query (range vectors) and /api/v1/query_range; Loki /loki/api/v1/query_rangenone—TraceQL nestedSetParent < 0 with spss=1; trace ids re-padded to 32 hex digits because Tempo drops leading zeros
Uptrace/internal/v1 in two dialects (2.1: /spans/{p}, /logs/{p}, /metrics/{p}, ms times, ::str keys; 2.0: everything under /tracing/{p}); MQL for metricsa user token (user_token_env), not the DSN's project token2.0 vs 2.1, probed once per process via /internal/v1/logs/{p}/systems (2.0 answers with the SPA's HTML) (#243, #298)2.1: _parent_id = "" server-side; 2.0: client-side from parentId
OpenObservePOST /api/{org}/_search?type=traces|metrics|logs with SQL; stream schema endpoint for columnssame Basic pairthe parent column is named reference_parent_span_id, reference (a JSON string) or parent_span_id depending on the build; tried in order (#299)that column IS NULL OR = ''
LangSmith, Honeycomb, Weave, Elastic, Parseable, OpenLIT, MLflow, Opik, Laminar, LangWatch, Latitude, otlpno adapter: the tab shows them as unavailable and the Live source or the backend's own UI is the answer—Honeycomb's Query Data API is Enterprise-gated, asynchronous and limited to 7 days—

Metric reads have their own quirks, all about one-shot processes: PromQL increase() never counts a series' first sample, so the Prometheus adapter reads raw range-vector samples and counts a first sample when the series started inside the window (counter_increases, #296); SigNoz's own increase over-counts turns that share a step and stamps each step at its epoch-aligned start, so the adapter reads latest per process and takes the differences itself (#297); Uptrace forward-fills a counter's cumulative value and its histogram count($m) is the number of samples, not observations, so the adapter computes sum($m)/avg($m) (#298); OpenObserve returns numbers as digit strings and lists metric streams without a time bound (#299).

Log reads differ in where the event name lives: Loki and OpenObserve have an event_name column (OpenObserve errors on it until the first event is ingested; the adapter answers empty instead, #268), SigNoz keeps no column for it and uses the event.name attribute, Uptrace keeps it as the event_name attribute while its own eventName is the row kind.

I. Operations: footprint, first run, and upstream bits each folder patches​

Resident memory is after one turn (docker-compose/README.md, Disk and memory); time to healthy is from the folder READMEs.

BackendContainersResidentReady inFirst-run stepUpstream breakage this folder works aroundLicence
Phoenix1under 100 MB~10 snoneimage is latest, unpinned; SQLite dies with the container unless you add a volumeElastic License 2.0
Langfuse6~3 GB20-60 s; traces appear 10-20 s after export (async via MinIO)none (headless init seeds user and keys)LANGFUSE_VERSION=4 is a separate data set: down -v when switchingMIT core + EE folders
SigNoz4 (+2 one-shot jobs)—30-60 smandatory admin registration (POST /api/v1/register, 12+ char mixed password) or the collector stays nop and resets connections—OSS
Jaeger v1 / v21~25 MBsecondsnonev1 line ended at 1.76.0 (Dec 2025); v2 pinned 2.21.0Apache-2.0
Agenta11 (trimmed from 15-17)~2 GB (api ~900 MB)~1-2 minsign up + mint a key (mint-api-key.sh)runner (SYS_ADMIN + /dev/fuse), playground services, cron, mobile UI, Composio and Traefik (Docker socket) dropped; nginx front door; PostHog key emptied; ~4 GB of imagesMIT (ee/ separate)
Gigapipe2~16 MB + ~300 MB ClickHouse~10 snoneBULK_MAX_AGE_MS=100 so a one-shot turn is queryable at once; qryn-view UI built into the imageAGPL-3.0
Aspire Dashboard1~75 MB~10 snone (anonymous mode)the app user cannot write a volume mounted on a fresh path ("Access to the path is denied" crash loop) → the volume sits on /home/app; PERSISTENCE_MODE=Resume or the data vanishes on a clean shutdown; no shell for a healthcheckMIT
VictoriaTraces / Logs / Metrics3~35 MB totalseconds (/health → OK)noneimages built from scratch (no shell): Compose cannot health-check them; a 30 s -search.latencyOffset hides the newest samples; VictoriaTraces pre-1.0, pinned v0.12.0Apache-2.0
ClickStack1 (four processes)~1.1 GB~30 s to healthy; OTLP receivers ~20 s after the first accountcreate an account (no default login) and mint the ingestion key: mint-api-key.shupstream leaves ClickHouse unpublished and its default user localhost-only → the folder publishes 8124 and the docs use api / api; image 2.6 GB; 3-day TTLMIT
GreptimeDB1~95 MB~10 snonehost port 4398 (upstream 4000 collides with dev servers; a Node server on [::1]:4000 makes localhost:4000 hang); pinned v1.2.1 because v1.3 introduces an opt-in greptime_trace_v2 model in a separate tableApache-2.0
Zipkin1~300 MB~20 s (JVM)nonethe stock image has no OTLP: the folder runs ghcr.io/openzipkin-contrib/zipkin-otel 0.3.0 (Zipkin 3.6.0), marked experimental upstream; in-memory store capped at 500 000 spansApache-2.0
Tempo2~260 MBsearch lags 20-30 snonemetrics-generator removed from config; attribute limit raised (G)AGPL-3.0
Grafana LGTM1—~30 sGrafana admin/adminspanmetrics connector dimensions set to the plugin's attribute names; skipped by all.sh because of port 3000 / 4318AGPL-3.0
Uptrace4—~20 s (upstream's compose: ~60 s)none (DSN seeded)trimmed from upstream's 12 services; pinned 2.1.0-beta.5OSS + premium features
OpenObserve1—secondsnonehealthcheck is a false negativeOSS
Parseable OSS2~190 MBsecondsnonecollector in front: protobuf refused, no API keysAGPL-3.0
Elastic3~1.6 GBwait for cluster yellow/greennonedisk watermark disabled (a Docker VM above 90 % makes every shard unassigned with no plugin-side error); 90 s exporter timeout + queue in otel.yamlElastic
OpenLIT2~560 MB~30 slogin user@openlit.io2.1.0 image needs the collector config mounted or nothing listens on 4318 (upstream PR 1588); "Unknown table otel_traces" until the first spanApache-2.0
MLflow1~1.2 GB~20 snoneneeds MLflow ≥ 3.6 and a SQL store (the file store refuses OTLP); host port 5001 because macOS AirPlay owns 5000Apache-2.0
Comet Opik7 + a bucket job~2.1 GB~1-3 minnonepython-backend dropped; OPIK_USAGE_REPORT_ENABLED=falseApache-2.0
Laminar5 + init job~600 MB~1 minmint a key (B)published frontend skips creating the Quickwit spans_v2 index: a quickwit-init job creates it or span search stays emptyApache-2.0
LangWatch5~1.9 GB~5 min of ClickHouse migrations (/api/health → 204)onboarding wizard mints the keyNLP and evaluator services dropped; latest tagApache-2.0
Latitude13—~2 min after an ~18 GB pullmagic-link sign-in via Mailpit; create project; copy keyships its own Hermes plugin: running both double-traces every turnMIT
Langtrace3~780 MB—login admin@langtrace.aiupstream ClickHouse healthcheck on localhost resolves to ::1 and never passes; 4.45 GB image; last release 2025-04AGPL-3.0
Sigiro176 MBsecondsnoneclosed source, latest onlyproprietary
OpenTelemetry Collector1~40 MB~1 spick COLLECTOR_TARGET (default prints to docker logs)distroless image (health from the host); ${env:} works in values only, so a custom header name is a copied targetApache-2.0
otel-desktop-viewer1~105 MB~1 snonethe image's CMD is in-memory; the folder adds --db on a volume. No shell or curl in the image (healthcheck from the host)Apache-2.0
otel-tui1 (docker run -d -i -t, no compose)~30 MBsecondsdocker attachneeds a TTY and an open stdin; no ENTRYPOINT (/otel-tui)Apache-2.0
Maple Local1~340 MBseconds (up -d --build)nonebuilt from a release tarball; needs libchdb.so beside the binaryFSL-1.1 (source-available)

Anonymous usage telemetry is switched off in every file where upstream has a knob (OpenLIT, Opik, Laminar, Langtrace, Tempo, Grafana, Parseable; ClickStack's Next.js telemetry is off in the image).

J. Not in this directory, and why​

CandidateReasonIssue
AgentOpsbuilds from source, needs the Supabase CLI and a JWT exchange the plugin cannot do#231
Monoclean SDK, not a server#14
Helicone, Lunaryno OTLP ingest#232
Pydantic Logfireall three signals at logfire-{us,eu}.pydantic.dev/v1/*; the header is a bare write token; dashboards stay empty on cumulative metrics, so delta is needed; self-host is enterprise-only; unbuilt until there is an account to verify against#225
Datadogagentless intake otlp.<site>/v1/*, dd-api-key on every signal and dd-otlp-source: llmobs on traces to land in Agent Observability; metrics intake accepts delta only; payload caps 512 KiB / 5.1 MiB / 15 MiB; 3-5 min delay#226
New Relicotlp.nr-data.net/v1/*, api-key: <license key>; 1 MB payload cap, 4095-char values, 128 span attributes; prefers delta#227
Azure Monitorno static-key OTLP endpoint: needs a Microsoft Entra bearer from the Collector's azure_auth extension; requires delta and exponential histograms; docs-only through the collector/ folder's azuremonitor target (azure_monitor exporter, connection string, cumulative_to_delta), not verified#228
(all four)work today through generic otlp with metrics_temporality: delta; explicit types tracked#232
Honeycomb, W&B Weave, LangSmith, telemetry.devSaaS only; documented on the site, nothing to compose—

Part 2: each backend's quirks and the treatment​

Format: quirk → treatment (issue).

Phoenix (phoenix)​

  • Ingest and UI on one port, no redirect from the root → the endpoint must end in /v1/traces; the resolver uses it as given.
  • 405 on /v1/metrics and /v1/logs → a traces-only spec; no metrics or logs exporter is created; banner says "(traces only)" (#160). A collector in front can still take them: metrics: true on the entry overrides.
  • Reads OpenInference, not GenAI → the plugin emits both; tokens are on the api.* spans and Phoenix totals them itself.
  • No default endpoint, despite what early docs promised (#93) → requires endpoint.
  • status=error on Phoenix means the root span's status, while the Live source means any span errored (#293); documented as a semantic difference.
  • GraphQL is the only read API; Phoenix 20 dropped the rootSpansOnly argument (every search became a 502 on 20.20.0, #293) → the adapter introspects the Project type once per URL and switches to parent_id is None. Legacy builds default orphanSpanAsRootSpan on, which leaks api.* spans whose parent was paged out → forced off.
  • A project is the service → the service filter is none; a missing project_name is an error that lists the projects, never a silent fallback (#159).
  • attributes comes back as one nested JSON string → flattened to dotted keys by the adapter.

Langfuse (langfuse, v3 and v4)​

  • Vendor ingest prefix /api/public/otel → langfuse_traces_url completes a root URL, the prefix, or the full URL (a root URL used to be posted as-is and every export died with a 405 that only the opentelemetry logger saw; PR #169, and the reason #167 added the per-batch SUCCESS|FAILURE lines to debug.log).
  • Basic auth from two keys, plus x-langfuse-ingestion-version: 4 → built by the resolver; both keys required.
  • Vendor LANGFUSE_* variables are often set for other tools, and used to switch on export to Langfuse Cloud with full content capture → they fill credentials only; OTEL_LANGFUSE_* is the opt-in (#258, #259, PRs #255, #261, #263).
  • docker.io/minio/minio left Docker Hub → the compose uses cgr.dev/chainguard/minio, as upstream does (PR #340).
  • Traces only → metrics=False, logs=False in the spec.
  • v4 events_only removed the v3 trace endpoints (/api/public/traces, /observations, /sessions) → the adapter probes GET /api/public/traces?limit=1, treats a 404 naming events_only as v4, and reads /api/public/v2/observations plus /api/public/v2/metrics instead; query_api: v3|v4 pins (#246). v4 observations carry no input, output or per-span usage; trace totals come from the metrics query.
  • OTLP-ingested observations keep the span attributes verbatim under metadata.attributes; Langfuse's own usage is zero-filled on non-GENERATION types → the adapter reads the attributes first and skips usage on OTLP observations, which fixed the header showing 0 tokens (#346).
  • Langfuse drops the tool. prefix on TOOL observations → restored by the adapter.
  • Ingest is asynchronous through MinIO → traces appear 10-20 s after a SUCCESS.
  • v3 and v4 do not share data → down -v when switching LANGFUSE_VERSION. Port 3000 collides with Grafana (LGTM) and Latitude.

LangSmith (langsmith)​

  • OTLP since 1.26 (#415): POST /otel/v1/traces on api.smith.langchain.com with x-api-key and an optional Langsmith-Project header; traces only. A regular backends: entry, so it fans out with the rest.
  • The run-tree client (langsmith_backend.py, selected by LANGSMITH_TRACING=true) is gone; LANGSMITH_API_KEY / LANGSMITH_PROJECT / LANGSMITH_ENDPOINT are fallback values once OTEL_LANGSMITH_* or the entry names the backend.

SigNoz (signoz)​

  • Fresh volume: OpAMP pushes a nop pipeline to the collector until an admin exists, so every healthcheck is green and every OTLP POST gets a connection reset (#239) → the README's headless POST /api/v1/register recipe; check GET /api/v1/version → setupCompleted.
  • Cloud key travels as signoz-ingestion-key, not Authorization → the resolver sets it on the trace, metric and log exporters; self-hosted needs none.
  • Recommends delta; exponential histograms delta-only and self-hosted-only → temporality="delta" on the signoz spec (#233); SigNoz Cloud does not accept exponential histograms by default.
  • The query API sits on the UI port and requires auth even on localhost → query_port: 3301 plus api_key_env; the adapter sends the key as both SIGNOZ-API-KEY and Authorization: Bearer, because the community build has no PAT endpoint (/api/v1/pats answers the SPA page) and the only credential is the 30-minute session JWT from POST /api/v2/sessions/email_password (#297, #302).
  • Trace detail POST /api/v1/traces/{id} answers the SPA page on v0.119 → GET first, POST only after a 4xx (#297). Histogram parts are dotted (hermes.tool.duration.sum / .count) → the pair is folded into one histogram.
  • Its increase over-counts one-shot turns that share a step, and steps are stamped at an epoch-aligned start → the adapter reads timeAggregation=latest grouped by service.instance.id, opens the window on the step grid and computes increases itself (#297).
  • No column for the OTLP event_name field → events filter on the event.name attribute (#268). A list panel returns only the columns asked for, and numeric attributes come back empty unless requested as float64 (#297).
  • The autocomplete catalogue is one bounded page → one search per metric namespace.
  • Ports remapped (4318 → 4328, 4317 → 4327, 8080 → 3301) so it coexists with LGTM and Jaeger.

Jaeger (jaeger, v1 and v2)​

  • Traces only → metrics=False, logs=False in the spec; a forced metrics: true is expected to 404.
  • Not LLM-aware → tokens and messages are plain tags; fine for a span-tree look.
  • v2 serves only /api/v3 (OTLP-JSON) and 404s the classic routes; Jaeger 1.x also exposes an /api/v3 gateway with snake_case parameters → the adapter probes classic first, then v3; query_api: v1|v3 pins (#245, PR #338). v3 silently ignores attributes[k]=v, so the tag map goes JSON-encoded in query.attributes; an empty search is a 404 "No traces found".
  • tags= and operation= match any span of a trace → with roots-only the adapter re-checks the root itself (#295); the root is the span with no CHILD_OF reference.
  • No default endpoint (#93).
  • Every search must name a service → service_name, else resource_attributes.service.name, else hermes-agent (the viewed profile's name with service_name_from_profile, #395).
  • The query port is the UI port, not the OTLP port → set query_port (or ui_url) when the entry's endpoint is :4318.
  • Host port 4318 collides with LGTM.

Agenta (agenta)​

  • Traces only, protobuf only, behind a project key → type: agenta completes the origin to /api/otlp/v1/traces, sends Authorization: ApiKey, defaults to the cloud host for region, and skips the entry when no key resolves.
  • Sign-ups are open and sessions are header-based SuperTokens → mint-api-key.sh signs up (or in), reads st-access-token from the headers and mints a key with POST /api/keys.
  • Server-side adapters for OpenInference, OpenLLMetry and OTel GenAI type every span and compute ag.metrics.tokens cumulatively down the tree; the root's total equalled the sum of the api.* spans → no root_usage preset.
  • Attributes are stored as nested objects per namespace and resource attributes are dropped → the adapter flattens to dotted keys, drops Agenta's ag.data / ag.meta copies and labels the resource with the entry's service name.
  • POST /api/spans/query: parent_id is null for roots, attributes with a dotted key, status_code is/is_not, contains for free text, no duration field, and a nested or group is accepted but silently dropped → every search-bar field server-side except the minimum duration, with free text as two flat queries (input, then output) merged client-side; service none (the key is the project); ingest is asynchronous (a few seconds).
  • The upstream stack is 15-17 containers with a FUSE sandbox runner and Traefik on the Docker socket → the folder keeps eleven (nginx front door on 3060), enough for ingest and the UI's tracing views.
  • Verified 2026-10-11 (api v0.122.x): 7 spans typed agent / chat / tool from one turn; dashboard verify harness PASS 35 / SKIP 13 from a fresh four-turn batch (skips = UI deep link, metrics, logs); ~2 GB resident.

Gigapipe (gigapipe, formerly qryn)​

  • One port for OTLP ingest and the Tempo, Prometheus and Loki APIs → the lgtm adapter subclass defaults every sibling URL to the entry's origin; 3100 remapped to 3110 (LGTM's Loki).
  • /api/search with a TraceQL q returns one row per trace but select() yields no attributes (and without q, one row per span) → the card is built from /api/traces/{id}: the root, or the first matching span on a widened search.
  • /api/traces/{id} is pre-1.0 OTLP JSON: resourceSpans → instrumentationLibrarySpans, every attribute a stringValue, no kind, the resource reduced to collector=gigapipe with the service on serviceName → reshaped into batches / scopeSpans, values coerced, one batch per service.
  • A range-vector instant query (name[1h]) answers an empty matrix; query_range works and PromQL is served from the 15 s rollup → counters via query_range at step 15 with the five-minute look-behind and counter_increases (#1027 sidestepped by bucketing here); gauges unchanged.
  • Delta metric points are rejected per point → cumulative default, no preset.
  • Loki streams carry every attribute as a label (hermes_session_id, trace_id, span_id), a level label and no scope name → LogQL stages on those labels, level=~ for the severity floor, logger filter and loggers list unsupported; the nanosecond cursor as end dropped the rest of that second → the bound is rounded up to the next second and strictly_older() cuts.
  • Trace ids upper-case from a tag search, lower-case from TraceQL → lower-cased and padded.
  • Verified 2026-10-11 (v5.5.3): 7 spans, 28 log lines, 27 metric names from one turn; ~16 MB + ~300 MB resident. Dashboard-verify PASS 46 / SKIP 5 (UI link, loggers).

Aspire Dashboard (aspire)​

  • Standard /v1/* on the OTLP/HTTP port, UI and read API on another → endpoint is the OTLP port, query_port the UI port; anonymous mode needs no key, x-otlp-api-key from api_key otherwise.
  • Stores Gauge, Sum and Histogram points; exponential histograms are rejected → no preset, keep metrics_histogram: explicit.
  • A resource is one service.name + service.instance.id, i.e. one Hermes process; resource=hermes-agent works with one process and fails once there are two → the adapter lists /api/telemetry/resources and repeats resource=<displayName> for every instance of the service.
  • The read API (13.6+) answers OTLP JSON but has no time window and only resource / limit / offset (+ traceId / search on logs) filters, verified empirically on 13.6.1 → the adapter fetches a bounded page (search_fetch 500 traces, logs_fetch 5000 records) and applies the window, every structured filter, roots-only and the cursor itself; status = ok and free text are therefore client-side but possible. No metrics route → metrics: false on the source.
  • Severities arrive as Information / Warning / Critical → the level comes from severityNumber; records carry an aspire.log_id attribute, dropped on read.
  • The image runs as the unprivileged app user and a named volume on a fresh path is root-owned → the compose mounts the volume on /home/app and points ASPIRE_DASHBOARD_DATA_DIRECTORY under it; PERSISTENCE_MODE=Resume keeps the SQLite database across restarts.
  • /traces/detail/{traceId} is a real UI route (an unknown route is 404) → the "open in" link.
  • Verified 2026-10-11 (13.6.1): 7 spans, 28 log rows (12 with a trace id) from one turn, hasMetrics: true; about 75 MB resident. Dashboard-verify PASS 48 / SKIP 3 (metrics).

VictoriaTraces / VictoriaLogs / VictoriaMetrics (victoria)​

  • Three stores, three hosts, three non-standard paths → endpoint, metrics_endpoint, logs_endpoint (origins or full URLs) resolve to each store's OTLP route; this type introduced metrics_endpoint / logs_endpoint on _ResolvedBackend, honoured by both exporters (every other type still derives /v1/metrics / /v1/logs from the traces URL). A signal whose store is not named is off; an explicit true without it skips the entry with a hint.
  • VictoriaLogs keys streams on every resource attribute by default and service.instance.id changes per Hermes process → the type sends VL-Stream-Fields: service.name on logs (headers overrides).
  • VictoriaMetrics stores OTLP names "as is" unless -opentelemetry.usePrometheusNaming → the bundled stack sets the flag so the dashboard's Prometheus reader (and the LGTM conventions) apply; without it the names are dotted and nothing is a counter to the reader.
  • VictoriaTraces' Jaeger-compatible API types every tag string and keeps the status as an error tag ("true" / "false") → the adapter (the Jaeger adapter on /select/jaeger, pinned to v1) coerces values and reads error; limit ordering undocumented → search_fetch 500.
  • VictoriaLogs has no Loki query API → a LogsQL reader (_time:[…), field filters, _msg:~, sort by (_time desc), stats by (scope.name) count()); field names keep their dots (scope.name).
  • A 30 s -search.latencyOffset on VictoriaMetrics and VictoriaTraces delays the newest samples; VictoriaTraces is pre-1.0 (pinned v0.12.0).
  • Verified 2026-10-11: 7 spans, 28 log rows (one stream, 12 with a trace id), 27 metric names from one turn; about 35 MB resident for the three; dashboard-verify PASS 46 / SKIP 5.

ClickStack (clickstack)​

  • Ingest is gated by the team's ingestion key as a bare authorization header (bearertokenauth accepts Bearer too) → api_key / api_key_env with OTEL_CLICKSTACK_API_KEY / HYPERDX_API_KEY fallbacks; a missing key skips the entry, because every batch would be a 401.
  • The Collector's OTLP receivers do not exist until a team does (config arrives over OpAMP) → mint-api-key.sh registers the first account through POST /api/register/password, reads db.teams.findOne().apiKey from the all-in-one's MongoDB and waits for 4318 to answer.
  • All three signals in the standard OTel ClickHouse tables; metric names stay dotted, histograms are one table with Count and Sum → the adapter lists <name>.sum / <name>.count and reads raw cumulative samples with the five-minute look-behind (#296).
  • Every map value is a string → coerced back on read except on id-like keys (coerce_tag in dashboard/backends/shapes.py, which the Zipkin, Victoria and Gigapipe adapters use too).
  • The bundled default ClickHouse user answers only from inside the container → user: api, password: api (defined with open networks by the image) on the entry; ClickHouse answers a bad query with 400/404 and text, mapped to a backend error.
  • The Collector lower-cases SeverityText → upper-cased again on read; otelcol_* self-metrics land in otel_metrics_gauge → metrics_match keeps the plugin's namespaces.
  • Ports remapped to 3050 / 4407 / 4408 / 8124 (upstream 8080 / 4317 / 4318 / 8123 collide with hn.fm, LGTM, Jaeger and Uptrace); 3-day TTL; 2.6 GB image, about 1.1 GB resident.
  • Verified 2026-10-11 (2.40.0): 7 spans, 28 log rows, 6 sum and 7 histogram metrics from one turn; dashboard-verify PASS 51.

GreptimeDB (greptime)​

  • OTLP lives under /v1/otlp → the resolver completes an origin or the prefix to /v1/otlp/v1/traces; the other two URLs are derived.
  • Traces need X-Greptime-Pipeline-Name: greptime_trace_v1 and logs refuse it (400 both ways, verified 2026-10-11) → headers carries it for traces only; metrics_headers / logs_headers carry just the db and auth headers. The exporters used to fall back to the trace headers when a per-signal set was empty; since this type an empty set means "none" (#363).
  • X-Greptime-DB-Name from db / db_env on every signal; Basic auth from user + password only when both resolve.
  • Stores cumulative and delta as sent, rejects exponential histograms → no temporality preset; keep metrics_histogram: explicit.
  • Attributes become typed columns; metrics one table per name in Prometheus spelling with greptime_timestamp (ms) → the adapter reads raw samples with SQL and does the counter maths itself (counter_increases, first-sample counting), because the PromQL route answers a range-vector instant query with an empty body.
  • Jaeger-compatible API under /v1/jaeger speaks the v1 grammar with typed tags → the Jaeger adapter, mounted on a query_path, pinned to v1 (no v3 probe); limit keeps arbitrary traces, not the newest → search_fetch 500 and the page is cut client-side.
  • Logs in opentelemetry_logs with log_attributes JSON → dotted keys need the bracket path json_get_string(log_attributes, '["hermes.session_id"]'); a bare dotted path descends and finds nothing.
  • No retention by default (x-greptime-hints: ttl=7d under headers); no documented deep link into the embedded dashboard → no "open in" link unless ui_url is set.
  • Host port 4398, not upstream's 4000: dev servers own 4000 (a Node server on [::1]:4000 made localhost:4000 hang while the container answered on 127.0.0.1).
  • Verified 2026-10-11 (v1.2.1): 7 spans, 28 log rows and 27 metric tables from one turn; about 95 MB resident.

Zipkin (zipkin)​

  • The stock openzipkin/zipkin image has no OTLP endpoint → the folder runs the zipkin-otel contrib image (0.3.0 = Zipkin 3.6.0), where POST /v1/traces lives on the query port 9411 next to the UI and API v2. Marked experimental upstream; needs 128-bit trace ids, which OTLP always sends.
  • Traces only → metrics=False, logs=False in the spec: /v1/metrics is 404; /v1/logs is accepted (202) and stored only as an annotation on a span named by trace id, span id and an event.name attribute, which the plugin's log records do not carry.
  • Every attribute becomes a string tag ("7589", "true"); arrays lose their brackets; links are dropped; INTERNAL spans have no kind → the adapter coerces numbers and booleans back on read, except on id-like keys.
  • Status is the otel.status_code tag plus an error tag carrying the message on ERROR; events are annotations whose value is the event name or "name":{...} → mapped back to OTLP status and events.
  • Search is serviceName + annotationQuery (k=v and k2=v2, no quoting, so a value containing and cannot be expressed), spanName exact, minDuration in µs, window endTs + lookback in ms → model, session, tool, error status and the raw query go server-side; the kind prefix and roots-only are client-side, and the root is re-checked because tags match any span (#295).
  • A span without service.name has no localEndpoint and is invisible to search → the plugin always sets one.
  • In-memory store (STORAGE_TYPE=mem, 500 000 spans, gone on restart); UI default lookback 15 min, /api/v2/traces default 24 h (QUERY_LOOKBACK).
  • Verified 2026-10-11: one turn stored 7 spans (agent root, llm.*, three api.*, tool.terminal, tool.read_file), about 300 MB resident.

Grafana Tempo (tempo)​

  • Traces only, and the type keeps metrics and logs off by default even behind a collector that would take them → a traces-only spec; set metrics: true / logs: true explicitly, or use lgtm.
  • Attribute values truncated at 2048 bytes by default, which cut gen_ai.input.messages → tempo.yaml raises distributor.max_attribute_bytes to 131072.
  • Search lags 20-30 s behind ingest.
  • Search returns the first limit matches in block order, not the newest (verified on 3.0.3: limit=3 over four traces skipped a newer one about half the time) → the adapter over-fetches search_fetch (default 500) and sorts.
  • minDuration is ignored next to q (verified otel-lgtm 0.34) → duration >= Nms goes into the TraceQL, or client-side with a raw query.
  • grafana/otel-lgtm:latest on 2026-09-20 listened on the collector's exporter ports (4417 / 4418 / 9099) instead of the mapped ones, and Tempo took 541 s to start → the image is pinned to 0.34.0 (#194).
  • Span sets have no name field and are capped at three spans → roots via nestedSetParent < 0 with spss=1; free text as a second span set ANDed at trace level (#296).
  • Trace ids lose leading zeros → re-padded to 32 hex digits to match Loki and the live store.
  • Older builds reject | select(...) with a 4xx → retried once without it.
  • No UI of its own → ui_url pointing at Grafana gives an Explore link (grafana_datasource_uid, default tempo).

Grafana LGTM (lgtm)​

  • Functionally the generic otlp type → kept as a distinct type so the config says what it is and the banner reads LGTM.
  • Prometheus wants cumulative; delta is dropped unless otlp-deltatocumulative is on → SDK default; native histograms need Prometheus 3.8+ / Mimir ingestion flag.
  • Two processes with byte-identical Resources interleave into one Prometheus series (a sawtooth; rate() reported calls on an idle model, #79) → service.instance.id, service.version and process.pid on every Resource (PR #115).
  • increase() never counts a series' first sample, so a one-shot run reads 0; query_range can miss a sample between evaluation instants → the adapter reads raw range-vector samples with a 300 s look-behind and counts a first sample when the series is new (#296).
  • Names are mangled (hermes.token.usage → hermes_token_usage_total, units appended, {token} units dropped) → otlp_metric_name reverses it for display (#284).
  • Loki stores resource and record attributes as labels or structured metadata depending on otlp_config, underscored → every filter is a pipeline stage; windows clamped to 30 days (max_query_length).
  • A tempo entry and an lgtm entry feeding the same Tempo return duplicate span ids until compaction → de-duplicated in the detail.
  • The bundled collector's spanmetrics connector uses the plugin's attribute names (llm.provider, llm.model_name, openinference.span.kind, hermes.session.kind, hermes.tool.outcome) → RED metrics without Python emitting them.
  • Keeps 3000 and 4318, so it cannot run beside Langfuse / Latitude or Jaeger; all.sh skips it.

Uptrace (uptrace)​

  • Auth is a DSN in the uptrace-dsn header, not a key → dsn / dsn_env / OTEL_UPTRACE_DSN / UPTRACE_DSN; the resolver never parses it. The exporter still needs endpoint next to dsn (#298).
  • Prefers delta and converts cumulative → preset delta (#233); exponential histograms recommended.
  • The read side needs a user token, not the project token → user_token_env (or UPTRACE_USER_TOKEN).
  • A 401 on the dialect probe used to cache "2.1" for the process lifetime → auth errors cache nothing (#298, PR #309).
  • Two API dialects: 2.1 (/spans/{p}, ms times, ::str key suffixes) and 2.0 (/tracing/{p}/..., bare keys); 2.0 answers the probe with the SPA's HTML → probed once per process, HTML counts as 2.0 (#243, #298).
  • Uptrace writes its own spans (service serve) and log lines into the same project → trace and log queries are pinned to the agent's service name; service_name: off removes the pin.
  • A counter's $m is forward-filled at each interval end and delta($m) loses the first interval; histogram count($m) is the number of samples, not observations → the adapter computes increases itself per service_instance_id and derives the count as sum($m)/avg($m) (#298).
  • _span_id is an "unsupported attr" for per-span logs → _parent_id is used; the end bound is floored to the second.
  • Trimmed from upstream's 12-service compose to 4 containers (20 s instead of 60 s). Host ports 5432, 8123, 9000 collide with other Postgres / ClickHouse stacks.

OpenObserve (openobserve)​

  • Org in the URL path (/api/<org>/v1/traces) and a stream-name header → the resolver builds Basic auth and the header; a wrong path 404s silently, a bad stream name is a 400.
  • Attribute names are flattened with underscores on storage, irreversibly; the first adapter reversed that with replace("_", ".") and invented llm.model.name → _attrs.KNOWN_ATTRIBUTES, a table of the documented names, drift-tested against span-attributes.md (#158, PR #165).
  • The parent-span column is named differently across builds (reference_parent_span_id, reference as a JSON string, parent_span_id) → tried in order; only a 4xx moves to the next (#299).
  • A query naming a column the stream has never seen is a 400 (event_name before the first event) → the adapter answers empty (#268, #299); free text ORs only over columns the schema has.
  • Each OTLP metric is its own stream; _total dropped; histograms split into _sum / _count / _bucket streams; the stream listing has no time bound → count_scope: stream.
  • Numbers sometimes come back as digit strings → converted.
  • A trace detail needs a window → trace_window_days (default 90).
  • One container, no port collisions: the safe default.

Parseable (parseable for Cloud / Enterprise, otlp + collector for OSS)​

  • OSS refuses protobuf (400 Protobuf ingestion is not supported in Parseable OSS) and has no API keys (any X-API-Key is a 401) → the folder runs an OTel Collector in front that re-encodes to OTLP/JSON and adds Basic auth; the plugin uses type: otlp against the collector (#238). Never type: parseable against OSS.
  • Cloud / Enterprise want X-API-Key plus one dataset per signal (X-P-Stream, X-P-Log-Source) → type: parseable carries three header sets, one per exporter (#57, PR #58); the ingestor URL, not the query URL. Env-var mode once sent the traces stream header to the metrics endpoint, so metrics landed in the wrong dataset (#90, PR #110).
  • Tag hermes-traces with agent-observability to get the Agents view.

Honeycomb (honeycomb)​

  • Key in x-honeycomb-team (#20, PR #21); keys are region-specific → region: us|eu picks the host; a US key on EU is a 401.
  • x-honeycomb-dataset is honoured only by Classic keys and ignored by Environments keys → dataset sent on all three exporters (correct for Classic, harmless no-op for modern); leave it unset on a modern key.
  • Query Data API is Enterprise-only, asynchronous and 7-day limited → no adapter.
  • Docs once showed x-honeycomb-team: ${HONEYCOMB_API_KEY} with no implementation, so users sent the literal string and got 401 → ${VAR} expansion in header values (#92, PR #112). A global headers: block used to override the resolver's auth → per-backend headers win (#105, PR #145).

W&B Weave (weave)​

  • Had no plugin-namespaced key, so WANDB_API_KEY + entity + project alone used to switch export on → OTEL_WEAVE_API_KEY / _ENDPOINT / _BASE_URL opt in (PR #255).
  • Routes by resource attributes wandb.entity / wandb.project → copied from the entry (or resource_attributes); both must be present for env-var mode to select it; one Weave project per process.
  • Dedicated Cloud uses a different path prefix (/traces/otel/v1/traces) than SaaS (/otel/v1/traces) → _weave_endpoint_from_base.
  • W&B documents the endpoint for traces → a traces-only spec.
  • Expects wandb.thread_id and wandb.is_turn=true on the root → set by the plugin.

Elastic (elastic)​

  • Authorization: ApiKey, not Bearer; a self-hosted EDOT needs none → header omitted when no key; a key alone never opts in because there is no default host.
  • Routes by data_stream.dataset / data_stream.namespace resource attributes with a strict character set and an appended .otel → validated ([a-z0-9_.], no -); invalid value skips the entry; conflicting entries fail at init.
  • ES does not handle cumulative histograms and converts cumulative counters lossily → preset delta.
  • Silent drop: the exporter reports SUCCESS while the collector's bulk indexer fails on a red cluster; a Docker VM above 90 % full leaves every shard unassigned → the compose disables the disk watermark, otel.yaml adds a 90 s timeout, retries and a queue; wait for yellow/green before the first export.
  • hermes.tool.duration is mapped as histogram through EDOT but as exponential_histogram on Elastic Cloud managed OTLP, so no Lens median works on both → the Kibana panel is ES|QL over the tool spans' duration (#333, PR #345).
  • No adapter; Kibana is the UI (dashboards.ndjson ships a five-panel dashboard). Elastic Cloud managed OTLP was verified by the contributor (#333), not re-run by the maintainer; both the .ingest. and the classic .apm. hosts accept the data (#318, PRs #319, #321).

OpenLIT (openlit)​

  • The 2.1.0 image does not listen on 4318 unless the collector config is mounted → otel-collector-config.yaml vendored (upstream PR 1588 will replace it); the UI port does not proxy /v1/*.
  • Optional Bearer key scopes to org / project / environment → omitted when none; data lands in the INIT_DB_* defaults.
  • Stores cumulative as-is, including exponential histograms → no preset.
  • Harmless "Unknown table otel_traces" errors until the first span.

MLflow (mlflow)​

  • Mandatory x-mlflow-experiment-id (400 without it) → default "0"; experiment_id / OTEL_MLFLOW_EXPERIMENT_ID / MLFLOW_EXPERIMENT_ID.
  • Needs MLflow ≥ 3.6 and a SQL tracking store; the file store refuses OTLP → the compose uses SQLite.
  • 404 on /v1/metrics and /v1/logs → a traces-only spec.
  • Computes trace token totals and cost itself (mlflow.trace.tokenUsage, mlflow.trace.cost) from the span attributes; reads the root → keeps the roll-up.
  • macOS AirPlay owns port 5000 → host port 5001.

Comet Opik (opik)​

  • Vendor prefix /api/v1/private/otel → _opik_traces_url completes the UI base, the SDK's …/api override, the prefix, or the full URL; cloud default host.
  • Cloud auth is the bare key in Authorization (no Bearer) plus Comet-Workspace; self-hosted needs nothing → every header optional and omitted when unresolved.
  • /metrics and /logs under the prefix 404 → a traces-only spec.
  • Sums gen_ai.usage.* over every span, so the root's roll-up doubled trace tokens and the estimated cost (28 560 shown for 14 280 real, #327) → root_usage: false preset; after it, the trace total equals the api.* spans (#353, 1.24.0; one turn re-run live went from 27 914 to 13 953 prompt tokens). Opik maps agent → general, api.* / llm.* → llm; HTTP only, gRPC errors.
  • First start takes one to three minutes while ClickHouse and MySQL migrate.

Laminar (laminar)​

  • Ingest is on the app-server (:8100 here, upstream 8000), not the frontend → endpoint is the app-server base; cloud default api.lmnr.ai.
  • A project key is required on both editions → resolver skips the entry with a clear message instead of a stream of 401s; mint-api-key.sh scripts the key.
  • /v1/metrics answers 200 and stores nothing → metrics=False, logs=True in the spec: metrics off by default, traces and logs on (verified: log records stored).
  • The published frontend skips creating its Quickwit spans_v2 index → a quickwit-init job creates it, or span search stays empty forever.

LangWatch (langwatch)​

  • Vendor prefix /api/otel for all three signals → _langwatch_traces_url completes the base or the prefix; derived metrics and logs URLs.
  • Project key required; a service key additionally needs X-Project-Id → project / LANGWATCH_PROJECT_ID.
  • Sums gen_ai.usage.* over every span (48 598 shown for a turn whose calls total 24 299, #327) → root_usage: false preset (#353, 1.24.0); not re-run after the fix. Common wrong paths (/v1/traces, /api/v1/traces) are rewritten server-side; 10 MiB per request.
  • About five minutes of ClickHouse migrations before /api/health answers 204; the key exists only after the onboarding wizard → mint-api-key.py or the Postgres read.

Latitude (latitude)​

  • Ingest is a separate ingest service (:3002), not the web or API ports → endpoint is its base; cloud default ingest.latitude.so.
  • Both the Bearer key and X-Latitude-Project: <slug> are mandatory; spans without a project are rejected → both required by the resolver; a missing one skips the entry.
  • Only /v1/traces and health routes exist → a traces-only spec. Ingest answers 202; 32 MiB per request by default. Per-span column values were not inspected (the stack was torn down to reclaim disk, PR #331).
  • Latitude ships its own Hermes plugin → running both double-traces every turn.
  • 13 containers, about 18 GB of images; the key is encrypted at rest and cannot be read back from Postgres.

telemetry.dev (generic otlp)​

  • Single Authorization: Bearer header, one key per project and environment → headers with ${VAR}.
  • Reads gen_ai.* and ignores OpenInference; sessions from gen_ai.conversation.id on the root → the plugin sets it.

Langtrace (generic otlp, evaluation only)​

  • Custom path /api/trace, x-api-key → type: otlp with the full URL and headers; URL derivation does not apply, so metrics: false, logs: false.
  • Drops bool and double attributes, reads only resourceSpans[0] of a batch → documented as "partial"; not promoted to a type (#224). AGPL-3.0, stale (last release 2025-04), 4.45 GB image.
  • Upstream ClickHouse healthcheck on localhost resolves to ::1 and never passes → 127.0.0.1 in the compose.

OpenTelemetry Collector (collector/, generic otlp, #368)​

  • Azure Monitor, AWS and Google Cloud want signed or exchanged auth a static-header exporter cannot produce → one contrib Collector (0.162.0) in front, type: otlp at :4448, COLLECTOR_TARGET picks the exporter file merged as a second --config (debug, otlphttp, azuremonitor, aws, googlecloud); all five validate, the first two verified with real turns, the cloud ones documented until there are accounts.
  • The plugin is one hop removed → its debug.log only ever records the Collector's 200; exporter errors, retries and drops live in docker logs hermes-otel-collector.
  • Application Insights expects delta metrics → the azuremonitor target runs cumulative_to_delta on the metrics pipeline (the plugin emits cumulative).
  • ${env:X} substitution works in values, not in header names → a target with a custom header is a copied file.
  • Verified 2026-10-11: debug printed Traces batches of 4 + 1 spans, a Metrics batch of 24 metrics and every log record's resource, scope and body; otlphttp into otel-desktop-viewer added 1 trace / 5 spans, 26 log records and 74 data points from one turn; ~40 MB resident.

otel-desktop-viewer and otel-tui (generic otlp, #367)​

  • Both are stock OTLP/HTTP receivers without auth → type: otlp with the full /v1/traces URL, metrics and logs derived and on; one container (viewer) or one docker run -d -i -t (otel-tui, which only works attached to a terminal, so no compose file).
  • The viewer's JSON-RPC (POST /rpc, v0.5.0) takes named parameters with times in Unix nanoseconds; its query parameter is the UI's parsed tree, a query string answers Invalid query → filter searchTraces results client-side (rootSpan.name, errorCount, spanCount).
  • Upstream main documents GET /export/traces/{id} and a read-only SQL query method; the v0.5.0 image has neither (unknown GET paths return the SPA's HTML with 200) → not relied on; pin ODV_VERSION.
  • otel-tui's Metrics tab lists one row per metric per export batch: 13 metric names appeared three times after one turn. Memory only with fixed caps; log records from outside a turn show N/A as the trace id; a trace with an error span is marked [!].
  • Verified 2026-10-11: viewer v0.5.0 stored 5 spans, 27 log records, 13 metric names (74 data points) from one turn; otel-tui 0.7.5 showed the same five-span timeline.

Sigiro and Maple Local (generic otlp)​

  • Standard paths, no auth, all three signals, SQL query endpoints (POST :9999/v1/query; POST :4388/local/query) → generic otlp; no adapter uses the SQL.
  • Sigiro is a closed-source binary; Maple is FSL-1.1 source-available and needs libchdb.so beside the binary.

Part 3: what the plugin normalizes​

The short version, for the overview page and the video:

  1. One URL field. Give the base URL; the type knows the vendor prefix (/api/public/otel, /api/v1/private/otel, /api/otel, /api/<org>, /otel) and derives the metrics and logs URLs.
  2. One key field. api_key / api_key_env becomes Basic, Bearer, ApiKey, a bare key, a DSN, x-honeycomb-team, wandb-api-key, signoz-ingestion-key or X-API-Key, whichever the backend reads, and the global headers: can never clobber it.
  3. Routing without reading vendor docs. project, workspace, experiment_id, dataset, namespace, stream_name, entity land in the right header or resource attribute per type; mandatory ones are checked at startup and a missing one skips the entry with one message.
  4. Only the signals the backend stores. Traces-only types get no metrics or logs exporter; Laminar's 200-and-drop gets metrics off; everything is overridable per entry. No 4xx spray in the debug log, no empty dashboards you cannot explain.
  5. The temporality the backend wants. Delta for SigNoz, Uptrace and Elastic; cumulative for the Prometheus family; per entry or global override.
  6. Token totals that are right in every UI. Both attribute conventions on every span; the root roll-up withheld from the two backends that sum every span.
  7. Isolation. One BatchSpanProcessor, one metric reader and one log processor per backend, each on its own queue, so a slow or dead backend never blocks the agent or the other backends.
  8. Headers that cannot misfire. ${VAR} expands in header values (#92); the per-backend headers win over the global block (#105); a vendor credential alone never switches export on (#259); a headers_helper command can mint a short-lived credential and is re-run on a 401/403 (#483).
  9. Reading it back the same way. Eight read APIs (GraphQL, TraceQL + PromQL + LogQL, SQL, a query builder, two REST dialects, two Jaeger generations, two Langfuse generations) behind one search bar, one trace detail shape and one metric panel, with the one-shot-process counter problem solved per backend.

Open as of 2026-10-11​

  • root_usage (#327, #353, 1.24.0) was verified live on Opik only; LangWatch was not re-run after it.
  • Elastic Cloud managed OTLP: contributor-verified only (#333).
  • Latitude per-span columns never inspected (PR #331); Uptrace 2.0 dialect not re-run since 2.0.2 (PR #277).
  • Logfire, Datadog, New Relic need accounts (#225-#227); Azure Monitor is documented through the Collector but unverified (#228); AgentOps needs a token-exchange design (#231).

Keeping this file honest​

When a backend changes, re-run one turn through it (docker-compose/README.md, "The loop"), read the debug log, query the store with the folder's Verify command, and update the row here with the date and version. If a statement here disagrees with backend_specs.py, backends.py or an adapter, the code is right and this file is wrong: fix it in the same PR.