OpenObserve

OpenObserve is an OpenTelemetry-native observability platform written in Rust. It ingests all three signals on one port, stores them on local disk (or S3 for production), and has no external dependencies — the default deployment is literally one container.

Signals: traces + metrics + logs. Deployment: local (single container) or self-hosted HA with S3 + etcd. Cost: OSS.

One-command quickstart

A compose file ships with the plugin:

cd ~/.hermes/plugins/hermes_otel
docker compose -f docker-compose/openobserve/docker-compose.yaml up -d

Open http://localhost:5080 and log in with the default admin:

• Email: root@example.com
• Password: Complexpass#123

Point the plugin at it — minimal hermes_otel.yaml:

backends:
  - type: openobserve
    endpoint: http://localhost:5080/api/default/v1/traces
    user: root@example.com
    password_env: OPENOBSERVE_PASSWORD
    # stream_name: default          # optional; defaults to "default"

logs:
  capture: true   # send Python log lines as OTel logs too

export OPENOBSERVE_PASSWORD='Complexpass#123'

The plugin encodes Basic base64(user:password) into an Authorization header and adds a stream-name header automatically. Restart hermes-agent; you should see [hermes-otel] ✓ OpenObserve at http://localhost:5080/api/default/v1/traces in the startup logs.

Endpoint URL structure

OpenObserve's OTLP endpoint is per-organization, per-signal:

Signal	URL template
Traces	http://HOST:5080/api/<org>/v1/traces
Logs	http://HOST:5080/api/<org>/v1/logs
Metrics	http://HOST:5080/api/<org>/v1/metrics

Configure endpoint: with the traces URL; the plugin derives the other two by swapping the final path segment. The default organization is default — if you create additional orgs in the UI, use their name in the URL.

A bare origin (endpoint: http://localhost:5080, no path) is completed to http://localhost:5080/api/<org>/v1/traces, with <org> from the entry's org: (or OTEL_OPENOBSERVE_ORG), default default. A URL with any path is used exactly as given, so a reverse-proxy prefix keeps working.

Why a dedicated type: openobserve

OpenObserve uses HTTP Basic auth plus a stream-name request header that type: otlp can't produce for you. Declaring type: openobserve asks the plugin to:

1. Read user / password, else the variables user_env / password_env name, else OTEL_OPENOBSERVE_USER / OTEL_OPENOBSERVE_PASSWORD, else OPENOBSERVE_USER / OPENOBSERVE_PASSWORD. Without both the entry is skipped at startup, unless it has a headers_helper that prints the Authorization header.
2. Build Authorization: Basic base64(user:password) automatically.
3. Add stream-name: <stream_name> (else OTEL_OPENOBSERVE_STREAM, default default).
4. Display ✓ OpenObserve at <endpoint> in startup logs.
5. Send all three signals (traces, metrics and logs) by default.

Ports

Port	Service	Purpose
5080	OpenObserve	UI + OTLP/HTTP ingestion — the plugin exports here
5081	OpenObserve	OTLP/gRPC ingestion (unused by the plugin)

No collisions with any other hermes-otel stack — OpenObserve can run concurrently with everything else.

What you'll see in the UI

Traces: Traces tab. Every span the plugin emits is indexed with its OpenInference / GenAI / hermes.* attributes. Filter by attribute, pivot to the waterfall, click into spans. OpenObserve's search syntax is SQL-flavoured (VRL under the hood).

Metrics: Metrics tab. The hermes_* metrics (session counts, token usage, tool-duration histograms) show up as time series with their resource attributes intact.

Logs: Logs tab. With logs: {capture: true}, every Python logger.info(...) record lands here. A line logged during a turn carries that turn's trace_id and span_id plus hermes_session_id (OpenObserve flattens the attribute name), and hermes_log_attribution says how the ids were found; lines logged between turns or from threads the plugin cannot attribute have an empty trace_id. Click a trace_id to jump to the corresponding trace. See OTel logs for the attribution rules.

Streams

OpenObserve partitions incoming data into named streams. Without a stream-name header everything lands in default; setting a different name routes telemetry to its own stream (useful for separating envs: hermes-prod, hermes-staging). Set it per-backend:

backends:
  - type: openobserve
    endpoint: http://localhost:5080/api/default/v1/traces
    user_env: OPENOBSERVE_USER
    password_env: OPENOBSERVE_PASSWORD
    stream_name: hermes-prod

Streams and their retention / index settings are managed in the OpenObserve UI under Settings → Streams.

Dashboard

The bundled dashboard's OpenObserve adapter reads traces, metrics and logs back with SQL over /api/<org>/_search, using the same user / password as the exporter. It reads them from the entry's inline values or user_env / password_env, not from the OTEL_OPENOBSERVE_* fallback variables, so name them on the entry when you want the dashboard to work. Dashboard-only keys:

Key	Meaning
query_port	Port of the search API (default 5080)
stream_name / stream	Trace stream to read (default default)
logs_stream	Log stream to read (default default)
trace_window_days	How far back a trace-detail lookup searches (default 90)
ui_url	Base URL for the "open in OpenObserve" links (default: the query origin)

OpenObserve stores attribute names with underscores (llm.model_name becomes llm_model_name); the adapter maps the plugin's known names back to the dotted form.

Organizations and users

The docker quickstart seeds one organization (default) and one root user. To split telemetry across environments or teams:

1. In the UI, go to Settings → Organizations → create new.
2. Put the new org's name in the endpoint path: http://HOST:5080/api/<new-org>/v1/traces.
3. Create additional users under Settings → Users → Invite (OpenObserve sends an email; for local dev the invite token shows up in the container logs).

Multi-backend fan-out

OpenObserve's port doesn't collide with anything, so it drops cleanly into a fan-out:

backends:
  - type: openobserve
    endpoint: http://localhost:5080/api/default/v1/traces
    user_env: OPENOBSERVE_USER
    password_env: OPENOBSERVE_PASSWORD

  - type: phoenix
    endpoint: http://localhost:6006/v1/traces

logs:
  capture: true

Traces fan out to both; metrics and logs go only to OpenObserve (Phoenix ingests traces only).

Troubleshooting

"401 Unauthorized in agent logs"

Basic-auth credentials don't match the root account. Confirm ZO_ROOT_USER_EMAIL / ZO_ROOT_USER_PASSWORD in docker-compose/openobserve/docker-compose.yaml match what you're passing to the plugin. A password that is rotated while Hermes runs can be followed with a headers_helper that prints the current Authorization header: the plugin re-runs it on the first 401 and retries that batch.

"I see ✓ OpenObserve at ... but no traces appear"

Check that the URL includes the org segment (/api/default/v1/traces, not /v1/traces); a bare origin such as http://localhost:5080 is completed for you. OpenObserve answers 404 on an unknown path, which the startup banner cannot see. Turn on HERMES_OTEL_DEBUG=true and look in debug.log for the failed batch and its status code.

"Container starts, UI is up, but POSTing traces returns 400"

Usually the stream-name header is missing or malformed. Check with docker logs hermes-otel-openobserve — bad stream names produce validation errors there.

"Data disappears after I recreate the container"

By default the compose uses a named volume oo_data. docker compose down preserves it; docker compose down -v drops it.

Production usage

This compose runs OpenObserve in standalone disk mode — everything in the oo_data named volume. For production you'd want:

• Object storage (S3, GCS, MinIO) for cold tier — configure via ZO_S3_* env vars.
• etcd for metadata replication across multiple OpenObserve nodes.
• Postgres for metadata (optional; alternative to etcd).
• A reverse proxy with TLS termination in front of port 5080.

See OpenObserve's HA + production docs for the full setup.

See also

• OTel logs — the log pipeline that makes trace-id correlation work.
• Multi-backend fan-out — adding OpenObserve alongside other backends.
• docker-compose/openobserve/README.md — the on-disk README with the same setup walkthrough.