com.worxbend.observability

Credential decisions, counted the same way by every service that has a credential.

This module, and not each service's own metrics façade. wolfram and cobalt verify tokens with two different implementations — one on jwt-scala, one on the JDK's JCA — and the whole point of a shared metric is that a panel reading auth_decisions_total{reason="bad-signature"} gets both. Two façades would have been two chances to spell a reason differently, and a Grafana row that silently covers one service.

The classifier is the interesting part. An auth layer's failures are only useful when you can tell which check refused: bad-signature at volume is an attack, expired at volume is a client that stopped refreshing, scope-missing at volume is a deployment that granted the wrong role. Those need three different responses and they all arrive as a 401 or 403 in http.server.requests.

Nothing attacker-controlled becomes a label. A malformed token is unbounded input; putting its text in a tag is how one client with a broken loop takes down a Prometheus instance. classify maps a verifier's own message onto the closed set in Meters.AuthReasons, and anything it does not recognise becomes malformed rather than a new label value.

Correlation between the log stream and the trace stream (ADR §7.3).

Logs and traces are exported by different pipelines to different backends. The only thing that joins them is the trace id printed on the log line, so putting it in the MDC is not decoration — without it, "find the logs for this slow request" is unanswerable and the trace tells you where time went but never why.

The MDC is a ThreadLocal. That is the whole hazard: Play actions, Pekko stream stages and Vert.x event-loop hops all move work to another thread, and the MDC does not follow. Every method here is therefore scoped — it restores what it found — and the scope must be re-established inside the stage that does the work, never assumed to have survived a Future boundary. A stale MDC is worse than an empty one: it attributes a log line to the wrong request.

The shared metric vocabulary (ADR §7.1).

Every meter name and tag key the three services have in common lives here as a constant, for one reason: a Grafana dashboard is written against strings, and a string typed twice is a string that eventually differs. If wolfram counts ingest.events.received and cobalt counts ingest.event.received, nothing fails — the panel just goes flat for one service, which is the failure mode that takes longest to notice.

Names follow Micrometer's dot-delimited convention, not Prometheus's underscores: the registry's naming convention performs that translation at scrape time, and hard-coding the Prometheus spelling here would break the day a second registry is added.

Cardinality is the standing hazard. Tag keys are listed here; the values a caller may attach are constrained either by an enumeration below or by a Micrometer filter in Telemetry. Anything derived from user input — a URI path, an event id, a device serial — is not a tag.

The observability composition root: one registry, one tracer, one identity, per process.

Why one registry is the whole point (ADR §7.1). Frameworks each want to bring their own metrics pipeline. Two pipelines mean two /metrics endpoints or two naming conventions, and then one Grafana dashboard stops working across the three services — which is the entire reason modules/observability exists. So exactly one PrometheusRegistry is constructed here and handed to PrometheusMeterRegistry; anything with a native Prometheus collector registers into that same object and appears in the same exposition, with no bridging and no second port.

Why scrape and not a getter for the registry. The three services mount metrics on three different HTTP stacks — Play, Vert.x/Tapir, Cask — and none of them should have to know that the exposition comes from Micrometer, let alone which Prometheus client version renders it. A String is the entire contract they need. (The registry is still reachable for instrumenting code; it is scrape that keeps the transport ignorant.)

The version trap this class exists to surface. micrometer-registry-prometheus is compiled against prometheus-metrics-core 1.7.0 while the build forces 1.8.0 (ADR §3.5, §3.11). Nothing about that mismatch is visible at compile time: it would appear as a NoSuchMethodError on the first scrape, in production, minutes after a green deploy. The unit tests call scrape for real for exactly this reason — do not replace that with an assertion on the meter list.

-Werror trap (ADR §7.4). MeterRegistry.Config#commonTags and #meterFilter both return Config, so calling them as statements fails under -Wnonunit-statement. They are bound with val _ = below.

Not a singleton and not global: one instance is constructed by each service's main and passed down. Micrometer has a static Metrics.globalRegistry, and this deliberately does not use it — a global makes the registry impossible to replace per test and turns meter registration order into a hidden coupling between suites.

Identity of the running process, as every signal sees it.

The three services must label metrics, resource attributes and log lines from the same three strings, because a dashboard filters on service, a trace search filters on service.name, and a log query filters on service — if those are derived independently, a rename lands in one place and silently orphans the other two. Constructing this once and threading it through Telemetry is the only mechanism that keeps them equal.

A minimal, transport-agnostic view of a string-keyed header set, used to carry W3C trace context across a process boundary.

This exists so that trace propagation lives in one place while the transports stay in the modules that own them. modules/eventing binds this to org.apache.kafka.common.header.Headers, a Play filter binds it to request headers, and neither obligation leaks a Kafka or Play dependency into this module (ADR §2 — observability has no Kafka on its classpath and must not acquire one).

Why put returns C rather than being Unit. The two carriers that matter have opposite mutability: a Scala Map is persistent and an update produces a new value, while Kafka's Headers is mutable and add returns this. A returning signature is the only one both can implement honestly — a Unit signature would force the Map instance to be secretly mutable, and a persistent-only signature would force needless copying of Kafka headers. Implementations are free to return the same instance.

Only the three operations OpenTelemetry's TextMapPropagator actually needs are here. Resist growing it: every method added is a method each transport binding must implement correctly for propagation to work at all.

Distributed tracing: a traces-only OpenTelemetry SDK plus the two operations the services actually perform.

Traces only. Metrics are Micrometer's, exclusively (ADR §7). If the OTel metrics SDK were also active, every counter would exist twice under two naming conventions in two backends, and the two would disagree the first time one pipeline dropped a batch — a discrepancy nobody can adjudicate. autoConfigured therefore forces the metrics and logs exporters off rather than merely defaulting them off; see the note there.

-Werror trap (ADR §7.4). Every OTel builder method returns this, so builder.setAttribute(k, v) written as a statement is a compile error under -Wnonunit-statement. Chain into one expression, or bind with val _ =. The same applies to Span#setStatus and Span#recordException.

Context is a ThreadLocal. Context.current() returns root — silently, never an error — on any thread the context was not entered on. Across a Future, a Pekko stream stage or a Vert.x event-loop hop, capture the Context explicitly and pass it to span as parent; do not rely on ambient state. An orphaned span is not a failure anything reports, it is simply a trace that is missing its middle.