Development¶

Everything needed to build, test and extend the observatory. Read docs/adr/0000-architecture.md first — it is the implementation contract, and this document only tells you how to work inside it.

1. Prerequisites¶

Tool	Version	Notes
JDK	25 (`25.0.1-tem`)	Pinned in `.sdkmanrc`. CI asks for `25` and the base image is `eclipse-temurin:25-jre-alpine`, both of which float to the newest 25.x — deliberately, so a JDK security release is picked up without a commit. `.sdkmanrc` is the only exact pin.
sbt	2.0.3	Pinned in `.sdkmanrc` and `project/build.properties`. sbt 1 will not load this build.
Docker	any recent daemon	Required for `sbt verifyIt` (Testcontainers) and for the images. Not needed for `sbt verify`.
Python + MkDocs	3.12, `mkdocs` 1.6.1 + `mkdocs-material` 9.7.7	Only for building the docs site locally. A ```mermaid fence needs no plugin — `pymdownx.superfences` hands it to Material, which loads mermaid 11 from unpkg at page load. So a syntax error renders as raw text rather than failing `--strict`, and an offline reader sees the source. Check a diagram in a browser, not in the build log.
Tailwind CSS CLI	4.3.3 standalone binary	Only for `ferrite/tailwind` — see §8. Absent, the task warns and leaves the committed stylesheet alone.
osv-scanner	2.4.0	Only for the dependency gate — see §9. Not needed by `verify`.

sdk env          # adopts the JDK and sbt from .sdkmanrc
java -version    # 25
sbt --version    # 2.0.3

The build is Scala 3.8.4 with -source:3.3. Build definitions under project/ are themselves Scala 3.

2. Command vocabulary¶

sbt verify      # fmtCheck + headerCheck + IT lint + IT/compile + Test/testFull. Fast; no Docker.
sbt verifyIt    # IT/testFull — the slow tier. Needs a working Docker daemon. CI runs this too.
sbt fmt         # scalafmt, build sources included
sbt fmtCheck    # scalafmtSbtCheck + scalafmtCheckAll
sbt headerCreate  # stamp licence headers — NEVER hand-write one
sbt doc         # Scaladoc. -Werror does NOT apply here — read the warnings, see §4
sbt scaladocSite  # doc for every module, collected under target/site/api/<module>

sbt cobalt/run    # :8080 (HTTP_PORT)
sbt wolfram/run   # :8080 (HTTP_PORT)
sbt ferrite/run   # :9000, Play dev mode

sbt ferrite/tailwind       # regenerate ferrite's committed stylesheet — see §8
sbt ferrite/tailwindCheck  # fail if it is stale

sbt sbom          # one CycloneDX document per deployable, into target/sbom — see §9

sbt "cobalt/testOnly com.worxbend.cobalt.BatchProcessorSuite"
sbt "persistence/IT/testOnly com.worxbend.persistence.MigrationIT"

# One quoted argument each, or a ";a;b;c" sequence: sbt 2 does not split a single space-joined string.
sbt ";ferrite/Docker/publishLocal;cobalt/Docker/publishLocal;wolfram/Docker/publishLocal"

That last rule is not theoretical. The package job in .github/workflows/scala.yml ran sbt -batch ferrite/Docker/stage cobalt/Docker/stage wolfram/Docker/stage and had therefore failed on every push since it was written, with Expected whitespace character — so the one job that proves the images still build had never built one.

Run sbt verify before handing work back. headerCheck fails on any file sbt-header has not stamped, and new files are only stamped once they have been compiled or sbt headerCreate has run.

Never run sbt clean — it throws away an incremental-compile state this build takes minutes to rebuild.

The sbt 2 `test` trap — read this once, remember it forever¶

sbt 2 inverted sbt 1's naming.

Task	What it does
`test`	Incremental. Runs only the tests affected by what changed. `testQuick` is merely an alias for it.
`testFull`	Runs everything.

Test/test still selects the incremental task, so a command alias written as Test/test reports success while executing zero tests. That is why verify is spelled Test/testFull and verifyIt is spelled IT/testFull.

verify also runs IT/compile, IT/headerCheck and IT/scalafmtCheck — the parts of the slow tier that need no Docker. Leaving them out meant verify could be green over an src/it tree that did not compile, and it was: scalafmt rewrote a block lambda's braces off, leaving ps => val _ = …, and nothing said so until somebody with a daemon ran verifyIt three merges later. Compiling the slow tier costs seconds. When you want a real answer, use testFull.

The two test tiers¶

Tier	Source root	Config	Needs Docker	Run by
unit	`src/test/scala`	`Test`	no	`sbt verify`
integration	`src/it/scala`	`IT` (declared in `project/ItConfig.scala`, extends `Test`)	yes	`sbt verifyIt`

sbt no longer ships an IntegrationTest configuration, so IT is declared by hand. ItConfig also wires headerSettings(IT) and scalafmtConfigSettings(IT) — without them IT/headerCheck and IT/scalafmtCheck simply would not exist and every file under src/it would escape the formatting and licence gates. Note that verify runs headerCheck, not IT/headerCheck: run sbt IT/headerCheck IT/scalafmtCheck after touching integration sources.

IT/parallelExecution is false (containers bind host ports and would collide) and IT/fork is true. The root project carries .configs(IT) so IT/testFull aggregates to every module; without it, it silently tests nothing.

Which tier touches which tree, and what is checked by neither. The interesting edges are the two dashed ones: verify reads src/it without ever running it, and one tree is verified by nothing at all.

flowchart LR
  main["src/main/scala<br/>src/main/twirl"]
  test["src/test/scala"]
  it["src/it/scala"]
  css["public/css/app.css<br/>committed Tailwind output"]
  dep["deploy/*.yml<br/>observability/rules"]
  js["public/js/app.js"]

  verify["sbt verify<br/>JDK only — laptop, air-gapped box and CI alike"]
  verifyIt["sbt verifyIt<br/>needs a Docker daemon"]
  ci1["stylesheet job<br/>CI only: the CLI is 112 MB and not on Maven"]
  ci2["deploy-config job<br/>CI only: needs docker and promtool"]
  nothing["nothing"]

  main --> verify
  test --> verify
  it -. "format, headers, compile — never executed" .-> verify
  it --> verifyIt
  main --> verifyIt
  css --> ci1
  dep --> ci2
  js -. "read, not run" .-> nothing

verify compiling src/it without running it is not a half-measure — it is the cheap half. scalafmt once rewrote a block lambda's braces off, leaving ps => val _ = … in an integration suite, and nothing said so until somebody with a Docker daemon ran verifyIt three merges later. §11 has what CI does with each of these.

3. Module layout and the dependency rule¶

modules/          libraries — no main, no image
  kernel          the domain: CloudEvents envelope, observation ADT, search filter grammar
  eventing        CloudEvents ↔ Kafka wire adapters, trace propagation
  persistence     Hikari pools, jsonb codecs, filter→SQL compiler, Flyway migrations
  observability   Micrometer registry, OTel tracing, Logback JSON, the metric vocabulary

applications/     exactly three deployable services
  wolfram         Tapir on Vert.x 5     — ingestion
  cobalt          Pekko Streams Kafka + Cask — consumer
  ferrite         Play 3 + Twirl/HTMX   — web UI and search

Dependency arrows point inward:

ferrite  -> kernel, persistence, observability
cobalt   -> kernel, eventing, persistence, observability
wolfram  -> kernel, eventing, observability
eventing -> kernel
persistence -> kernel
observability -> (nothing in this repo)

applications/ stays at exactly three. The three services must agree byte-for-byte on one wire contract and one metric taxonomy; duplication drift only ever surfaces in production. Shared code goes in a modules/ library.

modules/kernel must stay framework-free, and the build enforces it — not the reviewer. kernel is declared with domainLibrary, which gives it no ambient dependencies (not even the common ones), and build.sbt runs an assertion at build load:

val allowed = Set("io.circe", "org.scala-lang")
val foreign = declared.filter(m => m.configurations.isEmpty && !allowed(m.organization))
require(foreign.isEmpty, "modules/kernel must stay framework-free; …")

Adding a compile-scoped dependency on anything else — Play, Kafka, JDBC, even logging — fails sbt before a single file compiles. Test-scoped dependencies are allowed. That constraint is the whole value of the module: the domain compiles without a framework in scope, so nothing about the domain can depend on how it is transported.

Every other module automatically gets pureconfig, scala-logging and logback (main) plus munit, munit-scalacheck, scalacheck and scalatest (test) from commonDependencies/testDependencies — do not re-declare them. Keep that list short: anything in it ships in all three images whether or not a line of code references it. quicklens was in it and was referenced by nothing, in any module, so it was removed. Coordinates live in project/Dependencies.scala; Versions is public so build.sbt can reference it. Two traps already resolved there: pureconfig publishes no pureconfig_3 aggregate (Scala 3 derivation lives in pureconfig-core), and the scala-garden/scala-logging fork publishes nothing to Maven Central.

4. Compiler flags that will bite¶

project/BaseSettings.scala sets, and -Werror promotes to errors:

-new-syntax -indent — indentation syntax is mandatory. Braces are a compile error, not a style comment. .scalafmt.conf rewrites brace style away (rewrite.scala3.removeOptionalBraces = yes), so sbt fmt fixes most of it for you. Exception: ferrite removes -new-syntax (Compile / scalacOptions ~= (_.filterNot(_ == "-new-syntax"))) because Twirl generates code this build's rules do not govern. Enforcement in ferrite therefore rests entirely on scalafmt — write indentation syntax there anyway.
-Wunused:all — an unused import or parameter fails the build. This catches unused using ec: ExecutionContext parameters, which are easy to leave behind when a method stops returning a Future.
-Wvalue-discard / -Wnonunit-statement — a discarded non-Unit value fails the build. This is the flag you will meet most often, because almost every fluent builder returns this:

API	What returns non-`Unit`
Guice / Play modules	`bind[X].asEagerSingleton()`, `addBinding.to[Y]`
Micrometer	`MeterRegistry#counter/timer/summary`, `Config#commonTags`, `Config#meterFilter`
OpenTelemetry	every `spanBuilder.set…`, `Span#setStatus`, `Span#recordException`
Vert.x	`router.get(path).handler(…)`, `response().setStatusCode(…)…end(…)`
Kafka	`producer.send(…)`, `Headers#add`
Java executors	`scheduler.scheduleWithFixedDelay(…)`

Chain into one expression, or bind with val _ = …. The codebase uses val _ = consistently; follow it.

-Wnonunit-statement is removed in Test scope only, because ScalaTest's assert returns an Assertion and every multi-assertion test would otherwise fail. Do not re-add it there.
120 columns, enforced by scalafmt.
Scaladoc is not compiled with the same flags. doc logs Skipping unused scalacOptions: -Werror, -Wvalue-discard, -Wnonunit-statement, -source, -new-syntax, -indent and an unresolved [[link]] is a warning it walks past — modules/observability's Tracing.scala has two today. Read the sbt doc output; do not assume a green exit means the links resolve.
sbt 2.0.3 silently drops scalacOptions it does not recognise, and this is a trap. Adding, say, -Wsafe-init to BaseSettings.scalacFlags changes nothing: sbt "print kernel/scalacOptions" still reports the old list, and a deliberately invalid -Wbogus-probe-flag compiles clean even with -Werror. So a flag added here can look enabled and be doing nothing. Before relying on a new one, prove it fires — write a file that violates it and watch the build go red.
Never hard-code an sbt output path. sbt 2 writes to one shared root — target/out/jvm/scala-<v>/<project>/ — not to <module>/target/scala-<v>/. The Pages workflow used to copy Scaladoc from the sbt 1 location behind an [ -d "$src" ] && guard, so all seven modules were skipped in silence for as long as it existed. Ask sbt for the path (sbt scaladocSite, sbt "show <project>/Compile/doc") rather than reconstructing it.

Other toolchain facts worth knowing:

sbt 2 plugin artifacts use the _sbt2_3 suffix, not _3_2.0. When checking whether a plugin supports sbt 2, search Maven Central for <plugin>_sbt2_3.
Play 3 uses jakarta.inject, not javax.inject.
version comes from BUILD_VERSION and defaults to 0.1.0-SNAPSHOT. Keep it deterministic.
application.conf holds overrides only. Never inline a copy of an upstream reference configuration.

5. Adding an endpoint¶

The three services route in three deliberately different ways. Keep operational endpoints (/metrics, /health/live, /health/ready) separate from business endpoints in all three — they have a different audience, a different failure mode, and they must stay out of the http.server.requests timer.

5.1 wolfram — Tapir endpoint values on Vert.x¶

Endpoints are values, so the same description drives the Vert.x routes, the OpenAPI document and the tests.

Models — add request/response case classes to ApiModel.scala. Circe codecs plus Tapir's generic.auto schema derivation cover them.
Description — add a val to object Endpoints in Endpoints.scala:

val describeEvent: PublicEndpoint[String, IngestFailure, EventSummary, Any] =
  endpoint.get
    .in("events" / path[String]("id"))
    .out(jsonBody[EventSummary])
    .errorOut(failures)          // the shared oneOf — variants match on class, so status cannot drift
    .name("describeEvent")
    .summary("…")
    .description("…")

Register it — add it to Endpoints.all (that is what OpenApi documents), and to Endpoints.requestMediaTypes if it takes a body.
Logic — write it as a pure function on IngestionService (or a new service class), then bind it in IngestApi.routes with Endpoints.describeEvent.serverLogic(…). Descriptions and logic stay separate so the same endpoints could drive a client or a spec.
Tests — EndpointsSuite asserts on the description, OpenApiSuite on the generated document, and the logic is tested directly without a server.
Operational routes are not Tapir. They are plain Vert.x routes in AdminRoutes.mount, mounted outside the interpreter so HttpMetrics structurally cannot see them and no exclusion list can fall out of date.

5.2 cobalt — Cask annotations, one line each¶

cobalt has no business endpoints and never will. Events arrive over Kafka; an HTTP write path would be a second, unordered, uncommitted way into the same database. New routes here are operational only.

Add a pure method returning AdminReply to AdminHandlers in AdminRoutes.scala — every decision lives here, because Cask has no test kit and the alternative is binding a port in a unit test.
Add a one-line delegation to CobaltRoutes:

@cask.get("/health/startup")
def startup(): cask.Response[String] = CobaltRoutes.respond(handlers.startup())

The path must be a string literal (Cask's annotations are macros and want a constant). 3. Assert the literal against the corresponding constant in modules/observability's Meters in AdminRoutesSuite — that is what stops this file drifting from the shared vocabulary and quietly handing Prometheus a 404.

5.3 ferrite — SIRD routers, no `conf/routes`¶

There is no routes file and there must not be: Compile / routes / sources := Nil, and routing is AppRouter, selected by play.http.router in application.conf.

URL — add the path to com.worxbend.ferrite.web.Urls (and a builder function if it takes parameters). This is the single source of the string.
Pattern — add a PathExtractor to object Paths in routing/Routers.scala, built from the Urls constant via PathExtractor.cached, so the route and the URL builder are the same string by construction:

val Device: PathExtractor = PathExtractor.cached(Seq(s"${Urls.Devices}/", ""))

An empty trailing part matches one decoded segment; a part beginning with * matches greedily (asset paths). 3. Controller — add the action to EventsController (UI) or OpsController (operational), returning a Result. Blocking JDBC work goes through SearchService on the SearchExecutionContext, never on Play's default dispatcher. 4. Route — add a case to WebRouter.routes or OpsRouter.routes. AppRouter composes web → assets → ops with orElse; ops comes last because its paths are exact and cannot collide. 5. View — Twirl templates live in src/main/twirl/views/…. A page and its htmx fragment are separate templates; Hx.isFragment(request) decides which to render, and a fragment response should set Hx.PushUrlHeader when it changes what the URL should say. 6. Wiring — a new injectable type only needs a line in FerriteModule.bindings if Play cannot construct it (abstract, or with a shutdown obligation). Constructor injection everywhere; no global state. 7. Tests — UrlRoutingSuite proves each builder's output is matched by its extractor; EventsControllerSuite constructs the controller directly. Play's test helpers stream the result, so a suite calling Helpers.call needs a Materializer in scope and Helpers.writeableOf_AnyContentAsEmpty. 8. CSRF is enabled (the filter bar is a browser form); the token reaches every htmx request through one hx-headers attribute on <body>. The CSP allows script-src 'unsafe-eval' only because Alpine 3 compiles its x- expressions with new Function — keep Alpine to presentational state.

6. Adding a database migration¶

Migrations live in modules/persistence/src/main/resources/db/migration and ship inside the module's jar, so the schema and the code that queries it version together. cobalt applies them on boot — it is the write side, so it must not start against a schema older than its own inserts. ferrite never migrates.

Add a new file. Never edit an applied one. V2__add_alerts_view.sql, sequential version, snake-case description. validateOnMigrate is on, so editing V1__events.sql after it shipped turns every subsequent boot into a failure — which is the point. outOfOrder is off and baselineOnMigrate is off.
Follow the DDL conventions of V1__events.sql, which are asserted by tests:
Partition bounds carry an explicit +00 offset. A bare date is parsed in the session timezone and silently shifts every partition. Run Flyway with -Duser.timezone=UTC.
Every index gets a COMMENT ON INDEX events.<name> IS '…' naming the query shape it serves. MigrationScriptSuite fails without it — an index nobody can name is an index nobody can justify deleting.
Extraction functions used by generated columns must be IMMUTABLE, PARALLEL SAFE and non-throwing. A bare cast in a generated column aborts the entire 500-event batch on one malformed payload; return NULL instead so a bad payload is a missing dimension rather than an outage.
Storage parameters go on leaf partitions, never on the partitioned parent (Postgres rejects them there), and they are not inherited by new partitions.
Anything mirroring Scala — events.severity_rank vs com.worxbend.kernel.search.Severity, the reserved attribute list vs Envelope.ReservedAttributes — must stay identical, spellings and aliases included. MigrationScriptSuite checks the reserved list; keep new mirrors equally checked.
Update the tests that pin the schema:
MigrationScriptSuite (unit, no database) — text-level assertions about the script.
MigrationIT (integration, real Postgres) — assertEquals(Migrations.appliedVersions(…), Vector("1")) is a committed baseline. Add your version to it. That assertion is what catches a migration edited in place.
Run both tiers: sbt verify then sbt verifyIt (Docker required). The integration tier is where a generated column that Postgres refuses as non-IMMUTABLE actually fails — nothing about that is visible at compile time.
Live tables need care. Indexes on a partitioned parent cannot be built CONCURRENTLY and take ACCESS EXCLUSIVE on the whole hierarchy; see docs/operations.md §7.4 for the three-step alternative.
Partition creation deliberately does not belong in a migration: migrations are versioned and immutable, partitions are a rolling concern. See docs/operations.md §7.3.

7. Writing tests¶

munit leads; scalatest is kept for Play's test helpers. scalacheck for properties — the wire and filter generators (WireGenerators, FilterGenerators, Generators) already exist, reuse them.
Unit tests must not need Docker. Testcontainers is declared % IT only, exactly so the fast tier never needs a daemon.
Test the pure function, not the socket. Every service splits its decisions out of its framework surface for this reason: AdminHandlers (cobalt), the logic functions behind Endpoints (wolfram), Presenter/SearchQuery (ferrite). New logic goes there, not inline in an annotated method or an Action body.
src/it/resources/logback-test.xml sets org.testcontainers, com.github.dockerjava and tc-java to WARN. Without it, docker-java's wire logger buries every integration-test result.
Every IT suite provisions its own dependencies. One shared lazy Testcontainers singleton per forked JVM, in a companion object rather than a field — munit builds a fresh suite instance per test, so a lazy val on the suite would start one container per test. Ryuk reaps them at JVM exit, so there is no teardown hook to forget. PostgresSuite, KafkaWireIT, CobaltIT, WolframIngestIT and EventsPageIT each own one.
A suite skips only when Docker is unreachable, via munitIgnore. A red suite on a laptop with no Docker teaches people to ignore red suites. What this is not is the old behaviour: these suites used to skip on the absence of an environment variable nobody set, and WolframIngestIT did not even skip — it returned unit and reported three passing tests that had contacted no broker. If you add a fixture, make the no-dependency case loudly skipped, never quietly green.
Environment variables the IT tier reads: IT_POSTGRES_URL, IT_POSTGRES_USER, IT_POSTGRES_PASSWORD and KAFKA_BOOTSTRAP_SERVERS. These override the containers rather than gating the suites — that is how CI points every module at one database and one broker instead of starting five of each, and it is why the fixtures TRUNCATE before they seed. Note that deploy/docker-compose.yml publishes neither Postgres nor Kafka to the host, so pointing the suites at a compose stack needs a temporary port mapping first:

IT_POSTGRES_URL=jdbc:postgresql://localhost:5432/observatory \
IT_POSTGRES_USER=observatory IT_POSTGRES_PASSWORD=… \
KAFKA_BOOTSTRAP_SERVERS=localhost:9092 sbt verifyIt

8. Changing ferrite's stylesheet¶

applications/ferrite/src/main/resources/public/css/app.css is committed CLI output, not a hand-written file. It is what the standalone Tailwind CSS CLI v4.3.3 produces from applications/ferrite/src/main/assets/css/app.css, which imports Tailwind, imports the hand-written components.css, declares the theme tokens, and declares the class scan with @source. That arrangement is ADR §8.3's, and its whole purpose is that the runtime image never contains Node or Tailwind.

The cost of committing generated output is that it goes stale silently: a template gains class="mt-4", the CSS does not contain .mt-4, and the page renders without it. Nothing in a compile, a test or a page load says so.

sbt ferrite/tailwind        # regenerate the committed stylesheet in place
sbt ferrite/tailwindCheck   # fail if it differs from what the templates imply

Run ferrite/tailwind in the same change as any template edit that touches a class. Then sbt verify as usual — the regenerated file is a source file like any other.

Getting the binary¶

The CLI is a 112 MB self-contained binary from GitHub releases, not from Maven (org.webjars.npm:tailwindcss ships the @tailwindcss/oxide native compiler and needs Node). Tailwind.resolve in project/Tailwind.scala looks for it in this order — TAILWIND_BIN, then ~/.cache/worxbend/tailwindcss-<platform>-4.3.3, then tailwindcss on PATH — and if none is there, it warns with the exact curl command and changes nothing.

mkdir -p ~/.cache/worxbend
curl -fsSL -o ~/.cache/worxbend/tailwindcss-linux-x64-4.3.3 \
  https://github.com/tailwindlabs/tailwindcss/releases/download/v4.3.3/tailwindcss-linux-x64
chmod +x ~/.cache/worxbend/tailwindcss-linux-x64-4.3.3

PATH is searched last on purpose: a globally installed tailwindcss is very often a different major version, and the pinned copy in the cache is the one that reproduces the committed file byte for byte. On a musl host (Alpine) take the -musl asset from the same release and point TAILWIND_BIN at it; the glibc build will not run there. In CI, cache ~/.cache/worxbend beside ~/.cache/coursier.

Four decisions in that task worth not re-litigating¶

It is not a Compile / resourceGenerators entry, which is what ADR §8.3 sketched. A generator must produce a file on every build, so on a machine without the CLI it would have to invent one — an empty file, an un-compiled copy of components.css, or a build failure. All three are worse than the checked-in file, which is already correct. An explicit task can do the one right thing: change nothing, and say why.
Absence is a warning, never an error, and never a rewrite. The committed CSS is only ever replaced by bytes a successful, non-empty CLI run produced into a scratch file first — a CLI that died half-way through writing its --output would otherwise leave a truncated stylesheet behind.
tailwindCheck is in CI but not in verify. verify must stay runnable with nothing but a JDK, and without the CLI the check warns and passes, exactly like tailwind does — so a local verify will not tell you the stylesheet is stale. The stylesheet job in .github/workflows/scala.yml installs the pinned CLI and runs the check for real, so drift is caught before merge. The residual gap is the local one, recorded in docs/operations.md §8.
Both tasks are wrapped in Def.uncached. sbt 2 caches task results by declared inputs, and neither task declares the Twirl templates as one. Without it the first invocation is replayed for every later one and a template that gained a class reports "up to date" forever — the exact failure the tasks exist to catch. This was observed, not anticipated.

9. Dependencies, advisories and the supply-chain gate¶

9.1 The gate¶

.github/workflows/supply-chain.yml builds a CycloneDX SBOM per deployable and scans it against OSV. It fails the job on any advisory. It needs no account and no paid tier.

sbt sbom                                        # -> target/sbom/{ferrite,cobalt,wolfram}.cdx.json
osv-scanner scan --config osv-scanner.toml \
  --lockfile target/sbom/ferrite.cdx.json \
  --lockfile target/sbom/cobalt.cdx.json \
  --lockfile target/sbom/wolfram.cdx.json

Three things about that shape are deliberate:

SBOMs, not a source scan. osv-scanner has no sbt extractor — point it at this repository and it reports "No package sources found". project/plugins.sbt carries sbt-sbom purely to give it something to read.
*.cdx.json, under one directory. osv-scanner identifies an SBOM by filename and rejects sbt-sbom's default wolfram-0.1.0-SNAPSHOT.bom.json with "could not determine extractor suitable to this file". That is what bomFileName and bomOutputPath in build.sbt are for.
Three SBOMs, one per service, and none for the libraries. A service's Compile bom already contains the transitive closure of the modules/ it depends on. Test and IT dependencies are not covered — Testcontainers, and the Selenium/Appium tree play-test drags into ferrite, never leave CI, and gating on them would make main red for something nobody ships.

The scan is also on a weekly schedule. An advisory is published without anyone pushing a commit; without the cron, the person who finds out is whoever opens the next pull request.

9.2 When the gate goes red¶

Two places to fix it, and picking the wrong one is how a gate rots:

A fix exists upstream → add the coordinate to Dependencies.securityOverrides with the advisory id, the upstream that holds it back, and why the bump is safe. That list closed 22 of the 23 advisories this repository was carrying when the gate was written.
No fix exists, or the code path is unreachable → add an entry to osv-scanner.toml with an argument and an ignoreUntil date. The date is the point: the ignore expires, the gate goes red, and somebody re-reads the argument instead of inheriting it. There is exactly one entry today.

Never widen the gate to a severity threshold. A threshold accepts every medium for ever and writes nothing down.

9.3 Before you move a `dependencyOverride`¶

Forcing a transitive dependency is a bet that its consumer compiled against a compatible shape, and neither sbt evicted nor the test suites can settle that bet. missinglink can:

// project/plugins.sbt, temporarily
addSbtPlugin("ch.epfl.scala" % "sbt-missinglink" % "0.3.8")

SBT_OPTS=-Xmx4g sbt cobalt/missinglinkCheck

It earns the detour. Moving undertow-core to 2.3.26 brought jboss-threads 3.7.0 with it, and Undertow's own POM manages jboss-logging down to 3.4.3 — below the Messages.getBundle(MethodHandles.Lookup, Class) overload that jboss-threads calls from a static initialiser. sbt evicted printed nothing, because nothing was evicted: the low version simply won. verify and verifyIt were both green. The failure would have been a NoSuchMethodError the first time Undertow built its worker — that is, on cobalt's first boot. missinglinkCheck named the method, the caller and the jar; the fix is the jboss-logging entry in Dependencies.overrides.

It is not wired into verify, and that is a considered decision. On wolfram it cannot be made green without excluding vertx-core and three netty artifacts wholesale, because their optional integrations (Conscrypt, Brotli, JZlib, the native transports, Jackson 2) are absent by design and missinglink reports each as a conflict. Excluding the artifacts would mute the check exactly where the next real skew would appear. Run it by hand, read the "missing members" section — that is the part that matters — and ignore the "missing classes" noise.

10. Gotchas worth remembering¶

sbt verify is green at 709 unit tests and sbt verifyIt at 100 integration tests across the five modules that have a src/it. If your change makes a count drop, you probably renamed a suite into invisibility.
OpenTelemetry's Context is ThreadLocal-backed and Context.current() returns root silently on any thread it was not entered on. Across a Future, a Pekko stream stage or a Vert.x event-loop hop, capture the Context and pass it explicitly. An orphaned span is not an error anything reports.
Never add a -javaagent:opentelemetry-javaagent.jar "for free tracing": it duplicates every Kafka span and introduces a competing HTTP metric family.
Micrometer happily registers the same meter name twice with different tag sets, and Prometheus renders that as two unrelated series. That is why each service has a typed metrics façade (IngestMetrics, ConsumerMetrics) where a meter's tag set is fixed in exactly one place. Add meters there, and add names to Meters.
for over Future is sequential. SearchService starts its four queries as vals before awaiting any of them for exactly this reason; a for comprehension there turns one 40 ms page into four serial round trips.
Scala 3 has known -Wunused false positives for givens resolved inside the sql macro. Keep given-imports at the narrowest scope; a targeted @nowarn is permitted only with a comment naming the false positive.
Don't commit. The orchestrator commits.

11. What CI runs, and what it gates¶

Four workflows. Two of them are the real gates — scala.yml builds and tests, supply-chain.yml scans the dependencies — and both run on pushes and pull requests to main.

What runs when, and what a red result actually prevents.

flowchart LR
  trig(["push or pull_request on main"])
  wk(["schedule: Mondays 06:17 UTC"])
  pushonly(["push to main only"])

  subgraph scala["scala.yml — the build gate"]
    verify["verify<br/>fmtCheck · headerCheck · IT lint · IT/compile · Test/testFull"]
    integration["integration<br/>IT/testFull on Testcontainers"]
    stylesheet["stylesheet<br/>ferrite/tailwindCheck with the pinned CLI"]
    deployconf["deploy-config<br/>docker compose config · promtool check rules and config"]
    package["package<br/>Docker/stage for all three images"]
  end
  subgraph supply["supply-chain.yml — the dependency gate"]
    deps["dependencies<br/>sbt sbom · osv-scanner, fails on any advisory"]
  end
  subgraph docsw["docs.yml"]
    build["build<br/>mkdocs --strict · scaladocSite · per-module assertions"]
    pages["deploy<br/>GitHub Pages"]
  end

  merge(["a merge to main"])

  trig --> verify
  trig --> integration
  trig --> stylesheet
  trig --> deployconf
  trig --> deps
  wk --> deps
  pushonly --> package
  verify --> package
  pushonly --> build
  build --> pages
  verify -. advisory .-> merge
  deps -. advisory .-> merge

The two dashed edges are the honest part. main has no branch protection, so no job result blocks anything — a red build merges, and the gates gate a notification, not the branch. That is a repository setting nobody can fix from a file in the tree; docs/operations.md §8 tracks it.

Workflow	Trigger	What it is for
`scala.yml`	push, PR	The build. Five jobs, deliberately not chained: a formatting failure must not hide a broken migration, and only `package` has a `needs:` because staging an image you have not compiled proves nothing.
`supply-chain.yml`	push, PR, weekly cron, manual	One CycloneDX SBOM per deployable, scanned against OSV. The cron is the point of the weekly run: an advisory is published without anyone pushing a commit.
`docs.yml`	push to `main`, manual	Builds this site with `--strict` plus Scaladoc, and deploys to Pages. It gates nothing on a PR, so a broken doc link on a branch is found after the merge.
`stale.yml`	nightly cron	Labels and closes dormant issues and pull requests. Touches no code and gates nothing.

Three security workflows used to sit beside these and were deleted rather than fixed: scala-snyk.yml failed on every push because Snyk's Scala integration cannot read an sbt 2 build, sonar.yml failed in 1.1 s on a token from 2020, and security.yml ran a ZAP baseline scan against https://www.zaproxy.org — someone else's website — using a rules file that was never committed. Between them they had produced zero findings and a permanently red main, which is worse than no scanner: it teaches everyone to ignore red. supply-chain.yml replaced all three, needs no account, and fails the job on any advisory (§9).

Two jobs carry a guard worth knowing about, because both would otherwise be decorative:

stylesheet verifies the Tailwind CLI downloaded and then greps the sbt log for was not found. tailwindCheck warns and succeeds when the CLI is absent — right for a developer, useless for CI — so without both halves the job is a green tick that checked nothing.
deploy-config derives the mandatory compose variables by grepping ${VAR:?…} out of the file rather than listing them. The list was listed, and it went stale the day wolfram gained AUTH_SECRET: every push then failed on a missing variable, in a job whose subject was unrelated to whatever the pusher had changed.

Every job carries timeout-minutes. The default is six hours, and the failure that runs into it is never a test failure — it is an sbt server that never answers or a container that never becomes healthy, which is exactly the case where nobody is watching.