docs/ci/integration-schema-loss-investigation.md

@@ -37,7 +37,26 @@ Recorded so none of it is re-investigated.
 | Two jobs sharing one database | The paired runs per SHA are `lint` and `sonarqube`, and on a single runner they queue rather than overlap — run 444 ended 14:18:50, run 445 started 14:18:51 |
 | A mismatch between the two pools | `env.setup.ts` derives `PG*` from `TEST_PG*`, and the workflow sets both to the same service container |
 
-## The hypothesis
+## SUPERSEDED — read this first
+
+The hypothesis below (the container dying under memory pressure) is **wrong**, and the evidence that refutes it was in the original log all along.
+
+`dmesg` came back empty — done, A1 is complete and negative. More decisively: step 7 `Run migrations` succeeded, and then step 9's `globalSetup` applied **all six migrations again from scratch**. Both target the same database, since `migrate.js` reads `PGHOST`/`PGDATABASE` and the job points those and `TEST_PG*` at the same service and the same `redefined_test`. Had step 7 migrated it, `globalSetup` would have printed `No migrations to run!`, which is what a local run prints.
+
+It found an empty database. **The wipe was already happening before the tests started.**
+
+So this is not one container dying partway through a heavy run. The database is being reset repeatedly, and the schema loss the suites hit later is the same cycle coming round again. That also accounts for the empty `dmesg` — a restart is not a kill.
+
+**The suspect is now the runner's handling of `services:`**, where `act_runner` has been uneven across releases and "the service container is recreated between steps" is a known shape. Not memory.
+
+### What to check instead
+
+1. **In any failing run, compare step 7 and step 9 in the Gitea UI.** Both applying all six migrations confirms the wipe outright. Step 7 applying six while step 9 says `No migrations to run!` would mean this reading is wrong and the wipe happens later.
+2. **The `act_runner` version on the NAS**, and whether its config says anything about service container lifetime or reuse.
+
+A2 below is still worth running if that comparison is inconclusive, since watching the container is the only direct observation of a restart. A3 is no longer relevant — it was designed to catch memory starvation.
+
+## The original hypothesis, kept for the record
 
 The Postgres service container is being lost and recreated partway through the run. A recreated `postgres:16` with an empty data directory re-initialises `POSTGRES_DB` and comes back with no schema, which is exactly what the tail of the run sees.
 
@@ -49,18 +68,19 @@ The runner is the NAS, which also hosts Gitea, the QA stack and production. `jes
 
 # Action list
 
-## Now — diagnosis (yours, on the NAS)
+## Now — diagnosis
 
-- [ ] **A1.** Check for an OOM kill — cheapest, needs no timing, survives container cleanup
+- [x] **A1.** OOM kill — **done, negative.** `dmesg` empty
-- [ ] **A2.** Watch the container live during a failing run — the decisive one
+- [ ] **A5.** Compare step 7 and step 9's migration output in any failing run — Gitea UI, no NAS access needed
-- [ ] **A3.** Capture memory headroom during the integration step
+- [ ] **A6.** Record the `act_runner` version and any service-container settings in its config
-- [ ] **A4.** Paste the outputs below and hand back for review
+- [ ] **A2.** Watch the container live — only if A5 is inconclusive
+- [ ] ~~**A3.** Memory headroom~~ — dropped; the mechanism is not starvation
 
 ## Next — depends on what A1–A3 say
 
-- [ ] **B1.** If confirmed: cap memory on the QA and production stacks, or stop QA during CI runs
+- [ ] **B1.** If the service container is being recreated: pin or upgrade `act_runner`, or stop relying on `services:` and start Postgres as a step the job controls
-- [ ] **B2.** If confirmed: consider dropping `--coverage` from the integration step, accepting the loss of backend integration coverage in SonarQube
+- [ ] **B2.** If the wipe turns out to be later than step 7: fall back to A2 and watch it happen
-- [ ] **B3.** If **not** confirmed: reopen the investigation — the next suspect is a `pg_isready` health flap recreating the service container
+- [ ] ~~Cap memory on the QA and production stacks~~ — dropped with the OOM hypothesis
 - [ ] **B4.** Give the integration step `continue-on-error: true` so a database problem stops taking the SonarQube scan down with it
 
 ## Regardless of cause — diagnosability