diff --git a/docs/ci/integration-schema-loss-investigation.md b/docs/ci/integration-schema-loss-investigation.md index f0af447..122d538 100644 --- a/docs/ci/integration-schema-loss-investigation.md +++ b/docs/ci/integration-schema-loss-investigation.md @@ -37,7 +37,26 @@ Recorded so none of it is re-investigated. | Two jobs sharing one database | The paired runs per SHA are `lint` and `sonarqube`, and on a single runner they queue rather than overlap — run 444 ended 14:18:50, run 445 started 14:18:51 | | A mismatch between the two pools | `env.setup.ts` derives `PG*` from `TEST_PG*`, and the workflow sets both to the same service container | -## The hypothesis +## SUPERSEDED — read this first + +The hypothesis below (the container dying under memory pressure) is **wrong**, and the evidence that refutes it was in the original log all along. + +`dmesg` came back empty — done, A1 is complete and negative. More decisively: step 7 `Run migrations` succeeded, and then step 9's `globalSetup` applied **all six migrations again from scratch**. Both target the same database, since `migrate.js` reads `PGHOST`/`PGDATABASE` and the job points those and `TEST_PG*` at the same service and the same `redefined_test`. Had step 7 migrated it, `globalSetup` would have printed `No migrations to run!`, which is what a local run prints. + +It found an empty database. **The wipe was already happening before the tests started.** + +So this is not one container dying partway through a heavy run. The database is being reset repeatedly, and the schema loss the suites hit later is the same cycle coming round again. That also accounts for the empty `dmesg` — a restart is not a kill. + +**The suspect is now the runner's handling of `services:`**, where `act_runner` has been uneven across releases and "the service container is recreated between steps" is a known shape. Not memory. + +### What to check instead + +1. **In any failing run, compare step 7 and step 9 in the Gitea UI.** Both applying all six migrations confirms the wipe outright. Step 7 applying six while step 9 says `No migrations to run!` would mean this reading is wrong and the wipe happens later. +2. **The `act_runner` version on the NAS**, and whether its config says anything about service container lifetime or reuse. + +A2 below is still worth running if that comparison is inconclusive, since watching the container is the only direct observation of a restart. A3 is no longer relevant — it was designed to catch memory starvation. + +## The original hypothesis, kept for the record The Postgres service container is being lost and recreated partway through the run. A recreated `postgres:16` with an empty data directory re-initialises `POSTGRES_DB` and comes back with no schema, which is exactly what the tail of the run sees. @@ -49,18 +68,19 @@ The runner is the NAS, which also hosts Gitea, the QA stack and production. `jes # Action list -## Now — diagnosis (yours, on the NAS) +## Now — diagnosis -- [ ] **A1.** Check for an OOM kill — cheapest, needs no timing, survives container cleanup -- [ ] **A2.** Watch the container live during a failing run — the decisive one -- [ ] **A3.** Capture memory headroom during the integration step -- [ ] **A4.** Paste the outputs below and hand back for review +- [x] **A1.** OOM kill — **done, negative.** `dmesg` empty +- [ ] **A5.** Compare step 7 and step 9's migration output in any failing run — Gitea UI, no NAS access needed +- [ ] **A6.** Record the `act_runner` version and any service-container settings in its config +- [ ] **A2.** Watch the container live — only if A5 is inconclusive +- [ ] ~~**A3.** Memory headroom~~ — dropped; the mechanism is not starvation ## Next — depends on what A1–A3 say -- [ ] **B1.** If confirmed: cap memory on the QA and production stacks, or stop QA during CI runs -- [ ] **B2.** If confirmed: consider dropping `--coverage` from the integration step, accepting the loss of backend integration coverage in SonarQube -- [ ] **B3.** If **not** confirmed: reopen the investigation — the next suspect is a `pg_isready` health flap recreating the service container +- [ ] **B1.** If the service container is being recreated: pin or upgrade `act_runner`, or stop relying on `services:` and start Postgres as a step the job controls +- [ ] **B2.** If the wipe turns out to be later than step 7: fall back to A2 and watch it happen +- [ ] ~~Cap memory on the QA and production stacks~~ — dropped with the OOM hypothesis - [ ] **B4.** Give the integration step `continue-on-error: true` so a database problem stops taking the SonarQube scan down with it ## Regardless of cause — diagnosability