From 457ecbfb2be19634e0128d196ec16994751b1906 Mon Sep 17 00:00:00 2001 From: Thom Lamb Date: Mon, 24 Aug 2026 13:46:36 -0500 Subject: [PATCH] =?UTF-8?q?docs(ci):=20correct=20the=20#154=20hypothesis?= =?UTF-8?q?=20=E2=80=94=20the=20database=20is=20wiped=20before=20the=20tes?= =?UTF-8?q?ts=20start=20(#154)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dmesg came back empty, and the evidence that actually settles it was in the original log the whole time. Step 7 `Run migrations` succeeded, then step 9's globalSetup applied all six migrations again from scratch. Both point at the same database — migrate.js reads PGHOST/PGDATABASE and the job sets those and TEST_PG* to the same service and the same redefined_test — so had step 7 migrated it, globalSetup would have printed "No migrations to run!", which is what a local run prints. It found an empty database. The wipe was already happening before the tests started, which makes this a database being reset repeatedly rather than a container dying partway through a heavy run, and accounts cleanly for the empty dmesg: a restart is not a kill. The suspect moves from memory pressure to the runner's handling of `services:`, where act_runner has been uneven across releases. The diagnostics change with it: A3 is dropped because it was designed to catch starvation, A2 is demoted to a fallback, and the new first check needs nothing but the Gitea UI — compare step 7 and step 9's migration output in any failing run. The superseded hypothesis is kept rather than deleted. A future reader finding memory ruled out is better served by seeing why it was suspected and what refuted it than by a document that never mentions it. Refs #154 --- .../integration-schema-loss-investigation.md | 38 ++++++++++++++----- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/ci/integration-schema-loss-investigation.md b/docs/ci/integration-schema-loss-investigation.md index f0af447..122d538 100644 --- a/docs/ci/integration-schema-loss-investigation.md +++ b/docs/ci/integration-schema-loss-investigation.md @@ -37,7 +37,26 @@ Recorded so none of it is re-investigated. | Two jobs sharing one database | The paired runs per SHA are `lint` and `sonarqube`, and on a single runner they queue rather than overlap — run 444 ended 14:18:50, run 445 started 14:18:51 | | A mismatch between the two pools | `env.setup.ts` derives `PG*` from `TEST_PG*`, and the workflow sets both to the same service container | -## The hypothesis +## SUPERSEDED — read this first + +The hypothesis below (the container dying under memory pressure) is **wrong**, and the evidence that refutes it was in the original log all along. + +`dmesg` came back empty — done, A1 is complete and negative. More decisively: step 7 `Run migrations` succeeded, and then step 9's `globalSetup` applied **all six migrations again from scratch**. Both target the same database, since `migrate.js` reads `PGHOST`/`PGDATABASE` and the job points those and `TEST_PG*` at the same service and the same `redefined_test`. Had step 7 migrated it, `globalSetup` would have printed `No migrations to run!`, which is what a local run prints. + +It found an empty database. **The wipe was already happening before the tests started.** + +So this is not one container dying partway through a heavy run. The database is being reset repeatedly, and the schema loss the suites hit later is the same cycle coming round again. That also accounts for the empty `dmesg` — a restart is not a kill. + +**The suspect is now the runner's handling of `services:`**, where `act_runner` has been uneven across releases and "the service container is recreated between steps" is a known shape. Not memory. + +### What to check instead + +1. **In any failing run, compare step 7 and step 9 in the Gitea UI.** Both applying all six migrations confirms the wipe outright. Step 7 applying six while step 9 says `No migrations to run!` would mean this reading is wrong and the wipe happens later. +2. **The `act_runner` version on the NAS**, and whether its config says anything about service container lifetime or reuse. + +A2 below is still worth running if that comparison is inconclusive, since watching the container is the only direct observation of a restart. A3 is no longer relevant — it was designed to catch memory starvation. + +## The original hypothesis, kept for the record The Postgres service container is being lost and recreated partway through the run. A recreated `postgres:16` with an empty data directory re-initialises `POSTGRES_DB` and comes back with no schema, which is exactly what the tail of the run sees. @@ -49,18 +68,19 @@ The runner is the NAS, which also hosts Gitea, the QA stack and production. `jes # Action list -## Now — diagnosis (yours, on the NAS) +## Now — diagnosis -- [ ] **A1.** Check for an OOM kill — cheapest, needs no timing, survives container cleanup -- [ ] **A2.** Watch the container live during a failing run — the decisive one -- [ ] **A3.** Capture memory headroom during the integration step -- [ ] **A4.** Paste the outputs below and hand back for review +- [x] **A1.** OOM kill — **done, negative.** `dmesg` empty +- [ ] **A5.** Compare step 7 and step 9's migration output in any failing run — Gitea UI, no NAS access needed +- [ ] **A6.** Record the `act_runner` version and any service-container settings in its config +- [ ] **A2.** Watch the container live — only if A5 is inconclusive +- [ ] ~~**A3.** Memory headroom~~ — dropped; the mechanism is not starvation ## Next — depends on what A1–A3 say -- [ ] **B1.** If confirmed: cap memory on the QA and production stacks, or stop QA during CI runs -- [ ] **B2.** If confirmed: consider dropping `--coverage` from the integration step, accepting the loss of backend integration coverage in SonarQube -- [ ] **B3.** If **not** confirmed: reopen the investigation — the next suspect is a `pg_isready` health flap recreating the service container +- [ ] **B1.** If the service container is being recreated: pin or upgrade `act_runner`, or stop relying on `services:` and start Postgres as a step the job controls +- [ ] **B2.** If the wipe turns out to be later than step 7: fall back to A2 and watch it happen +- [ ] ~~Cap memory on the QA and production stacks~~ — dropped with the OOM hypothesis - [ ] **B4.** Give the integration step `continue-on-error: true` so a database problem stops taking the SonarQube scan down with it ## Regardless of cause — diagnosability -- 2.54.0