docs(ci): correct the #154 hypothesis — the database is wiped before the tests start (#154)

dmesg came back empty, and the evidence that actually settles it was in the original log the whole time. Step 7 `Run migrations` succeeded, then step 9's globalSetup applied all six migrations again from scratch. Both point at the same database — migrate.js reads PGHOST/PGDATABASE and the job sets those and TEST_PG* to the same service and the same redefined_test — so had step 7 migrated it, globalSetup would have printed "No migrations to run!", which is what a local run prints.

It found an empty database. The wipe was already happening before the tests started, which makes this a database being reset repeatedly rather than a container dying partway through a heavy run, and accounts cleanly for the empty dmesg: a restart is not a kill.

The suspect moves from memory pressure to the runner's handling of `services:`, where act_runner has been uneven across releases. The diagnostics change with it: A3 is dropped because it was designed to catch starvation, A2 is demoted to a fallback, and the new first check needs nothing but the Gitea UI — compare step 7 and step 9's migration output in any failing run.

The superseded hypothesis is kept rather than deleted. A future reader finding memory ruled out is better served by seeing why it was suspected and what refuted it than by a document that never mentions it.

Refs #154
This commit is contained in:
2026-08-24 15:03:08 -05:00
parent fc3a190ec2
commit ae20568601
@@ -37,7 +37,26 @@ Recorded so none of it is re-investigated.
| Two jobs sharing one database | The paired runs per SHA are `lint` and `sonarqube`, and on a single runner they queue rather than overlap — run 444 ended 14:18:50, run 445 started 14:18:51 |
| A mismatch between the two pools | `env.setup.ts` derives `PG*` from `TEST_PG*`, and the workflow sets both to the same service container |
## The hypothesis
## SUPERSEDED — read this first
The hypothesis below (the container dying under memory pressure) is **wrong**, and the evidence that refutes it was in the original log all along.
`dmesg` came back empty — done, A1 is complete and negative. More decisively: step 7 `Run migrations` succeeded, and then step 9's `globalSetup` applied **all six migrations again from scratch**. Both target the same database, since `migrate.js` reads `PGHOST`/`PGDATABASE` and the job points those and `TEST_PG*` at the same service and the same `redefined_test`. Had step 7 migrated it, `globalSetup` would have printed `No migrations to run!`, which is what a local run prints.
It found an empty database. **The wipe was already happening before the tests started.**
So this is not one container dying partway through a heavy run. The database is being reset repeatedly, and the schema loss the suites hit later is the same cycle coming round again. That also accounts for the empty `dmesg` — a restart is not a kill.
**The suspect is now the runner's handling of `services:`**, where `act_runner` has been uneven across releases and "the service container is recreated between steps" is a known shape. Not memory.
### What to check instead
1. **In any failing run, compare step 7 and step 9 in the Gitea UI.** Both applying all six migrations confirms the wipe outright. Step 7 applying six while step 9 says `No migrations to run!` would mean this reading is wrong and the wipe happens later.
2. **The `act_runner` version on the NAS**, and whether its config says anything about service container lifetime or reuse.
A2 below is still worth running if that comparison is inconclusive, since watching the container is the only direct observation of a restart. A3 is no longer relevant — it was designed to catch memory starvation.
## The original hypothesis, kept for the record
The Postgres service container is being lost and recreated partway through the run. A recreated `postgres:16` with an empty data directory re-initialises `POSTGRES_DB` and comes back with no schema, which is exactly what the tail of the run sees.
@@ -49,18 +68,19 @@ The runner is the NAS, which also hosts Gitea, the QA stack and production. `jes
# Action list
## Now — diagnosis (yours, on the NAS)
## Now — diagnosis
- [ ] **A1.** Check for an OOM kill — cheapest, needs no timing, survives container cleanup
- [ ] **A2.** Watch the container live during a failing run — the decisive one
- [ ] **A3.** Capture memory headroom during the integration step
- [ ] **A4.** Paste the outputs below and hand back for review
- [x] **A1.** OOM kill — **done, negative.** `dmesg` empty
- [ ] **A5.** Compare step 7 and step 9's migration output in any failing run — Gitea UI, no NAS access needed
- [ ] **A6.** Record the `act_runner` version and any service-container settings in its config
- [ ] **A2.** Watch the container live — only if A5 is inconclusive
- [ ] ~~**A3.** Memory headroom~~ — dropped; the mechanism is not starvation
## Next — depends on what A1A3 say
- [ ] **B1.** If confirmed: cap memory on the QA and production stacks, or stop QA during CI runs
- [ ] **B2.** If confirmed: consider dropping `--coverage` from the integration step, accepting the loss of backend integration coverage in SonarQube
- [ ] **B3.** If **not** confirmed: reopen the investigation — the next suspect is a `pg_isready` health flap recreating the service container
- [ ] **B1.** If the service container is being recreated: pin or upgrade `act_runner`, or stop relying on `services:` and start Postgres as a step the job controls
- [ ] **B2.** If the wipe turns out to be later than step 7: fall back to A2 and watch it happen
- [ ] ~~Cap memory on the QA and production stacks~~ — dropped with the OOM hypothesis
- [ ] **B4.** Give the integration step `continue-on-error: true` so a database problem stops taking the SonarQube scan down with it
## Regardless of cause — diagnosability