bermudalamb/redefined-designs
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects 3 Releases Wiki Activity

The integration suite's Postgres service loses its schema mid-run in CI, failing 12 of 17 suites #154

New Issue
Open
opened 2026-08-24 09:33:36 -05:00 by bermudalamb · 4 comments
bermudalamb commented 2026-08-24 09:33:36 -05:00
Owner
Copy Link
Copy Source

Backend integration tests with coverage has been failing in the SonarQube workflow since around 5ef97be. It takes the whole job with it: step 9 has no continue-on-error, so the end-to-end run, the coverage merge and the SonarQube scan are all skipped behind it.

Run 445 (job 700): 12 of 17 suites failed, 36 of 238 tests, in 145 seconds.

What is actually happening

The assertion failures are misleading. They look like data isolation problems — a category create returning 409 when the suite had just truncated, an admin price filter returning eleven items instead of one, favorite-sold notifications coming back empty. Chasing those leads nowhere.

The real error is further down the same log:

error: relation "items" does not exist
error: relation "orders" does not exist
    at resetDb (tests/integration/setup/testDb.ts:30:3)
    at Object.<anonymous> (tests/integration/errorHandling.integration.test.ts:7:3)

resetDb() is not failing to clear the tables. The tables are gone.

And there is a progression across the run, which is the part that identifies it:

  1. globalSetup migrates successfully — the log shows all six migrations applied.
  2. Early suites behave as though TRUNCATE had no effect: a beforeEach reset runs, and the next test still sees the previous test's rows. That is a database that is present but not the one just written to, or one that has been rolled back underneath the suite.
  3. Later suites cannot find the tables at all.

A schema that exists, then behaves oddly, then does not exist, over 145 seconds, is not something the test code can do. There is no globalTeardown, migrate() runs direction: 'up' only, and nothing anywhere issues a DROP. TRUNCATE ... CASCADE truncates; it never drops.

What it is not

Worth recording so nobody re-investigates:

  • Not a code regression. The suite passes locally, 238/238 across 17 suites, using the exact CI command including --coverage --forceExit.
  • Not the commit it correlates with. Failures begin at 5ef97be, which is frontend-only — src/cart/* and the Playwright suite. It cannot touch the backend integration tests. The correlation is real but coincidental.
  • Not two jobs sharing a database. Each SHA produces two runs, but they are lint and sonarqube, and on a single runner they queue rather than overlap: run 444 finished 14:18:50, run 445 started 14:18:51.
  • Not a mismatch between the two pools. env.setup.ts derives PG* from TEST_PG*, and the workflow sets both to the same service. The app's pool and testPool address the same database.

What it most likely is

The Postgres service container is being lost and recreated partway through the run. A recreated postgres:16 with an empty data directory re-initialises POSTGRES_DB and comes back with no schema, which is exactly what the tail of the run sees.

The runner is the NAS, which also hosts Gitea, the QA stack and production. jest --coverage on 17 suites is memory-hungry, the run takes 145s against a documented ~90s, and this suite already has a history of CI-specific misbehaviour — backend-integration.yml was made workflow_dispatch only after a run held the runner for 3h12m by hanging after the tests completed.

That is a hypothesis, not a conclusion. It cannot be confirmed from the workflow log, because the log only shows the client side of the connection.

Confirming it

On the NAS, during or immediately after a failing run:

  • docker ps -a for the job's postgres service container — a non-zero exit code, or a start time later than the job's, would settle it
  • dmesg -T | grep -i oom for a kill
  • runner host memory during the integration step

Making it diagnosable regardless

Independent of the cause, the failure should not present as 36 assertion errors about categories and price filters. resetDb() should check that the schema is there and fail with one clear message naming what is missing, so the next occurrence reads as "the database lost its schema" rather than as a suite full of unrelated logic bugs. globalSetup should assert the same thing after migrating.

Acceptance criteria

  • The cause of the schema loss is identified from runner-side evidence, not inferred
  • A run whose database disappears fails with a message that says so
  • The integration step's failure no longer silently skips the end-to-end run and the SonarQube scan without explanation — see #142, which covers the reporting half
`Backend integration tests with coverage` has been failing in the SonarQube workflow since around `5ef97be`. It takes the whole job with it: step 9 has no `continue-on-error`, so the end-to-end run, the coverage merge and the SonarQube scan are all skipped behind it. Run 445 (job 700): **12 of 17 suites failed, 36 of 238 tests**, in 145 seconds. ## What is actually happening The assertion failures are misleading. They look like data isolation problems — a category create returning 409 when the suite had just truncated, an admin price filter returning eleven items instead of one, favorite-sold notifications coming back empty. Chasing those leads nowhere. The real error is further down the same log: ``` error: relation "items" does not exist error: relation "orders" does not exist at resetDb (tests/integration/setup/testDb.ts:30:3) at Object.<anonymous> (tests/integration/errorHandling.integration.test.ts:7:3) ``` `resetDb()` is not failing to clear the tables. **The tables are gone.** And there is a progression across the run, which is the part that identifies it: 1. `globalSetup` migrates successfully — the log shows all six migrations applied. 2. Early suites behave as though `TRUNCATE` had no effect: a `beforeEach` reset runs, and the next test still sees the previous test's rows. That is a database that is present but not the one just written to, or one that has been rolled back underneath the suite. 3. Later suites cannot find the tables at all. A schema that exists, then behaves oddly, then does not exist, over 145 seconds, is not something the test code can do. There is no `globalTeardown`, `migrate()` runs `direction: 'up'` only, and nothing anywhere issues a `DROP`. `TRUNCATE ... CASCADE` truncates; it never drops. ## What it is not Worth recording so nobody re-investigates: - **Not a code regression.** The suite passes locally, 238/238 across 17 suites, using the exact CI command including `--coverage --forceExit`. - **Not the commit it correlates with.** Failures begin at `5ef97be`, which is frontend-only — `src/cart/*` and the Playwright suite. It cannot touch the backend integration tests. The correlation is real but coincidental. - **Not two jobs sharing a database.** Each SHA produces two runs, but they are `lint` and `sonarqube`, and on a single runner they queue rather than overlap: run 444 finished 14:18:50, run 445 started 14:18:51. - **Not a mismatch between the two pools.** `env.setup.ts` derives `PG*` from `TEST_PG*`, and the workflow sets both to the same service. The app's pool and `testPool` address the same database. ## What it most likely is The Postgres service container is being lost and recreated partway through the run. A recreated `postgres:16` with an empty data directory re-initialises `POSTGRES_DB` and comes back with no schema, which is exactly what the tail of the run sees. The runner is the NAS, which also hosts Gitea, the QA stack and production. `jest --coverage` on 17 suites is memory-hungry, the run takes 145s against a documented ~90s, and this suite already has a history of CI-specific misbehaviour — `backend-integration.yml` was made `workflow_dispatch` only after a run held the runner for 3h12m by hanging after the tests completed. That is a hypothesis, not a conclusion. It cannot be confirmed from the workflow log, because the log only shows the client side of the connection. ## Confirming it On the NAS, during or immediately after a failing run: - `docker ps -a` for the job's postgres service container — a non-zero exit code, or a start time later than the job's, would settle it - `dmesg -T | grep -i oom` for a kill - runner host memory during the integration step ## Making it diagnosable regardless Independent of the cause, the failure should not present as 36 assertion errors about categories and price filters. `resetDb()` should check that the schema is there and fail with one clear message naming what is missing, so the next occurrence reads as "the database lost its schema" rather than as a suite full of unrelated logic bugs. `globalSetup` should assert the same thing after migrating. ## Acceptance criteria - The cause of the schema loss is identified from runner-side evidence, not inferred - A run whose database disappears fails with a message that says so - The integration step's failure no longer silently skips the end-to-end run and the SonarQube scan without explanation — see #142, which covers the reporting half
bermudalamb added this to the Code Quality and Hardening 2 project 2026-08-24 12:20:20 -05:00
bermudalamb self-assigned this 2026-08-24 12:20:29 -05:00
bermudalamb commented 2026-08-24 13:44:19 -05:00
Author
Owner
Copy Link
Copy Source

Correction: the OOM hypothesis is wrong, and the evidence was already in the log

sudo dmesg -T | grep -iE "oom|killed process" came back empty. That is not conclusive on its own — a container can be recreated without a kill — but combined with what follows, memory pressure is the wrong suspect and this issue should stop pointing at it.

What was missed in the original log

Step 7 Run migrations runs node migrate.js up and succeeded. Then step 9's globalSetup applied all six migrations again, from scratch:

> Migrating files:
> - 1700000000000_baseline-schema
> - 1786974336000_add-categories-and-tags
> - 1787000000000_add-customer-disabled-at
> - 1787100000000_add-favorites
> - 1787300000000_default-items-to-pending
> - 1787400000000_split-customer-name
### MIGRATION 1700000000000_baseline-schema (UP) ###

Both target the same database. migrate.js reads PGHOST/PGPORT/PGUSER/PGDATABASE; the job sets those and TEST_PG* to the same postgres service and the same redefined_test database. There is no second database for one of them to be talking to.

So if step 7 had migrated that database, pgmigrations would already have been populated and globalSetup would have printed No migrations to run! — which is exactly what a local run prints, and what the tail of the CI run prints too.

It found an empty database instead.

What that changes

The database was already being wiped before the tests started. This is not a container dying partway through a long, memory-hungry run. It is being reset repeatedly, and the schema loss the suites hit later is the same thing happening again mid-run.

That also explains the empty dmesg cleanly: a restart is not a kill, and nothing would appear there.

It further explains the progression that was previously hard to account for — migrations succeed, then truncates appear to do nothing, then tables are gone. Those are three different points in the same repeating cycle rather than a single degradation.

Where to look now

Not at memory. The suspect is the runner's handling of services:, which in act_runner has been uneven across releases and where "the service container is recreated between steps" is a known shape.

Two things to check, both cheaper than the NAS diagnostics this issue previously asked for, and the first needs nothing but the Gitea UI:

  1. In any failing run, compare step 7 and step 9. Both applying all six migrations confirms the wipe outright. Step 7 applying six while step 9 says No migrations to run! would mean the reading above is wrong and the wipe happens later.
  2. The act_runner version on the NAS, and whether the runner config sets anything about service container lifetime or reuse.

Status of the earlier diagnostics

A1 (dmesg) is done and negative. A2 and A3 in docs/ci/integration-schema-loss-investigation.md are no longer the priority — they were designed to catch a container dying under memory pressure, and the mechanism now looks like recreation rather than death. A2 is still worth running if step 7 versus step 9 turns out inconclusive, since watching the container is the only direct observation of a restart.

The document will be updated to match rather than left describing a hypothesis the evidence no longer supports.

## Correction: the OOM hypothesis is wrong, and the evidence was already in the log `sudo dmesg -T | grep -iE "oom|killed process"` came back **empty**. That is not conclusive on its own — a container can be recreated without a kill — but combined with what follows, memory pressure is the wrong suspect and this issue should stop pointing at it. ### What was missed in the original log Step 7 `Run migrations` runs `node migrate.js up` and **succeeded**. Then step 9's `globalSetup` applied all six migrations again, from scratch: ``` > Migrating files: > - 1700000000000_baseline-schema > - 1786974336000_add-categories-and-tags > - 1787000000000_add-customer-disabled-at > - 1787100000000_add-favorites > - 1787300000000_default-items-to-pending > - 1787400000000_split-customer-name ### MIGRATION 1700000000000_baseline-schema (UP) ### ``` Both target the same database. `migrate.js` reads `PGHOST`/`PGPORT`/`PGUSER`/`PGDATABASE`; the job sets those and `TEST_PG*` to the same `postgres` service and the same `redefined_test` database. There is no second database for one of them to be talking to. So if step 7 had migrated that database, `pgmigrations` would already have been populated and `globalSetup` would have printed `No migrations to run!` — which is exactly what a local run prints, and what the tail of the CI run prints too. It found an empty database instead. ### What that changes **The database was already being wiped before the tests started.** This is not a container dying partway through a long, memory-hungry run. It is being reset repeatedly, and the schema loss the suites hit later is the same thing happening again mid-run. That also explains the empty `dmesg` cleanly: a restart is not a kill, and nothing would appear there. It further explains the progression that was previously hard to account for — migrations succeed, then truncates appear to do nothing, then tables are gone. Those are three different points in the same repeating cycle rather than a single degradation. ### Where to look now Not at memory. The suspect is the runner's handling of `services:`, which in `act_runner` has been uneven across releases and where "the service container is recreated between steps" is a known shape. Two things to check, both cheaper than the NAS diagnostics this issue previously asked for, and the first needs nothing but the Gitea UI: 1. **In any failing run, compare step 7 and step 9.** Both applying all six migrations confirms the wipe outright. Step 7 applying six while step 9 says `No migrations to run!` would mean the reading above is wrong and the wipe happens later. 2. **The `act_runner` version on the NAS**, and whether the runner config sets anything about service container lifetime or reuse. ### Status of the earlier diagnostics A1 (`dmesg`) is **done and negative**. A2 and A3 in `docs/ci/integration-schema-loss-investigation.md` are no longer the priority — they were designed to catch a container dying under memory pressure, and the mechanism now looks like recreation rather than death. A2 is still worth running if step 7 versus step 9 turns out inconclusive, since watching the container is the only direct observation of a restart. The document will be updated to match rather than left describing a hypothesis the evidence no longer supports.
bermudalamb commented 2026-08-24 14:49:18 -05:00
Author
Owner
Copy Link
Copy Source

Parked

Shelved deliberately rather than stalled: the remaining work needs runner-side access and attention that is not available right now. Everything below is so this can be picked up cold.

State of knowledge

Symptom. Backend integration tests with coverage fails, 12 of 17 suites and 36 of 238 tests. Step 9 has no continue-on-error, so it takes the end-to-end run, the coverage merge and the SonarQube scan with it. The SonarQube analysis has been stale since this started — that is the ongoing cost of leaving this parked, and it is the reason to unpark it sooner rather than later.

Cause, as far as it is understood. The redefined_test database is wiped repeatedly during the job. Not once, and not under load: step 7 Run migrations succeeds, and then step 9's globalSetup applies all six migrations again from scratch, which only happens against an empty database. The schema loss the later suites hit is the same cycle coming round again.

Ruled out, with evidence:

  • A code regression — the suite passes locally 238/238 using the exact CI command, --coverage --forceExit included
  • The commit it correlates with, 5ef97be, which is frontend-only and cannot touch backend integration tests
  • Two jobs sharing a database — the paired runs are lint and sonarqube, and they queue on the single runner rather than overlap
  • A mismatch between the app pool and testPool — both resolve to the same service and the same database
  • Memory pressure / an OOM kill — dmesg empty, and the wipe happens before the tests start rather than partway through a heavy run

Current suspect. The runner's handling of services:. act_runner has been uneven across releases, and "the service container is recreated between steps" is a known shape that fits every observation including the empty dmesg.

To resume, in order

  1. In any failing run, compare step 7 and step 9's migration output. Gitea UI, no NAS access. Both applying all six confirms the wipe outright. Step 7 applying six while step 9 says No migrations to run! means the reading above is wrong and the wipe happens later.
  2. Record the act_runner version and whether its config says anything about service container lifetime or reuse.
  3. Only if 1 is inconclusive: A2 in docs/ci/integration-schema-loss-investigation.md — watch the container during a run. It has a timing constraint that is easy to miss, since the service container is deleted when the job ends.

Worth doing while parked, and independent of the cause

  • continue-on-error: true on the integration step, with the gate extended to include its outcome. A database problem would then stop taking the SonarQube scan down with it, which is most of what this failure actually costs.
  • resetDb() and globalSetup asserting the schema is present, so this presents as "the database lost its schema" rather than as 36 assertion errors about categories and price filters. That misdirection is what made the original diagnosis expensive.

Neither depends on knowing the cause, and both make the next occurrence cheaper to read.

## Parked Shelved deliberately rather than stalled: the remaining work needs runner-side access and attention that is not available right now. Everything below is so this can be picked up cold. ### State of knowledge **Symptom.** `Backend integration tests with coverage` fails, 12 of 17 suites and 36 of 238 tests. Step 9 has no `continue-on-error`, so it takes the end-to-end run, the coverage merge and the SonarQube scan with it. **The SonarQube analysis has been stale since this started** — that is the ongoing cost of leaving this parked, and it is the reason to unpark it sooner rather than later. **Cause, as far as it is understood.** The `redefined_test` database is wiped repeatedly during the job. Not once, and not under load: step 7 `Run migrations` succeeds, and then step 9's `globalSetup` applies all six migrations again from scratch, which only happens against an empty database. The schema loss the later suites hit is the same cycle coming round again. **Ruled out, with evidence:** - A code regression — the suite passes locally 238/238 using the exact CI command, `--coverage --forceExit` included - The commit it correlates with, `5ef97be`, which is frontend-only and cannot touch backend integration tests - Two jobs sharing a database — the paired runs are `lint` and `sonarqube`, and they queue on the single runner rather than overlap - A mismatch between the app pool and `testPool` — both resolve to the same service and the same database - **Memory pressure / an OOM kill** — `dmesg` empty, and the wipe happens before the tests start rather than partway through a heavy run **Current suspect.** The runner's handling of `services:`. `act_runner` has been uneven across releases, and "the service container is recreated between steps" is a known shape that fits every observation including the empty `dmesg`. ### To resume, in order 1. **In any failing run, compare step 7 and step 9's migration output.** Gitea UI, no NAS access. Both applying all six confirms the wipe outright. Step 7 applying six while step 9 says `No migrations to run!` means the reading above is wrong and the wipe happens later. 2. **Record the `act_runner` version** and whether its config says anything about service container lifetime or reuse. 3. Only if 1 is inconclusive: **A2** in `docs/ci/integration-schema-loss-investigation.md` — watch the container during a run. It has a timing constraint that is easy to miss, since the service container is deleted when the job ends. ### Worth doing while parked, and independent of the cause - **`continue-on-error: true` on the integration step**, with the gate extended to include its outcome. A database problem would then stop taking the SonarQube scan down with it, which is most of what this failure actually costs. - **`resetDb()` and `globalSetup` asserting the schema is present**, so this presents as "the database lost its schema" rather than as 36 assertion errors about categories and price filters. That misdirection is what made the original diagnosis expensive. Neither depends on knowing the cause, and both make the next occurrence cheaper to read.
bermudalamb commented 2026-08-25 09:21:02 -05:00
Author
Owner
Copy Link
Copy Source

Resume step 1 is answered, and it refutes the reading this issue was parked on

The parked comment said:

Step 7 applying six while step 9 says No migrations to run! means the reading above is wrong and the wipe happens later.

That is what run 525 shows.

13:53:58  No migrations to run!
13:54:13  FAIL categoriesTags — relation "categories" does not exist

globalSetup found pgmigrations populated. The database was not empty when the tests started, so the "wiped before the run" conclusion is wrong and the earlier log was read incorrectly. Whatever happens, happens during the run.

The schema comes back, which rules out a wipe entirely

The order of suites, with --runInBand:

Time Suite Result
13:54:13 categoriesTags FAIL — categories does not exist
13:54:33 favorites FAIL — orders does not exist, in resetDb's TRUNCATE
13:54:45 adminInventory FAIL — categories, tags, items, admin_settings
13:55:00 passwordReset PASS

passwordReset calls resetDb(), and resetDb() truncates orders. It passed. So orders did not exist at 13:54:33 and did exist at 13:55:00.

A database being dropped does not un-drop itself. This is not one database losing its schema — it is more than one database answering to the same name, some migrated and some not, with connections landing on different ones.

That also explains, for the first time, the symptom this issue has always led with and never accounted for: "a failing set that reshuffles between identical runs, in specs that touch no passwords at all, and every one of them passing in isolation." Which suites fail depends on which backend their connections happened to reach. Nothing about the code is involved, which is why it passes locally with one container and why 5ef97be was always an innocent correlation.

It explains the empty dmesg too, and more cleanly than a restart does — nothing died and nothing was recreated.

What this does not yet tell us

Where the second Postgres comes from. Two candidates I could rule out from the repository:

  • lint.yml declares no services at all, so the paired run is not it.
  • backend-integration.yml does declare one, but it is workflow_dispatch only, so it is not running alongside.

That leaves the runner. A leftover service container from an earlier run keeping the postgres network alias would produce exactly this, since Docker's embedded DNS round-robins every container sharing an alias. That is consistent with the existing suspicion about act_runner's services: handling, but it is a hypothesis and this issue has already been wrong twice from reasoning ahead of evidence.

The measurement that settles it

One line, in globalSetup, before anything else runs: resolve postgres and print every address it returns.

  • More than one address is the answer, outright.
  • One address means this reading is wrong too, and the next thing to look at is whether the single container is being restarted with a fresh data directory mid-run.

Worth pairing with SELECT inet_server_addr() from both pool and testPool, so the log records which backend each one actually reached rather than which one it was configured to reach, and with a schema assertion in resetDb so the failure says "the schema is gone" instead of producing assertion errors about categories and price filters.

I will put that together next unless you would rather look at the runner first — the act_runner version and whether its config says anything about service container lifetime is still worth having either way, and it is the thing I cannot see from here.

Unrelated to the cause, but fixed today

Run 525 also confirmed #174 works: the integration suite failed, the end-to-end suite ran again, and SonarQube Scan succeeded — the first analysis since this started. The job still failed, at the gate, which is the intent.

Summarize integration tests failed in that run, which it should not be able to. That is #178, and it is fixed on feature/178-resilient-summarisers.

## Resume step 1 is answered, and it refutes the reading this issue was parked on The parked comment said: > Step 7 applying six while step 9 says `No migrations to run!` means the reading above is wrong and the wipe happens later. That is what run 525 shows. ``` 13:53:58 No migrations to run! 13:54:13 FAIL categoriesTags — relation "categories" does not exist ``` `globalSetup` found `pgmigrations` populated. The database was **not** empty when the tests started, so the "wiped before the run" conclusion is wrong and the earlier log was read incorrectly. Whatever happens, happens during the run. ## The schema comes back, which rules out a wipe entirely The order of suites, with `--runInBand`: | Time | Suite | Result | | --- | --- | --- | | 13:54:13 | categoriesTags | FAIL — `categories` does not exist | | 13:54:33 | favorites | FAIL — `orders` does not exist, in `resetDb`'s TRUNCATE | | 13:54:45 | adminInventory | FAIL — `categories`, `tags`, `items`, `admin_settings` | | 13:55:00 | passwordReset | **PASS** | `passwordReset` calls `resetDb()`, and `resetDb()` truncates `orders`. It passed. So `orders` did not exist at 13:54:33 and did exist at 13:55:00. A database being dropped does not un-drop itself. This is not one database losing its schema — it is **more than one database answering to the same name**, some migrated and some not, with connections landing on different ones. That also explains, for the first time, the symptom this issue has always led with and never accounted for: *"a failing set that reshuffles between identical runs, in specs that touch no passwords at all, and every one of them passing in isolation."* Which suites fail depends on which backend their connections happened to reach. Nothing about the code is involved, which is why it passes locally with one container and why `5ef97be` was always an innocent correlation. It explains the empty `dmesg` too, and more cleanly than a restart does — nothing died and nothing was recreated. ## What this does not yet tell us Where the second Postgres comes from. Two candidates I could rule out from the repository: - `lint.yml` declares no services at all, so the paired run is not it. - `backend-integration.yml` does declare one, but it is `workflow_dispatch` only, so it is not running alongside. That leaves the runner. A leftover service container from an earlier run keeping the `postgres` network alias would produce exactly this, since Docker's embedded DNS round-robins every container sharing an alias. That is consistent with the existing suspicion about `act_runner`'s `services:` handling, but it is a hypothesis and this issue has already been wrong twice from reasoning ahead of evidence. ## The measurement that settles it One line, in `globalSetup`, before anything else runs: resolve `postgres` and print every address it returns. - More than one address is the answer, outright. - One address means this reading is wrong too, and the next thing to look at is whether the single container is being restarted with a fresh data directory mid-run. Worth pairing with `SELECT inet_server_addr()` from both `pool` and `testPool`, so the log records which backend each one actually reached rather than which one it was configured to reach, and with a schema assertion in `resetDb` so the failure says "the schema is gone" instead of producing assertion errors about categories and price filters. I will put that together next unless you would rather look at the runner first — the `act_runner` version and whether its config says anything about service container lifetime is still worth having either way, and it is the thing I cannot see from here. ## Unrelated to the cause, but fixed today Run 525 also confirmed #174 works: the integration suite failed, the end-to-end suite ran again, and **`SonarQube Scan` succeeded** — the first analysis since this started. The job still failed, at the gate, which is the intent. `Summarize integration tests` failed in that run, which it should not be able to. That is #178, and it is fixed on `feature/178-resilient-summarisers`.
bermudalamb commented 2026-09-09 16:27:10 -05:00
Author
Owner
Copy Link
Copy Source

The measurement fired, and it is the answer

Run 917, job 1172, step 9. The instrumentation added for this issue printed:

[#154] "postgres" resolves to 2 address(es): 192.168.144.4, 192.168.144.5  <-- more than one server can answer; this is the bug
[#154] globalSetup reached: db=redefined_test addr=192.168.144.4/32 postmaster_start=2026-09-08 22:34:13.62096+00 pid=1341

The previous comment set the terms: "More than one address is the answer, outright." There are two. Two Postgres containers are answering to the alias postgres, and Docker's embedded DNS round-robins between them.

This is evidence, not inference, which closes the first acceptance criterion.

And the second address is a leftover, which the timestamps prove

The job started 2026-09-09T21:08:10Z. The server that answered globalSetup reports a postmaster start time of 2026-09-08 22:34:13 — twenty-two and a half hours earlier, the previous day.

That container is not this job's service container. It is a leftover from an earlier run that was never torn down, still attached to the network and still holding the postgres alias.

Everything this issue could not previously explain now falls out

  • "The schema comes back." It never went. orders was missing at 13:54:33 and present at 13:55:00 because those two connections landed on different servers. A pg pool opens several connections and each resolves the alias independently, so this varies within a single pool, not merely between suites.
  • The reshuffling failure set. Which suites fail depends on which backend their connections happened to reach. Nothing about the code is involved.
  • Passing locally, 238/238. One container, one address, no ambiguity.
  • 5ef97be was always innocent. The correlation was coincidental, as suspected.
  • The empty dmesg. Nothing died and nothing was recreated. Both servers were healthy the whole time.

It also explains itself, which is the part worth sitting with

Run 917 is hung right now, on step 13 Frontend end-to-end tests with coverage, and has been since it started. This issue already records a run that held the runner for 3h12m by hanging after its tests completed.

A hung job never reaches teardown, so its Postgres service container is never removed. That container is the leftover the next run trips over.

The hangs cause the schema-loss symptom. These were never two problems.

What is left

The first acceptance criterion is met. The remaining work is in two places:

Runner-side, and it needs NAS access:

  • Clear the hung run 917. It is holding the only runner, and 24 runs are queued behind it — every workflow for #38 through #42 is unrun.
  • Remove orphaned service containers, and find out why a hung job's containers survive. docker ps -a filtered to the postgres image will show how many are lying around.
  • The hang itself is the root cause and deserves its own issue once this one is closed.

Repository-side, and I can do it from here:

A run whose alias resolves to more than one address should refuse to start, naming both. It is meaningless to test against two databases at once, and the current behaviour is 36 assertion errors about categories and price filters — the misdirection that made this expensive the first three times.

Worth noting the guard has to compare within an address family: localhost legitimately resolves to both 127.0.0.1 and ::1 locally, and failing on that would break every local run.

A second, smaller defect in the same log

schemaLoss.integration.test.ts failed:

● when the database loses its schema › resetDb turns a failed truncate into the schema message
  Expected pattern: /has no schema \(while resetting between tests\)/
  Received message: "relation \"item_drafts\" does not exist"

That test drops the schema and expects resetDb to produce the readable message. It got the raw error, because missingTables() reported nothing missing — its connection reached the server that still had the schema, while the TRUNCATE reached the one that did not.

So the friendly-message guard is itself defeated by this bug. Worth knowing before anyone treats that failure as a separate problem.

## The measurement fired, and it is the answer Run 917, job 1172, step 9. The instrumentation added for this issue printed: ``` [#154] "postgres" resolves to 2 address(es): 192.168.144.4, 192.168.144.5 <-- more than one server can answer; this is the bug [#154] globalSetup reached: db=redefined_test addr=192.168.144.4/32 postmaster_start=2026-09-08 22:34:13.62096+00 pid=1341 ``` The previous comment set the terms: *"More than one address is the answer, outright."* There are two. **Two Postgres containers are answering to the alias `postgres`, and Docker's embedded DNS round-robins between them.** This is evidence, not inference, which closes the first acceptance criterion. ## And the second address is a leftover, which the timestamps prove The job started `2026-09-09T21:08:10Z`. The server that answered `globalSetup` reports a postmaster start time of **`2026-09-08 22:34:13`** — twenty-two and a half hours earlier, the previous day. That container is not this job's service container. It is a leftover from an earlier run that was never torn down, still attached to the network and still holding the `postgres` alias. ## Everything this issue could not previously explain now falls out - **"The schema comes back."** It never went. `orders` was missing at 13:54:33 and present at 13:55:00 because those two connections landed on different servers. A `pg` pool opens several connections and each resolves the alias independently, so this varies *within a single pool*, not merely between suites. - **The reshuffling failure set.** Which suites fail depends on which backend their connections happened to reach. Nothing about the code is involved. - **Passing locally, 238/238.** One container, one address, no ambiguity. - **`5ef97be` was always innocent.** The correlation was coincidental, as suspected. - **The empty `dmesg`.** Nothing died and nothing was recreated. Both servers were healthy the whole time. ## It also explains itself, which is the part worth sitting with Run 917 is **hung right now**, on step 13 `Frontend end-to-end tests with coverage`, and has been since it started. This issue already records a run that held the runner for 3h12m by hanging after its tests completed. A hung job never reaches teardown, so its Postgres service container is never removed. That container is the leftover the next run trips over. **The hangs cause the schema-loss symptom.** These were never two problems. ## What is left The first acceptance criterion is met. The remaining work is in two places: **Runner-side, and it needs NAS access:** - Clear the hung run 917. It is holding the only runner, and **24 runs are queued behind it** — every workflow for #38 through #42 is unrun. - Remove orphaned service containers, and find out why a hung job's containers survive. `docker ps -a` filtered to the postgres image will show how many are lying around. - The hang itself is the root cause and deserves its own issue once this one is closed. **Repository-side, and I can do it from here:** A run whose alias resolves to more than one address should refuse to start, naming both. It is meaningless to test against two databases at once, and the current behaviour is 36 assertion errors about categories and price filters — the misdirection that made this expensive the first three times. Worth noting the guard has to compare within an address family: `localhost` legitimately resolves to both `127.0.0.1` and `::1` locally, and failing on that would break every local run. ## A second, smaller defect in the same log `schemaLoss.integration.test.ts` failed: ``` ● when the database loses its schema › resetDb turns a failed truncate into the schema message Expected pattern: /has no schema \(while resetting between tests\)/ Received message: "relation \"item_drafts\" does not exist" ``` That test drops the schema and expects `resetDb` to produce the readable message. It got the raw error, because `missingTables()` reported nothing missing — its connection reached the server that still had the schema, while the `TRUNCATE` reached the one that did not. So the friendly-message guard is itself defeated by this bug. Worth knowing before anyone treats that failure as a separate problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Code Quality and Hardening 2
To Do
Assignees
Clear assignees
calamb (Chelsea Lamb) bermudalamb (Thom Lamb)
No Assignees bermudalamb
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: bermudalamb/redefined-designs#154