redefined-designs/docs/ci/integration-schema-loss-investigation.md

Investigating the integration suite's schema loss in CI (#154)

A working document for the open investigation into #154. Fill in the output blocks as you go — each command has a slot beneath it and a note saying what the answer would mean. Nothing here changes the repository; it is all read-only inspection on the NAS.

What is known

Backend integration tests with coverage fails in the SonarQube workflow, taking the whole job with it. Step 9 carries no continue-on-error, so the end-to-end run, the coverage merge and the SonarQube scan are all skipped behind it. The SonarQube analysis has been stale since this started.

Run 445 (job 700) failed 12 of 17 suites and 36 of 238 tests in 145 seconds.

The 36 assertion failures are misleading. They read as data-isolation problems — a category create returning 409 immediately after a truncate, an admin price filter returning eleven items instead of one, favorite-sold notifications coming back empty. Chasing those leads nowhere. The real error appears further down the same log:

error: relation "items" does not exist
error: relation "orders" does not exist
    at resetDb (tests/integration/setup/testDb.ts:30:3)

resetDb() is not failing to clear the tables. The tables are gone.

There is a progression across the run, and it is the progression rather than any single error that identifies the problem:

1. globalSetup migrates successfully — all six migrations apply.
2. Early suites behave as though TRUNCATE had no effect: a beforeEach reset runs, and the next test still sees the previous test's rows.
3. Later suites cannot find the tables at all.

A schema that exists, then behaves oddly, then does not exist, across 145 seconds, is not something the test code can cause. There is no globalTeardown, migrate() runs direction: 'up' only, nothing anywhere issues a DROP, and TRUNCATE ... CASCADE truncates but never drops.

What has been ruled out

Recorded so none of it is re-investigated.

Ruled out	How
A code regression	The suite passes locally, 238/238 across 17 suites, using the exact CI command including --coverage --forceExit
The commit it correlates with	Failures begin at 5ef97be, which touches only frontend/src/cart/* and the Playwright suite. It cannot affect backend integration tests. The correlation is real and coincidental
Two jobs sharing one database	The paired runs per SHA are lint and sonarqube, and on a single runner they queue rather than overlap — run 444 ended 14:18:50, run 445 started 14:18:51
A mismatch between the two pools	env.setup.ts derives PG* from TEST_PG*, and the workflow sets both to the same service container

The hypothesis

The Postgres service container is being lost and recreated partway through the run. A recreated postgres:16 with an empty data directory re-initialises POSTGRES_DB and comes back with no schema, which is exactly what the tail of the run sees.

The runner is the NAS, which also hosts Gitea, the QA stack and production. jest --coverage across 17 suites is memory-hungry, the run takes 145s against a documented ~90s, and this suite already has form: backend-integration.yml was made workflow_dispatch-only after a run held the runner for 3h12m by hanging after the tests had completed.

This is a hypothesis, not a conclusion. It cannot be confirmed from the workflow log, which only shows the client side of the connection. If the checks below show the container healthy throughout, the hypothesis is wrong and this document should say so rather than be quietly abandoned.

---

Action list

Now — diagnosis (yours, on the NAS)

• A1. Check for an OOM kill — cheapest, needs no timing, survives container cleanup
• A2. Watch the container live during a failing run — the decisive one
• A3. Capture memory headroom during the integration step
• A4. Paste the outputs below and hand back for review

Next — depends on what A1–A3 say

• B1. If confirmed: cap memory on the QA and production stacks, or stop QA during CI runs
• B2. If confirmed: consider dropping --coverage from the integration step, accepting the loss of backend integration coverage in SonarQube
• B3. If not confirmed: reopen the investigation — the next suspect is a pg_isready health flap recreating the service container
• B4. Give the integration step continue-on-error: true so a database problem stops taking the SonarQube scan down with it

Regardless of cause — diagnosability

• C1. resetDb() asserts the schema is present and fails with one clear message naming what is missing
• C2. globalSetup asserts the same thing immediately after migrating, so the run fails at setup rather than 90 seconds later
• C3. Confirm the change by dropping a table by hand and checking the message reads as "the database lost its schema" rather than as a suite full of logic errors

C1–C3 are mine to do and do not depend on the diagnosis. They only change what the failure says, never whether it happens.

---

Diagnostics to run

A1 — OOM kill

Run at any time after a failing run. This is kernel-level and survives the container being removed, which is why it comes first.

sudo dmesg -T | grep -iE "oom|killed process" | tail -20

Paste output:

Reading it: a line naming postgres or node at around the run's timestamp is conclusive — the hypothesis is confirmed and you can skip to B1. No output at all does not disprove it; a container can be recreated without an OOM kill, so continue to A2.

A2 — Watch the container during a failing run

The decisive check. Timing matters: the service container is deleted when the job finishes, so this has to run during the job, not after.

Trigger the workflow (any push, or re-run one from the Gitea UI), wait for the Backend integration tests with coverage step to start, then run this and leave it up for the ~2½ minutes the step takes:

watch -n 5 'docker ps -a --filter "name=postgres" --format "{{.Names}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}"'

Paste what you see, especially any change:

Reading it:

• Up 40 seconds resetting to a lower number — the container restarted. Confirmed.
• Status showing Restarting or Exited (137) — killed, 137 meaning SIGKILL, which is what an OOM looks like. Confirmed.
• Up climbing steadily the whole time, never resetting — the hypothesis is wrong. Go to B3.

A3 — Memory headroom

Run once while the integration step is in flight.

free -m && docker stats --no-stream --format "{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

Paste output:

Reading it: available memory near zero while both jest and postgres are running is the mechanism, and it tells us which of the other stacks to cap in B1. Comfortable headroom argues against the hypothesis even if A2 showed a restart, and would point at something recreating the container deliberately rather than it dying.

---

If you can only do one

A1. It needs no timing, survives cleanup, and a positive result settles the question outright.

Notes for the review

When you hand these back, worth saying alongside them:

• roughly how much memory the NAS has, and whether QA was up during the run
• whether the failing runs are every run now, or intermittent — the issue assumes every run since 5ef97be, and if it is actually intermittent that changes the shape of the answer