ci: fail at the end rather than part way through, so the scan still runs (#174)
Linting / lint (pull_request) Successful in 2m6s
SonarQube Analysis / sonarqube (pull_request) Failing after 19m53s

`sonarqube.yml` already had a documented design for this: run every suite, produce coverage, scan, summarise, then fail at the end from recorded step outcomes. The comments on the gate spell it out and #142 fixed it once already. The integration suite was never wired into it — no `id`, no `continue-on-error`, and absent from the gate, where the unit and end-to-end suites had all three.

So it was the one suite whose failure aborted the job. On every push since #154 started, step 9 failed and steps 10 through 15 were skipped, which means there has been no SonarQube analysis at all for the duration — not a degraded one, none. The end-to-end suite has not run in CI either, which is separately why #116 cannot be verified: the step that would demonstrate its fix is skipped rather than failing.

Fixing only that step would have left the same shape in four other places, so every step from the first suite to the scan is now guarded and named in the gate: starting the backend, installing browsers, merging frontend coverage, and the scan itself, which until now took the summaries down with it. The preconditions before the suites — checkout, installs, build checks, migrations — still fail hard, because when they fail there is genuinely nothing to analyse.

The job still fails. It fails at the end, having produced everything it could.

The integration suite also gains the summary the other two already had. It was the only suite without one, so the failure this workflow has been stuck on presented as 36 assertion errors about categories and price filters rather than as a count — #154 records how expensive that misdirection was to read. `summarize-jest.js` already takes a label, so this is reuse.

`tests/unit/workflowGate.test.ts` asserts the pairing that makes the design work: a step carries `continue-on-error` so it cannot abort the job, and the gate names it so it can still fail the job. Both halves are needed and nothing connected them, which is how this happened — and the other direction is worse, since a step guarded but unnamed cannot fail the job at all. The test checks the invariant rather than a list of names, for the same reason `composeEnvironment.test.ts` reads the real list rather than a copy. Verified by mutation: dropping the integration suite from the gate, un-guarding it, and removing `always()` each fail it.

Confirmed by running the new flag combination rather than assuming it: the integration suite writes `integration-results.json`, the summariser reads it and exits 0, and both that file and `coverage/integration/lcov.info` are still written when the suite fails — which is what makes scanning with a failing suite produce real coverage rather than a fabricated regression.

Closes #174
This commit is contained in:
2026-08-25 08:47:42 -05:00
parent 667a40c4eb
commit e3842b1a4c
2 changed files with 242 additions and 6 deletions
+59 -6
View File
@@ -106,11 +106,24 @@ jobs:
# Covers everything in src/routes, which the unit suite does not touch —
# without this the backend reports around 11% rather than the ~73% it
# actually has.
#
# Guarded like the other two suites. It was the only one that was not, so
# it was the only one whose failure aborted the job — taking the scan, the
# end-to-end run and the coverage merge with it. That is what #154 has
# cost on every push since it started: not a degraded analysis, none at
# all. See #174.
- name: Backend integration tests with coverage
run: npm run test:integration:cov
id: integration
continue-on-error: true
run: npm run test:integration:cov -- --json --outputFile=integration-results.json
working-directory: backend
# Guarded because the scan does not depend on it. A backend that will not
# start fails the end-to-end run below on its own, and the gate records
# both — there is no reason for it to also cost the analysis.
- name: Start backend for the end-to-end run
id: backend
continue-on-error: true
run: |
mkdir -p /tmp/redefined-uploads
node dist/server.js > /tmp/backend.log 2>&1 &
@@ -126,7 +139,10 @@ jobs:
exit 1
working-directory: backend
# A network fetch, and so the least interesting way to lose an analysis.
- name: Install Playwright browsers
id: browsers
continue-on-error: true
run: npx playwright install --with-deps chromium
working-directory: frontend
@@ -147,17 +163,32 @@ jobs:
# never fire and the log that explains an end-to-end failure would go
# unprinted precisely when it is wanted.
- name: Backend log
if: steps.e2e.outcome == 'failure'
if: steps.e2e.outcome == 'failure' || steps.backend.outcome == 'failure'
run: cat /tmp/backend.log
# Fails when nothing was collected rather than writing an empty report. An
# uninstrumented dev server lets every test pass while gathering nothing,
# and the resulting 0% reads as "the tests stopped covering things".
- name: Merge frontend coverage
id: coverage
continue-on-error: true
run: npm run coverage:report
working-directory: frontend
# Guarded so that a scanner error does not take the summaries below with
# it. The gate still records it, so a failed scan fails the job.
#
# If a coverage report is missing — the end-to-end run collecting nothing,
# say — this scans anyway and SonarQube reports those files as uncovered,
# which reads as a regression rather than as a missing input. Accepted:
# jest still writes coverage for a suite whose tests fail, so the failure
# actually occurring produces all three reports; there is no
# sonar.qualitygate.wait, so a degraded run marks the dashboard and is
# overwritten by the next good one rather than blocking anything; and the
# job fails regardless, so no run in this state reads as clean.
- name: SonarQube Scan
id: scan
continue-on-error: true
uses: sonarsource/sonarqube-scan-action@v4
env:
SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
@@ -170,15 +201,28 @@ jobs:
if: always()
run: node scripts/summarize-jest.js backend/unit-results.json "Backend Unit Test"
# The suite this workflow has been failing on for weeks, and the only one
# that had no summary — so it presented as 36 assertion errors about
# categories and price filters rather than as a count. #154 records how
# expensive that misdirection was to read.
- name: Summarize integration tests
if: always()
run: node scripts/summarize-jest.js backend/integration-results.json "Backend Integration Test"
- name: Summarize end-to-end tests
if: always()
run: node scripts/summarize-playwright.js frontend/playwright-results.json
# Last, so a failing suite still produces coverage, a scan and both
# summaries first. Without this step continue-on-error above would turn a
# Last, so a failing step still produces coverage, a scan and every
# summary first. Without this step continue-on-error above would turn a
# failing suite into a passing job, which is the one way this change could
# do real damage.
#
# Every guarded step is listed. That is the invariant — a step carrying
# continue-on-error and missing from here cannot fail the job at all — and
# tests/unit/workflowGate.test.ts asserts it, because the integration
# suite going unlisted is exactly how #174 happened.
#
# always() is load-bearing. A step whose `if:` omits it still implicitly
# requires every previous step to have succeeded, so this was skipped in
# exactly the case it exists for: the summarise step above used to exit 1
@@ -186,6 +230,15 @@ jobs:
# code. The job then reported its failure under a name that describes
# summarising rather than testing. The summarisers exit 0 now; this is what
# fails the run. See #142.
- name: Fail if either suite failed
if: always() && (steps.unit.outcome == 'failure' || steps.e2e.outcome == 'failure')
- name: Fail if any guarded step failed
if: >-
always() && (
steps.unit.outcome == 'failure' ||
steps.integration.outcome == 'failure' ||
steps.backend.outcome == 'failure' ||
steps.browsers.outcome == 'failure' ||
steps.e2e.outcome == 'failure' ||
steps.coverage.outcome == 'failure' ||
steps.scan.outcome == 'failure'
)
run: exit 1