bermudalamb
261d087a9c
Two standing documents rather than one, because they are different kinds of thing: one is a contract the pipeline must keep, the other is work not yet done. The coverage contract names the seven requirements that keep SonarQube's number real, and what specifically breaks if each lapses. It exists because coverage does not fail loudly — it reports a smaller number, which looks exactly like tests covering less. That is the third time this project has met a tool that succeeds while measuring nothing, after #67 and #60, so the failure mode is written down alongside how to check the guard still fires. The identity document covers CI authenticating to SonarQube as admin rather than a restricted account, raised as a "Related" note in #61 and split out so a permissions change is not buried in a CI-config commit. It spells out the revoke step explicitly, since the workflow goes green one step earlier and stopping there leaves the old credential valid.
4.3 KiB
CI Contract — Test Coverage
Status: Live as of #61
Applies to: .gitea/workflows/sonarqube.yml, sonar-project.properties, both workspaces' test tooling
What CI has to keep doing for SonarQube's coverage number to stay real. This exists because coverage does not fail loudly when it breaks — it reports a smaller number, which is indistinguishable from tests genuinely covering less.
The contract
Every one of these is load-bearing. Breaking any of them produces a plausible-looking number rather than an error.
| # | Requirement | What breaks if it lapses |
|---|---|---|
| 1 | Both backend suites run with coverage before the scan | The unit suite alone reports ~11%, because everything in src/routes is exercised only by the integration suite |
| 2 | The two backend reports go to separate directories | Jest writes coverage/lcov.info by default; the second run overwrites the first and half the coverage vanishes |
| 3 | The end-to-end run uses test:e2e:cov, not test:e2e |
An uninstrumented dev server collects nothing while every test still passes |
| 4 | coverage:report runs and is allowed to fail the job |
It is the only thing that notices an empty collection |
| 5 | sonar.javascript.lcov.reportPaths lists all three reports |
A dropped path silently removes that suite's contribution |
| 6 | The integration suite keeps --forceExit |
It hangs after completing; on 2026-08-18 that cost 3h12m of runner time |
| 7 | Nothing in the production path sets COVERAGE |
An instrumented bundle ships to customers: larger, slower, and publishing the source structure through window.__coverage__ |
The failure mode this is written against
This project has now been bitten three times by a tool succeeding while measuring nothing:
- #67 — SonarQube skipped all 34 frontend files because their tsconfig used
moduleResolution: "bundler", and still exitedEXECUTION SUCCESS. The quality gate reported on a third of the codebase for months while looking complete. - #60 — an ESLint matcher during development matched no files at all. The run was green because there was nothing to complain about.
- Coverage has the same shape by construction. If Playwright reuses an already-running, uninstrumented dev server — which
reuseExistingServermakes likely on a developer machine — every test passes,window.__coverage__is undefined, and the report is empty but valid.
The lesson each time was the same: a green tool is weak evidence. The specific defence here is frontend/scripts/coverage-report.js, which refuses to write a report when .nyc_output holds no samples and explains the two likely causes. It exists instead of calling nyc report directly, and that is the whole reason it exists.
Checking it still holds
After any change to the workflow, the Vite config, or the test tooling:
npm run buildinfrontend, then grep the bundle for__coverage__. Zero occurrences is required. This is the one that ships to customers if it regresses.- Run the coverage suites and confirm all three
lcov.infofiles exist and are non-empty. - Delete
.nyc_outputand runnpm run coverage:report. It must exit non-zero. A guard nobody has seen fire is a guard nobody knows works. - After a scan, check the coverage percentage moved in a direction the change explains. A sharp drop is far more likely to be broken collection than lost tests.
Reading the number
Backend and frontend coverage do not mean the same thing, and averaging them hides that.
Backend coverage comes from tests that assert on responses — a covered line is usually a checked line. Frontend coverage comes from Playwright driving an instrumented browser, and istanbul marks a line covered when it executes. A component rendered during an end-to-end test reports as covered with nothing asserting anything about it, so the frontend number reads considerably better than the testing behind it.
The practical consequence: the 80% gate on new code is easier to clear on frontend changes than backend ones. Treat a high frontend number as evidence the code ran, not that it works. The real fix is a frontend unit suite, which does not exist yet.
Related
docs/superpowers/specs/2026-08-20-coverage-import-design.md— the design and why each choice was madedocs/ci/sonarqube-ci-identity.md— the separate question of which account CI authenticates as