bermudalamb
261d087a9c
Two standing documents rather than one, because they are different kinds of thing: one is a contract the pipeline must keep, the other is work not yet done. The coverage contract names the seven requirements that keep SonarQube's number real, and what specifically breaks if each lapses. It exists because coverage does not fail loudly — it reports a smaller number, which looks exactly like tests covering less. That is the third time this project has met a tool that succeeds while measuring nothing, after #67 and #60, so the failure mode is written down alongside how to check the guard still fires. The identity document covers CI authenticating to SonarQube as admin rather than a restricted account, raised as a "Related" note in #61 and split out so a permissions change is not buried in a CI-config commit. It spells out the revoke step explicitly, since the workflow goes green one step earlier and stopping there leaves the old credential valid.
53 lines
4.3 KiB
Markdown
53 lines
4.3 KiB
Markdown
# CI Contract — Test Coverage
|
|
|
|
**Status:** Live as of #61
|
|
**Applies to:** `.gitea/workflows/sonarqube.yml`, `sonar-project.properties`, both workspaces' test tooling
|
|
|
|
What CI has to keep doing for SonarQube's coverage number to stay real. This exists because coverage does not fail loudly when it breaks — it reports a smaller number, which is indistinguishable from tests genuinely covering less.
|
|
|
|
## The contract
|
|
|
|
Every one of these is load-bearing. Breaking any of them produces a plausible-looking number rather than an error.
|
|
|
|
| # | Requirement | What breaks if it lapses |
|
|
| --- | --- | --- |
|
|
| 1 | Both backend suites run with coverage before the scan | The unit suite alone reports ~11%, because everything in `src/routes` is exercised only by the integration suite |
|
|
| 2 | The two backend reports go to separate directories | Jest writes `coverage/lcov.info` by default; the second run overwrites the first and half the coverage vanishes |
|
|
| 3 | The end-to-end run uses `test:e2e:cov`, not `test:e2e` | An uninstrumented dev server collects nothing while every test still passes |
|
|
| 4 | `coverage:report` runs and is allowed to fail the job | It is the only thing that notices an empty collection |
|
|
| 5 | `sonar.javascript.lcov.reportPaths` lists all three reports | A dropped path silently removes that suite's contribution |
|
|
| 6 | The integration suite keeps `--forceExit` | It hangs after completing; on 2026-08-18 that cost 3h12m of runner time |
|
|
| 7 | Nothing in the production path sets `COVERAGE` | An instrumented bundle ships to customers: larger, slower, and publishing the source structure through `window.__coverage__` |
|
|
|
|
## The failure mode this is written against
|
|
|
|
This project has now been bitten three times by a tool succeeding while measuring nothing:
|
|
|
|
- **#67** — SonarQube skipped all 34 frontend files because their tsconfig used `moduleResolution: "bundler"`, and still exited `EXECUTION SUCCESS`. The quality gate reported on a third of the codebase for months while looking complete.
|
|
- **#60** — an ESLint matcher during development matched no files at all. The run was green because there was nothing to complain about.
|
|
- **Coverage** has the same shape by construction. If Playwright reuses an already-running, uninstrumented dev server — which `reuseExistingServer` makes likely on a developer machine — every test passes, `window.__coverage__` is undefined, and the report is empty but valid.
|
|
|
|
The lesson each time was the same: a green tool is weak evidence. The specific defence here is `frontend/scripts/coverage-report.js`, which refuses to write a report when `.nyc_output` holds no samples and explains the two likely causes. It exists instead of calling `nyc report` directly, and that is the whole reason it exists.
|
|
|
|
## Checking it still holds
|
|
|
|
After any change to the workflow, the Vite config, or the test tooling:
|
|
|
|
1. `npm run build` in `frontend`, then grep the bundle for `__coverage__`. Zero occurrences is required. This is the one that ships to customers if it regresses.
|
|
2. Run the coverage suites and confirm all three `lcov.info` files exist and are non-empty.
|
|
3. Delete `.nyc_output` and run `npm run coverage:report`. It must exit non-zero. A guard nobody has seen fire is a guard nobody knows works.
|
|
4. After a scan, check the coverage percentage moved in a direction the change explains. A sharp drop is far more likely to be broken collection than lost tests.
|
|
|
|
## Reading the number
|
|
|
|
Backend and frontend coverage do not mean the same thing, and averaging them hides that.
|
|
|
|
Backend coverage comes from tests that assert on responses — a covered line is usually a checked line. Frontend coverage comes from Playwright driving an instrumented browser, and istanbul marks a line covered when it executes. A component rendered during an end-to-end test reports as covered with nothing asserting anything about it, so the frontend number reads considerably better than the testing behind it.
|
|
|
|
The practical consequence: the 80% gate on new code is easier to clear on frontend changes than backend ones. Treat a high frontend number as evidence the code ran, not that it works. The real fix is a frontend unit suite, which does not exist yet.
|
|
|
|
## Related
|
|
|
|
- `docs/superpowers/specs/2026-08-20-coverage-import-design.md` — the design and why each choice was made
|
|
- `docs/ci/sonarqube-ci-identity.md` — the separate question of which account CI authenticates as
|