docs(ci): record the coverage pipeline contract and the CI identity gap
SonarQube Analysis / sonarqube (pull_request) Successful in 14m41s
Tests / lint (pull_request) Successful in 2m5s
Tests / backend-unit (pull_request) Successful in 40s
Tests / frontend-e2e (pull_request) Failing after 8m57s

Two standing documents rather than one, because they are different kinds of thing: one is a contract the pipeline must keep, the other is work not yet done.

The coverage contract names the seven requirements that keep SonarQube's number real, and what specifically breaks if each lapses. It exists because coverage does not fail loudly — it reports a smaller number, which looks exactly like tests covering less. That is the third time this project has met a tool that succeeds while measuring nothing, after #67 and #60, so the failure mode is written down alongside how to check the guard still fires.

The identity document covers CI authenticating to SonarQube as admin rather than a restricted account, raised as a "Related" note in #61 and split out so a permissions change is not buried in a CI-config commit. It spells out the revoke step explicitly, since the workflow goes green one step earlier and stopping there leaves the old credential valid.
This commit is contained in:
2026-08-20 10:14:33 -05:00
parent 332c1e7cd0
commit 261d087a9c
2 changed files with 101 additions and 0 deletions
+52
View File
@@ -0,0 +1,52 @@
# CI Contract — Test Coverage
**Status:** Live as of #61
**Applies to:** `.gitea/workflows/sonarqube.yml`, `sonar-project.properties`, both workspaces' test tooling
What CI has to keep doing for SonarQube's coverage number to stay real. This exists because coverage does not fail loudly when it breaks — it reports a smaller number, which is indistinguishable from tests genuinely covering less.
## The contract
Every one of these is load-bearing. Breaking any of them produces a plausible-looking number rather than an error.
| # | Requirement | What breaks if it lapses |
| --- | --- | --- |
| 1 | Both backend suites run with coverage before the scan | The unit suite alone reports ~11%, because everything in `src/routes` is exercised only by the integration suite |
| 2 | The two backend reports go to separate directories | Jest writes `coverage/lcov.info` by default; the second run overwrites the first and half the coverage vanishes |
| 3 | The end-to-end run uses `test:e2e:cov`, not `test:e2e` | An uninstrumented dev server collects nothing while every test still passes |
| 4 | `coverage:report` runs and is allowed to fail the job | It is the only thing that notices an empty collection |
| 5 | `sonar.javascript.lcov.reportPaths` lists all three reports | A dropped path silently removes that suite's contribution |
| 6 | The integration suite keeps `--forceExit` | It hangs after completing; on 2026-08-18 that cost 3h12m of runner time |
| 7 | Nothing in the production path sets `COVERAGE` | An instrumented bundle ships to customers: larger, slower, and publishing the source structure through `window.__coverage__` |
## The failure mode this is written against
This project has now been bitten three times by a tool succeeding while measuring nothing:
- **#67** — SonarQube skipped all 34 frontend files because their tsconfig used `moduleResolution: "bundler"`, and still exited `EXECUTION SUCCESS`. The quality gate reported on a third of the codebase for months while looking complete.
- **#60** — an ESLint matcher during development matched no files at all. The run was green because there was nothing to complain about.
- **Coverage** has the same shape by construction. If Playwright reuses an already-running, uninstrumented dev server — which `reuseExistingServer` makes likely on a developer machine — every test passes, `window.__coverage__` is undefined, and the report is empty but valid.
The lesson each time was the same: a green tool is weak evidence. The specific defence here is `frontend/scripts/coverage-report.js`, which refuses to write a report when `.nyc_output` holds no samples and explains the two likely causes. It exists instead of calling `nyc report` directly, and that is the whole reason it exists.
## Checking it still holds
After any change to the workflow, the Vite config, or the test tooling:
1. `npm run build` in `frontend`, then grep the bundle for `__coverage__`. Zero occurrences is required. This is the one that ships to customers if it regresses.
2. Run the coverage suites and confirm all three `lcov.info` files exist and are non-empty.
3. Delete `.nyc_output` and run `npm run coverage:report`. It must exit non-zero. A guard nobody has seen fire is a guard nobody knows works.
4. After a scan, check the coverage percentage moved in a direction the change explains. A sharp drop is far more likely to be broken collection than lost tests.
## Reading the number
Backend and frontend coverage do not mean the same thing, and averaging them hides that.
Backend coverage comes from tests that assert on responses — a covered line is usually a checked line. Frontend coverage comes from Playwright driving an instrumented browser, and istanbul marks a line covered when it executes. A component rendered during an end-to-end test reports as covered with nothing asserting anything about it, so the frontend number reads considerably better than the testing behind it.
The practical consequence: the 80% gate on new code is easier to clear on frontend changes than backend ones. Treat a high frontend number as evidence the code ran, not that it works. The real fix is a frontend unit suite, which does not exist yet.
## Related
- `docs/superpowers/specs/2026-08-20-coverage-import-design.md` — the design and why each choice was made
- `docs/ci/sonarqube-ci-identity.md` — the separate question of which account CI authenticates as
+49
View File
@@ -0,0 +1,49 @@
# CI Identity — SonarQube
**Status:** Not done. Tracked as its own issue.
**Applies to:** the `SONAR_TOKEN` Gitea Actions secret, and the SonarQube account behind it
## Current state
Gitea Actions authenticates to SonarQube using a token belonging to the **`admin`** account. This was flagged early in the project and never revisited.
The token reaches the scanner only through the `SONAR_TOKEN` environment variable — #67 removed the `-Dsonar.login=` command-line copy, so it is no longer passed on a command line where it could reach a log. That part is already fixed. What remains is *whose* token it is.
## Why it should change
An analysis job needs one permission: submit an analysis report for one project. The `admin` token carries every permission the server has — creating and deleting projects, changing quality gates and profiles, managing users, reading every project including `sql-utilities`.
That gap matters in three ordinary situations, none of which require anyone to be malicious:
- **A leaked token is a leaked server.** CI secrets end up in more places than intended: a debug run with `set -x`, a third-party action, a fork's workflow. The blast radius of an analysis token is one project's analysis history. The blast radius of this one is everything.
- **A misconfigured scan can destroy history.** `sonar.projectKey` is a string in a properties file. A wrong value plus admin rights silently creates projects; other admin endpoints can delete them. A restricted token simply fails.
- **Rotation is currently painful.** Rotating `admin`'s token means finding every other place that account is used. A dedicated account can be rotated on its own.
There is also a plainer reason: when CI shows up as `admin` in the analysis history, the audit trail cannot distinguish an automated scan from a person making a change.
## The change
1. On SonarQube, create a user — `gitea-ci` — with **no** global permissions.
2. Grant it **Execute Analysis** on the `redefined-designs` project only. That is the single permission a scan needs.
3. Generate a **user token** for that account. Not a project or global analysis token: SonarSource's own MCP server, and other tooling, require the user type, and this server is old enough that the distinction matters.
4. Update the `SONAR_TOKEN` secret in the repository's Actions settings.
5. Run the SonarQube workflow and confirm it still passes.
6. **Revoke the `admin` token** that CI was using. Skipping this leaves the old credential valid and the change cosmetic.
Step 6 is the one worth naming explicitly, because the workflow will already be green after step 5 and it is easy to stop there.
## Also worth deciding at the same time
The scratch project `redefined-designs-local`, used by `scripts/scan-local.sh` so local scans do not overwrite CI's analysis of `main`, is currently written by a personal token from the developer's environment. If local scanning becomes a habit rather than an occasional check, it deserves the same treatment: its own restricted account rather than whichever token is to hand.
## Constraints on this server
SonarQube here is **9.9.8 LTA, Community edition**. Two consequences:
- It has no branch analysis, so every scan overwrites the single `main` analysis for whichever project key it is given. This is why the scratch project exists.
- It does not accept `Authorization: Bearer` — a token is supplied as the basic-auth username. Any script checking the new account's permissions must use `curl -u "$TOKEN:"`, not a bearer header, or it will look like the permissions are wrong when the auth scheme is.
## Related
- `docs/ci/coverage-pipeline-contract.md` — what the pipeline must do for coverage to stay honest
- #61 raised this as a "Related" note; it was split out so a permissions change would not be buried inside a CI-config commit