A failing e2e run is reported as a failure in the summary step, and the explicit gate never runs #142

New Issue

2026-08-23T09:07:32-05:00

bermudalamb commented

2026-08-23 09:07:32 -05:00

When the end-to-end suite fails, the SonarQube workflow reports it in a way that points at the wrong thing. The job log ends like this:

Run node scripts/summarize-playwright.js frontend/playwright-results.json
  ❌  Failure - Main Summarize end-to-end tests
exitcode '1': failure

That reads as a broken summary script. It is not — it is the summariser correctly reporting that tests failed, with no output to say so.

Two separate things make it confusing.

The summariser doubles as the gate. scripts/summarize-playwright.js ends with process.exit(stats.unexpected > 0 ? 1 : 0). The workflow already has a step for exactly this purpose:

- name: Fail if either suite failed
  if: steps.unit.outcome == 'failure' || steps.e2e.outcome == 'failure'
  run: exit 1

Its comment says it exists so continue-on-error on the suites cannot turn a failing suite into a passing job — "the one way this change could do real damage". But in run 397 that step was skipped while its condition was true, because a step whose if: omits always() still implicitly requires the previous steps to have succeeded, and the summariser had already failed the job. So the gate that was written to be the place the job fails is dead code, and the job fails one step earlier under a name that describes summarising rather than testing.

The explanation goes somewhere the log is not. report() writes to GITEA_STEP_SUMMARY when it is set, and only falls back to console.log when it is not. Under Actions it is always set, so the pass/fail table and the list of failing tests land in the job's Summary tab and the log shows a bare exit 1 with no output at all. The --reporter=list,json output from the e2e step is in the log further up, but nothing at the point of failure says to look there.

The fix is to let each script do one job: summarize-playwright.js (and summarize-jest.js, which has the same shape) summarise and exit 0, and the existing gate step fail the job — with if: always() && (steps.unit.outcome == 'failure' || steps.e2e.outcome == 'failure') so it actually runs. Then the failure is reported by a step named for what failed.

Worth also echoing a one-line count to stdout as well as to the step summary, so the log says how many tests failed at the point where it stops.

Acceptance criteria

A failing suite fails the job at Fail if either suite failed, not at a summarise step
Both summarise scripts exit 0 regardless of the results they are summarising
The gate step runs even when an earlier step has failed
The log shows the pass/fail counts at the point of failure, not only the Summary tab
A passing run is unaffected

Note on the run that prompted this

Run 397's underlying e2e failures were not a regression. Locally, full-parallel runs fail a set that changes between identical runs — favorites, favorites-filter, sold-filter, and the two bcrypt-heavy account-details tests — and every one of them passes in isolation. That is the shared-database interference already filed as #116, not something this issue needs to fix.

When the end-to-end suite fails, the SonarQube workflow reports it in a way that points at the wrong thing. The job log ends like this: ``` Run node scripts/summarize-playwright.js frontend/playwright-results.json ❌ Failure - Main Summarize end-to-end tests exitcode '1': failure ``` That reads as a broken summary script. It is not — it is the summariser correctly reporting that tests failed, with no output to say so. Two separate things make it confusing. **The summariser doubles as the gate.** `scripts/summarize-playwright.js` ends with `process.exit(stats.unexpected > 0 ? 1 : 0)`. The workflow already has a step for exactly this purpose: ```yaml - name: Fail if either suite failed if: steps.unit.outcome == 'failure' || steps.e2e.outcome == 'failure' run: exit 1 ``` Its comment says it exists so `continue-on-error` on the suites cannot turn a failing suite into a passing job — "the one way this change could do real damage". But in run 397 that step was **skipped** while its condition was true, because a step whose `if:` omits `always()` still implicitly requires the previous steps to have succeeded, and the summariser had already failed the job. So the gate that was written to be the place the job fails is dead code, and the job fails one step earlier under a name that describes summarising rather than testing. **The explanation goes somewhere the log is not.** `report()` writes to `GITEA_STEP_SUMMARY` when it is set, and only falls back to `console.log` when it is not. Under Actions it is always set, so the pass/fail table and the list of failing tests land in the job's Summary tab and the log shows a bare `exit 1` with no output at all. The `--reporter=list,json` output from the e2e step is in the log further up, but nothing at the point of failure says to look there. The fix is to let each script do one job: `summarize-playwright.js` (and `summarize-jest.js`, which has the same shape) summarise and exit 0, and the existing gate step fail the job — with `if: always() && (steps.unit.outcome == 'failure' || steps.e2e.outcome == 'failure')` so it actually runs. Then the failure is reported by a step named for what failed. Worth also echoing a one-line count to stdout as well as to the step summary, so the log says how many tests failed at the point where it stops. ### Acceptance criteria - A failing suite fails the job at `Fail if either suite failed`, not at a summarise step - Both summarise scripts exit 0 regardless of the results they are summarising - The gate step runs even when an earlier step has failed - The log shows the pass/fail counts at the point of failure, not only the Summary tab - A passing run is unaffected ### Note on the run that prompted this Run 397's underlying e2e failures were not a regression. Locally, full-parallel runs fail a set that changes between identical runs — `favorites`, `favorites-filter`, `sold-filter`, and the two bcrypt-heavy `account-details` tests — and every one of them passes in isolation. That is the shared-database interference already filed as #116, not something this issue needs to fix.

bermudalamb added this to the Code Quality and Hardening 2 project 2026-08-23 09:12:23 -05:00

bermudalamb self-assigned this 2026-08-23 09:12:32 -05:00

bermudalamb referenced this issue

2026-08-24 09:33:36 -05:00

The integration suite's Postgres service loses its schema mid-run in CI, failing 12 of 17 suites #154

bermudalamb referenced this issue from a commit

2026-08-24 09:48:36 -05:00

ci: let the summarisers summarise and the gate do the failing (#142)

bermudalamb referenced a pull request that will close this issue

2026-08-24 09:51:07 -05:00

ci: let the summarisers summarise and the gate do the failing (#142) #155

bermudalamb added reference feature/142-ci-failure-reporting