test(e2e): run the suite on one worker so a red run means something (#241)
The lever the plan deliberately held back, applied now that there is evidence it is needed. Better failure messages and a wider assertion timeout were not enough. Measured on this branch, same commit, same machine. Two parallel runs each failed 3 of 155, and not the same three: password-reset and admin-inventory-filters in one, admin-email-settings, admin-inventory-filters and resend-verification in the next. All 16 tests from those three specs then passed when run serially, and the full suite passed 155 of 155 twice in a row. Failures that move between runs of identical code are contention, not defects — specs asserting over tables other specs are concurrently writing to. The cost is about three minutes, roughly one minute parallel against 3.9 and 4.0 serially. A red run that means something is worth three minutes; the previous state was a suite whose result nobody could act on, which is what #245 had to skip a real test to work around. If that time is ever needed back, the cheaper fix is giving each worker its own database rather than raising this number and reopening #241. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,7 +2,28 @@ import { defineConfig, devices } from '@playwright/test';
|
||||
|
||||
export default defineConfig({
|
||||
testDir: './tests/e2e',
|
||||
// Kept true so a file's tests are independent units, but with one worker it
|
||||
// no longer interleaves them. See `workers` below.
|
||||
fullyParallel: true,
|
||||
// One worker, because the suite shares a single database.
|
||||
//
|
||||
// This was the lever #241 deliberately held back, to see first whether better
|
||||
// failure messages and a wider assertion timeout were enough. They were not.
|
||||
// Measured on this branch, same commit, same machine: parallel runs failed 3
|
||||
// of 155 twice over, and the three were not the same three — password-reset
|
||||
// and admin-inventory-filters in one run, admin-email-settings,
|
||||
// admin-inventory-filters and resend-verification in the next. Run serially,
|
||||
// all 16 of those same tests passed; the full suite then passed 155 of 155
|
||||
// twice in a row.
|
||||
//
|
||||
// So the failures were contention, not defects: specs asserting over tables
|
||||
// that other specs were concurrently writing to. The cost is about 3 minutes
|
||||
// — roughly 1 minute parallel against 3.9 and 4.0 serially — and a red run
|
||||
// that means something is worth three minutes.
|
||||
//
|
||||
// The cheaper fix, if that time is ever needed back, is giving each worker
|
||||
// its own database rather than raising this number and reopening #241.
|
||||
workers: 1,
|
||||
retries: process.env.CI ? 1 : 0,
|
||||
reporter: [['list']],
|
||||
use: {
|
||||
|
||||
Reference in New Issue
Block a user