We Scored 17 Real Playwright Test Suites: The State of Playwright Quality in 2026

Every managed QA vendor, including us, claims to write "good" Playwright tests. Almost none of that claim is measurable. So we built a scorer that doesn't take anyone's word for it — an open-source, deterministic, AI-free tool called playwright-score — and pointed it at 17 real, public Playwright suites we don't own, didn't write, and didn't curate for a good outcome.

Updated 2026-08-27: every number in this post was re-scored with playwright-score 0.5.0 and its new scoring model — a stricter, per-test model where your score is simply the weighted share of your tests that are clean. Two suites that passed under the original model now fail. The original 2026-08-19 run used package 0.1.x; the deltas and why we changed the model are covered below.

The result: 1,228 spec files, 5,664 tests, 171,602 lines of test code, and 6,415 individual rule violations, scored on 2026-08-27 against each project's live main/master branch. Every number in this post is reproducible from a single script against the public repos linked throughout — no surveys, no self-reported data, nothing from our own customers.

The headline numbers

The corpus

We didn't pick repos to make the tool — or the industry — look good or bad. The corpus mixes heavily-engineered platforms (Supabase, Grafana, Mattermost, n8n) with smaller, less mature projects found by searching for real @playwright/test usage. Every repo is scored on the exact subdirectory that holds its Playwright suite, via a fresh, shallow, sparse clone — nothing is vendored or cached.

Run on 2026-08-27 with playwright-score 0.5.0 (scoring model v3), standard profile, threshold 80. Twelve of seventeen pass. Reproduce it yourself with scripts/validate-corpus.sh in the GitHub repo.

What's actually wrong with real Playwright suites

Grades hide the interesting part. We also kept every individual finding — 6,415 of them — and ranked which specific anti-patterns show up most often across the corpus. This is the part that isn't on our own tool's landing page.

Two patterns dominate everything else. Locator anti-patterns account for 4,060 of the 6,415 total findings — 63% of every violation in the corpus — and no-raw-locators alone is 51% of all findings, present in 16 of the 17 repos. This is not a niche mistake. It's the default way most teams still write Playwright selectors, three years after Playwright's own docs started recommending role-based locators over raw CSS.

Zoomed out across every locator call in the corpus — not just findings, every single getByRole/getByTestId/getByText vs. page.locator()/frame.locator() call, 17,145 of them — 69.9% are role-based and 30.1% are raw selectors. Two repos in the corpus, livecodes and openplayerjs, use zero role-based locators: every single locator in both suites is a raw CSS or XPath selector. Both fail the default threshold.

The second-biggest category is one we're publishing with some humility: conditional logic inside tests — 877 occurrences (no-conditional-in-test, 677 in 12 repos, plus no-conditional-expect, 200 in 9). An if (await locator.isVisible()) branch means the test does different things on different runs — and an assertion inside that branch can silently never execute. Until late August, a config bug in our own scorer left these rules silently disabled: a suite built entirely out of conditional branches scored a perfect 100. We found it, fixed it, published the fix the same day, and the second-most-common anti-pattern in real-world Playwright appeared out of nowhere. The full account is in the changelog.

When we made the scorer stricter, two more suites failed

The original version of this study used a finding-density model: violations per line of test code, smoothed so one repeated mistake couldn't sink a big suite. It had a flaw we couldn't defend once we saw it clearly — a tiny suite where two of three tests had hard waits and no assertions still scored 78/C, because a handful of findings barely registers per-line. Density is also not how anyone reviews a test suite. Reviewers say "two of your three tests are broken," not "0.5 penalty units per 25 lines."

Scoring model v3 says exactly what a reviewer says: every finding is attributed to the test it sits in, a test with an error is a flawed test, and your score is the weighted share of your tests that are clean. A problem in a beforeEach counts against every test that runs through it. Under that model, this corpus has 4,580 clean tests out of 5,664 — 80.9%. Roughly one in five real-world Playwright tests carries at least one defect the scorer can prove.

The strictness has consequences: cal.com (73) and Immich (78) now fail the default threshold, joining the three suites that always did, and TheCyberHub drops to an F. cal.com is instructive: 1,258 findings across 253 tests, only 60% of its tests clean, and a locator ratio of 41.2% role-based (613 native vs. 875 raw) against a corpus average of 69.9%. The old density model forgave that concentration; the per-test model doesn't. Meanwhile the suites that were engineered carefully barely moved — n8n held its 95 exactly, and Playwright's own TodoMVC example its 98 — which is what you want from a stricter model: pressure on real problems, not noise on good suites.

Tests that assert nothing

playwright/expect-expect — a test block that runs, exits without an error, and never calls expect() — appeared 102 times across 12 of the 17 repos (71%). These aren't failing tests. They're passing tests that prove nothing, the exact failure mode we've written about before: a green suite that doesn't mean the product works. A scorer that only checked pass/fail would never catch a single one of these 102 cases — they all pass. Under model v3 each of them is, by definition, a fully flawed test in the assertions dimension: a test that asserts nothing can't be clean.

Methodology

Every score in this post comes from @qaguardian/playwright-score 0.5.0, scoring model v3, standard profile, threshold 80 — the same package anyone can npm install -D today. The scorer is fully deterministic and AI-free: it wraps eslint-plugin-playwright for community best practices, attributes every finding to the test it sits in via AST test spans, and adds a locator-ratio metric and assertion-delegation tracing through local imports. No LLM is in the scoring path.

Each repo was scored via a fresh, shallow, sparse git clone of the exact subdirectory holding its Playwright suite — nothing from any of these repos is vendored into ours. The rule-frequency breakdown in this post comes from the raw JSON findings output of that same run, aggregated by rule across all 17 reports. The full scoring model, weights, and a worked example are documented in METHODOLOGY.md. Scores will drift slightly over time as these projects' own suites change — that's expected, and the whole point of scoring live code instead of a frozen snapshot.

What to do with this if you maintain a Playwright suite

Run npx -p @qaguardian/playwright-score playwright-score ./tests --format text against your own suite before you assume it's fine. The output now tells you directly how many of your tests are clean. Given that 63% of every finding in this corpus was a locator anti-pattern, that's the highest-value place to look first — a raw page.locator('.btn-primary') breaks the moment a class name changes; getByRole('button', { name: 'Submit' }) survives a restyle. If your suite is CI-gated, the tool exits non-zero below your threshold, so this is a five-minute add to an existing pipeline, not a rewrite.

If you'd rather have engineers own that quality bar for you — the same bar we hold AI-generated Playwright to internally — book a demo and we'll score your actual suite live, on the call.