AI-Generated Tests: What They're Good At and Where They Still Fail

Every QA vendor now claims "AI-generated tests" somewhere on their homepage. The phrase covers an enormous range of what is actually happening behind the scenes — from a language model drafting selector boilerplate a human then rewrites, to a fully autonomous pipeline that writes, runs, and merges tests with no engineer in the loop. Those are very different products, and the gap between the pitch and what ships in your CI matters more than the label.

What AI Is Actually Good At

The genuinely useful part of AI-assisted testing is speed on the first draft, not judgment on the final one.

• Turning a recorded session into a first-pass script. Click through a flow once, and a model can produce a reasonable skeleton — page objects, initial selectors, basic assertions.
• Translating a plain-English flow description into code. "User adds an item, applies a promo code, and completes checkout" becomes a working script draft in seconds instead of an hour.
• Suggesting semantic selectors over brittle ones. A model trained on good Playwright patterns will reach for getByRole before it reaches for a nth-child CSS selector, which is a real improvement over how a lot of hand-written tests still get built.

Where It Still Needs a Human

The parts of test-writing that actually determine whether coverage is worth anything are judgment calls, and judgment is exactly what current models are weakest at.

• Deciding what's worth testing. A model can generate a script for any flow you point it at. It cannot tell you which flows are load-bearing for the business and which are edge cases that don't justify maintenance cost.
• Judging whether an assertion proves the right thing. An AI-drafted test that checks a success toast appeared is not the same as one that confirms the order actually landed in the backend. Both look green. Only one means anything — the same distinction that matters in why your app breaks when 100% of your tests pass.
• Handling multi-step state and auth correctly. Session setup, role-based permissions, and multi-tenant data are exactly where autogenerated drafts tend to take shortcuts that quietly weaken what the test proves.

The Fully-Autonomous Pitch vs. Reality

Some vendors market a pipeline with no human review step at all — AI writes the test, AI runs the test, AI merges the test. In practice, this tends to produce suites that grow fast and mean less over time, because nothing is checking whether new tests assert business outcomes or just re-confirm that a page renders. Volume without judgment is the same failure mode covered in our managed QA coverage comparison — a high test count is not evidence of high signal.

A Reasonable Bar: AI Drafts, Engineers Verify

The model that holds up under scrutiny is narrower than the marketing: AI accelerates the first draft, and a senior engineer verifies that the assertion proves the journey before it merges. That division of labor gets you the speed gain without inheriting the judgment gap. It also keeps the output in your hands — standard, readable Playwright code, not a black box you have to trust blindly. See what you actually own in modern QA for why that ownership matters independent of who — or what — wrote the first draft.

If you want to see what AI-assisted, human-verified flow coverage looks like on your own product, book a demo and we'll show you the actual pipeline, not just the pitch.