The Per-Test Trap: How Test Count Incentives Create QA Waste

Paying per test feels fair. You get a unit of work; you pay for a unit of work. The problem is that "a test" is not a unit of coverage — and vendors who bill that way are rewarded for maximizing the count, not the signal.

That incentive structure creates a specific failure mode: the Per-Test Trap. Your suite grows, your invoice grows, and your confidence in releases stays flat — or drops. This post is the diagnostic guide: how to recognize the trap, what it costs beyond the bill, and how to audit your way out. For the pricing model that fixes the incentive itself, see flow-based pricing.

What the Trap Looks Like in Practice

You ask a vendor to cover checkout. A thoughtful engineer would write one script that walks a real purchase from product page through confirmation.

Under per-test billing, that same journey often arrives as a dozen fragments: page loads, button presence, form validation, success-page render. Each fragment is billable. Almost none prove a customer can complete a purchase.

Twelve green checks. Zero end-to-end proof. That is the trap — volume that looks like coverage while the journey that matters stays untested.

The Invoice Is the Smaller Cost

Inflated billing is annoying. What you inherit is worse.

Every fragment is another file, another failure point, another thing to update when the UI changes. One layout tweak to checkout can break a dozen scripts that never exercised checkout as a journey. You multiplied maintenance without multiplying confidence.

Per-test pricing also pushes vendors toward presence checks over behavior. Asserting that a button exists is cheap to write and cheap to keep green. Asserting that clicking it completes a transaction takes real engineering — and does not pay more when you are billed by file count.

Five Signs You Are in the Trap

• Test count climbs every month, but release confidence does not
• Tests fail often when nothing in the product actually changed
• Names reference DOM nodes or components instead of user goals ("verify address form renders" vs. "complete checkout")
• Reports show hundreds of passes, but nobody can list which critical journeys are covered
• A single UI change requires edits across many test files for one flow

If three or more apply, you are not under-covered — you are over-fragmented. For why journeys beat slices, see flows vs. tests and why apps break when tests pass.

A Simple Coverage Audit

Getting out starts with mapping tests back to journeys — not with renegotiating the invoice first.

1. List your critical user journeys (checkout, signup, login, core create/edit flows). How to map critical user journeys walks through that model.
2. For each existing test, tag the journey it belongs to — or mark it "orphan" if it only asserts a render state.
3. For each journey, ask: if we replaced every fragment with one end-to-end flow, what would we lose?
4. Keep anything that proves a distinct business outcome. Cut or fold presence checks that never touch the path a user takes.

Most teams find the answer to step three is: noise. Fewer scripts, clearer ownership, faster CI, and maintenance that scales with journeys — not with how finely a vendor sliced the bill.

What Aligned Incentives Look Like

Once the audit shows the waste, change what you buy. Charge per journey covered — one flow, one credit — and splitting checkout into twelve files no longer increases revenue. The rational move becomes writing the complete journey once.

That alignment is structural. You do not need to hope the vendor "does the right thing." The model makes the right thing the cheapest thing. Details and economics are in Flow-Based Pricing: How Per-Test Billing Broke QA.

Want to run this audit on your current suite? Book a demo and we'll walk through it with your real flows.