How to Evaluate a Managed QA Vendor: The Questions That Actually Matter

Choosing a managed QA vendor is a multi-year operational decision. The wrong choice means rebuilding your test suite, re-training your team, or losing the test code entirely when you switch. Most buying processes focus on demo quality and pricing rather than the things that determine long-term value. This guide covers the evaluation criteria that actually matter.

Do You Own the Test Code?

This is the most important question and the one most often glossed over in sales conversations. Test ownership has three requirements — all three must be met:

• Standard, open-source format. Your tests must be written in a framework you can run without the vendor's platform — standard Playwright TypeScript, not a proprietary DSL. If running your tests requires their software, you do not own them.
• Self-serve export. You must be able to download your test code at any time without a support ticket. If export requires their help, the vendor controls the exit.
• Zero-modification portability. If you export and run the tests tomorrow, they should run without modification. If exported tests require significant rework to function outside the vendor's environment, portability is nominal, not real.

Ask vendors directly: "Can I export my test code and run it independently of your platform tomorrow? What format is it in?" Ask to see an example export. For more on this, see what you actually own in modern QA.

Pricing Transparency

Most managed QA vendors do not publish pricing. This is a structural disadvantage to the buyer: you cannot evaluate cost without a sales conversation, and the price you are quoted may depend on how much you appear to be willing to pay.

Published pricing is a signal of alignment. When a vendor publishes its rates, it cannot charge different customers different prices for the same scope. It also signals that the vendor is confident in its value proposition rather than relying on sales skill to close.

The second pricing question is incentive alignment: what behavior does the pricing model reward? Per-test pricing rewards writing more tests, not better ones. Flow-based or journey-based pricing rewards covering more real user scenarios efficiently. See managed QA pricing models for a full breakdown.

Time to First Coverage

Onboarding speed matters. Every week your critical paths are not covered is a week a regression can ship to production. Ask vendors for their specific timeline:

• How long until the first tests are running in my CI/CD pipeline?
• What does the onboarding process look like week by week? (Day-one access to systems, journey discovery, test writing, engineer review, CI integration)
• What are the prerequisites? What do I need to provide to hit the timeline?

Understand whether a vendor is quoting time to first tests or time tocomprehensive coverage. These are not the same. A 30-day timeline is strong for first critical flows; a 30-day timeline for comprehensive coverage is likely over-promised.

Flake and Maintenance Policy

Test maintenance is the hidden cost of managed QA. Tests break when UI changes, and every breakage requires someone to investigate and repair. The questions to ask:

• What is the SLA for fixing a broken test after a UI change?
• Does a human investigate every failure, or is it automated-only?
• How does the vendor distinguish between a real bug and a broken test?
• Is maintenance included in the base price or billed separately?

A vendor that auto-retries failures without human investigation is hiding flakiness, not fixing it. A red result should always mean a human looked at it.

Contract Terms and Exit Conditions

Before signing, understand:

• Minimum commitment. Annual contracts are standard; pilot periods vary. A paid pilot period before annual commitment is reasonable to ask for.
• Exit conditions. What happens to your test code if you cancel? Does the vendor transfer the code? Is there a wind-down period?
• Coverage guarantees. Does the vendor commit to specific coverage metrics, response times, or uptime? What are the remedies if they are not met?

For a comparison of how the leading vendors perform across these dimensions, see the managed QA services scorecard.