To choose a QA outsourcing vendor, score every shortlisted firm against the same weighted criteria, ask each the same written questions, and test them on the same real scenario before you sign. Weight test thinking, defect reporting and handover of assets above price, because a cheap team that reports unreproducible bugs costs more than it saves.
Most buyers compare QA vendors by hourly rate and a slide deck of logos. Neither predicts whether the team will find the defects that matter. This guide gives you a scorecard you can copy into a spreadsheet, a question list for your request for proposals (RFP), the red flags that show up in proposals, and a pilot method that compares vendors fairly. If you are still deciding whether to outsource at all, read our QA outsourcing guide first; this post assumes you have decided and need to pick a partner. The service side of the same decision is described on our QA outsourcing services page.
What should a QA vendor scorecard measure?
A QA vendor scorecard should measure the things that determine test quality (how the vendor thinks, reports and hands over), the things that determine working fit (communication, hours, tooling) and the commercial terms, in that order of weight. Price belongs on the card but rarely deserves the largest weight.
Copy the table below into a spreadsheet. The weights are a starting point, not a standard: change them to match what hurts most in your releases today. Score each criterion from 1 (weak) to 5 (strong) from evidence, not impressions, and multiply by the weight.
| Criterion | Suggested weight | What good looks like | Evidence to ask for |
|---|---|---|---|
| Test thinking | 20% | Identifies risks and edge cases in a sample feature, not just the happy path | A written test approach for one of your real features |
| Defect reporting quality | 15% | Reproducible steps, expected versus actual result, severity, environment, evidence | Three anonymized defect reports from earlier work |
| Automation approach | 10% | Automates stable, repeated checks first; scripts live in your repository; follows the test pyramid | Their automation priority list and framework choices |
| Independence and honesty | 10% | Reports what is true, including their own missed checks; no pressure to pass releases | How they handled a disagreement with a development team |
| Communication and overlap hours | 10% | Agreed overlap windows, response times and a daily or weekly summary | A written communication plan |
| Domain and technology fit | 10% | Has tested systems like yours (ERP, mobile, API, web) and knows the failure modes | Described experience, with references you can call |
| Handover of assets | 10% | Test cases, scripts, data and defect history stored in your accounts and handed over on exit | Contract wording on ownership and exit |
| Security and confidentiality | 5% | NDA before details are shared, role-based access, anonymized data | Written data-handling approach |
| Commercial terms | 10% | Clear rate card, defined scope, notice period, no hidden tooling charges | The full rate card and a quote for your scenario |
Worked example: scoring three vendors
The vendors and scores below are invented to show the arithmetic. They are not real firms and not a recommendation.
| Criterion (weight) | Vendor A | Vendor B | Vendor C |
|---|---|---|---|
| Test thinking (20%) | 4 | 5 | 2 |
| Defect reporting (15%) | 4 | 4 | 3 |
| Automation approach (10%) | 3 | 4 | 5 |
| Independence (10%) | 4 | 4 | 3 |
| Communication (10%) | 5 | 3 | 4 |
| Domain fit (10%) | 3 | 4 | 3 |
| Handover (10%) | 5 | 4 | 2 |
| Security (5%) | 4 | 4 | 3 |
| Commercial terms (10%) | 3 | 3 | 5 |
| Weighted score (out of 5) | 3.90 | 4.00 | 3.20 |
Each weighted score is the sum of weight times score. Vendor B wins on test thinking and automation even though Vendor A communicates better and Vendor C is the cheapest. The point of the card is not the number itself; it is that the team has to agree, criterion by criterion, why one vendor scored higher than another.
Which questions should go in a QA outsourcing RFP?
Put the same written questions to every vendor and ask for answers in writing, so the proposals can be compared line by line. Group the questions by what they reveal.
Approach and thinking
- How would you test this sample feature of ours? List the ten checks you would run first and why.
- How do you decide what to test more deeply and what to test lightly?
- How do you handle requirements that are incomplete or contradictory?
- What do you do in the first two weeks, before you have run a single test?
- How do you find problems that are not in the written requirements?
Reporting and communication
- Show a redacted example of a defect report and a weekly summary.
- How do you agree severity, and who has the final say when we disagree?
- How quickly do you acknowledge a blocking defect, and during which hours?
- What do you need from our team each week to keep working at speed?
- How do you report on tests that could not be run, and why?
Automation and tooling
- Which checks would you automate first, and which would you leave manual?
- Which frameworks and tools do you propose, and who pays for licences?
- Where do scripts live, and can we run them in our own pipeline?
- How do you keep automated tests from becoming flaky and ignored?
- How do you measure whether automation is paying back?
People and continuity
- Who exactly will test for us, and may we interview them?
- What happens if a tester leaves during the engagement?
- How do you transfer knowledge between testers?
- What is the ratio of senior to junior testers on the proposed team?
- How do you handle peaks, for example before a major launch?
Security, ownership and exit
- Do you sign a non-disclosure agreement before we share details?
- Who owns test cases, scripts and data, and how are they handed over on exit?
- How is access to our environments granted, limited and removed?
- How do you handle test data that comes from production?
- What is the notice period, and is there a minimum term?
Measures and accountability
- Which measures would you report: defects found, defects escaped, regression cycle time?
- How do you avoid being measured only by the number of test cases run?
- What would make you advise us not to release?
- Can we speak to two clients with similar systems?
- What would a time-boxed pilot look like, and what would count as success?