Automate the checks that repeat on every release and rarely change: smoke tests, regression of the core journeys, API checks and calculation checks. Keep exploratory testing, usability judgement and screens that are still changing manual. Automation pays back through repetition, so release frequency and script maintenance decide whether it is worth the effort.
Manual versus automated testing is usually argued as a choice. It is better treated as a budget: a fixed amount of testing effort, spent where it earns the most. This guide gives a priority order for what to automate first, a worked return-on-investment (ROI) example you can adapt with your own numbers, a sensitivity table showing how release frequency and maintenance change the answer, and the pitfalls that turn automation into a liability. All figures in the worked example are illustrative assumptions we chose to show the method, not benchmarks, not market data and not a Timeline Digital price. For how outsourced teams handle the split, see our QA outsourcing services page.
What is the difference between manual and automated testing?
Manual testing means a person runs the checks, observes the result and judges it. Automated testing means a script runs the checks and compares the result with an expected value. Both are needed, because they find different problems.
| Manual testing | Automated testing |
|---|
| Who runs the check | A tester, step by step | A script, in a pipeline or on a schedule |
| Strength | Notices what nobody thought to check; judges clarity and usability | Fast, identical, runs on every change at any hour |
| Weakness | Slow and error-prone when repeated; cost grows with every release | Needs building and maintaining; cannot judge look and feel |
| Best at | Exploratory testing, new features, usability, acceptance testing | Regression, smoke checks, API and calculation checks |
| Cost shape | Pay again on every cycle | Pay up front, then smaller amounts per run |
The ISTQB glossary describes exploratory testing as designing, running and evaluating tests at the same time while learning about the system. That is a human activity by definition, which is why a sensible plan never aims at "fully automated".
What should you automate first?
Automate in this order, and stop when the next item no longer pays back. The order follows two questions: how often does the check repeat, and how stable is the thing being checked?
- Smoke tests. A handful of checks after every deployment that confirm the build is usable at all: it starts, a user can log in, the main page loads, a record can be saved. They are quick to write and run on every build.
- Regression of the core journeys. The few paths the business cannot afford to break: ordering, payment, posting, approval, reporting. These repeat every release and are the most expensive to re-check by hand.
- API and integration checks. Calls between services and to third parties are faster to test than screens and less brittle. They also catch broken contracts before the user interface shows a symptom.
- Calculation and data-rule checks. Tax, pricing, payroll, discount and stock-valuation rules are repetitive and exact. Given a table of inputs and expected outputs, a script can check hundreds of cases in seconds.
- Cross-browser or cross-device repeats of stable journeys, where the same journey must be checked on several platforms.
- Screens that are still changing: last, or not at all. Scripts written against a design that changes weekly are rewritten weekly.
Ham Vocke's "The Practical Test Pyramid", published on martinfowler.com, gives the proportions behind this order: many small, fast tests at the bottom (units and services), a thinner layer of broader tests, and very few slow end-to-end tests at the top, because those are the ones that break and cost most to maintain. If a proposed automation plan is mostly browser-driven scripts, ask why.
What should stay manual?
Keep these manual, at least until they settle:
- Exploratory testing of new features, where a skilled tester follows hunches and learns the system.
- Usability and clarity checks: whether a message makes sense, a workflow feels right, a screen is readable.
- One-off scenarios that will not repeat.
- Visual and layout judgement that a script cannot make.
- User acceptance testing, where your own staff confirm the system fits their daily work. Our guide to the ERP UAT testing plan shows how that is structured for enterprise programs.
What is the ROI of test automation? A worked example
The example below compares running a regression set by hand with automating part of it. Every number is an illustrative assumption, chosen to make the arithmetic easy to follow. Replace them with your own measurements.
Assumptions
- The regression set has 150 checks. A tester needs 5 minutes per check on average, so a full manual pass takes 12.5 hours.
- 100 of the 150 checks are stable and suitable for automation. The other 50 stay manual.
- Building one automated check takes 1.5 hours on average (writing, data setup, review), so the build effort is 150 hours.
- After the scripts exist, each release needs 2 hours of maintenance (fixing scripts after intentional changes) and 0.5 hours to review results and triage failures.
- The comparison counts hours only. Multiply by your own loaded hourly cost to convert to money.
Calculation
- Manual time saved per release: 100 checks multiplied by 5 minutes is 500 minutes, or about 8.3 hours.
- Automation running cost per release: 2 hours of maintenance plus 0.5 hours of review is 2.5 hours.
- Net hours saved per release: 8.3 minus 2.5, about 5.8 hours.
- Break-even point: 150 build hours divided by 5.8 net hours per release is about 26 releases.
How release frequency changes the answer
| Release cadence | Releases per year | Net hours saved per year | Year-one result after the 150-hour build | Break-even |
|---|
| Weekly | 52 | about 303 | about +153 hours | after about 26 weeks |
| Every two weeks | 26 | about 152 | about +2 hours | after about one year |
| Monthly | 12 | 70 | about -80 hours | after about two years |
The pattern is the point: the same suite is a clear win for a product that ships weekly, roughly neutral in year one for fortnightly releases, and slow to repay for one that ships monthly.
How maintenance changes the answer
Keep the weekly cadence and change one assumption. If script maintenance is 4 hours per release instead of 2, net saving falls from 5.8 to about 3.8 hours per release and break-even moves from about 26 releases to about 39. If maintenance reaches 6 hours per release, net saving drops to about 1.8 hours and break-even stretches past 80 releases. Maintenance is the assumption that most often ruins an automation business case, because teams estimate the build and forget the upkeep.
What the example leaves out
The model counts testing hours only. Real returns also include faster feedback to developers, regression checks that actually happen under deadline pressure, and defects caught earlier. They also include costs the model ignores: the pipeline, test environments, flaky-test investigation and tool licences. Treat the example as a method for asking the right questions about your own numbers, not as a forecast.
When does test automation not pay back?
Automation is the wrong investment in these situations:
- The product changes shape every week. Scripts are rewritten as fast as they are written.
- Releases are rare. There are too few repeats to recover the build effort.
- The system is short-lived. A one-off campaign site does not need a regression suite.
- Nobody owns the suite. Unmaintained scripts decay into noise that people learn to ignore.
- The environment is unstable. Failures caused by the environment, not the software, destroy trust in the results.
In these cases, invest in good manual testing and in the practices that cost less, such as code review and unit tests written by developers.
What are the common pitfalls of test automation?
- Automating through the user interface only. Screen-driven scripts are slow and fragile. Push checks down to API and unit level where possible.
- Ignoring flaky tests. A test that fails randomly teaches the team to ignore failures. Review and fix or delete flaky tests regularly.
- Measuring by the number of scripts. Count the risk covered and the defects caught, not scripts written.
- Scripts that only the vendor can run. Keep scripts in your repository, in a common framework, running in your pipeline.
- No ownership. Name the person responsible for the suite and give them time to maintain it.
How should an outsourced team split manual and automated work?
A sensible outsourced plan starts manual, because that finds defects on day one while automation is being built, then moves the stable core to automation over the first months and keeps humans on exploratory and acceptance work. Ask the vendor for its automation priority list in the proposal, and for the framework and repository arrangements. Our guide to choosing a QA outsourcing vendor lists the questions to ask, and the guide to QA outsourcing cost drivers explains how the automation share changes the price shape.
How Timeline Digital approaches the manual and automated split
Timeline Digital is a custom software development company founded in 2013 and headquartered in Islamabad, Pakistan. On projects we build, testing runs inside each sprint, starting with the pilot modules, and automation covers the stable core while manual testing covers new features and judgement. If automation would not repay the effort on your system, we say so. Automation scripts and test cases transfer to you on full payment. To discuss your regression suite, Discuss Your Project.