A fair software pilot is a limited build of two or three real modules, run on your data against written acceptance criteria, with the code ownership position and exit terms agreed in writing before work starts. It tests whether this vendor can deliver your work, which a polished demo cannot.
Plenty of vendors say "we start with a pilot". In one vendor's mouth it means working software your team uses; in another's, a slide deck and a click-through prototype. This guide is for CTOs, founders and heads of engineering about to run a pilot who want to judge the result fairly. It is about custom software vendors, so it differs from our guide to evaluating an AI pilot, which deals with test sets and model metrics. If you are still building the shortlist, our guide to the best custom software development companies explains how to compare them, and this guide helps you design the test.
Key takeaways
- A pilot is evidence-gathering, not a sales step. Write down what decision it will support before the vendor writes a line of code.
- Scope it to two or three modules that are representative and risky, not the easiest ones.
- Use real data (minimised or masked where needed) and written acceptance criteria that a colleague could check without the vendor in the room.
- Agree code ownership, repository access and exit terms in writing before the pilot starts, even when it is free. Ownership terms are something to agree, not a fixed standard.
- No honest general duration exists for a pilot. Ask each vendor for a dated plan and judge the plan.
- Score every vendor on the same sheet, and set minimum thresholds on ownership and code quality that no total can override.
What is a free software pilot, and how does it differ from a demo or a proof of concept?
A pilot is a working, limited version of the real system built to be used by your people on your data. A demo shows scripted screens; a proof of concept tests whether one risky idea is technically possible; a minimum viable product is a first sellable release.
Each answers a different question. A prototype answers "do we like how this looks?" A PoC answers "can this integration be done at all?" A pilot answers "can this team build what we actually need, and will our staff use it?" Vendors sometimes call a prototype a pilot because it sounds more substantial. Ask which of the four it is.
What should a fair vendor pilot include?
A fair pilot has seven parts: a scope of two or three modules, real data, written acceptance criteria, a named test group on your side, an agreed position on ownership of the pilot code, a review meeting, and written exit terms. If any part is missing, the pilot proves less than it appears to.
| Part | What fair looks like | Question to ask the vendor |
|---|
| Scope | Two or three modules, one of them hard | "Which modules would you pick, and why?" |
| Data | Real records, masked or minimised if sensitive; NDA in place | "What data do you need and how do you protect it?" |
| Acceptance criteria | Pass or fail statements per module, agreed before work starts | "Can I see the criteria in writing?" |
| Test group | Named users on your side with time reserved | "What do you need from my team, and how many hours?" |
| Code and ownership | Repository access during the pilot; ownership position stated | "Who owns the pilot code, and can my engineer read it?" |
| Review | A scheduled walkthrough with a go or no-go decision | "When do we meet, and who decides?" |
| Exit | No obligation to continue; what happens to data and code if you stop | "What do I owe if I stop here?" |
Write these into a one-page pilot brief that both sides sign; it is also your record if a dispute arises. Our software requirements brief template can seed the scope section.
Which two or three modules should you pilot?
Pick modules that represent the real project and carry real risk: one core workflow your staff use daily, one that touches your existing data or another system, and, if possible, one with awkward business rules. An easy module proves little; a representative one predicts the full build.
Ask what you are most worried about in the full project. If it is integration with an old system, include a real connection to it. If it is permissions and approvals, include a workflow with three roles. If it is reporting accuracy, include one report your finance team will reconcile by hand. Vendors who steer you towards the easiest screens, such as a login page and a dashboard of dummy charts, are optimising for a good demo, not for your learning.
How long should a pilot take?
No honest general duration exists. A pilot's length depends on the modules chosen, how ready your data is, whether an integration is involved, and how quickly your testers can respond. Ask each vendor for a dated plan, and judge the quality of that plan rather than the shortest promise.
Buyers often ask for a typical pilot duration. Treat any single number as incomplete unless it comes with a scope. Two vendors can both say "a week" and mean different things: one builds a login and a list view, the other a working workflow with validation and a report. Ask what will be demonstrably working at each date, what your team must supply and by when, and what happens if your data arrives late. A pilot that slips because your testers were unavailable is not evidence against the vendor, so record dependencies on both sides.
What does a "free" pilot actually cost you?
A free pilot costs your time and carries commercial conditions. Time goes to preparing data, attending reviews and testing. Conditions can include exclusivity, a deadline to sign, or a minimum commitment. Ask for all of them in writing before accepting the offer.
"Free" is only free if nothing is attached. Ask five things: Is there any fee if I stop after the pilot? Am I barred from showing the output to other vendors? Is there a date by which I must decide? Does accepting the pilot commit me to a rate card or a minimum team size? Who pays if the pilot needs a third-party licence? Sensible vendors answer plainly. Paid pilots are also legitimate when the work is substantial, if deliverables and ownership are as clear.
How do you judge a pilot? A scoring sheet
Judge a pilot on seven weighted criteria using the same sheet for every vendor: scope fit, handling of real data, acceptance criteria met, code quality, ownership and access, communication and honesty, and terms after the pilot. Set minimum scores for ownership and code quality that no total can override.
| Criterion | Weight | What to check |
|---|
| Scope fit | 15 | Did it cover the modules agreed, including the hard one? |
| Real data handled | 15 | Did it cope with your messy records, edge cases and volumes? |
| Acceptance criteria met | 20 | Count criteria passed; independent testers, not the vendor |
| Code quality | 15 | An engineer you trust reads the repository: structure, tests, readability |
| Ownership and access | 10 | Repository access, dependency list, ownership and exit terms agreed in writing |
| Communication and honesty | 15 | Did the vendor flag gaps early, or hide them until the review? |
| Terms after the pilot | 10 | Clear scope, milestones and exit terms for the full project |
Illustrative worked example (invented numbers, to show the method). Three vendors are scored out of 100.
| Vendor | Scope | Data | Criteria | Code | Ownership | Communication | Terms | Total |
|---|
| A | 12 | 10 | 16 | 11 | 10 | 12 | 8 | 79 |
| B | 14 | 14 | 18 | 6 | 4 | 9 | 5 | 70 |
| C | 8 | 6 | 10 | 9 | 10 | 13 | 9 | 65 |
Vendor B has the best functional scores, but a code score of 6 out of 15 and an ownership score of 4 out of 10 suggest a polished front end on throwaway code with vague terms. The ownership score rates how clear, accessible and fair the agreed terms are, not whether you own the code from day one: a transfer on full payment scores well if it is written down and you can read the code meanwhile. With a rule such as "ownership must score at least 6 and code at least 8", B is eliminated despite 70 points, and A is the clear choice. The thresholds stop attractive screens outweighing structural risk.
What are the red flags in a vendor pilot?
The main red flags are a demo dressed as a pilot, hidden commitments, throwaway code, no access to the repository, vague acceptance criteria, and pressure to sign quickly. Any one can be fixed by a direct question; a vendor who resists the question is giving you the answer.
| Red flag | What it looks like | What to do |
|---|
| Demo dressed as a pilot | Screens with hard-coded data, no real workflow | Ask to enter your own records and run a full cycle |
| Hidden commitments | Exclusivity, signing deadline or minimum team buried in terms | Request every condition in one page, in writing |
| Throwaway code | Pilot is "just for show" and will be rebuilt | Ask whether the full build extends the pilot code |
| No code visibility | Vendor refuses repository access | Require read access for your engineer |
| Moving criteria | Pass conditions change during the pilot | Freeze criteria in the signed brief |
| Pressure | "The offer ends Friday" | A fair pilot does not expire before you can evaluate it |
| No named people | You cannot say who built it | Ask for the names and roles of the team |
Who owns the code from a pilot?
Ownership of pilot code must be stated before it starts, even when the pilot is free. You may own it from creation, own it on a later event such as full payment, or receive only a licence. No one position is the standard; each is workable if agreed in writing and you know which applies.
Copyright in software generally starts with whoever wrote it, so the vendor holds the pilot code unless a contract says otherwise. That matters if the full project builds on it. Ask whether the pilot code becomes part of the delivered system, and if so, on what terms it is assigned. Our guide to vendor lock-in clauses and the IP handover checklist goes through the wording in detail. This is general information, not legal advice; have counsel review the pilot agreement.
What should happen after the pilot?
After the pilot, hold a structured review: demonstrate each module against its criteria, share your scores, list open issues, and decide go, adjust or stop. If you go, scope, milestones, team and terms for the full project are agreed in writing, informed by what the pilot revealed.
Good vendors use the pilot to improve their plan, because they now know how messy your data really is. Be wary of a vendor whose plan is unchanged after the pilot: either it told them nothing, or they were not listening. After a "go", expect the findings to appear in the scope; after a "stop", expect your data to be returned or deleted as agreed. The wider selection steps are in how to outsource software development.
When is a pilot the wrong approach?
A pilot is the wrong tool when the requirement is unclear, nobody on your side can test it, the project is too small to justify the effort, or the real risk is not in the software. Fix those problems first, or use a paid discovery phase instead.
- Your requirements are unwritten. A pilot will reveal that, expensively. Write the brief first.
- No testers. Without real users on real data, you will end up judging screens.
- The risk is organisational. A pilot will not fix adoption, budget or approvals.
- A single trivial module. Reference checks and a code sample may be enough.
- Regulated data you cannot share yet. Use masked or synthetic data, or wait for contract and security review.
Glossary: pilot, a limited real build for testing with real users; proof of concept (PoC), a narrow technical experiment, often thrown away; prototype, a visual or clickable model with little or no working logic; acceptance criteria, written pass or fail conditions for each module.
How does Timeline Digital run its free pilot?
Timeline Digital is a custom software development company founded in 2013 and headquartered in Islamabad, Pakistan. Delivery is remote. We start by learning your requirements, then choose 2 to 3 key modules together. We build those modules as real, working software, free of charge. Your team uses the pilot, and you decide on the final agreement after testing. Project source code, designs and documentation transfer to you on full payment, and an NDA can be signed before you share sensitive details. Our process page shows each step, and schedules are set from your requirements in the Plan step rather than quoted as a fixed number.