IDP accuracy is measured by comparing what the system extracts with correct values your own people have labelled, field by field and document by document. The straight-through processing (STP) rate is the share of documents posted with no human touch. Report both together with the error rate among the documents that went straight through.
Most IDP evaluations quote one number, "accuracy", which hides what a CFO or COO cares about: how many wrong values reach the ERP unseen. This guide is the measurement protocol, not how the pipeline works, which our intelligent document processing guide covers. If you are choosing a supplier for AI document processing, these are the numbers to ask for, measured on your documents.
Key takeaways
- Measure field-level accuracy per field, not one blended figure. An amount and a "notes" field do not carry the same risk.
- Measure document-level accuracy too: the share of documents where every critical field is right. It is always lower than field accuracy.
- STP rate alone is meaningless. Pair it with the error rate inside the straight-through group, called false acceptance.
- Confidence thresholds trade STP against errors. Choose them from a sweep on your own labelled data, field by field.
- Validation against master data and business rules is a separate layer. Measure what it catches that the model got wrong, and what it flags needlessly.
- A test set drawn from vendor demos or from your easiest suppliers will overstate every metric.
How do you measure the accuracy of an IDP solution?
You measure it by running the system on a test set of real documents whose correct values were labelled in advance, then scoring each extracted value as correct, wrong or missing. Report the result per field and per document, and break it down by supplier, layout and scan quality.
Three levels of accuracy answer three different questions.
| Metric | How it is calculated | Question it answers | What it hides |
|---|
| Field-level accuracy | Correct values for one field divided by documents where that field exists | How reliable is the invoice number, the total, the date? | Which documents were wrong, and whether errors cluster |
| Document-level accuracy | Documents with every critical field correct divided by all documents | How often is a document fully right? | Which field fails, and how costly the wrong one is |
| Line-item accuracy | Lines matched and fully correct divided by true lines | Can the tables be used for matching and coding? | Missing lines versus misread lines |
Field accuracy is the number vendors quote; document accuracy drives workload. If six critical fields are each 98 percent accurate and errors were independent, only about 89 percent of documents would have all six right (0.98 to the sixth power). Errors are rarely independent, so measure it directly.
Separate three outcomes for each field: correct, wrong (a value was returned and it is not right) and missing (nothing returned). A wrong value is usually worse than a missing one, because a missing value triggers review and a wrong value may not.
A value is correct when it matches the labelled answer under rules fixed in writing before scoring: exact match for amounts, identifiers and bank details after agreed normalisation of separators, spaces and case, and a defined conversion for dates. Fuzzy matching is allowed only for free text where small differences change no outcome.
| Field type | Typical matching rule | Watch for |
|---|
| Amounts | Exact numeric match after removing currency symbols and thousand separators | Decimal comma versus point, negative amounts on credit notes |
| Dates | Match after converting to one format | Day-month versus month-day ambiguity |
| Identifiers (invoice number, PO, tax number) | Exact match after trimming spaces; case rule agreed | Leading zeros, hyphens, letters read as digits |
| Line items | Match each true line to an extracted line, then score its fields | Merged lines, split lines, extra lines |
What is the straight-through processing (STP) rate?
The straight-through processing rate is the share of documents that move from receipt to posting in the target system with no human touching them. The label is used loosely, so state which version you mean: some vendors count a document as straight-through when extraction needed no correction, while the business cares whether it was posted without a person.
| Version | Counts a document as straight-through when | Use it for |
|---|
| Extraction STP | All fields were above their confidence thresholds | Tuning the model |
| Validated STP | Extraction passed and every validation rule passed | Estimating real workload |
| Touchless to posting | Validated and posted to the ERP with no human step | Business reporting |
STP rate = documents processed with no human touch divided by documents received, over the same period and the same document types. Say which documents are excluded, such as statements, duplicates and unreadable files. A high STP rate on an easy subset is not an STP rate for your business.
The number only means something next to its partner:
False acceptance rate = documents that went straight through with at least one wrong critical value, divided by documents that went straight through.
A system with 70 percent STP and 3 percent false acceptance has put wrong data into your ledger on about one in 33 automated documents. A system with 45 percent STP and 0.2 percent false acceptance is slower and safer. Set the target from the cost of a wrong value, not from a brochure. There is no universal "good" STP rate, and a figure quoted without the document mix and the error rate cannot be compared with yours.
How do confidence thresholds change accuracy and STP?
A confidence threshold decides which extracted values are accepted without review. Raising it sends more documents to a person, so STP falls and false acceptance falls. Lowering it does the reverse. Choose the point from a sweep on labelled data, not by default.
Run the test set, record each critical field's confidence, then calculate the outcome at several thresholds. The numbers below are illustrative, for one field (invoice total) on 500 documents.
| Threshold | Documents auto-accepted | Wrong among auto-accepted | False acceptance | Sent to review |
|---|
| 0.70 | 450 | 18 | 4.0% | 50 |
| 0.85 | 380 | 5 | 1.3% | 120 |
| 0.95 | 290 | 1 | 0.3% | 210 |
| 0.99 | 180 | 0 | 0% observed | 320 |
Three points. The choice is a business one: finance decides what error rate on totals is tolerable. A confidence score is not a probability until you compare each score band with its real error rate. And "0 observed" does not mean zero: with zero errors in n independent cases, the 95 percent upper bound on the true error rate is roughly 3 divided by n (the "rule of three"), so 180 clean documents supports only "probably below about 1.7 percent". Demonstrating 0.1 percent needs thousands of cases, or validation rules that catch the remainder.
Set thresholds per field, as our human-in-the-loop design guide argues for AI actions generally: a total, a bank account and a description deserve different ones.
How do you build an IDP test set?
Build the set from real documents sampled across suppliers, layouts, languages and scan quality, over-sample the hard cases, have two people label independently, and keep a held-back portion untouched until the decision. It must resemble your worst month, not the supplier's sample.
- Sample across the dimensions that cause failures: supplier or sender, layout, document type, language and script, scan versus native PDF versus photo, single-page versus multi-page, and months of the year if formats change at year-end.
- Over-sample the hard cases, then report easy and hard groups separately so the blended figure does not mislead.
- Label by two people independently for at least a sample, and resolve differences; disagreement means the field definition is unclear.
- Split the set. A development set is for tuning rules and thresholds; a test set is scored once for the decision. Tuning on the test set inflates the score.
- Include documents the system should reject: statements, remittance advices, blank pages, duplicates, unrelated email attachments.
- Record the source of each document to see which supplier or channel performs worst, and protect the data: test sets hold bank details.
There is no magic size. Each segment you report on needs enough documents that one error does not swing the result, and to claim a low error rate you need the volume the rule of three implies. Our guide on how to evaluate an AI pilot covers test sets and go/no-go scoring for any AI task.
How do you measure what validation catches?
Measure validation as its own component. Count how many wrong extractions in your test set a rule flagged before posting (the catch rate), and how many correct values it flagged needlessly (rule noise). Seed known errors into copies of correct documents, so the rules can be tested without waiting for real mistakes.
The rule checklist itself is in the pipeline guide linked above. For measurement, seed one error per rule family and record whether it was flagged or missed.
| Rule family | Error to seed |
|---|
| Arithmetic | Change a line amount so the total no longer adds up |
| Vendor identity | Replace the bank account with a new one |
| Purchase order and receipt | Swap the PO number or raise the quantity |
| Duplicate check | Re-submit a posted invoice with a changed number |
| Date and period | Move the date into a closed period |
| Tax | Change the tax code or rate |
Validation is only as good as its master data: duplicate vendors, missing tax numbers and stale PO status make rules misfire or stay silent. Count data-quality exceptions separately; they are a finding for your data owners, not a defect of the IDP tool.
What belongs on an IDP evaluation scorecard?
A fair scorecard uses your documents, reports results by segment, shows false acceptance beside STP, includes validation and the ERP connection, and states what was excluded. It is repeatable: you can re-run the same set after any change. The criteria apply to any product or build.
| Criterion | What a weak evaluation does | What a good one does |
|---|
| Documents | Uses vendor-chosen samples | Uses your own, including suppliers unseen in tuning |
| Reporting | One blended accuracy figure | Per field, per segment, per document-level result |
| STP | Quotes a rate with no error rate | Quotes STP with false acceptance and the document mix |
| Thresholds | Fixed by the vendor | Chosen from a sweep on your development set |
| Validation | Not tested | Tested with seeded errors; catch rate and noise reported |
| New layouts | Assumes none | Measures the effort and accuracy for a layout it has not seen |
Worked example (illustrative)
A distributor receives 2,000 supplier invoices a month. It builds a test set of 400 invoices across 60 suppliers, 80 of them poor scans, labelled by two clerks.
The system reads the total correctly on 372 documents (93 percent), the invoice number on 380 (95 percent) and the PO number on 340 (85 percent). With total, number, date, vendor and PO all required, 301 invoices (75 percent) are fully correct: that is document-level accuracy. At the chosen thresholds, 240 invoices pass extraction. Validation then flags 30 of them, 22 correctly and 8 needlessly, leaving 210 for posting without review: validated STP of 52 percent. Auditing those 210 by hand finds 2 with a wrong PO number that no rule caught, so false acceptance is 1 percent.
The team's reading: tighten the PO check against open orders, which should catch both errors, then re-run. The figures are illustrative; the pattern is three measures, one decision and a fix aimed at the failing field.
How do you keep measuring after go-live?
Audit a random sample of straight-through documents every period; it is the only way to see false acceptance in production. Track exceptions by reason and the share of new suppliers, and re-run the test set before any model or rule change.
When this is the wrong approach
A formal measurement programme is not worth building when volumes are very low, when one clerk can check every document in an hour, or when structured data arrives by EDI, CSV or API. If every document is reviewed anyway, STP is zero by design and the useful metric is review time per document. If master data is badly out of date, fix that first.
Glossary: STP, straight-through processing with no human touch; false acceptance, a wrong value that passed without review; field-level accuracy, correct values per field; document-level accuracy, documents with all critical fields right; master data, your records of vendors, customers, items and tax codes; test set, the labelled documents used for scoring.
How Timeline Digital approaches IDP measurement
Timeline Digital, founded in 2013 and based in Islamabad, builds document-processing software for clients remotely. It starts from a free pilot of 2 to 3 key modules, which your team tests on your own documents before the final agreement, so these metrics are measured on your suppliers and not on a demo. The schedule is set from your requirements in the Plan step, and source code transfers to you on full payment. To scope a measured pilot, Discuss Your Project.