How accurate is AI document extraction, and how do you catch the mistakes?

Accuracy is usually quoted per field, but your team experiences it per document. If each of twenty fields is right 98 times in 100, only about two documents in three come through with every field correct, so a figure that sounds nearly perfect can still mean someone checks everything. Asking the model for a confidence score is the weakest way to find the errors, because language models are not reliably calibrated about their own mistakes and the damaging errors are the confident ones. The strong signal is validation against data you already hold. Line items should sum to the subtotal and the total should add up. The supplier should exist in your records, the purchase order should be open and the amount should sit near that supplier's normal range. A repeated invoice number is a flag rather than a second payable. Keep a link from every field back to where on the page it came from, and never let a document change bank details automatically, since that is the classic invoice fraud pattern.

Should we buy an off-the-shelf document processing tool or build a custom one?

Off-the-shelf tools are genuinely good for common documents going into common systems, such as receipts and standard invoices into mainstream accounting software, and you should start there if that describes you. A custom pipeline earns its cost in three situations: the documents are specific to your industry and do not match the templates a general tool expects, the validation that catches errors depends on your own data such as open purchase orders or customer records, or the destination is a system you built rather than a mainstream product. Five things move the cost more than the choice of model: how many different layouts arrive, input quality from generated PDFs down to phone photos and handwriting, how many languages, where the data lands and whether it touches money, and how sensitive the documents are. Whichever route you take, ask to clear ten flagged documents on the review screen yourself and time it, because the review screen decides the savings more than the model does.

AI Document Processing: Automating Data Entry From Invoices, Forms and Email

Q: Can AI automate data entry from invoices, forms and PDFs?

Yes, and reading the document is the easy part. Modern models extract fields such as supplier, date, invoice number, line items, tax and total from most clean documents well. The work that decides whether data entry actually disappears is everything around that: classifying what arrived, posting the result into your accounting or case system without duplicates, and handling the documents the system is unsure about. Judge a project on straight-through rate, meaning the share of documents that land in your system correctly with nobody touching them, and on how long a person spends on a flagged document compared with typing it from scratch today. Time your current process on a real week of real documents before the build so there is a baseline. And ask first whether the document needs to exist at all, because if the information comes from people you can reach, capturing it as structured data at the source through a web form, SMS or voice intake beats extracting it from paper later.

TL;DR: Getting an AI model to read a clean invoice is the easy part, and it is what every demo shows. Whether document automation actually removes data entry depends on the documents it is unsure about: how many go straight through with nobody touching them, how wrong fields get caught, and how fast a person can check the rest. Judge it on straight-through rate rather than accuracy, check every extracted field against data you already hold instead of asking the model how confident it is, treat the review screen as the real product, and never let a document change bank details on its own.

Most businesses have a person, or a whole room, whose job is reading documents and typing what they say into another system. Supplier invoices into accounting. Application forms into a case system. Purchase orders, delivery notes, claim forms, onboarding packets, the PDF a customer emailed at 11pm. It is slow, it is error-prone, and it is exactly the kind of work modern AI models are good at.

That last part is true, which is why the demo is so convincing. Ten sample invoices go in, twenty neat fields come out. Then the system goes live and the savings are a fraction of what the demo implied, for reasons that have little to do with the model.

Three jobs, and the one you can sometimes skip

Document automation is three jobs sold as one.

Classify. Work out what arrived: an invoice, a credit note, a statement that looks like an invoice, a signed contract, a photo of a delivery note, or three of those stapled into one PDF. Getting this wrong sends a correct extraction to the wrong place.

Extract. Pull out the specific fields: supplier, date, invoice number, line items, tax, total. This is the part the demo shows.

Post. Write the result into the system that runs the business, matched to the right supplier, customer or case, without creating duplicates. That is an integration project, and it usually takes more of the build than extraction does.

Before commissioning any of it, ask a question vendors rarely raise: does this document need to exist at all? Paper often survives because nobody redesigned the intake. If the information comes from people you can reach, capturing it as structured data at the source beats reading it back out of a form later. That is the idea behind Fortell AI, where we build the voice and SMS systems that help Community Action Agencies simplify intake in over 100 languages, so information arrives as data rather than as paperwork somebody has to key in. Extraction is the right answer for documents you do not control: supplier invoices, third-party forms, anything sent in someone else's format.

Measure straight-through rate, not accuracy

"Our extraction is 98 percent accurate" is the number most proposals lead with, and it hides the one that matters.

Accuracy is usually quoted per field. Your team experiences it per document. If each of twenty fields is right 98 times in 100, only about two documents in three come through with every field correct. A figure that sounds nearly perfect can describe a process where someone still checks everything, because nobody knows which third is wrong.

The number that decides the business case is straight-through rate: the share of documents that land in your system correctly with nobody touching them. The second is time to handle the rest: how long a person spends on a flagged document compared with typing it from scratch today.

Both need a baseline before the build, which means timing the current process on a real week of real documents, including the ugly ones. Without that, nobody can say afterwards whether the system saved anything.

Expect the rate to vary enormously by sender. A regular supplier with a clean, consistent layout may go straight through almost every time. A one-off sender with a scanned, stamped, handwritten form may never go straight through, and that is fine as long as the system knows which is which.

Do not ask the model how sure it is

The obvious way to decide what goes straight through is to ask the model for a confidence score and route anything low to a person. It is also the weakest way. Language models are not reliably calibrated about their own mistakes, and the errors that hurt are the confident ones: a total read from the wrong column, a date in the wrong format, an invoice number with two digits swapped.

The strong signal is checking extracted values against what you already know.

Arithmetic. Line items should sum to the subtotal, tax should match the rate, the total should add up. A document that fails its own math is flagged, whatever the model believes.
Master data. The supplier should exist in your records, the purchase order should be open, the amount should sit near that supplier's normal range.
Duplicates. The same invoice number from the same supplier twice is a flag, not a second payable.
Format and range. Dates that cannot be right, IDs with the wrong structure, quantities no order ever contained.

This is also where extraction differs from summarizing. LectureNotes AI, one of our own products, turns a recorded lecture into takeaways and clean outlines, and paraphrase is the point there. Extraction has no room for paraphrase. The number in your system has to be the number on the page, which is why every field should keep a link back to where on the document it came from.

Some fields should never be written automatically, whatever the checks say. A document asking to change a supplier's bank details is the classic invoice fraud pattern, and text inside a document should never authorize anything. That principle is covered in AI agents with write access, and it applies to every PDF a pipeline reads.

The review screen is the product

In the first months, a meaningful share of documents will need a person. How quickly that person can clear them decides the savings more than any model choice. If checking a flagged document is slower than typing it, the project has failed while the model succeeded.

A review screen worth building has four properties.

Document and fields side by side, with the source of each value highlighted on the page, so checking is looking rather than searching.

Only the doubtful fields flagged. If the arithmetic passed and the supplier matched, the reviewer should not re-read the whole thing. Asking people to verify everything trains them to verify nothing.

Built for speed. Keyboard first, one document after another, a fix in a few keystrokes.

Every correction captured. A fixed field is a free test case. Collected corrections become the regression set you rerun whenever the model, the prompt or a supplier's layout changes, the same discipline we apply in AI voice agent monitoring.

The queue also needs a named owner. Flagged documents with nobody responsible become a backlog, and a backlog of unposted invoices is worse than the manual process it replaced, because at least the manual process was visible.

Roll it out the way you would any system that writes into your records: run it alongside the current process and compare, then draft and approve, then let clean document types post on their own, one at a time.

What makes your version hard

Five things move the cost more than the choice of model.

Layout variety. Five suppliers with consistent invoices is a small project. Five hundred senders in five hundred layouts is a different one, because the long tail never stops producing new cases.

Input quality. A PDF generated by software is the best case. Scans, phone photos, faxes, stamps over text and handwriting each lower the straight-through rate and raise the review load.

Languages. Documents in several languages need a test set per language and someone who can read each one, the same lesson as multilingual voice AI.

Where it lands. Posting into a mainstream accounting tool through a documented API is simpler than posting into a custom system, and posting money is stricter than posting a case note. Records that feed a ledger inherit the rules in building financial software.

Sensitivity. Identity documents, bank details and health information change who may see the review queue, how long originals are kept and where they are processed. Settle that before the build, not during the security review.

On build versus buy: off-the-shelf tools are genuinely good for common documents going into common systems, such as receipts into mainstream accounting software. A custom pipeline earns its money when the documents are specific to your industry, when validation depends on your own data, or when the destination is a system you built. The wider trade-off is in vertical SaaS: build vs buy.

Questions to ask before you commission one

What is the straight-through rate on our documents? Measured on a real sample you supply, including the worst ones.
How is a wrong field caught? If the answer is a confidence score alone, ask what gets checked against your own data.
What does the review screen look like? Clear ten flagged documents yourself and time it.
Which fields can never post automatically? Bank details belong on that list.
What happens when a sender changes its layout? Who notices, and how quickly.
Where do the originals live, and for how long? And who can see them.
Who owns the validation rules and the test set? They are the valuable part and they should stay with you, as covered in software ownership and handover.

The bottom line

Document automation pays when it is judged on the documents nobody has to touch and the minutes saved on the ones somebody does. The model reading the page is the least uncertain part of the project. What decides the result is validation against data you already trust, a review screen fast enough that people use it properly, and a clear line around the fields that always wait for a person. We build AI systems that write real records into the tools businesses run on, from the intake behind Fortell AI to the booking behind CallGuard AI, and we build the systems those records land in, like Fyuel, where customers, suppliers, tankers, ledgers and banks have to agree in real time.

Have documents somebody retypes by hand every day? Book a demo and bring a real sample. We will tell you honestly how much of it could go straight through, and which part should stay with a person. See our work: Fortell AI, CallGuard AI, Fyuel and more.