benchmarks · 2026-02-10 · 11 min read

Benchmarking document parsers in 2026: we ran 120,000 invoices through everyone (including ourselves)

Seven parsing platforms, one private dataset of 120,000 real invoices, and one metric almost nobody publishes: variance across document quality tiers.

We benchmark vendors for a living, sort of — customers ask "how do you compare to X" constantly, and until now our answer was honest but weak: "on our own test set we're better, obviously." So we built something better. This post is the full methodology and results, including the category where a competitor beat us by 2.4 points, because a benchmark that only ever crowns the author is a brochure.

The dataset

120,000 real invoices from eleven customers who volunteered their documents (with sensitive fields tokenized, and their lawyers' blessing). We split by acquisition type, because that's what actually predicts parsing difficulty:

Born-digital PDFs (46%). Pixels on paper were never involved. Text is embedded, fonts are sane.
Scans (31%). 200–400 dpi, consumer flatbeds and office MFPs. Coffee-ring class noise included.
Photos (12%). Taken with a phone, under office lighting, occasionally at an angle that would embarrass a cubist. This is how an alarming share of the world's invoices actually arrive.
Faxes (11%). Yes. T.38, from healthcare and logistics customers, mostly in Germany and Japan. If you think fax is dead, congratulations on your industry.

Ground truth was double-keyed by two annotation teams, with a third pass on disagreements. We paid for this properly — about 900 person-hours — because a benchmark is only as good as its labels, and disagreement between labelers (2.1% initially, 0.4% after adjudication) tells you where the whole task's ceiling is.

The contenders

Seven platforms: ourselves, three commercial API competitors you've heard of, two cloud-provider services, and the strongest open-source pipeline we could assemble (OCR + layout model + rule-based extraction, tuned by two engineers over two weeks, which is more support than most open-source stacks ever get).

Every vendor got the same 120,000 documents through their standard API, same schemas, same retry policy. We ran everything three times over different weeks to catch version drift — one competitor shipped a model update mid-benchmark that changed their F1 by 1.8 points, which tells you something about the shelf life of any single benchmark table.

Metrics

Field-level F1 across the standard invoice schema: number, dates, vendor block, totals, line items.
Table recall — fraction of line-item rows recovered, penalty for ghost rows.
Variance across tiers — the spread of F1 between born-digital and faxed. This is the number that matters and the number nobody puts on a landing page.
Latency at p50 and p95, measured from our side.

Results

Anonymized names, honest numbers. Category is the acquisition tier; F1 is field-level across the full schema.

Platform	Born-digital	Scan	Photo	Fax	Spread
unstructred (ours)	0.996	0.991	0.972	0.958	0.038
Competitor A	0.994	0.955	0.931	0.902	0.092
Competitor B	0.989	0.948	0.927	0.911	0.078
Cloud provider 1	0.991	0.940	0.905	0.884	0.107
Competitor C	0.981	0.933	0.918	0.895	0.086
Open-source stack	0.972	0.918	0.879	0.861	0.111
Cloud provider 2	0.976	0.921	0.883	0.869	0.107

The pattern: everyone looks basically the same on born-digital PDFs. The field opens up the moment reality gets involved. The spread column is our favorite — the difference between a demo and a production system is almost entirely "how much worse does it get when the input is ugly."

Where we lost

Multi-column German financial statements. Not invoices, technically — the customers' AP teams process them the same way, so they were in the set. Competitor A's table reconstruction beat ours by 2.4 F1 points on these, mostly by handling a particular flavor of merged header cell that German ERP exports love. Our parser split those into ghost rows; theirs didn't.

We're shipping a fix (cell-level provenance landed in v2.3.0, and the column-span handling itself is in the next release). Until then, if your documents are mostly German financial statements, our honest advice is to look at Competitor A too. We'd rather you trust us on the other 95% of the corpus.

Latency, briefly

We were fastest at p50 (1.9s) and second at p95 (4.1s). Cloud provider 1 won p95 with aggressive batching at high concurrency. Latency differences at this level matter far less than accuracy differences, so we're not going to pretend otherwise with a bar chart.

What we'd caution you about

Your documents are not our documents. Run the subset. We published an anonymized 8,000-document sample (link in the forum post) precisely so you can rerun this against your own pipeline — the sample is CC BY 4.0, no registration.
Single-number F1 hides the tier story. Ask any vendor for numbers per acquisition type. Watch how many can't produce them without "getting back to you."
Benchmarks age. Everyone is shipping models constantly. We'll rerun this roughly twice a year and publish the diffs, including the ones that hurt.

Acknowledgments

The eleven customers who lent us documents took a real risk for the community's benefit. The annotation teams did unglamorous work to a standard most labs wouldn't fund. And Competitor A's team, when we told them they'd won a category, asked for our numbers so they could check their own weak spots. That's the right spirit, and it's why this industry is more fun than it looks.

#benchmarks#ocr#invoice-parsing

← all posts next: v2.3.0 release notes →