Short answer. Document annotation is four jobs people habitually treat as one: OCR correction, handwriting transcription, layout annotation, and key-value extraction. Modern systems handle all four well on clean, modern, printed pages. On a faded 1890s parish register written by three different clerks, they do not — and that gap between benchmark accuracy and real archival material is where the human work lives.
Key takeaways
- Document annotation is four distinct jobs: OCR correction, handwriting transcription, layout annotation and key-value extraction.
- Handwritten Text Recognition (HTR) is not OCR — it reads whole lines in context because cursive offers no reliable letter boundaries to segment.
- Character error rates on handwriting range from under 2% on trained datasets to roughly 36% on general unconstrained handwriting.
- Frontier LLMs reached 5.7–7% character error rate and 8.9–15.9% word error rate on 18th–19th century English handwriting out of the box, beating specialised HTR software by 14% and 32% respectively.
- Character accuracy, field accuracy and document accuracy are three different numbers — at 97% accuracy per field, a 20-field record is fully correct only about half the time.
What are the four jobs, and why separate them?
Each of the four jobs fails in a different way, and a pipeline that treats them as one step cannot tell you which part broke.
OCR (optical character recognition) turns pixels into text — it finds characters and words and keeps a rough reading order, but it will not tell you that a "Total" should equal the sum of the line items above it, or that a date has landed in the wrong format. A typical pipeline runs ingestion and normalisation, document classification, layout-aware OCR, key-value and table extraction, entity linking, confidence-threshold validation, human review, then export.
Layout annotation — marking columns, tables, headers and reading order — is the step buyers underestimate. A two-column page read straight across produces text that is individually correct and collectively meaningless. Even mature commercial systems acknowledge this: multi-column or irregular documents commonly need post-processing to restore reading order, and handwritten or cursive input remains inconsistent. In genealogy work this shows up constantly — a ledger where the surname column, the baptism date and the officiating minister sit in a grid that no reading-order heuristic gets right without being told.
This is the part of the business Lifewood grew up in. Long before "document AI" was a category, Lifewood was scanning fragile records, correcting OCR line by line, and turning them into indexed, searchable genealogical data, a capability described further in our AI data services. The technology around that work has changed enormously; the judgement it needs has not.
How good is handwriting recognition really?
It is extremely good on trained material and considerably worse than the headline figures suggest on everything else.
Handwritten Text Recognition (HTR) is a genuinely different technology from OCR, not a setting within it: classical OCR classifies printed glyphs one at a time, while HTR reads a whole line of connected script in context because cursive has no reliable letter boundaries to segment. Quality is reported as Character Error Rate (CER) — the share of characters wrong through insertion, deletion or substitution — and Word Error Rate, which runs three to four times higher than CER because one wrong character ruins a whole word.
A 2024 study tested mainstream multimodal models "out of the box" on a corpus of 18th and 19th century English handwriting captured the way historians and genealogists actually work — varied hands, phone and hand-held camera shots, black-and-white microfilm. The models achieved CER of 5.7–7% and WER of 8.9–15.9%, improvements of 14% and 32% respectively over specialised HTR software, while running faster and cheaper. The other end of the range is roughly 36% error on general, unconstrained handwriting — which is to say, most of what sits in an uncatalogued archive box. The distance between under 2% and 36% is the entire argument for treating document annotation as skilled work rather than a procurement line item, a point that shapes how we describe human-in-the-loop annotation routing.
There is also a subtler risk with LLM transcription that anyone working with historical sources should know about: a general model tends to produce a clean, confident, grammatical reading that happens to be wrong, inserting plausible archaic spellings the source never contained — documented as "over-historicising" in evaluations of these pipelines. For genealogy that is worse than a visible error, because a garbled name gets flagged for review while a plausible wrong name gets indexed, propagated and cited for the next thirty years.
Four jobs, four failure modes
| Job | What the annotator does | Where it goes wrong | Who should do it |
|---|---|---|---|
| OCR correction | Fixes misread characters against the page image | Confident substitutions in proper nouns, numerals and dates | Machine first, human verify |
| Handwriting | Transcribes script the OCR layer cannot touch | Unusual hands, abbreviations, faded ink, over-historicising by LLMs | Human-led, model-assisted |
| Layout | Marks columns, tables, headers, reading order and region types | Multi-column and irregular pages reassembled in the wrong sequence | Model with human spot-check |
| Key-value | Maps text to named fields: name, date, place, relationship | Right value, wrong field — invisible to character-level metrics | Rules plus confidence-gated review |
Grounding matters across all four jobs: linking every extracted value back to its exact page, region and text span is what lets a reviewer verify it in seconds instead of minutes.
Why do the accuracy numbers mislead?
Vendors report character accuracy, buyers hear document accuracy, and the two are separated by an exponent.
Character accuracy is the share of characters correct. Field accuracy asks whether the right value landed in each named field. Document accuracy asks what share of documents came through with zero errors — the number that actually governs whether a workflow can run without review. A 99% character-accuracy claim says very little about whether an extracted record is correct: a typical genealogical record — given name, surname, alternate spellings, sex, birth date, birth place, baptism date, parish, father, mother, occupation, witnesses, page and film references — runs past twenty fields, and the arithmetic at 97% field accuracy can still deliver a majority of records with at least one thing wrong. Understanding how these layers of error compound is central to any gold-sets, audit sampling and consensus approach to quality control.
None of which makes benchmarks worthless. Agentic parsing has reached 99.16% on the DocVQA validation split — 5,286 correct out of 5,331, with only 18 of the 45 misses attributable to genuine parsing shortcomings — and fine-tuned vision-language models reach around 99% in production when paired with human-in-the-loop workflows. But DocVQA is a scored benchmark of document images with question-answer pairs, and it is a useful signal of capability rather than a reliable predictor of how a parser will behave on a specific customer's documents. Whether it transfers depends on how closely that material resembles the benchmark's.
What does a workflow that holds up look like?
A workflow that holds up is machine-first, confidence-gated and grounded, with humans spending their time only on the pages that need them.
Classifying documents before extraction, tuning confidence thresholds per field rather than applying one threshold across every field, and grounding every value to its page, region and span are the practices that separate a workflow that scales from one that quietly accumulates errors. Double-keying names, dates and numerals, running a dedicated pass for proper nouns against local name lists, and reconciling totals and date ranges against validation rules all catch mistakes that a single confidence score misses. Reviewers who cannot see the source region beside the extracted value, and no record of which model version produced which output, are the two quiet failure points we see most often — both fixable, and both a matter of layered quality control before delivery rather than a smarter model.
Two things make the economics work. First, not every page deserves the same treatment: confidence thresholds decide what passes straight through and what gets a quick check, and the savings are substantial — switching from manual or OCR-based workflows to vision-AI processing has been reported to cut document processing costs by 75–92%. Second, the human effort has to land where it changes the answer. A reviewer verifying a printed form field the model got right with 0.99 confidence is expensive noise; the same reviewer resolving whether a faded surname reads "Mainwaring" or "Mannering" is doing something no model can do reliably today.
That is the shape of the work across scanning and genealogy programmes: machines handle volume, trained teams handle ambiguity, and the routing between them is where the quality comes from. Add languages and scripts to the picture — Gothic hands, Cyrillic, historical orthographies, name conventions that vary by region — and native-speaker review stops being a nice-to-have; a name misread in a language nobody on the review team speaks stays misread. Consistency across annotators on judgement calls like these is exactly what inter-annotator agreement is designed to measure.
Practical checklist for a workflow that holds up:
- Measure document accuracy from the start — sample records, count those with zero errors, and report that figure alongside any character-level metric.
- Set confidence thresholds field by field; a misread witness name and a misread birth date do not carry the same cost.
- Ground every extracted value so a reviewer can see the source region beside it.
- Never accept an LLM transcription unverified against the image — fluency is not accuracy, and over-historicising errors read as authentic.
- Annotate layout explicitly for tabular and multi-column material, where inferred reading order fails most often.
- Give proper nouns their own review pass and staff reviewers who read the language and period of the source.