Skip to main content
← All articles
AI Data

Document Annotation: OCR Correction, Handwriting, Layout and Key-Value Extraction

Short answer. Document annotation is four jobs that people habitually treat as one. OCR correction fixes what the machine misread. Handwriting transcription handles what OCR was never…

Mumu D. · August 2026 · 10 min read

Download PDF

Short answer. Document annotation is four jobs that people habitually treat as one. OCR correction fixes what the machine misread. Handwriting transcription handles what OCR was never built for. Layout annotation records the structure — columns, tables, reading order — that gives the text its meaning. Key-value extraction pulls named fields into a record you can actually query. Modern systems are genuinely good at all four on clean, modern, printed pages.

On a faded 1890s parish register written by three different clerks, they are not, and that gap is where the human work lives.

What are the four jobs, and why separate them?

Because each one fails differently, and a pipeline that treats them as a single step cannot tell you which part broke.

The cleanest description of the stack we have seen puts it in a sequence: ingestion and normalisation, document classification, layout-aware OCR, key-value and table extraction, entity linking, validation against confidence thresholds, human review, then export. OCR turns pixels into text — it finds characters and words, keeps a rough reading order, and hands you something searchable. Useful, and it stops there. OCR will not tell you that a "Total" ought to equal the sum of the line items above it, or that a date has landed in the wrong format.

Layout is the step people underestimate. A two-column page read straight across produces text that is individually correct and collectively meaningless. Even mature commercial systems acknowledge this: multi-column or irregular documents commonly need post-processing to restore reading order, and handwritten or cursive input remains inconsistent. In genealogy work this shows up constantly — a ledger where the surname column, the baptism date and the officiating minister sit in a grid that no reading-order heuristic gets right without being told.

This is the part of the business Lifewood grew up in. Long before "document AI" was a category, we were scanning fragile records, correcting OCR line by line and turning them into indexed, searchable genealogical data. The technology around that work has changed enormously. The judgement it needs has not.

How good is handwriting recognition really?

Extremely good on trained material, and much worse than the headline figures suggest on everything else.

Handwritten Text Recognition is a genuinely different technology from OCR, not a setting within it. Classical OCR classifies printed glyphs one at a time; HTR reads a whole line of connected script in context, because cursive has no reliable letter boundaries to segment. Quality is reported as Character Error Rate — the share of characters wrong through insertion, deletion or substitution — and Word Error Rate, which always runs higher, typically three to four times CER, because one wrong character ruins a whole word.

Two results in that chart deserve unpacking. The first is the LLM finding. A 2024 study tested mainstream multimodal models "out of the box" on a corpus of 18th and 19th century English handwriting, deliberately captured the way historians and genealogists actually work — varied hands, phone and hand-held camera shots, black-and-white microfilm. The models achieved CER of 5.7–7% and WER of 8.9–15.9%, improvements of 14% and 32% respectively over specialised HTR software, while being faster and cheaper.

The second is the 36%. That figure is for general, unconstrained handwriting — which is to say, most of what sits in an uncatalogued archive box. The distance between under 2% and 36% is the entire argument for treating document annotation as skilled work rather than a procurement line item.

There is also a subtler risk with LLM transcription that anyone working with historical sources should know about. A general model tends to produce a clean, confident, grammatical reading that happens to be wrong, and will insert plausible archaic spellings the source never contained — documented as "over-historicising" in evaluations of these pipelines. For genealogy that is worse than a visible error, because a garbled name gets flagged and a plausible wrong name gets indexed, propagated and cited for the next thirty years.

Four jobs, four failure modes JOB WHAT THE ANNOTATOR DOES WHERE IT GOES WRONG WHO SHOULD DO IT OCR CORRECTION Fixes misread characters against the page image Confident substitutions in proper nouns, numerals and dates Machine first, human verify HANDWRITING Transcribes script the OCR layer cannot touch Unusual hands, abbreviations, faded ink, over-historicising by LLMs Human-led, model-assisted LAYOUT Marks columns, tables, headers, reading order and region types Multi-column and irregular pages reassembled in the wrong sequence Model with human spot-check KEY-VALUE Maps text to named fields: name, date, place, relationship Right value, wrong field — invisible to character-level metrics Rules plus confidence-gated review Grounding matters across all four: linking every extracted value back to its exact page, region and text span is what lets a reviewer verify it in seconds instead of minutes.

Why do the accuracy numbers mislead?

Because vendors report character accuracy, buyers hear document accuracy, and those are separated by an exponent.

The distinction is worth learning properly. Character accuracy is the share of characters correct. Field accuracy asks whether the right value landed in each named field. Document accuracy asks what share of documents came through with zero errors — and that is the number that actually governs whether a workflow can run without review. A 99% character-accuracy claim tells you very little about whether an extracted record is correct.

A typical genealogical record — given name, surname, alternate spellings, sex, birth date, birth place, baptism date, parish, father, mother, occupation, witnesses, page and film references — runs comfortably past twenty fields. The arithmetic in can still deliver a majority of records with at least one thing wrong.

None of which means the benchmarks are worthless. Agentic parsing has reached 99.16% on the DocVQA validation split — 5,286 correct out of 5,331, with only 18 of the 45 misses attributable to genuine parsing shortcomings — and fine-tuned vision-language models reach around 99% in production when paired with human-in-the-loop workflows. But DocVQA is 12,767 document images with question-answer pairs scored by string similarity, and it is a useful signal of capability rather than a reliable predictor of how a parser will behave on your documents. Whether it transfers depends entirely on how much your material resembles the benchmark's.

What does a workflow that holds up look like?

Machine-first, confidence-gated, and grounded — with humans spending their time on the pages that need them.

WHAT WE'VE SEEN WORK WHAT QUIETLY GOES WRONG

  • Classify before extracting — different document types, different pipelines

  • A single confidence threshold applied across every field

  • Confidence thresholds per field, tuned by field risk

  • Trusting a fluent LLM transcription that was never checked against the image

  • Every value grounded to its page, region and span

  • Reading order assumed rather than annotated

  • Double-keying for names, dates and numerals

  • Names transliterated inconsistently across a collection

  • A dedicated pass for proper nouns against local name lists

  • Reviewers who cannot see the source region beside the value

  • Validation rules: totals reconcile, dates fall in plausible ranges

  • No record of which model version produced which output Report document accuracy, not character accuracy.

It is the number that predicts rework.

Systems fall into patterns — the same mistake repeated across every similar document.

Two things make the economics work. The first is that not every page deserves the same treatment: confidence thresholds decide what passes straight through and what gets a quick check, and the savings are substantial — switching from manual or OCR-based workflows to vision-AI processing has been reported to cut document processing costs by 75–92%. The second is that the human effort has to land where it changes the answer. A reviewer verifying a printed form field that the model got right with 0.99 confidence is expensive noise. The same reviewer resolving whether a faded surname reads "Mainwaring" or "Mannering" is doing something no model can do reliably today.

That is the shape of the work across our own scanning and genealogy programmes: machines handle volume, our teams handle ambiguity, and the routing between them is where the quality actually comes from. Add languages and scripts to the picture — Gothic hands, Cyrillic, historical orthographies, name conventions that vary by region — and the value of native-speaker reviewers stops being a nice-to-have. A name misread in a language nobody on the review team speaks is a name that stays misread.

Measure document accuracy from the start. Sample records, count those with zero errors, and report that alongside any character-level figure.

Set confidence thresholds field by field. A misread witness name and a misread birth date do not carry the same cost.

Ground every extracted value. If a reviewer cannot see the source region next to the value, verification takes minutes instead of seconds.

Never accept an LLM transcription unverified against the image. Fluency is not accuracy, and over-historicising errors read as authentic.

Annotate layout explicitly for tabular and multi-column material. Registers, ledgers and census returns are exactly where inferred reading order fails.

Give proper nouns their own pass. Names carry the most downstream weight and the least contextual redundancy.

Version everything. Which model, which ruleset, which export — so a correction three years later can be traced and reapplied.

Staff reviewers who read the language and the period. Script conventions, abbreviations and naming customs are learned knowledge, not general literacy.

Key takeaways

  • Document annotation is four distinct jobs: OCR correction, handwriting transcription, layout annotation and key-value extraction.
  • HTR is not OCR — it reads whole lines in context because cursive offers no reliable letter boundaries.
  • CER ranges from under 2% on trained datasets to roughly 36% error on general unconstrained handwriting; historical documents typically sit at 3–5% when legible and 10–15% when damaged.
  • Frontier LLMs reached 5.7–7% CER and 8.9–15.9% WER on 18th–19th century English handwriting out of the box, beating specialised HTR software by 14% and 32%.
  • LLMs carry a specific risk: fluent, confident, wrong readings with invented archaic spellings.
  • Character accuracy, field accuracy and document accuracy are three different numbers — at 97% per field, a 20-field record is clean only about half the time.
  • DocVQA scores such as 99.16% signal capability, not production performance on your documents.
  • Vision-AI processing has been reported to cut document processing costs by 75–92% versus manual or OCR-based workflows, with fine-tuned VLMs near 99% when paired with human review.

Sources and further reading

Frequently asked questions

For low-risk fields on clean documents, often yes — that is what confidence thresholds are for. For names, dates and anything that becomes a permanent identifier, no. Confidence measures the model's certainty, not its correctness.

Roughly 50–100 transcribed pages for a custom model, with larger sets improving accuracy. Transfer learning from a base model trained on millions of pages shortens this considerably.

On the tested corpus of 18th–19th century English, yes — lower error rates, faster and cheaper. But they fail differently, producing plausible inventions rather than obvious garble, so verification against the page image matters more, not less.

Because meaning lives in position. In a register, which column a date sits in determines whether it is a birth or a baptism.

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team