mlsanityData to 7 Oct 2026

From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

arXiv 2604.16504 ↗First posted 14 Apr 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Manual digitisation of structured handwritten documents is slow and costly. We benchmark 17 leading frontier multi-modal large language models and open-source models against a very challenging real-world medical form that mixes dates; structured, printed text; hand-written responses and significant variability challenges. None of the smaller or older models perform well but the latest Google and OpenAI models reach accuracies around 85% with weighted F1 scores simeq 90% across the discrete or predefined fields despite the very challenging nature of the responses. Clear task specific strengths emerge: GPT 5.4 excels in noisy date extraction as well as reliability with the lowest hallucination rate (6%). Claude Sonnet 4.6 had the best average performance across formatted fields (dates and numerical values), while Gemini 3.1 delivered the best overall performance, with the lowest free text error rates (WER = 0.50 and CER = 0.31) and the strongest results across discrete classification metrics. We further show that prompt optimisation dramatically improves macro precision, recall and F1 by over 60%, but has little impact on weighted metrics (only ~2-5% improvement). These results provide evidence that the rapid improvements of multimodal large language models offer a compelling pathway toward fully automated digitisation of complex handwritten workflows that is particularly relevant in low- and middle-income countries.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 12 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

6 of 12 references.

References · 6

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  3. 2021OCR-free Document Understanding Transformercited by 400
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  5. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  6. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
  2. 2026OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
  3. 2025Advancing Offline Handwritten Text Recognition: A Systematic Review of Data Augmentation and Generation Techniques
  4. 2026Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application
  5. 2025Benchmarking Large Language Models for Handwritten Text Recognition
  6. 2026ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts
  7. 2025NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding
  8. 2026Exploring In-Context Learning for Handwritten Text Recognition