mlsanityData to 7 Oct 2026

KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records

arXiv 2609.03597 ↗First posted 3 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs, no mainstream font, and no coverage in any OCR pipeline or tokenizer. The handwritten records that carry these fractions, RS Khatians, are the authoritative title record for millions of parcels and a frequent subject of civil litigation, yet no benchmark has asked whether a machine can read one. We introduce KhatianDoc, a four-task benchmark built from 107 real RS Khatian records from the Vumi (land) Office of Munshiganj, Bangladesh: symbol recognition, base-16-to-decimal conversion, structured field extraction, and legal document question answering over 1,634 QA pairs. Ground truth was transcribed by hand, verified by a land-law practitioner to full agreement, and anonymized through positional tokens that keep the referential distinctions multi-hop questions depend on. We evaluate six multimodal LLMs (8B to 72B+, open and closed) under a fixed zero-shot protocol. Five QA categories, 39.3% of our stratified set, return zero correct answers from every model; on the arithmetic task, every model that emits a number does worse than a constant-mean baseline, with exact- and near-match scores coinciding: decorrelation, not approximation. Auditing our own metrics surfaced two artifacts in opposite directions: we correct a refusal-scoring bug and report the fixed scores beside the originals, and flag an inflated metadata metric as an upper bound. KhatianDoc documents not a performance gap but the absence of a capability, with verified ground truth for future systems. Code and data, with a redacted image release, are publicly available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

17 of 24 references.

References · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  6. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  7. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  8. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  9. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  10. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  11. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  12. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
5 more
  1. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
  2. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  3. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  4. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  5. 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ILSIC: Corpora for Identifying Indian Legal Statutes from Queries by Laypeople
  2. 2025Assessing the Reliability of Large Language Models in the Bengali Legal Context: A Comparative Evaluation Using LLM-as-Judge and Legal Experts
  3. 2026Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model
  4. 2025NitiBench: A Comprehensive Study of LLM Framework Capabilities for Thai Legal Question Answering
  5. 2024Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice
  6. 2026BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension
  7. 2026NepKANUN: A RAG-Based Nepali Legal Assistant
  8. 2026BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models