mlsanityData to 14 Oct 2026

Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models

arXiv 2504.13945 ↗First posted 16 Apr 2025cs.LG cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

The rapid advancement of large vision-language models (LVLMs) has significantly propelled applications in document understanding, particularly in optical character recognition (OCR) and multilingual translation. However, current evaluations of LVLMs, like the widely used OCRBench, mainly focus on verifying the correctness of their short-text responses and long-text responses with simple layout, while the evaluation of their ability to understand long texts with complex layout design is highly significant but largely overlooked. In this paper, we propose Menu OCR and Translation Benchmark (MOTBench), a specialized evaluation framework emphasizing the pivotal role of menu translation in cross-cultural communication. MOTBench requires LVLMs to accurately recognize and translate each dish, along with its price and unit items on a menu, providing a comprehensive assessment of their visual understanding and language processing capabilities. Our benchmark is comprised of a collection of Chinese and English menus, characterized by intricate layouts, a variety of fonts, and culturally specific elements across different languages, along with precise human annotations. Experiments show that our automatic evaluation results are highly consistent with professional human evaluation. We evaluate a range of publicly available state-of-the-art LVLMs, and through analyzing their output to identify the strengths and weaknesses in their performance, offering valuable insights to guide future advancements in LVLM development. MOTBench is available at https://github.com/gitwzl/MOTBench.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 44 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

31 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond15 Mar

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
18 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  7. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  10. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  11. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  13. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  14. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  15. 2024Cultural Adaptation of Menus: A Fine-Grained Approachcited by 8
  16. 2024MULTI: Multimodal Understanding Leaderboard with Text and Imagescited by 7
  17. 2024A Novel Paradigm Boosting Translation Capabilities of Large Language Modelscited by 3
  18. 2023Text Style Transfer Back-Translationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Modelcited by 0
  2. 2025PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Languagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
  2. 2025MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding
  3. 2026JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
  4. 2026Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
  5. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  6. 2025Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments
  7. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
  8. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models