mlsanityData to 7 Oct 2026

Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency

arXiv 2507.08309 ↗First posted 11 Jul 2025cs.CL cs.AI cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 49 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

23 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  8. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  9. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  10. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  11. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  12. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
10 more
  1. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  2. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  3. 2024TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Modelscited by 27
  4. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  5. 2022Improving End-to-End Text Image Translation From the Auxiliary Text Translation Taskcited by 20
  6. 2024MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Modelscited by 9
  7. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
  8. 2023DocXChain: A Powerful Open-Source Toolchain for Document Parsing and Beyondcited by 8
  9. 2024Boosting LLM Translation Skills without General Ability Loss via Rationale Distillationcited by 5
  10. 2024Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translationcited by 8
  2. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  3. 2026MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translationcited by 4
  4. 2025HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discoverycited by 3
  5. 2026Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
  2. 2025Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
  3. 2025DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model
  4. 2025Evaluating Multimodal Large Language Models on Vertically Written Japanese Text
  5. 2026Comparative Evaluation of Machine Translation Systems on Images with Text
  6. 2026Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
  7. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models
  8. 2025MT3: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning