mlsanityData to 14 Oct 2026

Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

arXiv 2507.07572 ↗First posted 10 Jul 2025cs.CL cs.AI cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 40 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

21 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2016Sequence-Level Knowledge Distillationcited by 801
  5. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  6. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  7. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  8. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  9. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  10. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  11. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  12. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
8 more
  1. 2020Towards Fully Automated Manga Translationcited by 22
  2. 2022Improving End-to-End Text Image Translation From the Auxiliary Text Translation Taskcited by 20
  3. 2024Translatotron-V(ison): An End-to-End Model for In-Image Machine Translationcited by 14
  4. 2023SentAlign: Accurate and Scalable Sentence Alignmentcited by 9
  5. 2023DocXChain: A Powerful Open-Source Toolchain for Document Parsing and Beyondcited by 8
  6. 2023E2TIMT: Efficient and Effective Modal Adapter for Text Image Machine Translationcited by 5
  7. 2023Efficient End-to-End Visual Document Understanding with Rationale Distillationcited by 5
  8. 2023Multi-Teacher Knowledge Distillation For Text Image Machine Translationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translationcited by 9
  2. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  3. 2026MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translationcited by 4
  4. 2026Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantic Alignment for Multimodal Large Language Models
  2. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
  3. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  4. 2025VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization
  5. 2025DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model
  6. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
  7. 2025AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
  8. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models