mlsanityData to 14 Oct 2026

Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation

arXiv 2412.12627 ↗First posted 17 Dec 2024cs.CLcited by 5

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Visual information has been introduced for enhancing machine translation (MT), and its effectiveness heavily relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations. In this paper, we introduce a stable diffusion-based imagination network into a multimodal large language model (MLLM) to explicitly generate an image for each source sentence, thereby advancing the multimodel MT. Particularly, we build heuristic human feedback with reinforcement learning to ensure the consistency of the generated image with the source sentence without the supervision of image annotation, which breaks the bottleneck of using visual information in MT. Furthermore, the proposed method enables imaginative visual information to be integrated into large-scale text-only MT in addition to multimodal MT. Experimental results show that our model significantly outperforms existing multimodal MT and text-only MT, especially achieving an average improvement of more than 14 BLEU points on Multi30K multimodal MT benchmarks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 21 of its 83 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

21 of 83 references.

References · 21

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  9. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  10. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  11. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  12. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
9 more
  1. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  2. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  3. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  4. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  5. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
  6. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  7. 2024Large Language Models for Classical Chinese Poetry Translation: Benchmarking, Evaluating, and Improvingcited by 7
  8. 2024LLM-based Translation Inference with Iterative Bilingual Understandingcited by 6
  9. 2024Aligning Translation-Specific Understanding to General Understanding in Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  2. 2025Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awarenesscited by 2
  3. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  4. 2026VC-Bench: Pioneering the Video Connecting Benchmark with a Dataset and Evaluation Metricscited by 0
  5. 2026Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2025Dual-branch Prompting for Multimodal Machine Translation
  5. 2023Learning to Imagine: Visually-Augmented Natural Language Generation
  6. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  7. 2023LLMGA: Multimodal Large Language Model based Generation Assistant
  8. 2024TIE: Revolutionizing Text-based Image Editing for Complex-Prompt Following and High-Fidelity Editing