mlsanityData to 7 Oct 2026

Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

arXiv 2603.01696 ↗First posted 2 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 85 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

74 of 85 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning26 Sep
  2. 2025Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning18 Jun
  3. 2024BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs3 Jul
  4. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioning2 Apr
  5. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions28 May

Other references · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
38 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2025DINOv3cited by 1,297
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  11. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  13. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  14. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  15. 2019Attention on Attention for Image Captioningcited by 289
  16. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  17. 2019Connecting Vision and Language with Localized Narrativescited by 225
  18. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  19. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  20. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  21. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  22. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  23. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  24. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  25. 2024Benchmarking and Improving Detail Image Captioncited by 81
  26. 2018Unsupervised Image Captioningcited by 79
  27. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  28. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  29. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  30. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
  31. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  32. 2018Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Datacited by 37
  33. 2016Attention Correctness in Neural Image Captioningcited by 35
  34. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  35. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  36. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  37. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  38. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22

50 most cited of 69 All 69 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
  2. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
  3. 2026Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning
  4. 2026Re3Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning
  5. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
  6. 2024Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning
  7. 2025RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models
  8. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning