mlsanityData to 14 Oct 2026

Exploiting Pseudo Image Captions for Multimodal Summarization

arXiv 2305.05496 ↗First posted 9 May 2023cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives. In this paper, we study this problem from the perspective of Mutual Information (MI) optimization. It is common sense that InfoNCE loss used in contrastive learning will maximize the lower bound of MI between anchors and their positives, while we theoretically prove that MI involving negatives also matters when noises commonly exist. Guided by a more general lower bound form for optimization, we propose a contrastive learning strategy regulated by progressively refined cross-modal similarity, to more accurately optimize MI between an image/text anchor and its negative texts/images instead of improperly minimizing it. Our method performs competitively on four downstream cross-modal tasks and systematically balances the beneficial and harmful effects of (partial) false negative samples under theoretical guidance.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 41 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  9. 2019Text Summarization with Pretrained Encoderscited by 556
  10. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  11. 2020Extractive Summarization as Text Matchingcited by 189
  12. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
13 more
  1. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  2. 2020Heterogeneous Graph Neural Networks for Extractive Document Summarizationcited by 91
  3. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
  4. 2020VMSMO: Learning to Generate Multimodal Summary for Video-based News Articlescited by 44
  5. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  6. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  7. 2021UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillationcited by 23
  8. 2021Self-Supervised Multimodal Opinion Summarizationcited by 18
  9. 2021Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarizationcited by 18
  10. 2023TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selectioncited by 18
  11. 2020Multi-modal Summarization for Video-containing Documentscited by 13
  12. 2020Multi-Modal Summary Generation using Multi-Objective Optimizationcited by 8
  13. 2021DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 3; 2026: 2.

10 papers.

  1. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  2. 2024Fine-grained and Explainable Factuality Evaluation for Multimodal Summarizationcited by 10
  3. 2023BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarizationcited by 5
  4. 2023COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignmentcited by 5
  5. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  6. 2023Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filteringcited by 2
  7. 2025Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correctioncited by 1
  8. 2025MMCIG: Multimodal Cover Image Generation for Text-only Documents and Its Dataset Construction via Pseudo-labelingcited by 0
  9. 2026Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversitycited by 0
  10. 2026Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training
  2. 2022Vision-Language Pre-Training with Triple Contrastive Learning
  3. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models
  4. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillation
  5. 2024CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training
  6. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens
  7. 2026Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning
  8. 2024AmCLR: Unified Augmented Learning for Cross-Modal Representations