MASS: Overcoming Language Bias in Image-Text Matching
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Pretrained visual-language models have made significant advancements in multimodal tasks, including image-text retrieval. However, a major challenge in image-text matching lies in language bias, where models predominantly rely on language priors and neglect to adequately consider the visual content. We thus present Multimodal ASsociation Score (MASS), a framework that reduces the reliance on language priors for better visual accuracy in image-text matching problems. It can be seamlessly incorporated into existing visual-language models without necessitating additional training. Our experiments have shown that MASS effectively lessens language bias without losing an understanding of linguistic compositionality. Overall, MASS offers a promising solution for enhancing image-text matching performance in visual-language models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 54 references that are in the map. First posted on arXiv 20 Jan 2025.
Builds on
41 of 54 references.
References · 41
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
29 more
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
- 2018Stacked Cross Attention for Image-Text Matchingcited by 414
- 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
- 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
- 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
- 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
- 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
- 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
- 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
- 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
- 2021Probing Image-Language Transformers for Verb Understandingcited by 109
- 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
- 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
- 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
- 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
- 2023Image Captioners Are Scalable Vision Learners Toocited by 89
- 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
- 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
- 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
- 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
- 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
- 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
- 2020Image Colorization: A Survey and Datasetcited by 23
- 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?cited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Models
- 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs
- 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
- 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
- 2023RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models
- 2026COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
- 2024Bringing Multimodality to Amazon Visual Search System
- 2026The Alignment Illusion in Multimodal Large Language Models