mlsanityData to 7 Oct 2026

Adventures of Trustworthy Vision-Language Models: A Survey

arXiv 2312.04231 ↗First posted 7 Dec 2023cs.CV cs.AIcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recently, transformers have become incredibly popular in computer vision and vision-language tasks. This notable rise in their usage can be primarily attributed to the capabilities offered by attention mechanisms and the outstanding ability of transformers to adapt and apply themselves to a variety of tasks and domains. Their versatility and state-of-the-art performance have established them as indispensable tools for a wide array of applications. However, in the constantly changing landscape of machine learning, the assurance of the trustworthiness of transformers holds utmost importance. This paper conducts a thorough examination of vision-language transformers, employing three fundamental principles of responsible AI: Bias, Robustness, and Interpretability. The primary objective of this paper is to delve into the intricacies and complexities associated with the practical use of transformers, with the overarching goal of advancing our comprehension of how to enhance their reliability and accountability.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 63 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

47 of 63 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2019Attention is not Explanationcited by 916
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
35 more
  1. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  2. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  3. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  4. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  5. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  6. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  7. 2021Are Transformers More Robust Than CNNs?cited by 187
  8. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  9. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  10. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  11. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  12. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  13. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  14. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  15. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  16. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  17. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  18. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 51
  19. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  20. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  21. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  22. 2020A Closer Look at the Robustness of Vision-and-Language Pre-trained Modelscited by 37
  23. 2022Quantifying Societal Bias Amplification in Image Captioningcited by 37
  24. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
  25. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Modelscited by 28
  26. 2022Men Also Do Laundry: Multi-Attribute Bias Amplificationcited by 21
  27. 2020On the Robustness of Face Recognition Algorithms Against Attacks and Biascited by 16
  28. 2021Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removalcited by 16
  29. 2018Anonymizing k-Facial Attributes via Adversarial Perturbationscited by 8
  30. 2023ImageCaptioner2: Image Captioner for Image Captioning Bias Amplification Assessmentcited by 5
  31. 2023Mitigating Bias in Visual Transformers via Targeted Alignmentcited by 4
  32. 2023VISION DIFFMASK: Faithful Interpretation of Vision Transformers with Differentiable Patch Maskingcited by 4
  33. 2022Pretrained Transformers Do not Always Improve Robustnesscited by 2
  34. 2023Variation of Gender Biases in Visual Recognition Models Before and After Finetuningcited by 2
  35. 2023On Responsible Machine Learning Datasets with Fairness, Privacy, and Regulatory Normscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  2. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  3. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  4. 2026Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Modelscited by 3
  5. 2025Robustness Feature Adapter for Efficient Adversarial Trainingcited by 2
  6. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  7. 2025Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Modelscited by 0
  8. 2025Trust in Vision-Language Models: Insights from a Participatory User Workshopcited by 0
  9. 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generationcited by 0
  10. 2026Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Vision Language Transformers: A Survey
  2. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Tasks
  3. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptions
  4. 2026A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers
  5. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms
  6. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  7. 2024Scaling Capability in Token Space: An Analysis of Large Vision Language Model
  8. 2024Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Models