mlsanityData to 14 Oct 2026

e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks

arXiv 2105.03761 ↗First posted 8 May 2021cs.CV cs.CL cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Recently, there has been an increasing number of efforts to introduce models capable of generating natural language explanations (NLEs) for their predictions on vision-language (VL) tasks. Such models are appealing, because they can provide human-friendly and comprehensive explanations. However, there is a lack of comparison between existing methods, which is due to a lack of re-usable evaluation frameworks and a scarcity of datasets. In this work, we introduce e-ViL and e-SNLI-VE. e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks. It spans four models and three datasets and both automatic metrics and human evaluation are used to assess model-generated explanations. e-SNLI-VE is currently the largest existing VL dataset with NLEs (over 430k instances). We also propose a new model that combines UNITER, which learns joint embeddings of images and text, and GPT-2, a pre-trained language model that is well-suited for text generation. It surpasses the previous state of the art by a large margin across all datasets. Code and data are available here: https://github.com/maximek3/e-ViL.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 8 May 2021.

Builds on

39 of 53 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2016Layer Normalizationcited by 5,669
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
27 more
  1. 2018Datasheets for Datasetscited by 1,386
  2. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  6. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  7. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  8. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2019Abductive Commonsense Reasoningcited by 301
  11. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  12. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  13. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  14. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  15. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  16. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  17. 2016Generating Visual Explanationscited by 158
  18. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  19. 2020Generating Fact Checking Explanationscited by 132
  20. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  21. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  22. 2014The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classificationcited by 67
  23. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  24. 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
  25. 2020Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphscited by 46
  26. 2020Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questionscited by 19
  27. 2020The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsetscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 21; 2023: 26; 2024: 17; 2025: 10; 2026: 4.

84 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 400
  2. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  3. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  4. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  5. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  6. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  7. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  8. 2022Language Models are General-Purpose Interfacescited by 97
  9. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  10. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 79
  11. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  12. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
38 more
  1. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  2. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  3. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  4. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learningcited by 36
  5. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  6. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  7. 2024Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarkscited by 28
  8. 2022Explaining Chest X-ray Pathologies in Natural Languagecited by 27
  9. 2021Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanationcited by 25
  10. 2021Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanationscited by 24
  11. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24
  12. 2022ExPUNations: Augmenting Puns with Keywords and Explanationscited by 21
  13. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  14. 2024ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?cited by 17
  15. 2021Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setupcited by 16
  16. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  17. 2022Investigating the Benefits of Free-Form Rationalescited by 13
  18. 2023Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Taskscited by 12
  19. 2023Learning to Follow Object-Centric Image Editing Instructions Faithfullycited by 12
  20. 2024Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledgecited by 12
  21. 2022Logical Satisfiability of Counterfactuals for Faithful Explanations in NLIcited by 11
  22. 2024LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologiescited by 11
  23. 2023DeViL: Decoding Vision features into Languagecited by 10
  24. 2023S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learningcited by 10
  25. 2024CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumescited by 10
  26. 2025ChartQA-X: Generating Explanations for Visual Chart Reasoningcited by 10
  27. 2023Characterizing Large Language Models as Rationalizers of Knowledge-intensive Taskscited by 9
  28. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  29. 2022Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanationscited by 8
  30. 2023ZARA: Improving Few-Shot Self-Rationalization for Small Language Modelscited by 8
  31. 2023Active Reasoning in an Open-World Environmentcited by 8
  32. 2022Explicit Image Caption Editingcited by 7
  33. 2023Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyondcited by 7
  34. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Modelscited by 6
  35. 2022Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanationscited by 6
  36. 2024The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systemscited by 6
  37. 2021Self-Supervised Bot Play for Conversational Recommendation with Justificationscited by 5
  38. 2023Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQAcited by 5

50 most cited of 84 All 84 (JSON)