# The Ability of Image-Language Explainable Models to Resemble Domain Expertise

arXiv: https://arxiv.org/abs/2209.09310 · HTML: https://mlsanity.com/paper/2209.09310/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Clinically Accurate Chest X-Ray Report Generation](https://mlsanity.com/paper/1904.02633/) (2019)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems](https://mlsanity.com/paper/2008.09100/) (2020)
- [BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis](https://mlsanity.com/paper/2108.04938/) (2021)
- [Optimising Chest X-Rays for Image Analysis by Identifying and Removing Confounding Factors](https://mlsanity.com/paper/2208.10320/) (2022)
- [EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision](https://mlsanity.com/paper/2007.05597/) (2020)

## Built on by (most cited first, 1 of 1)

- [Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability](https://mlsanity.com/paper/2406.12897/) (2024)

## Abstract (arXiv)

Recent advances in vision and language (V+L) models have a promising impact in the healthcare field. However, such models struggle to explain how and why a particular decision was made. In addition, model transparency and involvement of domain expertise are critical success factors for machine learning models to make an entrance into the field. In this work, we study the use of the local surrogate explainability technique to overcome the problem of black-box deep learning models. We explore the feasibility of resembling domain expertise using the local surrogates in combination with an underlying V+L to generate multi-modal visual and language explanations. We demonstrate that such explanations can serve as helpful feedback in guiding model training for data scientists and machine learning engineers in the field.