# A study on the Interpretability of Neural Retrieval Models using DeepSHAP

arXiv: https://arxiv.org/abs/1907.06484 · HTML: https://mlsanity.com/paper/1907.06484/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2019.

## Built on by (most cited first, 19 of 19)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [Rank-LIME: Local Model-Agnostic Feature Attribution for Learning to Rank](https://mlsanity.com/paper/2212.12722/) (2022)
- [Query Understanding in the Age of Large Language Models](https://mlsanity.com/paper/2306.16004/) (2023)
- [Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning](https://mlsanity.com/paper/2103.00370/) (2021)
- [Diagnosing BERT with Retrieval Heuristics](https://mlsanity.com/paper/2201.04458/) (2022)
- [Explain like I am BM25: Interpreting a Dense Model's Ranked-List with a Sparse Approximation](https://mlsanity.com/paper/2304.12631/) (2023)
- [FaxPlainAC: A Fact-Checking Tool Based on EXPLAINable Models with HumAn Correction in the Loop](https://mlsanity.com/paper/2110.10144/) (2021)
- [RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank](https://mlsanity.com/paper/2405.01848/) (2024)
- [ShaRP: Explaining Rankings and Preferences with Shapley Values](https://mlsanity.com/paper/2401.16744/) (2024)
- [ExaRanker: Explanation-Augmented Neural Ranker](https://mlsanity.com/paper/2301.10521/) (2023)
- [AdaptGrad: Adaptive Sampling to Reduce Noise](https://mlsanity.com/paper/2410.07711/) (2024)
- [Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval](https://mlsanity.com/paper/2410.18527/) (2024)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)
- [Axiomatization of Gradient Smoothing in Neural Networks](https://mlsanity.com/paper/2407.00371/) (2024)
- [Quantifying Retriever-Generator Alignment in RAG with Local Explanations](https://mlsanity.com/paper/2601.21803/) (2026)
- [Enhancing AI Interpretability with Localised Architectures](https://mlsanity.com/paper/2606.07998/) (2026)
- [A unified framework for global and local interpretability using adaptive derivative-ordered random explanation](https://mlsanity.com/paper/2609.17171/) (2026)
- [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026)

## Abstract (arXiv)

A recent trend in IR has been the usage of neural networks to learn retrieval models for text based adhoc search. While various approaches and architectures have yielded significantly better performance than traditional retrieval models such as BM25, it is still difficult to understand exactly why a document is relevant to a query. In the ML community several approaches for explaining decisions made by deep neural networks have been proposed -- including DeepSHAP which modifies the DeepLift algorithm to estimate the relative importance (shapley values) of input features for a given decision by comparing the activations in the network for a given image against the activations caused by a reference input. In image classification, the reference input tends to be a plain black image. While DeepSHAP has been well studied for image classification tasks, it remains to be seen how we can adapt it to explain the output of Neural Retrieval Models (NRMs). In particular, what is a good "black" image in the context of IR? In this paper we explored various reference input document construction techniques. Additionally, we compared the explanations generated by DeepSHAP to LIME (a model agnostic approach) and found that the explanations differ considerably. Our study raises concerns regarding the robustness and accuracy of explanations produced for NRMs. With this paper we aim to shed light on interesting problems surrounding interpretability in NRMs and highlight areas of future work.