# Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models

arXiv: https://arxiv.org/abs/2505.12509 · HTML: https://mlsanity.com/paper/2505.12509/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Automatic Detection of Fake News](https://mlsanity.com/paper/1708.07104/) (2017)
- [FastSHAP: Real-Time Shapley Value Estimation](https://mlsanity.com/paper/2107.07436/) (2021)
- [Sampling Permutations for Shapley Value Estimation](https://mlsanity.com/paper/2104.12199/) (2021)
- [Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey](https://mlsanity.com/paper/2104.00950/) (2021)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- [The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations](https://mlsanity.com/paper/2205.03295/) (2022)
- [groupShapley: Efficient prediction explanation with Shapley values for feature groups](https://mlsanity.com/paper/2106.12228/) (2021)
- [Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution](https://mlsanity.com/paper/2401.15866/) (2024)
- [Data Poisoning for In-context Learning](https://mlsanity.com/paper/2402.02160/) (2024)
- [CoRTX: Contrastive Framework for Real-time Explanation](https://mlsanity.com/paper/2303.02794/) (2023)
- [TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents](https://mlsanity.com/paper/2312.01279/) (2023)
- [Efficient XAI Techniques: A Taxonomic Survey](https://mlsanity.com/paper/2302.03225/) (2023)
- [Multi-Level Explanations for Generative Language Models](https://mlsanity.com/paper/2403.14459/) (2024)
- [AcME -- Accelerated Model-agnostic Explanations: Fast Whitening of the Machine-Learning Black Box](https://mlsanity.com/paper/2112.12635/) (2021)
- [Word Importance Explains How Prompts Affect Language Model Outputs](https://mlsanity.com/paper/2403.03028/) (2024)
- [Interactive Prompt Debugging with Sequence Salience](https://mlsanity.com/paper/2404.07498/) (2024)
- [ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniques](https://mlsanity.com/paper/2209.03798/) (2022)
- [MAnchors: Memorization-Based Acceleration of Anchors via Rule Reuse and Transformation](https://mlsanity.com/paper/2502.11068/) (2025)

## Built on by (most cited first, 2 of 2)

- [Explaining Large Language Models with gSMILE](https://mlsanity.com/paper/2505.21657/) (2025)
- [Focus-LIME: Surgical Interpretation of Long-Context Large Language Models via Proxy-Based Neighborhood Selection](https://mlsanity.com/paper/2602.04607/) (2026)

## Abstract (arXiv)

Post-hoc explanations provide transparency and are essential for guiding model optimization, such as prompt engineering and data sanitation. However, applying model-agnostic techniques to Large Language Models (LLMs) is hindered by prohibitive computational costs, rendering these tools dormant for real-world applications. To revitalize model-agnostic interpretability, we propose a budget-friendly proxy framework that leverages efficient models to approximate the decision boundaries of expensive LLMs. We introduce a screen-and-apply mechanism to statistically verify local alignment before deployment. Our empirical evaluation confirms that proxy explanations achieve over 90% fidelity with only 11% of the oracle's cost. Building on this foundation, we demonstrate the actionable utility of our framework in prompt compression and poisoned example removal. Results show that reliable proxy explanations effectively guide optimization, transforming interpretability from a passive observation tool into a scalable primitive for LLM development. Additionally, we open-source code and datasets to facilitate future research.