# LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations

arXiv: https://arxiv.org/abs/2401.12576 · HTML: https://mlsanity.com/paper/2401.12576/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 62 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 30 of 45)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [Petals: Collaborative Inference and Fine-tuning of Large Models](https://mlsanity.com/paper/2209.01188/) (2022)

## Built on by (most cited first, 16 of 16)

- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty](https://mlsanity.com/paper/2412.06771/) (2024)
- [Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification](https://mlsanity.com/paper/2510.11408/) (2025)
- [Large Language Models Are Cross-Lingual Knowledge-Free Reasoners](https://mlsanity.com/paper/2406.16655/) (2024)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem](https://mlsanity.com/paper/2409.07123/) (2024)
- [Interactive Prompt Debugging with Sequence Salience](https://mlsanity.com/paper/2404.07498/) (2024)
- [FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation](https://mlsanity.com/paper/2501.00777/) (2025)
- [CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systems](https://mlsanity.com/paper/2406.08101/) (2024)
- [Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals](https://mlsanity.com/paper/2505.13972/) (2025)
- [Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanations](https://mlsanity.com/paper/2601.00282/) (2026)
- [Visual-Conversational Interface for Evidence-Based Explanation of Diabetes Risk Prediction](https://mlsanity.com/paper/2507.02920/) (2025)
- [Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems](https://mlsanity.com/paper/2508.14982/) (2025)
- [Interpretative Interfaces: Designing for AI-Mediated Reading Practices and the Knowledge Commons](https://mlsanity.com/paper/2603.15863/) (2026)
- [Persistent and Conversational Multi-Method Explainability for Trustworthy Financial AI](https://mlsanity.com/paper/2605.11687/) (2026)
- [Explainability Assistant: A Conversational XAI Interface for Interpreting Energy Consumption Models](https://mlsanity.com/paper/2609.11860/) (2026)

## Abstract (arXiv)

Interpretability tools that offer explanations in the form of a dialogue have demonstrated their efficacy in enhancing users' understanding (Slack et al., 2023; Shen et al., 2023), as one-off explanations may fall short in providing sufficient information to the user. Current solutions for dialogue-based explanations, however, often require external tools and modules and are not easily transferable to tasks they were not designed for. With LLMCheckup, we present an easily accessible tool that allows users to chat with any state-of-the-art large language model (LLM) about its behavior. We enable LLMs to generate explanations and perform user intent recognition without fine-tuning, by connecting them with a broad spectrum of Explainable AI (XAI) methods, including white-box explainability tools such as feature attributions, and self-explanations (e.g., for rationale generation). LLM-based (self-)explanations are presented as an interactive dialogue that supports follow-up questions and generates suggestions. LLMCheckupprovides tutorials for operations available in the system, catering to individuals with varying levels of expertise in XAI and supporting multiple input modalities. We introduce a new parsing strategy that substantially enhances the user intent recognition accuracy of the LLM. Finally, we showcase LLMCheckup for the tasks of fact checking and commonsense question answering.