mlsanityData to 7 Oct 2026

Rethinking Interpretability in the Era of Large Language Models

arXiv 2402.01761 ↗First posted 30 Jan 2024cs.CL cs.AI cs.LGcited by 110

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Interpretable machine learning has exploded as an area of interest over the last decade, sparked by the rise of increasingly large datasets and deep neural networks. Simultaneously, large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks, offering a chance to rethink opportunities in interpretable machine learning. Notably, the capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. However, these new capabilities raise new challenges, such as hallucinated explanations and immense computational costs. In this position paper, we start by reviewing existing methods to evaluate the emerging field of LLM interpretation (both interpreting LLMs and using LLMs for explanation). We contend that, despite their limitations, LLMs hold the opportunity to redefine interpretability with a more ambitious scope across many applications, including in auditing LLMs themselves. We highlight two emerging research priorities for LLM interpretation: using LLMs to directly analyze new datasets and to generate interactive explanations.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 122 of its 144 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

122 of 144 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Measuring and Narrowing the Compositionality Gap in Language Models7 Oct
  2. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning6 May

Other references · 120

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  5. 2018Sanity Checks for Saliency Mapscited by 1,149
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2022In-context Learning and Induction Headscited by 930
  11. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  13. 2022Galactica: A Large Language Model for Sciencecited by 736
  14. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  15. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  16. 2019Attention is not not Explanationcited by 629
  17. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  18. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  19. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  20. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  21. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  22. 2020Artificial Intelligence, Values and Alignmentcited by 443
  23. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  24. 2023Larger language models do in-context learning differentlycited by 386
  25. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  26. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  27. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  28. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  29. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  30. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 343
  31. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  32. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  33. 2023Explainability for Large Language Models: A Surveycited by 331
  34. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  35. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  36. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
  37. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  38. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248

50 most cited of 120 All 120 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 39; 2026: 27.

110 papers.

Sort:

By area:

  1. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  2. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  3. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  4. 2024Language Repository for Long Video Understandingcited by 55
  5. 2024On Large Language Model Continual Unlearningcited by 51
  6. 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 45
  7. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  8. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  9. 2025Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Modelscited by 29
  10. 2024Perturbation-Restrained Sequential Model Editingcited by 27
  11. 2024Finding Visual Task Vectorscited by 24
  12. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
38 more
  1. 2024Materials science in the era of large language models: a perspectivecited by 23
  2. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
  3. 2025From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaningcited by 23
  4. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  5. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  6. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  7. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  8. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  9. 2024A Survey on LLM Inference-Time Self-Improvementcited by 17
  10. 2024A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoningcited by 15
  11. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 15
  12. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  13. 2026Controllable LLM Reasoning via Sparse Autoencoder-Based Steeringcited by 12
  14. 2024Crafting Interpretable Embeddings by Asking LLMs Questionscited by 11
  15. 2025LLM-Based Social Simulations Require a Boundarycited by 11
  16. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  17. 2024Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Modelscited by 9
  18. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  19. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  20. 2025GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generationcited by 9
  21. 2024Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Surveycited by 8
  22. 2024Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaborationcited by 8
  23. 2024Large Language Models for Combinatorial Optimization of Design Structure Matrixcited by 8
  24. 2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMscited by 8
  25. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
  26. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  27. 2024Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approachcited by 6
  28. 2024PromptExp: Multi-granularity Prompt Explanation of Large Language Modelscited by 6
  29. 2024Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracycited by 5
  30. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  31. 2025Layerwise Recall and the Geometry of Interwoven Knowledge in LLMscited by 5
  32. 2025KnowThyself: An Agentic Assistant for LLM Interpretabilitycited by 5
  33. 2025First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimationcited by 5
  34. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  35. 2026Do explanations generalize across large reasoning models?cited by 5
  36. 2024Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summariescited by 4
  37. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  38. 2025Towards Understanding Graphical Perception in Large Multimodal Modelscited by 4

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspective
  2. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  3. 2024LLMs' Understanding of Natural Language Revealed
  4. 2025Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models
  5. 2023In-Context Explainers: Harnessing LLMs for Explaining Black Box Models
  6. 2026Understanding Large Language Models
  7. 2023XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs
  8. 2026Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions