mlsanityData to 14 Oct 2026

Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures

arXiv 2604.16042 ↗First posted 17 Apr 2026cs.CL cs.AI cs.LGcited by 4

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

While Large Language Models (LLMs) have achieved strong performance across many NLP tasks, their opaque internal mechanisms hinder trustworthiness and safe deployment. Existing surveys in explainable AI largely focus on post-hoc explanation methods that interpret trained models through external approximations. In contrast, intrinsic interpretability, which builds transparency directly into model architectures and computations, has recently emerged as a promising alternative. This paper presents a systematic review of the recent advances in intrinsic interpretability for LLMs, categorizing existing approaches into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction. We further discuss open challenges and outline future research directions in this emerging field. The paper list is available at: https://github.com/PKU-PILLAR-Group/Survey-Intrinsic-Interpretability-of-LLMs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 39 of its 68 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

39 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Character-level Chinese Backpack Language Models19 Oct

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
26 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  2. 2020Concept Bottleneck Modelscited by 1,032
  3. 2016Language Modeling with Gated Convolutional Networkscited by 702
  4. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  5. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  6. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  7. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 246
  8. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  9. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  10. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  11. 2021Promises and Pitfalls of Black-Box Concept Learning Modelscited by 131
  12. 2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
  13. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  14. 2020GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactionscited by 82
  15. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 48
  16. 2025Weight-sparse transformers have interpretable circuitscited by 44
  17. 2025Advancing Expert Specialization for Better MoEcited by 39
  18. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  19. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  20. 2024Bilinear MLPs enable weight-based mechanistic interpretabilitycited by 28
  21. 2025Mixture of Experts Made Intrinsically Interpretablecited by 24
  22. 2024Kolmogorov-Arnold Networks: A Critical Assessment of Claims, Performance, and Practical Viabilitycited by 23
  23. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  24. 2023Backpack Language Modelscited by 15
  25. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 13
  26. 2025Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoderscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Sparsely gated tiny linear expertscited by 0
  2. 2026Faithful by Definition: Emotion Analysis via Natural Semantic Metalanguage Explicationscited by 0
  3. 2026RepBench: Compiling Benchmarks into Capability Representations for Large Language Modelscited by 0
  4. 2026Integrating Feature Understandability into Feature Selection for Human-Centred Tabular Explanationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  2. 2025Towards Transparent AI: A Survey on Explainable Large Language Models
  3. 2023Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models
  4. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspective
  5. 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
  6. 2024Rethinking Interpretability in the Era of Large Language Models
  7. 2025BELL: Benchmarking the Explainability of Large Language Models
  8. 2022Towards Faithful Model Explanation in NLP: A Survey