# Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models

arXiv: https://arxiv.org/abs/2311.01732 · HTML: https://mlsanity.com/paper/2311.01732/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Drug discovery with explainable artificial intelligence](https://mlsanity.com/paper/2007.00523/) (2020)
- [Evaluating Explanations: How much do explanations from the teacher aid students?](https://mlsanity.com/paper/2012.00893/) (2020)
- [Aligning Faithful Interpretations with their Social Attribution](https://mlsanity.com/paper/2006.01067/) (2020)
- [Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs](https://mlsanity.com/paper/1801.05453/) (2018)
- [Interpretable Image Recognition with Hierarchical Prototypes](https://mlsanity.com/paper/1906.10651/) (2019)
- [Framework for Evaluating Faithfulness of Local Explanations](https://mlsanity.com/paper/2202.00734/) (2022)
- [UNIREX: A Unified Learning Framework for Language Model Rationale Extraction](https://mlsanity.com/paper/2112.08802/) (2021)
- [Self-Explaining Structures Improve NLP Models](https://mlsanity.com/paper/2012.01786/) (2020)
- [ProtoTEx: Explaining Model Decisions with Prototype Tensors](https://mlsanity.com/paper/2204.05426/) (2022)
- [Learning to Scaffold: Optimizing Model Explanations for Teaching](https://mlsanity.com/paper/2204.10810/) (2022)

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages](https://mlsanity.com/paper/2506.07249/) (2025)
- [Protocode: Prototype-Driven Interpretability for Code Generation in LLMs](https://mlsanity.com/paper/2509.25247/) (2025)
- [The Mysterious Case of Neuron 1512: Injectable Realignment Architectures Reveal Internal Characteristics of Meta's Llama 2 Model](https://mlsanity.com/paper/2407.03621/) (2024)
- [ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classification](https://mlsanity.com/paper/2510.12534/) (2025)
- [ProtoTopic: Prototypical Network for Few-Shot Medical Topic Modeling](https://mlsanity.com/paper/2510.13542/) (2025)
- [Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learning](https://mlsanity.com/paper/2603.27971/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), but their lack of interpretability has been a major concern. Current methods for interpreting LLMs are post hoc, applied after inference time, and have limitations such as their focus on low-level features and lack of explainability at higher level text units. In this work, we introduce proto-lm, a prototypical network-based white-box framework that allows LLMs to learn immediately interpretable embeddings during the fine-tuning stage while maintaining competitive performance. Our method's applicability and interpretability are demonstrated through experiments on a wide range of NLP tasks, and our results indicate a new possibility of creating interpretable models without sacrificing performance. This novel approach to interpretability in LLMs can pave the way for more interpretable models without the need to sacrifice performance.