# Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models

arXiv: https://arxiv.org/abs/2506.02132 · HTML: https://mlsanity.com/paper/2506.02132/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 58 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs](https://mlsanity.com/paper/2403.17299/) (2024)
- [The Dual-Route Model of Induction](https://mlsanity.com/paper/2504.03022/) (2025)
- [Can Unconditional Language Models Recover Arbitrary Sentences?](https://mlsanity.com/paper/1907.04944/) (2019)
- [Outlier Dimensions Encode Task-Specific Knowledge](https://mlsanity.com/paper/2310.17715/) (2023)
- [MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools](https://mlsanity.com/paper/2504.20168/) (2025)

## Built on by (most cited first, 5 of 5)

- [LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization](https://mlsanity.com/paper/2510.04013/) (2025)
- [Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations](https://mlsanity.com/paper/2601.03779/) (2026)
- [Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion](https://mlsanity.com/paper/2604.16656/) (2026)
- [Capsule Lens: Locating and Tracking Concept Geometry in Model Representations](https://mlsanity.com/paper/2609.05575/) (2026)
- [Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings](https://mlsanity.com/paper/2609.28290/) (2026)

## Abstract (arXiv)

Large transformer-based language models dominate modern NLP, yet our understanding of how they encode linguistic information relies primarily on studies of early models like BERT and GPT-2. We systematically probe 25 models from BERT Base to Qwen2.5-7B focusing on two linguistic properties: lexical identity and inflectional features across 6 diverse languages. We find a consistent pattern: inflectional features are linearly decodable throughout the model, while lexical identity is prominent early but increasingly weakens with depth. Further analysis of the representation geometry reveals that models with aggressive mid-layer dimensionality compression show reduced steering effectiveness in those layers, despite probe accuracy remaining high. Pretraining analysis shows that inflectional structure stabilizes early while lexical identity representations continue evolving. Taken together, our findings suggest that transformers maintain inflectional features across layers, while trading off lexical identity for compact, predictive representations. Our code is available at https://github.com/ml5885/model_internal_sleuthing