# Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval

arXiv: https://arxiv.org/abs/2410.18527 · HTML: https://mlsanity.com/paper/2410.18527/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 30 of 34)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Object Detectors Emerge in Deep Scene CNNs](https://mlsanity.com/paper/1412.6856/) (2014)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching](https://mlsanity.com/paper/2311.17030/) (2023)
- [Revisiting the Importance of Individual Units in CNNs via Ablation](https://mlsanity.com/paper/1806.02891/) (2018)
- [Is Sparse Attention more Interpretable?](https://mlsanity.com/paper/2106.01087/) (2021)
- [ABNIRML: Analyzing the Behavior of Neural IR Models](https://mlsanity.com/paper/2011.00696/) (2020)
- [A study on the Interpretability of Neural Retrieval Models using DeepSHAP](https://mlsanity.com/paper/1907.06484/) (2019)
- [EXS: Explainable Search Using Local Model Agnostic Interpretability](https://mlsanity.com/paper/1809.03857/) (2018)
- [A White Box Analysis of ColBERT](https://mlsanity.com/paper/2012.09650/) (2020)
- [Match Your Words! A Study of Lexical Matching in Neural Information Retrieval](https://mlsanity.com/paper/2112.05662/) (2021)
- [Axiomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models](https://mlsanity.com/paper/2405.02503/) (2024)

## Built on by (most cited first, 2 of 2)

- [Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs](https://mlsanity.com/paper/2509.23684/) (2025)
- [The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning](https://mlsanity.com/paper/2608.23338/) (2026)

## Abstract (arXiv)

Transformer networks, particularly those achieving performance comparable to GPT models, are well known for their robust feature extraction abilities. However, the nature of these extracted features and their alignment with human-engineered ones remain unexplored. In this work, we investigate the internal mechanisms of state-of-the-art, fine-tuned LLMs for passage reranking. We employ a probing-based analysis to examine neuron activations in ranking LLMs, identifying the presence of known human-engineered and semantic features. Our study spans a broad range of feature categories, including lexical signals, document structure, query-document interactions, and complex semantic representations, to uncover underlying patterns influencing ranking decisions. Through experiments on four different ranking LLMs, we identify statistical IR features that are prominently encoded in LLM activations, as well as others that are notably missing. Furthermore, we analyze how these models respond to out-of-distribution queries and documents, revealing distinct generalization behaviors. By dissecting the latent representations within LLM activations, we aim to improve both the interpretability and effectiveness of ranking models. Our findings offer crucial insights for developing more transparent and reliable retrieval systems, and we release all necessary scripts and code to support further exploration.