# How Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs

arXiv: https://arxiv.org/abs/2504.08780 · HTML: https://mlsanity.com/paper/2504.08780/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 5 Apr 2025.

## Builds on (most cited first, 4 of 4)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)

## Built on by (most cited first, 3 of 3)

- [Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations](https://mlsanity.com/paper/2508.17032/) (2025)
- [Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs](https://mlsanity.com/paper/2509.23684/) (2025)
- [The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning](https://mlsanity.com/paper/2608.23338/) (2026)

## Abstract (arXiv)

We conduct a behavioral exploration of LoRA fine-tuned LLMs for Passage Reranking to understand how relevance signals are learned and deployed by Large Language Models. By fine-tuning Mistral-7B, LLaMA3.1-8B, and Pythia-6.9B on MS MARCO under diverse LoRA configurations, we investigate how relevance modeling evolves across checkpoints, the impact of LoRA rank (1, 2, 8, 32), and the relative importance of updated MHA vs. MLP components. Our ablations reveal which layers and projections within LoRA transformations are most critical for reranking accuracy. These findings offer fresh explanations into LoRA's adaptation mechanisms, setting the stage for deeper mechanistic studies in Information Retrieval. All models used in this study have been shared.