# Don't "Overthink" Passage Reranking: Is Reasoning Truly Necessary?

arXiv: https://arxiv.org/abs/2505.16886 · HTML: https://mlsanity.com/paper/2505.16886/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [TokenSkip: Controllable Chain-of-Thought Compression in LLMs](https://mlsanity.com/paper/2502.12067/) (2025)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [Distilling System 2 into System 1](https://mlsanity.com/paper/2407.06023/) (2024)
- [Self-Training Elicits Concise Reasoning in Large Language Models](https://mlsanity.com/paper/2502.20122/) (2025)
- [ReasonIR: Training Retrievers for Reasoning Tasks](https://mlsanity.com/paper/2504.20595/) (2025)
- [Rank1: Test-Time Compute for Reranking in Information Retrieval](https://mlsanity.com/paper/2502.18418/) (2025)
- [Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning](https://mlsanity.com/paper/2503.06034/) (2025)
- [O1 Embedder: Let Retrievers Think Before Action](https://mlsanity.com/paper/2502.07555/) (2025)
- [Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking](https://mlsanity.com/paper/2502.12974/) (2025)

## Built on by (most cited first, 7 of 7)

- [Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short](https://mlsanity.com/paper/2510.08985/) (2025)
- [LELA: an LLM-based Entity Linking Approach with Zero-Shot Domain Adaptation](https://mlsanity.com/paper/2601.05192/) (2026)
- [TFRank: Think-Free Reasoning Enables Practical Pointwise LLM Ranking](https://mlsanity.com/paper/2508.09539/) (2025)
- [Investigating Thinking Behaviours of Reasoning-Based Language Models for Social Bias Mitigation](https://mlsanity.com/paper/2510.17062/) (2025)
- [Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval](https://mlsanity.com/paper/2604.23734/) (2026)
- [TabRank: Chain-of-Thought Distillation for Table Re-Rankers](https://mlsanity.com/paper/2607.25182/) (2026)
- [Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking](https://mlsanity.com/paper/2608.30398/) (2026)

## Abstract (arXiv)

With the growing success of reasoning models across complex natural language tasks, researchers in the Information Retrieval (IR) community have begun exploring how similar reasoning capabilities can be integrated into passage rerankers built on Large Language Models (LLMs). These methods typically employ an LLM to produce an explicit, step-by-step reasoning process before arriving at a final relevance prediction. But, does reasoning actually improve reranking accuracy? In this paper, we dive deeper into this question, studying the impact of the reasoning process by comparing reasoning-based pointwise rerankers (ReasonRR) to standard, non-reasoning pointwise rerankers (StandardRR) under identical training conditions, and observe that StandardRR generally outperforms ReasonRR. Building on this observation, we then study the importance of reasoning to ReasonRR by disabling its reasoning process (ReasonRR-NoReason), and find that ReasonRR-NoReason is surprisingly more effective than ReasonRR. Examining the cause of this result, our findings reveal that reasoning-based rerankers are limited by the LLM's reasoning process, which pushes it toward polarized relevance scores and thus fails to consider the partial relevance of passages, a key factor for the accuracy of pointwise rerankers.