# CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring

arXiv: https://arxiv.org/abs/2606.11700 · HTML: https://mlsanity.com/paper/2606.11700/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models](https://mlsanity.com/paper/2309.15088/) (2023)
- [A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models](https://mlsanity.com/paper/2310.09497/) (2023)
- [FIRST: Faster Improved Listwise Reranking with Single Token Decoding](https://mlsanity.com/paper/2406.15657/) (2024)
- [Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers](https://mlsanity.com/paper/2410.02642/) (2024)
- [Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking](https://mlsanity.com/paper/2506.09944/) (2025)
- [Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models](https://mlsanity.com/paper/2406.14848/) (2024)
- [RankLLM: A Python Package for Reranking with LLMs](https://mlsanity.com/paper/2505.19284/) (2025)
- [Tools are under-documented: Simple Document Expansion Boosts Tool Retrieval](https://mlsanity.com/paper/2510.22670/) (2025)
- [MultiConIR: Towards multi-condition Information Retrieval](https://mlsanity.com/paper/2503.08046/) (2025)
- [HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse](https://mlsanity.com/paper/2504.02921/) (2025)
- [Scalable In-context Ranking with Generative Models](https://mlsanity.com/paper/2510.05396/) (2025)
- [MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models](https://mlsanity.com/paper/2604.23321/) (2026)
- [Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short](https://mlsanity.com/paper/2510.08985/) (2025)
- [Efficient Long-Document Reranking via Block-Level Embeddings and Top-k Interaction Refinement](https://mlsanity.com/paper/2501.17039/) (2025)
- [HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads](https://mlsanity.com/paper/2604.17237/) (2026)
- [When Should Queries Be Decomposed? A Stage-Aware Study of Query Decomposition for Multi-Condition Retrieval](https://mlsanity.com/paper/2606.08577/) (2026)
- [miniReranker: Efficient Multimodal Reranking through Visual Cache Reuse and Interaction Sparsity](https://mlsanity.com/paper/2606.10759/) (2026)

## Abstract (arXiv)

Large language model (LLM) rerankers have become an important component of modern retrieval and retrieval-augmented generation pipelines, but their high computational cost limits their applicability to long candidate lists. In this paper, we propose CompRank, a token-efficient reranking framework that reduces redundant computation by aligning reranker design with the sparsity of ranking signals. CompRank decouples document representations from candidate order and query context, enabling reusable document-side states; applies segment-wise token compression to reduce query--document interaction cost; and introduces a CopyNet-style objective that directly aligns attention-based document scoring with training supervision. Experiments on seven BEIR datasets show that CompRank achieves strong reranking performance while retaining only 10.2% of document tokens, reaching an average NDCG@10 of 39.2 compared with 39.7 under full-token attention. Further scaling experiments on TREC-COVID show that CompRank remains stable when evaluated on candidate lists of up to 500 documents after training on 30-document lists, while achieving 4.9×--9.5× end-to-end speedup over generation-based listwise reranking and approximately 1.3× speedup over the full-token CompRank variant. These results suggest that token-level compression and decoding-free attention scoring provide an effective path toward scalable LLM-based reranking.