mlsanityData to 14 Oct 2026

RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedback

arXiv 2508.05512 ↗First posted 7 Aug 2025cs.IRcited by 5

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Evaluating the quality of retrieval-augmented generation (RAG) and document reranking systems remains challenging due to the lack of scalable, user-centric, and multi-perspective evaluation tools. We introduce RankArena, a unified platform for comparing and analysing the performance of retrieval pipelines, rerankers, and RAG systems using structured human and LLM-based feedback as well as for collecting such feedback. RankArena supports multiple evaluation modes: direct reranking visualisation, blind pairwise comparisons with human or LLM voting, supervised manual document annotation, and end-to-end RAG answer quality assessment. It captures fine-grained relevance feedback through both pairwise preferences and full-list annotations, along with auxiliary metadata such as movement metrics, annotation time, and quality ratings. The platform also integrates LLM-as-a-judge evaluation, enabling comparison between model-generated rankings and human ground truth annotations. All interactions are stored as structured evaluation datasets that can be used to train rerankers, reward models, judgment agents, or retrieval strategy selectors. Our platform is publicly available at https://rankarena.ngrok.io/, and the Demo video is provided https://youtu.be/jIYAP4PaSSI.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 48 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

34 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers3 Oct
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 32

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  5. 2024A Survey on LLM-as-a-Judgecited by 1,423
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  10. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  12. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
20 more
  1. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  2. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  3. 2019Multi-Stage Document Ranking with BERTcited by 201
  4. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178
  5. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 165
  6. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  7. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  8. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  9. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  10. 2024ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrievalcited by 30
  11. 2023Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Modelscited by 26
  12. 2024TWOLAR: a TWO-step LLM-Augmented distillation method for passage Rerankingcited by 11
  13. 2025A Study into Investigating Temporal Robustness of LLMscited by 11
  14. 2025ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrievalcited by 9
  15. 2024InRanker: Distilled Rankers for Zero-shot Information Retrievalcited by 8
  16. 2024Exploring Hint Generation Approaches in Open-Domain Question Answeringcited by 7
  17. 2024An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Rerankingcited by 4
  18. 2024DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classificationcited by 2
  19. 2025From Retrieval to Generation: Comparing Different Approachescited by 2
  20. 2024Detecting Temporal Ambiguity in Questionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillationcited by 11
  2. 2025How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Modelscited by 4
  3. 2026Rethinking RAG in Long Videos: What to Retrieve and How to Use It?cited by 1
  4. 2025SPAR: Session-based Pipeline for Adaptive Retrieval on Legacy File Systemscited by 0
  5. 2026Coordinated Semantic Alignment and Evidence Constraints for Retrieval-Augmented Generation with Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-time Corpus Feedback: From Retrieval to RAG
  2. 2026Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?
  3. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
  4. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking
  5. 2025RankLLM: A Python Package for Reranking with LLMs
  6. 2025Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion
  7. 2026LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAG
  8. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems