mlsanityData to 7 Oct 2026

A Theoretical Analysis of NDCG Type Ranking Measures

arXiv 1304.6480 ↗First posted 24 Apr 2013cs.LG cs.IR stat.MLcited by 119

Not on the map. Why?

Cited by 20 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A central problem in ranking is to design a ranking measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the widely used Normalized Discounted Cumulative Gain (NDCG)-type ranking measures. Although there are extensive empirical studies of NDCG, little is known about its theoretical properties. We first show that, whatever the ranking function is, the standard NDCG which adopts a logarithmic discount, converges to 1 as the number of items to rank goes to infinity. On the first sight, this result is very surprising. It seems to imply that NDCG cannot differentiate good and bad ranking functions, contradicting to the empirical success of NDCG in many applications. In order to have a deeper understanding of ranking measures in general, we propose a notion referred to as consistent distinguishability. This notion captures the intuition that a ranking measure should have such a property: For every pair of substantially different ranking functions, the ranking measure can decide which one is better in a consistent manner on almost all datasets. We show that NDCG with logarithmic discount has consistent distinguishability although it converges to the same limit for all ranking functions. We next characterize the set of all feasible discount functions for NDCG according to the concept of consistent distinguishability. Specifically we show that whether NDCG has consistent distinguishability depends on how fast the discount decays, and 1/r is a critical point. We then turn to the cut-off version of NDCG, i.e., NDCG@k. We analyze the distinguishability of NDCG@k for various choices of k and the discount functions. Experimental results on real Web search datasets agree well with the theory.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Apr 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 5; 2023: 14; 2024: 28; 2025: 32; 2026: 29.

119 papers.

Sort:

By area:

  1. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  2. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrieverscited by 83
  3. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  4. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  5. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  6. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  7. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 49
  8. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 44
  9. 2021Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similaritycited by 35
  10. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  11. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  12. 2025Region-Adaptive Sampling for Diffusion Transformerscited by 31
38 more
  1. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  2. 2022Unsupervised Model Selection for Time-series Anomaly Detectioncited by 24
  3. 2024LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendationscited by 24
  4. 2023BookGPT: A General Framework for Book Recommendation Empowered by Large Language Modelcited by 22
  5. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  6. 2021CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Examplecited by 19
  7. 2021Boosting Search Engines with Interactive Agentscited by 19
  8. 2024C-FedRAG: A Confidential Federated Retrieval-Augmented Generation Systemcited by 19
  9. 2022Rethinking Knowledge Graph Evaluation Under the Open-World Assumptioncited by 17
  10. 2024Optimal Design for Human Preference Elicitationcited by 17
  11. 2026MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalizationcited by 15
  12. 2023A Survey on Fairness-aware Recommender Systemscited by 14
  13. 2025Tool-integrated Reinforcement Learning for Repo Deep Searchcited by 14
  14. 2023Unsupervised Contrast-Consistent Ranking with Language Modelscited by 13
  15. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  16. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  17. 2021One Billion Audio Sounds from GPU-enabled Modular Synthesiscited by 11
  18. 2024CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agentscited by 11
  19. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  20. 2025Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systemscited by 9
  21. 2025AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Miningcited by 9
  22. 2021AcME -- Accelerated Model-agnostic Explanations: Fast Whitening of the Machine-Learning Black Boxcited by 8
  23. 2024PhysBERT: A Text Embedding Model for Physics Scientific Literaturecited by 8
  24. 2025RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understandingcited by 8
  25. 2024Better RAG using Relevant Information Gaincited by 7
  26. 2023Sketch-based Medical Image Retrievalcited by 6
  27. 2023Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systemscited by 6
  28. 2024INDUS: Effective and Efficient Language Models for Scientific Applicationscited by 6
  29. 2025RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practicecited by 6
  30. 2026Structural Anchor Pruning: Training-Free Multi-Vector Compression for Visual Document Retrievalcited by 6
  31. 2022Simpson's Paradox in Recommender Fairness: Reconciling differences between per-user and aggregated evaluationscited by 5
  32. 2023Policy-Gradient Training of Language Models for Rankingcited by 5
  33. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Frameworkcited by 5
  34. 2023Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabiccited by 4
  35. 2024DEXTER: A Benchmark for open-domain Complex Question Answering using LLMscited by 4
  36. 2025Are Bias Evaluation Methods Biased ?cited by 4
  37. 2025Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Searchcited by 4
  38. 2021Least Square Calibration for Peer Reviewcited by 3

50 most cited of 119 All 119 (JSON)