# RewardRank: Optimizing True Learning-to-Rank Utility

arXiv: https://arxiv.org/abs/2508.14180 · HTML: https://mlsanity.com/paper/2508.14180/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Stochastic Optimization of Sorting Networks via Continuous Relaxations](https://mlsanity.com/paper/1903.08850/) (2019)
- [SoftSort: A Continuous Relaxation for the argsort Operator](https://mlsanity.com/paper/2006.16038/) (2020)
- [Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search](https://mlsanity.com/paper/2206.06588/) (2022)
- [NeuralNDCG: Direct Optimisation of a Ranking Metric via Differentiable Relaxation of Sorting](https://mlsanity.com/paper/2102.07831/) (2021)
- [Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness](https://mlsanity.com/paper/2105.00855/) (2021)
- [PiRank: Scalable Learning To Rank via Differentiable Sorting](https://mlsanity.com/paper/2012.06731/) (2020)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Non-autoregressive Generative Models for Reranking Recommendation](https://mlsanity.com/paper/2402.06871/) (2024)
- [Revisit Recommender System in the Permutation Prospective](https://mlsanity.com/paper/2102.12057/) (2021)
- [A Large Scale Search Dataset for Unbiased Learning to Rank](https://mlsanity.com/paper/2207.03051/) (2022)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [PIER: Permutation-Level Interest-Based End-to-End Re-ranking Framework in E-commerce](https://mlsanity.com/paper/2302.03487/) (2023)
- [Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank](https://mlsanity.com/paper/2407.19943/) (2024)
- [Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability](https://mlsanity.com/paper/2403.18462/) (2024)
- [Policy-Gradient Training of Language Models for Ranking](https://mlsanity.com/paper/2310.04407/) (2023)
- [NLGR: Utilizing Neighbor Lists for Generative Rerank in Personalized Recommendation Systems](https://mlsanity.com/paper/2502.06097/) (2025)
- [Reducing Popularity Influence by Addressing Position Bias](https://mlsanity.com/paper/2412.08780/) (2024)
- [Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity](https://mlsanity.com/paper/2308.00177/) (2023)
- [Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank](https://mlsanity.com/paper/2409.09881/) (2024)

## Abstract (arXiv)

Traditional ranking systems optimize offline proxy objectives that rely on oversimplified assumptions about user behavior, often neglecting factors such as position bias and item diversity. Consequently, these models fail to improve true counterfactual utilities such as such as click-through rate or purchase probability, when evaluated in online A/B tests. We introduce RewardRank, a data-driven learning-to-rank (LTR) framework for counterfactual utility maximization. RewardRank first learns a reward model that predicts the utility of any ranking directly from logged user interactions, and then trains a ranker to maximize this reward using a differentiable soft permutation operator. To enable rigorous and reproducible evaluation, we further propose two benchmark suites: (i) Parametric Oracle Evaluation (PO-Eval), which employs an open-source click model as a counterfactual oracle on the Baidu-ULTR dataset, and (ii) LLM-as-User Evaluation (LAU-Eval), which simulates realistic user behavior via large language models on the Amazon-KDD-Cup dataset. RewardRank achieves the highest counterfactual utility across both benchmarks and demonstrates that optimizing classical metrics such as NDCG is sub-optimal for maximizing true user utility. Finally, using real user feedback from the Baidu-ULTR dataset, RewardRank establishes a new state of the art in offline relevance performance. Overall, our results show that learning-to-rank can be reformulated as direct optimization of counterfactual utility, achieved in a purely data-driven manner without relying on explicit modeling assumptions such as position bias. Our code is available at: https://github.com/GauravBh1010tt/RewardRank