# Unbiased Learning to Rank: Online or Offline?

arXiv: https://arxiv.org/abs/2004.13574 · HTML: https://mlsanity.com/paper/2004.13574/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2020.

## Built on by (most cited first, 8 of 8)

- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Towards Disentangling Relevance and Bias in Unbiased Learning to Rank](https://mlsanity.com/paper/2212.13937/) (2022)
- [ULTRA: An Unbiased Learning To Rank Algorithm Toolbox](https://mlsanity.com/paper/2108.05073/) (2021)
- [Exploring the Escalation of Source Bias in User, Data, and Recommender System Feedback Loop](https://mlsanity.com/paper/2405.17998/) (2024)
- [Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank](https://mlsanity.com/paper/2206.01702/) (2022)
- [Learning to Rank for Maps at Airbnb](https://mlsanity.com/paper/2407.00091/) (2024)
- [A Causal Information-Flow Framework for Unbiased Learning-to-Rank](https://mlsanity.com/paper/2601.05590/) (2026)
- [Localization Boosting for Growth Markets: Mitigating Cross-Locale Behavioral Bias in Learning-to-Rank](https://mlsanity.com/paper/2605.11272/) (2026)

## Abstract (arXiv)

How to obtain an unbiased ranking model by learning to rank with biased user feedback is an important research question for IR. Existing work on unbiased learning to rank (ULTR) can be broadly categorized into two groups -- the studies on unbiased learning algorithms with logged data, namely the textitoffline unbiased learning, and the studies on unbiased parameters estimation with real-time user interactions, namely the textitonline learning to rank. While their definitions of textitunbiasness are different, these two types of ULTR algorithms share the same goal -- to find the best models that rank documents based on their intrinsic relevance or utility. However, most studies on offline and online unbiased learning to rank are carried in parallel without detailed comparisons on their background theories and empirical performance. In this paper, we formalize the task of unbiased learning to rank and show that existing algorithms for offline unbiased learning and online learning to rank are just the two sides of the same coin. We evaluate six state-of-the-art ULTR algorithms and find that most of them can be used in both offline settings and online environments with or without minor modifications. Further, we analyze how different offline and online learning paradigms would affect the theoretical foundation and empirical effectiveness of each algorithm on both synthetic and real search data. Our findings could provide important insights and guideline for choosing and deploying ULTR algorithms in practice.