# ULTRA: An Unbiased Learning To Rank Algorithm Toolbox

arXiv: https://arxiv.org/abs/2108.05073 · HTML: https://mlsanity.com/paper/2108.05073/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 11 of 11)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Learning a Deep Listwise Context Model for Ranking Refinement](https://mlsanity.com/paper/1804.05936/) (2018)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval](https://mlsanity.com/paper/1912.05891/) (2019)
- [Estimating Position Bias without Intrusive Interventions](https://mlsanity.com/paper/1812.05161/) (2018)
- [Differentiable Unbiased Online Learning to Rank](https://mlsanity.com/paper/1809.08415/) (2018)
- [Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks](https://mlsanity.com/paper/1811.04415/) (2018)
- [Efficient Exploration of Gradient Space for Online Learning to Rank](https://mlsanity.com/paper/1805.07317/) (2018)
- [Unbiased Learning to Rank: Online or Offline?](https://mlsanity.com/paper/2004.13574/) (2020)
- [Self-Attentive Document Interaction Networks for Permutation Equivariant Ranking](https://mlsanity.com/paper/1910.09676/) (2019)

## Built on by (most cited first, 6 of 6)

- [A Large Scale Search Dataset for Unbiased Learning to Rank](https://mlsanity.com/paper/2207.03051/) (2022)
- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective](https://mlsanity.com/paper/2306.07528/) (2023)
- [Whole Page Unbiased Learning to Rank](https://mlsanity.com/paper/2210.10718/) (2022)
- [Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank](https://mlsanity.com/paper/2408.09817/) (2024)
- [Structure-aware Relative Policy Optimization for Ranking](https://mlsanity.com/paper/2607.25268/) (2026)

## Abstract (arXiv)

Learning to rank systems has become an important aspect of our daily life. However, the implicit user feedback that is used to train many learning to rank models is usually noisy and suffered from user bias (i.e., position bias). Thus, obtaining an unbiased model using biased feedback has become an important research field for IR. Existing studies on unbiased learning to rank (ULTR) can be generalized into two families-algorithms that attain unbiasedness with logged data, offline learning, and algorithms that achieve unbiasedness by estimating unbiased parameters with real-time user interactions, namely online learning. While there exist many algorithms from both families, there lacks a unified way to compare and benchmark them. As a result, it can be challenging for researchers to choose the right technique for their problems or for people who are new to the field to learn and understand existing algorithms. To solve this problem, we introduced ULTRA, which is a flexible, extensible, and easily configure ULTR toolbox. Its key features include support for multiple ULTR algorithms with configurable hyperparameters, a variety of built-in click models that can be used separately to simulate clicks, different ranking model architecture and evaluation metrics, and simple learning to rank pipeline creation. In this paper, we discuss the general framework of ULTR, briefly describe the algorithms in ULTRA, detailed the structure, and pipeline of the toolbox. We experimented on all the algorithms supported by ultra and showed that the toolbox performance is reasonable. Our toolbox is an important resource for researchers to conduct experiments on ULTR algorithms with different configurations as well as testing their own algorithms with the supported features.