# A Large-Scale Web Search Dataset for Federated Online Learning to Rank

arXiv: https://arxiv.org/abs/2508.12353 · HTML: https://mlsanity.com/paper/2508.12353/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 17 Aug 2025.

## Builds on (most cited first, 15 of 15)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge](https://mlsanity.com/paper/1804.08333/) (2018)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Asynchronous Federated Learning on Heterogeneous Devices: A Survey](https://mlsanity.com/paper/2109.04269/) (2021)
- [ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search](https://mlsanity.com/paper/2006.05324/) (2020)
- [Simplified Data Wrangling with ir_datasets](https://mlsanity.com/paper/2103.02280/) (2021)
- [Differentiable Unbiased Online Learning to Rank](https://mlsanity.com/paper/1809.08415/) (2018)
- [Is Non-IID Data a Threat in Federated Online Learning to Rank?](https://mlsanity.com/paper/2204.09272/) (2022)
- [Privacy in Open Search: A Review of Challenges and Solutions](https://mlsanity.com/paper/2110.10720/) (2021)
- [SwarmSearch: Decentralized Search Engine with Self-Funding Economy](https://mlsanity.com/paper/2505.07452/) (2025)

## Abstract (arXiv)

The centralized collection of search interaction logs for training ranking models raises significant privacy concerns. Federated Online Learning to Rank (FOLTR) offers a privacy-preserving alternative by enabling collaborative model training without sharing raw user data. However, benchmarks in FOLTR are largely based on random partitioning of classical learning-to-rank datasets, simulated user clicks, and the assumption of synchronous client participation. This oversimplifies real-world dynamics and undermines the realism of experimental results. We present AOL4FOLTR, a large-scale web search dataset with 2.6 million queries from 10,000 users. Our dataset addresses key limitations of existing benchmarks by including user identifiers, real click data, and query timestamps, enabling realistic user partitioning, behavior modeling, and asynchronous federated learning scenarios.