# Pretraining De-Biased Language Model with Large-scale Click Logs for Document Ranking

arXiv: https://arxiv.org/abs/2302.13498 · HTML: https://mlsanity.com/paper/2302.13498/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2023.

## Built on by (most cited first, 2 of 2)

- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [CLAX: Fast and Flexible Neural Click Models in JAX](https://mlsanity.com/paper/2511.03620/) (2025)

## Abstract (arXiv)

Pre-trained language models have achieved great success in various large-scale information retrieval tasks. However, most of pretraining tasks are based on counterfeit retrieval data where the query produced by the tailored rule is assumed as the user's issued query on the given document or passage. Therefore, we explore to use large-scale click logs to pretrain a language model instead of replying on the simulated queries. Specifically, we propose to use user behavior features to pretrain a debiased language model for document ranking. Extensive experiments on Baidu desensitization click logs validate the effectiveness of our method. Our team on WSDM Cup 2023 Pre-training for Web Search won the 1st place with a Discounted Cumulative Gain @ 10 (DCG@10) score of 12.16525 on the final leaderboard.