# Pre-trained Language Model for Web-scale Retrieval in Baidu Search

arXiv: https://arxiv.org/abs/2106.03373 · HTML: https://mlsanity.com/paper/2106.03373/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2021.

## Built on by (most cited first, 30 of 31)

- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval](https://mlsanity.com/paper/2205.09153/) (2022)
- [Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training](https://mlsanity.com/paper/2306.03166/) (2023)
- [HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking](https://mlsanity.com/paper/2205.10569/) (2022)
- [Enhancing Recommendation Diversity by Re-ranking with Large Language Models](https://mlsanity.com/paper/2401.11506/) (2024)
- [Uni-Retriever: Towards Learning The Unified Embedding Based Retriever in Bing Sponsored Search](https://mlsanity.com/paper/2202.06212/) (2022)
- [Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings](https://mlsanity.com/paper/2204.00185/) (2022)
- [TongSearch-QR: Reinforced Query Reasoning for Retrieval](https://mlsanity.com/paper/2506.11603/) (2025)
- [Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking](https://mlsanity.com/paper/2204.11673/) (2022)
- [Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2402.04867/) (2024)
- [ESPN: Memory-Efficient Multi-Vector Information Retrieval](https://mlsanity.com/paper/2312.05417/) (2023)
- [LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services](https://mlsanity.com/paper/2512.07436/) (2025)
- [Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval](https://mlsanity.com/paper/2201.05409/) (2022)
- [Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval](https://mlsanity.com/paper/2308.08285/) (2023)
- [Modality-Balanced Embedding for Video Retrieval](https://mlsanity.com/paper/2204.08182/) (2022)
- [NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval](https://mlsanity.com/paper/2409.02343/) (2024)
- [PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding](https://mlsanity.com/paper/2210.12308/) (2022)
- [Centrality-aware Product Retrieval and Ranking](https://mlsanity.com/paper/2410.15930/) (2024)
- [Best Practices for Distilling Large Language Models into BERT for Web Search Ranking](https://mlsanity.com/paper/2411.04539/) (2024)
- [CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems](https://mlsanity.com/paper/2506.19993/) (2025)
- [Integrity and Junkiness Failure Handling for Embedding-based Retrieval: A Case Study in Social Network Search](https://mlsanity.com/paper/2304.09287/) (2023)
- [LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Bias](https://mlsanity.com/paper/2307.02912/) (2023)
- [Intent-based Product Collections for E-commerce using Pretrained Language Models](https://mlsanity.com/paper/2110.08241/) (2021)
- [NeuralSearchX: Serving a Multi-billion-parameter Reranker for Multilingual Metasearch at a Low Cost](https://mlsanity.com/paper/2210.14837/) (2022)
- [Event-driven Real-time Retrieval in Web Search](https://mlsanity.com/paper/2312.00372/) (2023)
- [Layout-aware Webpage Quality Assessment](https://mlsanity.com/paper/2301.12152/) (2023)
- [Generate to Understand for Representation](https://mlsanity.com/paper/2306.10056/) (2023)
- [Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation](https://mlsanity.com/paper/2604.21536/) (2026)
- [PLUME: Parameter-Efficient Personalization of Large Language Models via Low-Rank User Modulation in Shared Subspaces](https://mlsanity.com/paper/2609.04715/) (2026)

## Abstract (arXiv)

Retrieval is a crucial stage in web search that identifies a small set of query-relevant candidates from a billion-scale corpus. Discovering more semantically-related candidates in the retrieval stage is very promising to expose more high-quality results to the end users. However, it still remains non-trivial challenges of building and deploying effective retrieval models for semantic matching in real search engine. In this paper, we describe the retrieval system that we developed and deployed in Baidu Search. The system exploits the recent state-of-the-art Chinese pretrained language model, namely Enhanced Representation through kNowledge IntEgration (ERNIE), which facilitates the system with expressive semantic matching. In particular, we developed an ERNIE-based retrieval model, which is equipped with 1) expressive Transformer-based semantic encoders, and 2) a comprehensive multi-stage training paradigm. More importantly, we present a practical system workflow for deploying the model in web-scale retrieval. Eventually, the system is fully deployed into production, where rigorous offline and online experiments were conducted. The results show that the system can perform high-quality candidate retrieval, especially for those tail queries with uncommon demands. Overall, the new retrieval system facilitated by pretrained language model (i.e., ERNIE) can largely improve the usability and applicability of our search engine.