# SPM: Structured Pretraining and Matching Architectures for Relevance Modeling in Meituan Search

arXiv: https://arxiv.org/abs/2308.07711 · HTML: https://mlsanity.com/paper/2308.07711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [Convolutional Neural Network Architectures for Matching Natural Language Sentences](https://mlsanity.com/paper/1503.03244/) (2015)
- [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://mlsanity.com/paper/1610.08136/) (2016)
- [Text Matching as Image Recognition](https://mlsanity.com/paper/1602.06359/) (2016)
- [Semantic Product Search](https://mlsanity.com/paper/1907.00937/) (2019)
- [Pre-trained Language Model based Ranking in Baidu Search](https://mlsanity.com/paper/2105.11108/) (2021)
- [A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations](https://mlsanity.com/paper/1511.08277/) (2015)
- [Learning a Product Relevance Model from Click-Through Data in E-Commerce](https://mlsanity.com/paper/2102.07098/) (2021)
- [E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](https://mlsanity.com/paper/2009.02835/) (2020)
- [A unified Neural Network Approach to E-CommerceRelevance Learning](https://mlsanity.com/paper/2104.12302/) (2021)
- [BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search](https://mlsanity.com/paper/2010.10442/) (2020)
- [ACE-BERT: Adversarial Cross-modal Enhanced BERT for E-commerce Retrieval](https://mlsanity.com/paper/2112.07209/) (2021)

## Built on by (most cited first, 2 of 2)

- [Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning](https://mlsanity.com/paper/2412.12504/) (2024)
- [Improving Topic Relevance Model by Mix-structured Summarization and LLM-based Data Augmentation](https://mlsanity.com/paper/2404.02616/) (2024)

## Abstract (arXiv)

In e-commerce search, relevance between query and documents is an essential requirement for satisfying user experience. Different from traditional e-commerce platforms that offer products, users search on life service platforms such as Meituan mainly for product providers, which usually have abundant structured information, e.g. name, address, category, thousands of products. Modeling search relevance with these rich structured contents is challenging due to the following issues: (1) there is language distribution discrepancy among different fields of structured document, making it difficult to directly adopt off-the-shelf pretrained language model based methods like BERT. (2) different fields usually have different importance and their length vary greatly, making it difficult to extract document information helpful for relevance matching. To tackle these issues, in this paper we propose a novel two-stage pretraining and matching architecture for relevance matching with rich structured documents. At pretraining stage, we propose an effective pretraining method that employs both query and multiple fields of document as inputs, including an effective information compression method for lengthy fields. At relevance matching stage, a novel matching method is proposed by leveraging domain knowledge in search query to generate more effective document representations for relevance scoring. Extensive offline experiments and online A/B tests on millions of users verify that the proposed architectures effectively improve the performance of relevance modeling. The model has already been deployed online, serving the search traffic of Meituan for over a year.