# YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker

arXiv: https://arxiv.org/abs/2109.06436 · HTML: https://mlsanity.com/paper/2109.06436/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Embedding-based Retrieval in Facebook Search](https://mlsanity.com/paper/2006.11632/) (2020)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval](https://mlsanity.com/paper/1910.10687/) (2019)
- [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://mlsanity.com/paper/2101.08751/) (2021)
- [Universal Language Model Fine-tuning for Text Classification](https://mlsanity.com/paper/1801.06146/) (2018)
- [PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2010.10137/) (2020)
- [A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations](https://mlsanity.com/paper/1511.08277/) (2015)
- [Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN](https://mlsanity.com/paper/1604.04378/) (2016)
- [B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2104.09791/) (2021)
- [PACRR: A Position-Aware Neural IR Model for Relevance Matching](https://mlsanity.com/paper/1704.03940/) (2017)
- [DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval](https://mlsanity.com/paper/1710.05649/) (2017)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [Revisit Recommender System in the Permutation Prospective](https://mlsanity.com/paper/2102.12057/) (2021)

## Built on by (most cited first, 1 of 1)

- [HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking](https://mlsanity.com/paper/2205.10569/) (2022)

## Abstract (arXiv)

Pre-trained model such as BERT has been proved to be an effective tool for dealing with Information Retrieval (IR) problems. Due to its inspiring performance, it has been widely used to tackle with real-world IR problems such as document ranking. Recently, researchers have found that selecting "hard" rather than "random" negative samples would be beneficial for fine-tuning pre-trained models on ranking tasks. However, it remains elusive how to leverage hard negative samples in a principled way. To address the aforementioned issues, we propose a fine-tuning strategy for document ranking, namely Self-Involvement Ranker (SIR), to dynamically select hard negative samples to construct high-quality semantic space for training a high-quality ranking model. Specifically, SIR consists of sequential compressors implemented with pre-trained models. Front compressor selects hard negative samples for rear compressor. Moreover, SIR leverages supervisory signal to adaptively adjust semantic space of negative samples. Finally, supervisory signal in rear compressor is computed based on condition probability and thus can control sample dynamic and further enhance the model performance. SIR is a lightweight and general framework for pre-trained models, which simplifies the ranking process in industry practice. We test our proposed solution on MS MARCO with document ranking setting, and the results show that SIR can significantly improve the ranking performance of various pre-trained models. Moreover, our method became the new SOTA model anonymously on MS MARCO Document ranking leaderboard in May 2021.