# Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

arXiv: https://arxiv.org/abs/2010.08240 · HTML: https://mlsanity.com/paper/2010.08240/

## Key facts

- Cited by 138 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 30 of 138)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [SGPT: GPT Sentence Embeddings for Semantic Search](https://mlsanity.com/paper/2202.08904/) (2022)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval](https://mlsanity.com/paper/2112.07577/) (2021)
- [How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval](https://mlsanity.com/paper/2302.07452/) (2023)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [GuardT2I: Defending Text-to-Image Models from Adversarial Prompts](https://mlsanity.com/paper/2403.01446/) (2024)
- [UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers](https://mlsanity.com/paper/2303.00807/) (2023)
- [The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design](https://mlsanity.com/paper/2110.04541/) (2021)
- [SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features](https://mlsanity.com/paper/2206.07023/) (2022)
- [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023)
- [Pairwise Supervised Contrastive Learning of Sentence Representations](https://mlsanity.com/paper/2109.05424/) (2021)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring](https://mlsanity.com/paper/2106.06719/) (2021)
- [Improving Contrastive Learning of Sentence Embeddings from AI Feedback](https://mlsanity.com/paper/2305.01918/) (2023)
- [PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT](https://mlsanity.com/paper/2103.11933/) (2021)
- [CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation](https://mlsanity.com/paper/2210.04873/) (2022)
- [Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective](https://mlsanity.com/paper/2505.15045/) (2025)
- [On Cross-Lingual Retrieval with Multilingual Text Encoders](https://mlsanity.com/paper/2112.11031/) (2021)
- [A Thorough Examination on Zero-shot Dense Retrieval](https://mlsanity.com/paper/2204.12755/) (2022)
- [Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations](https://mlsanity.com/paper/2109.13059/) (2021)
- [Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence](https://mlsanity.com/paper/2105.12936/) (2021)
- [Question Answering Infused Pre-training of General-Purpose Contextualized Representations](https://mlsanity.com/paper/2106.08190/) (2021)
- [Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework](https://mlsanity.com/paper/2210.16798/) (2022)

## Abstract (arXiv)

There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While cross-encoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance.