# On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks

arXiv: https://arxiv.org/abs/2010.05103 · HTML: https://mlsanity.com/paper/2010.05103/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2020.

## Built on by (most cited first, 11 of 11)

- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review](https://mlsanity.com/paper/2204.10805/) (2022)
- [Active Learning Helps Pretrained Models Learn the Intended Task](https://mlsanity.com/paper/2204.08491/) (2022)
- [ALLSH: Active Learning Guided by Local Sensitivity and Hardness](https://mlsanity.com/paper/2205.04980/) (2022)
- [Question Answering Infused Pre-training of General-Purpose Contextualized Representations](https://mlsanity.com/paper/2106.08190/) (2021)
- [Deep Indexed Active Learning for Matching Heterogeneous Entity Representations](https://mlsanity.com/paper/2104.03986/) (2021)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)
- [Multilingual Detection of Personal Employment Status on Twitter](https://mlsanity.com/paper/2203.09178/) (2022)
- [Interactive Ontology Matching with Cost-Efficient Learning](https://mlsanity.com/paper/2404.07663/) (2024)
- [Benchmarking Multi-Domain Active Learning on Image Classification](https://mlsanity.com/paper/2312.00364/) (2023)
- [Active Learning for Finding a Needle in a Haystack](https://mlsanity.com/paper/2610.15165/) (2026)

## Abstract (arXiv)

Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., 99.99% of examples are negatives). In contrast, many recent datasets heuristically choose examples to ensure label balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only 2.4% average precision when evaluated on realistically imbalanced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently retrieve uncertain points from a very large pool of unlabeled utterance pairs. By creating balanced training data with more informative negative examples, active learning greatly improves average precision to 32.5% on QQP and 20.1% on WikiQA.