mlsanityData to 7 Oct 2026

Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

arXiv 2210.17167 ↗First posted 31 Oct 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model. We show the catastrophic forgetting phenomena behind the training instability, where models learn and forget different negative groups during training iterations. We then propose ANCE-Tele, which accumulates momentum negatives from past iterations and approximates future iterations using lookahead negatives, as "teleportations" along the time axis to smooth the learning process. On web search and OpenQA, ANCE-Tele outperforms previous state-of-the-art systems of similar size, eliminates the dependency on sparse retrieval negatives, and is competitive among systems using significantly more (50x) parameters. Our analysis demonstrates that teleportation negatives reduce catastrophic forgetting and improve convergence speed for dense retrieval training. Our code is available at https://github.com/OpenMatch/ANCE-Tele.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

34 of 48 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  12. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
22 more
  1. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  2. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  3. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  4. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  5. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  6. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  7. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  8. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  9. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  10. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  11. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  12. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  13. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  14. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  15. 2021Few-Shot Conversational Dense Retrievalcited by 71
  16. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  17. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  18. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  19. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  20. 2021Boosted Dense Retrievercited by 10
  21. 2021OpenMatch: An Open Source Library for Neu-IR Researchcited by 6
  22. 2021Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotationcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 1.

5 papers.

  1. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  2. 2023Policy-Gradient Training of Language Models for Rankingcited by 5
  3. 2023Rethinking Dense Retrieval's Few-Shot Abilitycited by 1
  4. 2024LLM-Oriented Retrieval Tunercited by 0
  5. 2025RRRA: Resampling and Reranking through a Retriever Adaptercited by 0