mlsanityData to 7 Oct 2026

Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup

arXiv 2101.06983 ↗First posted 18 Jan 2021cs.LG cs.CL cs.IRcited by 123

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

13 of 17 references.

References · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  9. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  10. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  11. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  12. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
1 more
  1. 2018Reversible Recurrent Neural Networkscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 16; 2023: 9; 2024: 19; 2025: 39; 2026: 36.

123 papers.

Sort:

By area:

  1. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  3. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  4. 2024Generative Representational Instruction Tuningcited by 227
  5. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  6. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  7. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  8. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  9. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  10. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  11. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  12. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
38 more
  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  2. 2024Repetition Improves Language Model Embeddingscited by 70
  3. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  4. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  5. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  6. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  7. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  8. 2021Scaling Law for Recommendation Models: Towards General-purpose User Representationscited by 26
  9. 2024IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFTcited by 26
  10. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  11. 2022i-Code: An Integrative and Composable Multimodal Learning Frameworkcited by 24
  12. 2022CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typoscited by 22
  13. 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
  14. 2022SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrievalcited by 21
  15. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  16. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 21
  17. 2024Contextual Document Embeddingscited by 19
  18. 2023GLEN: Generative Retrieval via Lexical Index Learningcited by 17
  19. 2025PyLate: Flexible Training and Retrieval for Late Interaction Modelscited by 17
  20. 2023DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Trainingcited by 15
  21. 2022C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrievalcited by 14
  22. 2024OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMscited by 14
  23. 2024Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrievalcited by 13
  24. 2022Conditional Supervised Contrastive Learning for Fair Text Classificationcited by 11
  25. 2023Language Models are Universal Embedderscited by 11
  26. 2026Representation Distribution Matching for One-Step Visual Generationcited by 10
  27. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  28. 2024PhysBERT: A Text Embedding Model for Physics Scientific Literaturecited by 8
  29. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  30. 2025SDTagNet: Leveraging Text-Annotated Navigation Maps for Online HD Map Constructioncited by 8
  31. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
  32. 2023Dual-Encoders for Extreme Multi-Label Classificationcited by 7
  33. 2022Federated Momentum Contrastive Clusteringcited by 6
  34. 2023CONVERSER: Few-Shot Conversational Dense Retrieval with Synthetic Data Generationcited by 6
  35. 2024ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learningcited by 6
  36. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  37. 2022Generation-Augmented Query Expansion For Code Retrievalcited by 5
  38. 2025Efficient Text Encoders for Labor Market Analysiscited by 5

50 most cited of 123 All 123 (JSON)