# A Gradient Accumulation Method for Dense Retriever under Memory Constraint

arXiv: https://arxiv.org/abs/2406.12356 · HTML: https://mlsanity.com/paper/2406.12356/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning](https://mlsanity.com/paper/2509.07252/) (2025)
- [MobileFineTuner: A Mobile-Native Framework for On-Device LLM Fine-Tuning in Real-World Embedded AI Applications](https://mlsanity.com/paper/2512.08211/) (2025)
- [KunlunBaize: LLM with Multi-Scale Convolution and Multi-Token Prediction Under TransformerX Framework](https://mlsanity.com/paper/2503.04784/) (2025)
- [Similarity Choice and Negative Scaling in Supervised Contrastive Learning for Deepfake Audio Detection](https://mlsanity.com/paper/2604.26057/) (2026)

## Abstract (arXiv)

InfoNCE loss is commonly used to train dense retriever in information retrieval tasks. It is well known that a large batch is essential to stable and effective training with InfoNCE loss, which requires significant hardware resources. Due to the dependency of large batch, dense retriever has bottleneck of application and research. Recently, memory reduction methods have been broadly adopted to resolve the hardware bottleneck by decomposing forward and backward or using a memory bank. However, current methods still suffer from slow and unstable training. To address these issues, we propose Contrastive Accumulation (ContAccum), a stable and efficient memory reduction method for dense retriever trains that uses a dual memory bank structure to leverage previously generated query and passage representations. Experiments on widely used five information retrieval datasets indicate that ContAccum can surpass not only existing memory reduction methods but also high-resource scenario. Moreover, theoretical analysis and experimental results confirm that ContAccum provides more stable dual-encoder training than current memory bank utilization methods.