mlsanityData to 7 Oct 2026

DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training

arXiv 2202.13808 ↗First posted 28 Feb 2022cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

A standard hardware bottleneck when training deep neural networks is GPU memory. The bulk of memory is occupied by caching intermediate tensors for gradient computation in the backward pass. We propose a novel method to reduce this footprint - Dropping Intermediate Tensors (DropIT). DropIT drops min-k elements of the intermediate tensors and approximates gradients from the sparsified tensors in the backward pass. Theoretically, DropIT reduces noise on estimated gradients and therefore has a higher rate of convergence than vanilla-SGD. Experiments show that we can drop up to 90% of the intermediate tensor elements in fully-connected and convolutional layers while achieving higher testing accuracy for Visual Transformers and Convolutional Neural Networks on various tasks (e.g., classification, object detection, instance segmentation). Our code and models are available at https://github.com/chenjoya/dropit.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 28 Feb 2022.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2017Mask R-CNNcited by 7,259
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
20 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  9. 2018Sparsified SGD with Memorycited by 373
  10. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  11. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  12. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  13. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  14. 2017In-Place Activated BatchNorm for Memory-Optimized Training of DNNscited by 81
  15. 2016Memory-Efficient Backpropagation Through Timecited by 78
  16. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  17. 2019Low-Memory Neural Network Training: A Technical Reportcited by 56
  18. 2021MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learningcited by 21
  19. 2019Backprop with Approximate Activations for Memory-efficient Network Trainingcited by 19
  20. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 3; 2026: 1.

9 papers.

  1. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  2. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  3. 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
  4. 2025SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsitycited by 7
  5. 2025PaCA: Partial Connection Adaptation for Efficient Fine-Tuningcited by 6
  6. 2023SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamicscited by 4
  7. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 2
  8. 2026Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuningcited by 1
  9. 2024UnifiedNN: Efficient Neural Network Training on the Cloudcited by 0