mlsanityData to 14 Oct 2026

Survey: Exploiting Data Redundancy for Optimization of Deep Learning

arXiv 2208.13363 ↗First posted 29 Aug 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Data redundancy is ubiquitous in the inputs and intermediate results of Deep Neural Networks (DNN). It offers many significant opportunities for improving DNN performance and efficiency and has been explored in a large body of work. These studies have scattered in many venues across several years. The targets they focus on range from images to videos and texts, and the techniques they use to detect and exploit data redundancy also vary in many aspects. There is not yet a systematic examination and summary of the many efforts, making it difficult for researchers to get a comprehensive view of the prior work, the state of the art, differences and shared principles, and the areas and directions yet to explore. This article tries to fill the void. It surveys hundreds of recent papers on the topic, introduces a novel taxonomy to put the various techniques into a single categorization framework, offers a comprehensive description of the main methods used for exploiting data redundancy in improving multiple kinds of DNNs on data, and points out a set of research opportunities for future to explore.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 83 of its 164 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

83 of 164 references.

References · 83

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2020Reformer: The Efficient Transformercited by 1,668
38 more
  1. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  2. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  3. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  4. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  9. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  10. 2013Network In Networkcited by 605
  11. 2020What is the State of Neural Network Pruning?cited by 583
  12. 2019Deep Learning for 3D Point Clouds: A Surveycited by 558
  13. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  15. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  16. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  17. 2020Binary Neural Networks: A Surveycited by 190
  18. 2016Deep Feature Flow for Video Recognitioncited by 186
  19. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  20. 2017Flow-Guided Feature Aggregation for Video Object Detectioncited by 177
  21. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  22. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  23. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  24. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  25. 2018AdaFrame: Adaptive Frame Selection for Fast Video Recognitioncited by 134
  26. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 134
  27. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  28. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  29. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  30. 2015End-to-end Learning of Action Detection from Frame Glimpses in Videoscited by 101
  31. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  32. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  33. 2016Seq-NMS for Video Object Detectioncited by 68
  34. 2016T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videoscited by 67
  35. 2019Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognitioncited by 66
  36. 2017Towards High Performance Video Object Detectioncited by 60
  37. 2019Resource Efficient 3D Convolutional Neural Networkscited by 50
  38. 2015Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networkscited by 45

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 2.

6 papers.

  1. 2023Hybrid Representation-Enhanced Sampling for Bayesian Active Learning in Musculoskeletal Segmentation of Lower Extremitiescited by 4
  2. 2025Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Corescited by 2
  3. 2025A Novel Multi-layer Task-centric and Data Quality Framework for Autonomous Drivingcited by 2
  4. 2024Hard-Thresholding Meets Evolution Strategies in Reinforcement Learningcited by 0
  5. 2026Modeling and Measuring Redundancy in Multisource Multimodal Data for Autonomous Drivingcited by 0
  6. 2026Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysiscited by 0