mlsanityData to 7 Oct 2026

Differentiable Patch Selection for Image Recognition

arXiv 2104.03059 ↗First posted 7 Apr 2021cs.CV cs.AI cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Neural Networks require large amounts of memory and compute to process high resolution images, even when only a small part of the image is actually informative for the task at hand. We propose a method based on a differentiable Top-K operator to select the most relevant parts of the input to efficiently process high resolution images. Our method may be interfaced with any downstream neural network, is able to aggregate information from different patches in a flexible way, and allows the whole model to be trained end-to-end using backpropagation. We show results for traffic sign recognition, inter-patch relationship reasoning, and fine-grained recognition without using object/part bounding box annotations during training.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 7 Apr 2021.

Builds on

37 of 50 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2017Mask R-CNNcited by 7,259
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2015Fast R-CNNcited by 2,930
  10. 2015Spatial Transformer Networkscited by 1,423
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
25 more
  1. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  2. 2017Residual Attention Network for Image Classificationcited by 385
  3. 2014Recurrent Models of Visual Attentioncited by 344
  4. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  5. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  6. 2015Bilinear CNNs for Fine-grained Visual Recognitioncited by 228
  7. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  8. 2020Fast Differentiable Sorting and Rankingcited by 196
  9. 2015Attention to Scale: Scale-aware Semantic Image Segmentationcited by 169
  10. 2018Tell Me Where to Look: Guided Attention Inference Networkcited by 168
  11. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  12. 2017Learning non-maximum suppressioncited by 112
  13. 2018Learning to Navigate for Fine-grained Classificationcited by 103
  14. 2020Learning with Differentiable Perturbed Optimizerscited by 81
  15. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 69
  16. 2015Action Recognition using Visual Attentioncited by 49
  17. 2019Saccader: Improving Accuracy of Hard Attention Models for Visioncited by 35
  18. 2019Processing Megapixel Images with Deep Attention-Sampling Modelscited by 29
  19. 2017A-Lamp: Adaptive Layout-Aware Multi-Patch Deep Convolutional Neural Network for Photo Aesthetic Assessmentcited by 28
  20. 2017Dynamic Computational Time for Visual Attentioncited by 23
  21. 2020Differentiable Top-k Operator with Optimal Transportcited by 17
  22. 2019Patch Transformer for Multi-tagging Whole Slide Histopathology Imagescited by 13
  23. 2016Recurrent Attentional Networks for Saliency Detectioncited by 11
  24. 2016Fully Convolutional Attention Networks for Fine-Grained Recognitioncited by 8
  25. 2018MIST: Multiple Instance Spatial Transformer Networkcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 21; 2023: 12; 2024: 11; 2025: 10; 2026: 7.

68 papers.

Sort:

By area:

  1. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  2. 2023Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunitiescited by 129
  3. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  4. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  5. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 58
  6. 2022Differentiable Zooming for Multiple Instance Learning on Whole-Slide Imagescited by 43
  7. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  8. 2024KVQ: Kwai Video Quality Assessment for Short-form Videoscited by 38
  9. 2022Vision Transformer with Attentive Pooling for Robust Facial Expression Recognitioncited by 35
  10. 2024Adaptive Slot Attention: Object Discovery with Dynamic Slot Numbercited by 34
  11. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspectivecited by 26
  12. 2022Monotonic Differentiable Sorting Networkscited by 25
38 more
  1. 2022Token Turing Machinescited by 25
  2. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
  3. 2022ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classificationcited by 21
  4. 2022Differentiable Top-k Classification Learningcited by 21
  5. 2021Efficient Classification of Very Large Images with Tiny Objectscited by 17
  6. 2023Differentiable Clustering with Perturbed Spanning Forestscited by 14
  7. 2022Iterative Patch Selection for High-Resolution Image Recognitioncited by 13
  8. 2023Learning by Sorting: Self-supervised Learning with Group Ordering Constraintscited by 13
  9. 2025Positional Encoding in Transformer-Based Time Series Models: A Surveycited by 13
  10. 2021A Probabilistic Hard Attention Model For Sequentially Observed Scenescited by 12
  11. 2022EmbryosFormer: Deformable Transformer and Collaborative Encoding-Decoding for Embryos Stage Development Classificationcited by 12
  12. 2022Consistency driven Sequential Transformers Attention Model for Partially Observable Scenescited by 11
  13. 2022Knowledge Mining with Scene Text for Fine-Grained Recognitioncited by 10
  14. 2022Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognitioncited by 9
  15. 2024StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Modelscited by 8
  16. 2025Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathologycited by 8
  17. 2021Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spacescited by 7
  18. 2023Few-shot Action Recognition via Intra- and Inter-Video Information Maximizationcited by 7
  19. 2024PATHS: A Hierarchical Transformer for Efficient Whole Slide Image Analysiscited by 7
  20. 2024Learning to Rank Patches for Unbiased Image Redundancy Reductioncited by 6
  21. 2022Dynamics-aware Adversarial Attack of Adaptive Neural Networkscited by 5
  22. 2022Mirror Sinkhorn: Fast Online Optimization on Transport Polytopescited by 5
  23. 2025No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-k Patch Samplingcited by 5
  24. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervisioncited by 5
  25. 2023Token Sparsification for Faster Medical Image Segmentationcited by 4
  26. 2025VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligencecited by 4
  27. 2024Soft Condorcet Optimization for Ranking of General Agentscited by 3
  28. 2021Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognitioncited by 2
  29. 2022Weakly-Supervised End-to-End CAD Retrieval to Scan Objectscited by 2
  30. 2022NUNet: Deep Learning for Non-Uniform Super-Resolution of Turbulent Flowscited by 2
  31. 2023Self-Ordering Point Cloudscited by 2
  32. 2023Set Learning for Accurate and Calibrated Modelscited by 2
  33. 2024MST: Adaptive Multi-Scale Tokens Guided Interactive Segmentationcited by 2
  34. 2024Newton Losses: Using Curvature Information for Learning with Differentiable Algorithmscited by 2
  35. 2025Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classificationcited by 2
  36. 2025REEF: Relevance-Aware and Efficient LLM Adapter for Video Understandingcited by 2
  37. 2022Problem-dependent attention and effort in neural networks with applications to image resolution and model selectioncited by 1
  38. 2023Learning to Summarize Videos by Contrasting Clipscited by 1

50 most cited of 68 All 68 (JSON)