Deep is a Luxury We Don't Have
Not on the map. Why?
Abstract (arXiv)
Medical images come in high resolutions. A high resolution is vital for finding malignant tissues at an early stage. Yet, this resolution presents a challenge in terms of modeling long range dependencies. Shallow transformers eliminate this problem, but they suffer from quadratic complexity. In this paper, we tackle this complexity by leveraging a linear self-attention approximation. Through this approximation, we propose an efficient vision model called HCT that stands for High resolution Convolutional Transformer. HCT brings transformers' merits to high resolution images at a significantly lower cost. We evaluate HCT using a high resolution mammography dataset. HCT is significantly superior to its CNN counterpart. Furthermore, we demonstrate HCT's fitness for medical images by evaluating its effective receptive field.Code available at https://bit.ly/3ykBhhf
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 46 references that are in the map. First posted on arXiv 11 Aug 2022.
Builds on
42 of 46 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2020Reformer: The Efficient Transformercited by 1,668
- 2020Rethinking Attention with Performerscited by 1,601
- 2017Deformable Convolutional Networkscited by 1,488
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
30 more
- 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
- 2021Florence: A New Foundation Model for Computer Visioncited by 804
- 2020Efficient Transformers: A Surveycited by 790
- 2018Image Transformercited by 772
- 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
- 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
- 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
- 2021Early Convolutions Help Transformers See Bettercited by 396
- 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 320
- 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
- 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
- 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
- 2021TransFuse: Fusing Transformers and CNNs for Medical Image Segmentationcited by 258
- 2021CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentationcited by 227
- 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
- 2019Blockwise Self-Attention for Long Document Understandingcited by 159
- 2021UTNet: A Hybrid Transformer Architecture for Medical Image Segmentationcited by 116
- 2021Is it Time to Replace CNNs with Transformers for Medical Images?cited by 60
- 2021Convolution-Free Medical Image Segmentation using Transformerscited by 49
- 2020An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localizationcited by 43
- 2020OPTIMAM Mammography Image Database: a large scale resource of mammography images and clinical datacited by 38
- 2021Multi-view analysis of unregistered medical images using cross-view transformerscited by 36
- 2021TransCT: Dual-path Transformer for Low Dose Computed Tomographycited by 35
- 2021Eformer: Edge Enhancement based Transformer for Medical Image Denoisingcited by 30
- 2019Robust breast cancer detection in mammography and digital breast tomosynthesis using annotation-efficient deep learning approachcited by 27
- 2017High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networkscited by 25
- 2019Globally-Aware Multiple Instance Classifier for Breast Cancer Screeningcited by 12
Built on by
Citing papers per year: 2023: 1.
1 paper.