# Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers

arXiv: https://arxiv.org/abs/2308.13442 · HTML: https://mlsanity.com/paper/2308.13442/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice](https://mlsanity.com/paper/2203.05962/) (2022)
- [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2207.08518/) (2022)
- [Implicit Neural Representation in Medical Imaging: A Comparative Survey](https://mlsanity.com/paper/2307.16142/) (2023)
- [LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2107.08623/) (2021)
- [Bi-Directional ConvLSTM U-Net with Densley Connected Convolutions](https://mlsanity.com/paper/1909.00166/) (2019)
- [Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning](https://mlsanity.com/paper/2207.04978/) (2022)
- [Learning to Merge Tokens in Vision Transformers](https://mlsanity.com/paper/2202.12015/) (2022)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [TransClaw U-Net: Claw U-Net with Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2107.05188/) (2021)
- [Contextual Attention Network: Transformer Meets U-Net](https://mlsanity.com/paper/2203.01932/) (2022)
- [Beyond Fixation: Dynamic Window Visual Transformer](https://mlsanity.com/paper/2203.12856/) (2022)
- [Multi-level Context Gating of Embedded Collective Knowledge for Medical Image Segmentation](https://mlsanity.com/paper/2003.05056/) (2020)
- [DermoSegDiff: A Boundary-aware Segmentation Diffusion Model for Skin Lesion Delineation](https://mlsanity.com/paper/2308.02959/) (2023)
- [Self-supervised Few-shot Learning for Semantic Segmentation: An Annotation-free Approach](https://mlsanity.com/paper/2307.14446/) (2023)

## Built on by (most cited first, 3 of 3)

- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [Wavelet Policy: Imitation Learning in the Scale Domain with World Prior Memory](https://mlsanity.com/paper/2504.04991/) (2025)
- [Event-to-Video Reconstruction using Spatio-Temporal and Frequency-Enhanced Deep Neural Networks](https://mlsanity.com/paper/2605.25804/) (2026)

## Abstract (arXiv)

Medical image segmentation is a critical task that plays a vital role in diagnosis, treatment planning, and disease monitoring. Accurate segmentation of anatomical structures and abnormalities from medical images can aid in the early detection and treatment of various diseases. In this paper, we address the local feature deficiency of the Transformer model by carefully re-designing the self-attention map to produce accurate dense prediction in medical images. To this end, we first apply the wavelet transformation to decompose the input feature map into low-frequency (LF) and high-frequency (HF) subbands. The LF segment is associated with coarse-grained features while the HF components preserve fine-grained features such as texture and edge information. Next, we reformulate the self-attention operation using the efficient Transformer to perform both spatial and context attention on top of the frequency representation. Furthermore, to intensify the importance of the boundary information, we impose an additional attention map by creating a Gaussian pyramid on top of the HF components. Moreover, we propose a multi-scale context enhancement block within skip connections to adaptively model inter-scale dependencies to overcome the semantic gap among stages of the encoder and decoder modules. Throughout comprehensive experiments, we demonstrate the effectiveness of our strategy on multi-organ and skin lesion segmentation benchmarks. The implementation code will be available upon acceptance. hrefhttps://github.com/mindflow-institue/WaveFormerGitHub.