# Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention

arXiv: https://arxiv.org/abs/2504.08801 · HTML: https://mlsanity.com/paper/2504.08801/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [A Hybrid Wavelet-Fourier Method for Next-Generation Conditional Diffusion Models](https://mlsanity.com/paper/2504.03821/) (2025)
- [State Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling](https://mlsanity.com/paper/2503.17382/) (2025)

## Built on by (most cited first, 7 of 7)

- [From Attention to Atoms: Spectral Dictionary Learning for Fast, Interpretable Language Models](https://mlsanity.com/paper/2505.00033/) (2025)
- [Wavelet-based Variational Autoencoders for High-Resolution Image Generation](https://mlsanity.com/paper/2504.13214/) (2025)
- [Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition](https://mlsanity.com/paper/2505.07862/) (2025)
- [WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering](https://mlsanity.com/paper/2606.00724/) (2026)
- [Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention](https://mlsanity.com/paper/2506.15714/) (2025)
- [Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks](https://mlsanity.com/paper/2507.19514/) (2025)
- [WNet: Discrete Wavelets Transform for Efficient Token Mixing](https://mlsanity.com/paper/2610.04720/) (2026)

## Abstract (arXiv)

Transformer architectures, underpinned by the self-attention mechanism, have achieved state-of-the-art results across numerous natural language processing (NLP) tasks by effectively modeling long-range dependencies. However, the computational complexity of self-attention, scaling quadratically with input sequence length, presents significant challenges for processing very long sequences or operating under resource constraints. This paper introduces the Learnable Multi-Scale Wavelet Transformer (LMWT), a novel architecture that replaces the standard dot-product self-attention with a learnable multi-scale Haar wavelet transform module. Leveraging the intrinsic multi-resolution properties of wavelets, the LMWT efficiently captures both local details and global context. Crucially, the parameters of the wavelet transform, including scale-specific coefficients, are learned end-to-end during training, allowing the model to adapt its decomposition strategy to the data and task. We present the detailed mathematical formulation of the learnable Haar wavelet module and its integration into the transformer framework, supplemented by an architectural diagram. We conduct a comprehensive experimental evaluation on a standard machine translation benchmark (WMT16 En-De), comparing the LMWT against a baseline self-attention transformer using metrics like BLEU score, perplexity, and token accuracy. Furthermore, we analyze the computational complexity, highlighting the linear scaling of our approach, discuss its novelty in the context of related work, and explore the interpretability offered by visualizing the learned Haar coefficients. Our results indicate that the LMWT achieves competitive performance while offering substantial computational advantages, positioning it as a promising and novel alternative for efficient sequence modeling.