mlsanityData to 7 Oct 2026

Focus Your Attention (with Adaptive IIR Filters)

arXiv 2305.14952 ↗First posted 24 May 2023cs.LG eess.SPcited by 12

Not on the map. Why?

Abstract (arXiv)

We present a new layer in which dynamic (i.e.,input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. The input is split into chunks, and the coefficients of these filters are determined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus attention on the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-the-art networks, with a fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable to layers such as Heyna, GPT2, and Mega, both with respect to the number of parameters and the obtained level of performance on multiple long-range sequence problems.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

51 of 70 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2017Decoupled Weight Decay Regularizationcited by 1,722
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  10. 2016HyperNetworkscited by 1,003
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  12. 2020Efficient Transformers: A Surveycited by 784
38 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  2. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  6. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  7. 2019Continual learning with hypernetworkscited by 267
  8. 2022Long Range Language Modeling via Gated State Spacescited by 265
  9. 2017Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Networkcited by 208
  10. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  11. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  12. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  13. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
  14. 2021KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamicscited by 128
  15. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  16. 2022Block-Recurrent Transformerscited by 107
  17. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  18. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  19. 2016Learning feed-forward one-shot learnerscited by 93
  20. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  21. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  22. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  23. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  24. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  25. 2023Effectively Modeling Time Series with Simple Discrete State Spacescited by 54
  26. 2022Pretraining Without Attentioncited by 51
  27. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  28. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  29. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  30. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  31. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  32. 2023Diagonal State Space Augmented Transformers for Speech Recognitioncited by 25
  33. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNscited by 23
  34. 2022HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworkscited by 12
  35. 2022OCD: Learning to Overfit with Conditional Diffusion Modelscited by 9
  36. 2023State Spaces Aren't Enough: Machine Translation Needs Attentioncited by 9
  37. 2022Global memory transformer for processing long documentscited by 4
  38. 2019AdaNN: Adaptive Neural Network-based Equalizer via Online Semi-supervised Learningcited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 2; 2026: 1.

12 papers.

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024The Hidden Attention of Mamba Modelscited by 88
  3. 2023LLMZip: Lossless Text Compression using Large Language Modelscited by 44
  4. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  5. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  6. 2023On the Long Range Abilities of Transformerscited by 10
  7. 2025Mechanistic evaluation of Transformers and state space modelscited by 8
  8. 2023Multi-Dimensional Hyena for Spatial Inductive Biascited by 5
  9. 2025Overflow Prevention Enhances Long-Context Recurrent LLMscited by 5
  10. 2024Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucinationcited by 4
  11. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3
  12. 2026Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulationcited by 0