Myosotis: structured computation for attention like layer
Not on the map. Why?
Abstract (arXiv)
Attention layers apply a sequence-to-sequence mapping whose parameters depend on the pairwise interactions of the input elements. However, without any structural assumptions, memory and compute scale quadratically with the sequence length. The two main ways to mitigate this are to introduce sparsity by ignoring a sufficient amount of pairwise interactions or to introduce recurrent dependence along them, as SSM does. Although both approaches are reasonable, they both have disadvantages. We propose a novel algorithm that combines the advantages of both concepts. Our idea is based on the efficient inversion of tree-structured matrices.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 24 Sep 2025.
Builds on
28 of 33 references.
References · 28
- 2017Attention Is All You Needcited by 60,361
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2020Big Bird: Transformers for Longer Sequencescited by 1,684
- 2020Rethinking Attention with Performerscited by 1,601
- 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
- 2022Simplified State Space Layers for Sequence Modelingcited by 788
- 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
- 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
- 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
- 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
16 more
- 2022It's Raw! Audio Generation with State-Space Modelscited by 171
- 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 143
- 2022Liquid Structural State-Space Modelscited by 136
- 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
- 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
- 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
- 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
- 2020Lipschitz Recurrent Neural Networkscited by 89
- 2017Dilated Recurrent Neural Networkscited by 83
- 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
- 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
- 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
- 2018Trellis Networks for Sequence Modelingcited by 52
- 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
- 2021Parallelizing Legendre Memory Unit Trainingcited by 23
- 2022Towards a General Purpose CNN for Long Range Dependencies in NDcited by 16
Built on by
Citing papers per year: 2026: 2.
2 papers.
- 2026HLA: Hadamard Linear Attentioncited by 1
- 2026The Key to Going Linear: Analysis-Driven Transformer Linearizationcited by 1
- 2026The Key to Going Linear: Analysis-Driven Transformer Linearization8 Jul 2026
- 2026HLA: Hadamard Linear Attention12 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026SMat-Attention: Structured Long-Context Sequence Modeling
- 2025How Many Heads Make an SSM? A Unified Framework for Attention and State Space Models
- 2025On Structured State-Space Duality
- 2025Eigenvalues as a Metric for Memory Dynamics in Sequence Models
- 2025Element-wise Attention Is All You Need
- 2024ATP: Enabling Fast LLM Serving via Attention on Top Principal Keys
- 2024The Hidden Attention of Mamba Models
- 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs