mlsanityData to 7 Oct 2026

Comba: Improving Bilinear RNNs with Closed-loop Control

arXiv 2506.02475 ↗First posted 3 Jun 2025cs.LG cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Recent efficient sequence modeling methods such as Gated DeltaNet, TTT, and RWKV-7 have achieved performance improvements by supervising the recurrent memory management through Delta learning rule. Unlike previous state-space models (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, structurally resembling bilinear systems. In this paper, we first introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models. Then, based on closed-loop control theory, we propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus. Comba demonstrates superior performance and computation efficiency in both language and vision modeling.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 80 of its 116 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

80 of 116 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
38 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  5. 2024VMamba: Visual State Space Modelcited by 955
  6. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  11. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  12. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  13. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  14. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  15. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  16. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  17. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  18. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  19. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  20. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  21. 2024Titans: Learning to Memorize at Test Timecited by 303
  22. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  23. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  24. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  25. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  26. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  27. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 134
  28. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  29. 2020LaSOT: A High-quality Large-scale Single Object Tracking Benchmarkcited by 130
  30. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  31. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  32. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  33. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  34. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  35. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  36. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  37. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  38. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  2. 2025Selective Rotary Position Embeddingcited by 8
  3. 2026Mem3R: Streaming 3D Reconstruction with Hybrid Memory via Test-Time Trainingcited by 5
  4. 2026Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrencescited by 5
  5. 2026Morphing into Hybrid Attention Modelscited by 2
  6. 2026Memory for Large Language Modelscited by 2
  7. 2026WriteSAE: Sparse Autoencoders for Recurrent Statecited by 1
  8. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representationscited by 1
  9. 2026Proteus: Incremental Memory Activation for Long-Context Sequence Modelingcited by 1
  10. 2026Kalman Delta Networks: Uncertainty-aware Associative Memorycited by 1
  11. 2026DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Modelscited by 0
  12. 2026Q-Delta: Beyond Key-Value Associative State Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
  2. 2023Resurrecting Recurrent Neural Networks for Long Sequences
  3. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
  4. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  5. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  6. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the story
  7. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
  8. 2023State space models can express n-gram languages