mlsanityData to 14 Oct 2026

Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling

arXiv 2403.03234 ↗First posted 5 Mar 2024q-bio.GN cs.LGcited by 113

Not on the map. Why?

Cited by 25 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale sequence modeling has sparked rapid advances that now extend into biology and genomics. However, modeling genomic sequences introduces challenges such as the need to model long-range token interactions, the effects of upstream and downstream regions of the genome, and the reverse complementarity (RC) of DNA. Here, we propose an architecture motivated by these challenges that builds off the long-range Mamba block, and extends it to a BiMamba component that supports bi-directionality, and to a MambaDNA block that additionally supports RC equivariance. We use MambaDNA as the basis of Caduceus, the first family of RC equivariant bi-directional long-range DNA language models, and we introduce pre-training and fine-tuning strategies that yield Caduceus DNA foundation models. Caduceus outperforms previous long-range models on downstream benchmarks; on a challenging long-range variant effect prediction task, Caduceus exceeds the performance of 10x larger models that do not leverage bi-directionality or equivariance.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Builds on 19 of its 63 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

19 of 63 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  9. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  10. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  11. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  12. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
7 more
  1. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  2. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  3. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
  4. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 143
  5. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  6. 2017Semi-supervised sequence tagging with bidirectional language modelscited by 86
  7. 2022Pretraining Without Attentioncited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 35; 2025: 47; 2026: 31.

113 papers.

Sort:

By area:

  1. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  3. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  4. 2024Is Mamba Effective for Time Series Forecasting?cited by 114
  5. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  6. 2025Transformers without Normalizationcited by 70
  7. 2024TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecastingcited by 64
  8. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  9. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 49
  10. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 44
  11. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  12. 2024A Survey of Mambacited by 39
38 more
  1. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  2. 2024EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Recordscited by 32
  3. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  4. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 24
  5. 2025GENERator: A Long-Context Generative Genomic Foundation Modelcited by 22
  6. 2024Mambular: A Sequential Model for Tabular Deep Learningcited by 21
  7. 2024Genomic Language Models: Opportunities and Challengescited by 20
  8. 2024Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theorycited by 19
  9. 2024Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequencescited by 13
  10. 2025Stronger Normalization-Free Transformerscited by 13
  11. 2024Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNAcited by 12
  12. 2024GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Modelscited by 10
  13. 2024TSCMamba: Mamba Meets Multi-View Learning for Time Series Classificationcited by 10
  14. 2025Contextualizing biological perturbation experiments through languagecited by 10
  15. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  16. 2024Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Modelscited by 8
  17. 2024Brain-to-Text Benchmark '24: Lessons Learnedcited by 8
  18. 2025Fast and Low-Cost Genomic Foundation Models via Outlier Removalcited by 8
  19. 2024Multi-modal Transfer Learning between Biological Foundation Modelscited by 7
  20. 2025HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Modelcited by 7
  21. 2024SE(3)-Hyena Operator for Scalable Equivariant Learningcited by 6
  22. 2024DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNAcited by 6
  23. 2025Regulatory DNA sequence Design with Reinforcement Learningcited by 6
  24. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 6
  25. 2026D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generationcited by 6
  26. 2024Sequential Order-Robust Mamba for Time Series Forecastingcited by 5
  27. 2025JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Modelcited by 5
  28. 2025SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Modelcited by 5
  29. 2024BarcodeMamba: State Space Models for Biodiversity Analysiscited by 4
  30. 2025Learning to Discover Regulatory Elements for Gene Expression Predictioncited by 4
  31. 2025Linear Attention for Efficient Bidirectional Sequence Modelingcited by 4
  32. 2025Geometric Hyena Networks for Large-scale Equivariant Learningcited by 4
  33. 2024AI-driven View Guidance System in Intra-cardiac Echocardiography Imagingcited by 3
  34. 2025Revisiting Convolution Architecture in the Realm of DNA Foundation Modelscited by 3
  35. 2025Hyperbolic Genome Embeddingscited by 3
  36. 2026Improved Sampling Schedules for Discrete Diffusion Modelscited by 3
  37. 2024Multi-Agent Reinforcement Learning with Selective State-Space Modelscited by 2
  38. 2025Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unificationcited by 2

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The limits of bio-molecular modeling with large language models : a cross-scale evaluation
  2. 2026dnaHNet: A Scalable and Hierarchical Foundation Model for Genomic Sequence Learning
  3. 2025Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs
  4. 2026Flexible Flows for Biological Sequence Design
  5. 2023Embed-Search-Align: DNA Sequence Alignment using Transformer Models
  6. 2023PoET: A generative model of protein families as sequences-of-sequences
  7. 2024Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Research
  8. 2024Chaining thoughts and LLMs to learn DNA structural biophysics