# GeneMamba: An Efficient and Effective Foundation Model on Single Cell Data

arXiv: https://arxiv.org/abs/2504.16956 · HTML: https://mlsanity.com/paper/2504.16956/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [LangCell: Language-Cell Pre-training for Cell Identity Understanding](https://mlsanity.com/paper/2405.06708/) (2024)
- [xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data](https://mlsanity.com/paper/2311.15156/) (2023)
- [Large-Scale Cell Representation Learning via Divide-and-Conquer Contrastive Learning](https://mlsanity.com/paper/2306.04371/) (2023)
- [Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation](https://mlsanity.com/paper/2302.03038/) (2023)
- [scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis in Brain](https://mlsanity.com/paper/2310.02713/) (2023)
- [scRDiT: Generating single-cell RNA-seq data by diffusion transformers and accelerating sampling](https://mlsanity.com/paper/2404.06153/) (2024)
- [White-Box Diffusion Transformer for single-cell RNA-seq generation](https://mlsanity.com/paper/2411.06785/) (2024)

## Built on by (most cited first, 5 of 5)

- [LLM4Cell: A Survey of Large Language and Agentic Models for Single-Cell Biology](https://mlsanity.com/paper/2510.07793/) (2025)
- [Clustering with Communication: A Variational Framework for Single Cell Representation Learning](https://mlsanity.com/paper/2505.04891/) (2025)
- [PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer](https://mlsanity.com/paper/2512.03111/) (2025)
- [CytoBERT: A Foundation Model for Cytometry Data](https://mlsanity.com/paper/2608.14414/) (2026)
- [scHFBiMamba: a bidirectional Mamba autoencoder for cardiac disease-state classification from single-nucleus RNA-seq data](https://mlsanity.com/paper/2610.15117/) (2026)

## Abstract (arXiv)

Single-cell RNA sequencing (scRNA-seq) enables high-resolution analysis of cellular heterogeneity, but its complexity, which is marked by high dimensionality, sparsity, and batch effects, which poses major computational challenges. Transformer-based models have made significant advances in this domain but are often limited by their quadratic complexity and suboptimal handling of long-range dependencies. In this work, we introduce GeneMamba, a scalable and efficient foundation model for single-cell transcriptomics built on state space modeling. Leveraging the Bi-Mamba architecture, GeneMamba captures bidirectional gene context with linear-time complexity, offering substantial computational gains over transformer baselines. The model is pretrained on nearly 30 million cells and incorporates biologically informed objectives, including pathway-aware contrastive loss and rank-based gene encoding. We evaluate GeneMamba across diverse tasks, including multi-batch integration, cell type annotation, and gene-gene correlation, demonstrating strong performance, interpretability, and robustness. These results position GeneMamba as a practical and powerful alternative to transformer-based methods, advancing the development of biologically grounded, scalable tools for large-scale single-cell data analysis.