# PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer

arXiv: https://arxiv.org/abs/2512.03111 · HTML: https://mlsanity.com/paper/2512.03111/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 31 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 6 of 6)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Point Cloud Pre-training with Diffusion Models](https://mlsanity.com/paper/2311.14960/) (2023)
- [EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encoder](https://mlsanity.com/paper/2212.04098/) (2022)
- [GeneFormer: Learned Gene Compression using Transformer-based Context Modeling](https://mlsanity.com/paper/2212.08379/) (2022)
- [Unsupervised Point Cloud Pre-Training via Contrasting and Clustering](https://mlsanity.com/paper/2202.02543/) (2022)
- [GeneMamba: An Efficient and Effective Foundation Model on Single Cell Data](https://mlsanity.com/paper/2504.16956/) (2025)

## Abstract (arXiv)

Single-cell RNA sequencing (scRNA-seq) is essential for decoding tumor heterogeneity. However, pan-cancer research still faces two key challenges: learning discriminative and efficient single-cell representations, and establishing a comprehensive evaluation benchmark. In this paper, we introduce PanFoMa, a lightweight hybrid neural network that combines the strengths of Transformers and state-space models to achieve a balance between performance and efficiency. PanFoMa consists of a front-end local-context encoder with shared self-attention layers to capture complex, order-independent gene interactions; and a back-end global sequential feature decoder that efficiently integrates global context using a linear-time state-space model. This modular design preserves the expressive power of Transformers while leveraging the scalability of Mamba to enable transcriptome modeling, effectively capturing both local and global regulatory signals. To enable robust evaluation, we also construct a large-scale pan-cancer single-cell benchmark, PanFoMaBench, containing over 3.5 million high-quality cells across 33 cancer subtypes, curated through a rigorous preprocessing pipeline. Experimental results show that PanFoMa outperforms state-of-the-art models on our pan-cancer benchmark (+4.0%) and across multiple public tasks, including cell type annotation (+7.4%), batch integration (+4.0%) and multi-omics integration (+3.1%). The code is available at https://github.com/Xiaoshui-Huang/PanFoMa.