# A single-cell gene expression language model

arXiv: https://arxiv.org/abs/2210.14330 · HTML: https://mlsanity.com/paper/2210.14330/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 13 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 4 of 4)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)

## Built on by (most cited first, 3 of 3)

- [LangCell: Language-Cell Pre-training for Cell Identity Understanding](https://mlsanity.com/paper/2405.06708/) (2024)
- [Cell-ontology guided transcriptome foundation model](https://mlsanity.com/paper/2408.12373/) (2024)
- [Scalable Single-Cell Gene Expression Generation with Latent Diffusion Models](https://mlsanity.com/paper/2511.02986/) (2025)

## Abstract (arXiv)

Gene regulation is a dynamic process that connects genotype and phenotype. Given the difficulty of physically mapping mammalian gene circuitry, we require new computational methods to learn regulatory rules. Natural language is a valuable analogy to the communication of regulatory control. Machine learning systems model natural language by explicitly learning context dependencies between words. We propose a similar system applied to single-cell RNA expression profiles to learn context dependencies between genes. Our model, Exceiver, is trained across a diversity of cell types using a self-supervised task formulated for discrete count data, accounting for feature sparsity. We found agreement between the similarity profiles of latent sample representations and learned gene embeddings with respect to biological annotations. We evaluated Exceiver on a new dataset and a downstream prediction task and found that pretraining supports transfer learning. Our work provides a framework to model gene regulation on a single-cell level and transfer knowledge to downstream tasks.