# Cell-ontology guided transcriptome foundation model

arXiv: https://arxiv.org/abs/2408.12373 · HTML: https://mlsanity.com/paper/2408.12373/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [A Survey on Oversmoothing in Graph Neural Networks](https://mlsanity.com/paper/2303.10993/) (2023)
- [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](https://mlsanity.com/paper/2105.02605/) (2021)
- [A Survey on Graph Representation Learning Methods](https://mlsanity.com/paper/2204.01855/) (2022)
- [Dissecting Supervised Contrastive Learning](https://mlsanity.com/paper/2102.08817/) (2021)
- [Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning](https://mlsanity.com/paper/2204.07596/) (2022)
- [Generalized Neural Collapse for a Large Number of Classes](https://mlsanity.com/paper/2310.05351/) (2023)
- [Personalized PageRank to a Target Node, Revisited](https://mlsanity.com/paper/2006.11876/) (2020)
- [Large-Scale Cell Representation Learning via Divide-and-Conquer Contrastive Learning](https://mlsanity.com/paper/2306.04371/) (2023)
- [scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis in Brain](https://mlsanity.com/paper/2310.02713/) (2023)
- [A single-cell gene expression language model](https://mlsanity.com/paper/2210.14330/) (2022)
- [Targeted sampling from massive block model graphs with personalized PageRank](https://mlsanity.com/paper/1910.12937/) (2019)

## Built on by (most cited first, 7 of 7)

- [SToFM: a Multi-scale Foundation Model for Spatial Transcriptomics](https://mlsanity.com/paper/2507.11588/) (2025)
- [Cell2Text: Multimodal LLM for Generating Single-Cell Descriptions from RNA-Seq Data](https://mlsanity.com/paper/2509.24840/) (2025)
- [The cell as a token: high-dimensional geometry in language models and cell embeddings](https://mlsanity.com/paper/2503.20278/) (2025)
- [OmniCellTOSG: The First Cell Text-Omic Signaling Graphs Dataset for Graph Language Foundation Modeling](https://mlsanity.com/paper/2504.02148/) (2025)
- [BioVERSE: Representation Alignment of Biomedical Modalities to LLMs for Multi-Modal Reasoning](https://mlsanity.com/paper/2510.01428/) (2025)
- [DOGMA: Weaving Structural Information into Data-centric Single-cell Transcriptomics Analysis](https://mlsanity.com/paper/2602.01839/) (2026)
- [Effective Biological Representation Learning by Masking Gene Expression](https://mlsanity.com/paper/2605.31562/) (2026)

## Abstract (arXiv)

Transcriptome foundation models TFMs hold great promises of deciphering the transcriptomic language that dictate diverse cell functions by self-supervised learning on large-scale single-cell gene expression data, and ultimately unraveling the complex mechanisms of human diseases. However, current TFMs treat cells as independent samples and ignore the taxonomic relationships between cell types, which are available in cell ontology graphs. We argue that effectively leveraging this ontology information during the TFM pre-training can improve learning biologically meaningful gene co-expression patterns while preserving TFM as a general purpose foundation model for downstream zero-shot and fine-tuning tasks. To this end, we present single cell, Cell-ontology guided TFM scCello. We introduce cell-type coherence loss and ontology alignment loss, which are minimized along with the masked gene expression prediction loss during the pre-training. The novel loss component guide scCello to learn the cell-type-specific representation and the structural relation between cell types from the cell ontology graph, respectively. We pre-trained scCello on 22 million cells from CellxGene database leveraging their cell-type labels mapped to the cell ontology graph from Open Biological and Biomedical Ontology Foundry. Our TFM demonstrates competitive generalization and transferability performance over the existing TFMs on biologically important tasks including identifying novel cell types of unseen cells, prediction of cell-type-specific marker genes, and cancer drug responses.