# BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models

arXiv: https://arxiv.org/abs/2506.14861 · HTML: https://mlsanity.com/paper/2506.14861/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2025.

## Built on by (most cited first, 1 of 1)

- [BioVERSE: Representation Alignment of Biomedical Modalities to LLMs for Multi-Modal Reasoning](https://mlsanity.com/paper/2510.01428/) (2025)

## Abstract (arXiv)

Transcriptomic foundation models pretrained with masked language modeling can achieve low pretraining loss yet produce poor cell representations for downstream tasks. We introduce whole-cell expression decoding (WCED), where models reconstruct the entire gene vocabulary from a single CLS token embedding, even with limited inputs, creating a maximally informative bottleneck. WCED consistently outperforms MLM on all downstream metrics despite higher reconstruction error during training. Gene-level error tracking reveals that both methods preferentially learn genes whose expression co-varies with stable transcriptional programs rather than those driven by transient factors. We further add hierarchical cross-entropy loss that exploits Cell Ontology structure for zero-shot annotation at multiple granularity levels. Models trained with these objectives achieve best overall performance across CZI benchmarks, on zero-shot batch integration and linear probing cell-type annotation. Methods are implemented in biomed-multi-omic ( https://github.com/BiomedSciAI/biomed-multi-omic ), an open-source framework for transcriptomic foundation model development.