# Language-Enhanced Representation Learning for Single-Cell Transcriptomics

arXiv: https://arxiv.org/abs/2503.09427 · HTML: https://mlsanity.com/paper/2503.09427/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 85 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery](https://mlsanity.com/paper/2311.16208/) (2023)
- [ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts](https://mlsanity.com/paper/2301.12040/) (2023)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules](https://mlsanity.com/paper/2310.14753/) (2023)
- [ProtT3: Protein-to-Text Generation for Text-based Protein Understanding](https://mlsanity.com/paper/2405.12564/) (2024)
- [scDiffusion: conditional generation of high-quality single-cell data using diffusion model](https://mlsanity.com/paper/2401.03968/) (2024)
- [MolTC: Towards Molecular Relational Modeling In Language Models](https://mlsanity.com/paper/2402.03781/) (2024)
- [ReactXT: Understanding Molecular "Reaction-ship" via Reaction-Contextualized Molecule-Text Pretraining](https://mlsanity.com/paper/2405.14225/) (2024)
- [NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation](https://mlsanity.com/paper/2502.12638/) (2025)
- [LangCell: Language-Cell Pre-training for Cell Identity Understanding](https://mlsanity.com/paper/2405.06708/) (2024)
- [Large-Scale Cell Representation Learning via Divide-and-Conquer Contrastive Learning](https://mlsanity.com/paper/2306.04371/) (2023)
- [Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation](https://mlsanity.com/paper/2302.03038/) (2023)
- [A Multi-Modal AI Copilot for Single-Cell Analysis with Instruction Following](https://mlsanity.com/paper/2501.08187/) (2025)

## Built on by (most cited first, 7 of 7)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Towards Unified and Lossless Latent Space for 3D Molecular Latent Diffusion Modeling](https://mlsanity.com/paper/2503.15567/) (2025)
- [Cell-o1: Training LLMs to Solve Single-Cell Reasoning Puzzles with Reinforcement Learning](https://mlsanity.com/paper/2506.02911/) (2025)
- [EnzyControl: Adding Functional and Substrate-Specific Control for Enzyme Backbone Generation](https://mlsanity.com/paper/2510.25132/) (2025)
- [BioVERSE: Representation Alignment of Biomedical Modalities to LLMs for Multi-Modal Reasoning](https://mlsanity.com/paper/2510.01428/) (2025)
- [3D-GSRD: 3D Molecular Graph Auto-Encoder with Selective Re-mask Decoding](https://mlsanity.com/paper/2510.16780/) (2025)
- [Learning 3D Anisotropic Noise Distributions Improves Molecular Force Field Modeling](https://mlsanity.com/paper/2510.22123/) (2025)

## Abstract (arXiv)

Single-cell RNA sequencing (scRNA-seq) offers detailed insights into cellular heterogeneity. Recent advancements leverage single-cell large language models (scLLMs) for effective representation learning. These models focus exclusively on transcriptomic data, neglecting complementary biological knowledge from textual descriptions. To overcome this limitation, we propose scMMGPT, a novel multimodal framework designed for language-enhanced representation learning in single-cell transcriptomics. Unlike existing methods, scMMGPT employs robust cell representation extraction, preserving quantitative gene expression data, and introduces an innovative two-stage pre-training strategy combining discriminative precision with generative flexibility. Extensive experiments demonstrate that scMMGPT significantly outperforms unimodal and multimodal baselines across key downstream tasks, including cell annotation and clustering, and exhibits superior generalization in out-of-distribution scenarios.