# Large-Scale Cell Representation Learning via Divide-and-Conquer Contrastive Learning

arXiv: https://arxiv.org/abs/2306.04371 · HTML: https://mlsanity.com/paper/2306.04371/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2023.

## Built on by (most cited first, 11 of 11)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [SToFM: a Multi-scale Foundation Model for Spatial Transcriptomics](https://mlsanity.com/paper/2507.11588/) (2025)
- [Cell-ontology guided transcriptome foundation model](https://mlsanity.com/paper/2408.12373/) (2024)
- [Language-Enhanced Representation Learning for Single-Cell Transcriptomics](https://mlsanity.com/paper/2503.09427/) (2025)
- [GeneMamba: An Efficient and Effective Foundation Model on Single Cell Data](https://mlsanity.com/paper/2504.16956/) (2025)
- [ChatCell: Facilitating Single-Cell Analysis with Natural Language](https://mlsanity.com/paper/2402.08303/) (2024)
- [scDrugMap: Benchmarking Large Foundation Models for Drug Response Prediction](https://mlsanity.com/paper/2505.05612/) (2025)
- [LLM4Cell: A Survey of Large Language and Agentic Models for Single-Cell Biology](https://mlsanity.com/paper/2510.07793/) (2025)
- [Unsupervised Learning of Molecular Embeddings for Enhanced Clustering and Emergent Properties for Chemical Compounds](https://mlsanity.com/paper/2310.18367/) (2023)
- [Human-Guided Causal Knowledge Injection for Virtual Cells](https://mlsanity.com/paper/2608.08430/) (2026)

## Abstract (arXiv)

Single-cell RNA sequencing (scRNA-seq) data is a potent tool for comprehending the "language of life" and can provide insights into various downstream biomedical tasks. Large-scale language models (LLMs) are starting to be used for cell representation learning. However, current LLM-based cell representation learning methods depend solely on the BERT architecture, causing an anisotropic embedding space that leads to inefficient semantic representation. Contrastive learning alleviates this problem by distributing the embeddings uniformly. As a larger batch size in contrastive learning results in better representation, the practical application of contrastive learning in cell representation learning is hampered by the high dimensionality of scRNA-seq data and the large parameter volume of LLMs. To address the batch size limitation, we propose a novel divide-and-conquer contrastive learning approach to decouple the batch size from the GPU memory size for cell representation learning. Based on our divide-and-conquer contrastive learning approach, we introduce Single-Cell Language Model CellLM, a large-scale cell representation learning model to handle high-dimensional scRNA-seq data with tens of thousands of genes. CellLM has over 50 million parameters trained with 2 million scRNA-seq data and makes the first attempt to learn cell language models from both normal cells and cancer cells. CellLM achieves new state-of-the-art (SOTA) results in all evaluated downstream tasks: including a 71.8 F_1-score for cell type annotation (a 3.0% absolute improvement over scBERT), an average F_1-score of 88.9 for single-cell drug sensitivity prediction in a few-shot scenario (an 8.3% absolute improvement), and a 93.4 Pearson's correlation for single-omics cell line drug sensitivity prediction (a 6.2% absolute improvement).