mlsanityData to 14 Oct 2026

Long-range gene expression prediction with token alignment of large language model

arXiv 2410.01858 ↗First posted 2 Oct 2024q-bio.CB cs.LG q-bio.GNcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Gene expression is a cellular process that plays a fundamental role in human phenotypical variations and diseases. Despite advances of deep learning models for gene expression prediction, recent benchmarks have revealed their inability to learn distal regulatory grammar. Here, we address this challenge by leveraging a pretrained large language model to enhance gene expression prediction. We introduce Genetic sequence Token Alignment (GTA), which aligns genetic sequence features with natural language tokens, allowing for symbolic reasoning of genomic sequence features via the frozen language model. This cross-modal adaptation learns the regulatory grammar and allows us to further incorporate gene-specific human annotations as prompts, enabling in-context learning that is not possible with existing models. Trained on lymphoblastoid cells, GTA was evaluated on cells from the Geuvadis consortium and outperforms state-of-the-art models such as Enformer, achieving a Spearman correlation of 0.65, a 10% improvement. Additionally, GTA offers improved interpretation of long-range interactions through the identification of the most meaningful sections of the input genetic context. GTA represents a powerful and novel cross-modal approach to gene expression prediction by utilizing a pretrained language model, in a paradigm shift from conventional gene expression models trained only on sequence data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 41 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

12 of 41 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  11. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  12. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Research
  2. 2025Genomic Next-Token Predictors are In-Context Learners
  3. 2025JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
  4. 2026Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models
  5. 2025Gene42: Long-Range Genomic Foundation Model With Dense Attention
  6. 2026Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment
  7. 2023PoET: A generative model of protein families as sequences-of-sequences
  8. 2026Extending Sequence Length is Not All You Need: Effective Integration of Multimodal Signals for Gene Expression Prediction