mlsanityData to 14 Oct 2026

mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design

arXiv 2408.09048 ↗First posted 16 Aug 2024q-bio.QM cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Messenger RNA (mRNA)-based vaccines are accelerating the discovery of new drugs and revolutionizing the pharmaceutical industry. However, selecting particular mRNA sequences for vaccines and therapeutics from extensive mRNA libraries is costly. Effective mRNA therapeutics require carefully designed sequences with optimized expression levels and stability. This paper proposes a novel contextual language model (LM)-based embedding method: mRNA2vec. In contrast to existing mRNA embedding approaches, our method is based on the self-supervised teacher-student learning framework of data2vec. We jointly use the 5' untranslated region (UTR) and coding sequence (CDS) region as the input sequences. We adapt our LM-based approach specifically to mRNA by 1) considering the importance of location on the mRNA sequence with probabilistic masking, 2) using Minimum Free Energy (MFE) prediction and Secondary Structure (SS) classification as additional pretext tasks. mRNA2vec demonstrates significant improvements in translation efficiency (TE) and expression level (EL) prediction tasks in UTR compared to SOTA methods such as UTR-LM. It also gives a competitive performance in mRNA stability and protein production level tasks in CDS such as CodonBERT.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 12 of its 30 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

12 of 30 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2022Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Seriescited by 54
  9. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  10. 2023Searching Large Neighborhoods for Integer Linear Programs with Contrastive Learningcited by 37
  11. 2024RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Taskscited by 33
  12. 2017dna2vec: Consistent vector representations of variable-length k-merscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Propertiescited by 1
  2. 2025Decoding Translation-Related Functional Sequences in 5'UTRs Using Interpretable Deep Learning Modelscited by 0
  3. 2026An Evolutionary Approach for Designing Stable and Highly Expressible Low-Immunogenicity Therapeutic mRNA Sequencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Helix-mRNA: A Hybrid Foundation Model For Full Sequence mRNA Therapeutics
  2. 2026ctELM: Decoding and Manipulating Embeddings of Clinical Trials with Embedding Language Models
  3. 2026AMix-2: Establishing Protein as a Native Modality in Large Language Models
  4. 2024Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Research
  5. 2026CrossLLM-Mamba: Multimodal State Space Fusion of LLMs for RNA Interaction Prediction
  6. 2024Diffusion on language model encodings for protein sequence generation
  7. 2024TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineering
  8. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models