mlsanityData to 14 Oct 2026

HELM: Hierarchical Encoding for mRNA Language Modeling

arXiv 2410.12459 ↗First posted 16 Oct 2024cs.LG cs.CEcited by 10

Not on the map. Why?

Abstract (arXiv)

Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on seven diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 79 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

24 of 79 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  5. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  6. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
  7. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  8. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 143
  9. 2024Dirichlet Flow Matching with Applications to DNA Sequence Designcited by 115
  10. 2019Making Better Mistakes: Leveraging Class Hierarchies with Deep Networkscited by 113
  11. 2022Hyperbolic Image Segmentationcited by 102
  12. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 75
12 more
  1. 2021Deciphering antibody affinity maturation with language models and weakly supervised learningcited by 53
  2. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 50
  3. 2018Hierarchy-based Image Embeddings for Semantic Image Retrievalcited by 41
  4. 2024RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Taskscited by 33
  5. 2024Language Models as Hierarchy Encoderscited by 28
  6. 2022Learning Hierarchy Aware Features for Reducing Mistake Severitycited by 26
  7. 2023A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictionscited by 20
  8. 2024Large scale paired antibody language modelscited by 16
  9. 2021Deep learning models for predicting RNA degradation via dual crowdsourcingcited by 8
  10. 2024Beyond Sequence: Impact of Geometric Context for RNA Property Predictioncited by 8
  11. 2024SE(3)-Hyena Operator for Scalable Equivariant Learningcited by 6
  12. 2023Global Hierarchical Neural Networks using Hierarchical Softmaxcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 2.

10 papers.

  1. 2024HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classificationcited by 13
  2. 2025DualEquiNet: A Dual-Space Hierarchical Equivariant Network for Large Biomoleculescited by 5
  3. 2025Geometric Hyena Networks for Large-scale Equivariant Learningcited by 4
  4. 2025BioBO: Biology-informed Bayesian Optimization for Perturbation Designcited by 2
  5. 2025Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Modelscited by 1
  6. 2026mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Propertiescited by 1
  7. 2025Multimodal Modeling of CRISPR-Cas12 Activity Using Foundation Models and Chromatin Accessibility Datacited by 0
  8. 2025Generative Foundation Model for Structured and Unstructured Electronic Health Recordscited by 0
  9. 2025HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modelingcited by 0
  10. 2026Predictor-Guided Latent Space Codon Optimization for Maximizing Protein Expressioncited by 0