# HELM: Hierarchical Encoding for mRNA Language Modeling

arXiv: https://arxiv.org/abs/2410.12459 · HTML: https://mlsanity.com/paper/2410.12459/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 79 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [Dirichlet Flow Matching with Applications to DNA Sequence Design](https://mlsanity.com/paper/2402.05841/) (2024)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Hyperbolic Image Segmentation](https://mlsanity.com/paper/2203.05898/) (2022)
- [Dirichlet Diffusion Score Model for Biological Sequence Generation](https://mlsanity.com/paper/2305.10699/) (2023)
- [Deciphering antibody affinity maturation with language models and weakly supervised learning](https://mlsanity.com/paper/2112.07782/) (2021)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [Hierarchy-based Image Embeddings for Semantic Image Retrieval](https://mlsanity.com/paper/1809.09924/) (2018)
- [RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks](https://mlsanity.com/paper/2403.00043/) (2024)
- [Language Models as Hierarchy Encoders](https://mlsanity.com/paper/2401.11374/) (2024)
- [Learning Hierarchy Aware Features for Reducing Mistake Severity](https://mlsanity.com/paper/2207.12646/) (2022)
- [A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions](https://mlsanity.com/paper/2310.03281/) (2023)
- [Large scale paired antibody language models](https://mlsanity.com/paper/2403.17889/) (2024)
- [Deep learning models for predicting RNA degradation via dual crowdsourcing](https://mlsanity.com/paper/2110.07531/) (2021)
- [Beyond Sequence: Impact of Geometric Context for RNA Property Prediction](https://mlsanity.com/paper/2410.11933/) (2024)
- [SE(3)-Hyena Operator for Scalable Equivariant Learning](https://mlsanity.com/paper/2407.01049/) (2024)
- [Global Hierarchical Neural Networks using Hierarchical Softmax](https://mlsanity.com/paper/2308.01210/) (2023)

## Built on by (most cited first, 10 of 10)

- [HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification](https://mlsanity.com/paper/2411.07660/) (2024)
- [DualEquiNet: A Dual-Space Hierarchical Equivariant Network for Large Biomolecules](https://mlsanity.com/paper/2506.19862/) (2025)
- [Geometric Hyena Networks for Large-scale Equivariant Learning](https://mlsanity.com/paper/2505.22560/) (2025)
- [BioBO: Biology-informed Bayesian Optimization for Perturbation Design](https://mlsanity.com/paper/2509.19988/) (2025)
- [Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models](https://mlsanity.com/paper/2508.15103/) (2025)
- [mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Properties](https://mlsanity.com/paper/2605.31296/) (2026)
- [Multimodal Modeling of CRISPR-Cas12 Activity Using Foundation Models and Chromatin Accessibility Data](https://mlsanity.com/paper/2506.11182/) (2025)
- [Generative Foundation Model for Structured and Unstructured Electronic Health Records](https://mlsanity.com/paper/2508.16054/) (2025)
- [HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2509.24655/) (2025)
- [Predictor-Guided Latent Space Codon Optimization for Maximizing Protein Expression](https://mlsanity.com/paper/2610.03098/) (2026)

## Abstract (arXiv)

Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on seven diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.