# A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions

arXiv: https://arxiv.org/abs/2310.03281 · HTML: https://mlsanity.com/paper/2310.03281/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 37 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 2 of 2)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)

## Built on by (most cited first, 20 of 20)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks](https://mlsanity.com/paper/2403.00043/) (2024)
- [Genomic Language Models: Opportunities and Challenges](https://mlsanity.com/paper/2407.11435/) (2024)
- [Flow Matching Meets Biology and Life Science: A Survey](https://mlsanity.com/paper/2507.17731/) (2025)
- [HELM: Hierarchical Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2410.12459/) (2024)
- [BEACON: Benchmark for Comprehensive RNA Tasks and Language Models](https://mlsanity.com/paper/2406.10391/) (2024)
- [Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learning](https://mlsanity.com/paper/2505.19501/) (2025)
- [Latent Diffusion Models for Controllable RNA Sequence Generation](https://mlsanity.com/paper/2409.09828/) (2024)
- [mRNA Design and Optimization with Deep Knowledge-Infused Approach](https://mlsanity.com/paper/2505.23862/) (2025)
- [RNACG: A Universal RNA Sequence Conditional Generation model based on Flow-Matching](https://mlsanity.com/paper/2407.19838/) (2024)
- [Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models](https://mlsanity.com/paper/2411.11808/) (2024)
- [mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Properties](https://mlsanity.com/paper/2605.31296/) (2026)
- [OmniGenBench: Automating Large-scale in-silico Benchmarking for Genomic Foundation Models](https://mlsanity.com/paper/2410.01784/) (2024)
- [Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics](https://mlsanity.com/paper/2506.02212/) (2025)
- [Evaluating DNA function understanding in genomic language models using evolutionarily implausible sequences](https://mlsanity.com/paper/2506.10271/) (2025)
- [Decoding Translation-Related Functional Sequences in 5'UTRs Using Interpretable Deep Learning Models](https://mlsanity.com/paper/2507.16801/) (2025)
- [HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2509.24655/) (2025)
- [MultiMolecule: a modular ecosystem for biomolecular sequence-model workflows](https://mlsanity.com/paper/2606.16540/) (2026)
- [VANDAM: Viewing a nucleotide sequence with DNA molecular priors](https://mlsanity.com/paper/2610.00411/) (2026)

## Abstract (arXiv)

The 5' UTR, a regulatory region at the beginning of an mRNA molecule, plays a crucial role in regulating the translation process and impacts the protein expression level. Language models have showcased their effectiveness in decoding the functions of protein and genome sequences. Here, we introduced a language model for 5' UTR, which we refer to as the UTR-LM. The UTR-LM is pre-trained on endogenous 5' UTRs from multiple species and is further augmented with supervised information including secondary structure and minimum free energy. We fine-tuned the UTR-LM in a variety of downstream tasks. The model outperformed the best-known benchmark by up to 42% for predicting the Mean Ribosome Loading, and by up to 60% for predicting the Translation Efficiency and the mRNA Expression Level. The model also applies to identifying unannotated Internal Ribosome Entry Sites within the untranslated region and improves the AUPR from 0.37 to 0.52 compared to the best baseline. Further, we designed a library of 211 novel 5' UTRs with high predicted values of translation efficiency and evaluated them via a wet-lab assay. Experiment results confirmed that our top designs achieved a 32.5% increase in protein production level relative to well-established 5' UTR optimized for therapeutics.