# RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks

arXiv: https://arxiv.org/abs/2403.00043 · HTML: https://mlsanity.com/paper/2403.00043/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 83 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions](https://mlsanity.com/paper/2310.03281/) (2023)

## Built on by (most cited first, 30 of 33)

- [Path Planning for Masked Diffusion Model Sampling](https://mlsanity.com/paper/2502.03540/) (2025)
- [HELM: Hierarchical Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2410.12459/) (2024)
- [Beyond Sequence: Impact of Geometric Context for RNA Property Prediction](https://mlsanity.com/paper/2410.11933/) (2024)
- [Large Language Models in Bioinformatics: A Survey](https://mlsanity.com/paper/2503.04490/) (2025)
- [RNA-FrameFlow: Flow Matching for de novo 3D RNA Backbone Design](https://mlsanity.com/paper/2406.13839/) (2024)
- [NeuralPLexer3: Accurate Biomolecular Complex Structure Prediction with Flow Models](https://mlsanity.com/paper/2412.10743/) (2024)
- [mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design](https://mlsanity.com/paper/2408.09048/) (2024)
- [CoPRA: Bridging Cross-domain Pretrained Sequence Models with Complex Structures for Protein-RNA Binding Affinity Prediction](https://mlsanity.com/paper/2409.03773/) (2024)
- [Comprehensive benchmarking of large language models for RNA secondary structure prediction](https://mlsanity.com/paper/2410.16212/) (2024)
- [Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification](https://mlsanity.com/paper/2502.07299/) (2025)
- [Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models](https://mlsanity.com/paper/2411.11808/) (2024)
- [In-Context Learning can distort the relationship between sequence likelihoods and biological fitness](https://mlsanity.com/paper/2504.17068/) (2025)
- [Large Language Models Meet Virtual Cell: A Survey](https://mlsanity.com/paper/2510.07706/) (2025)
- [NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction](https://mlsanity.com/paper/2511.02888/) (2025)
- [Structure-based RNA Design by Step-wise Optimization of Latent Diffusion Model](https://mlsanity.com/paper/2601.19232/) (2026)
- [Designing RNAs with Language Models](https://mlsanity.com/paper/2602.12470/) (2026)
- [Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction](https://mlsanity.com/paper/2603.22330/) (2026)
- [mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Properties](https://mlsanity.com/paper/2605.31296/) (2026)
- [Multimodal Modeling of CRISPR-Cas12 Activity Using Foundation Models and Chromatin Accessibility Data](https://mlsanity.com/paper/2506.11182/) (2025)
- [A HyperGraphMamba-Based Multichannel Adaptive Model for ncRNA Classification](https://mlsanity.com/paper/2509.20240/) (2025)
- [HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2509.24655/) (2025)
- [SAE-RNA: A Sparse Autoencoder Model for Interpreting RNA Language Model Representations](https://mlsanity.com/paper/2510.02734/) (2025)
- [Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances](https://mlsanity.com/paper/2511.03354/) (2025)
- [CrossLLM-Mamba: Multimodal State Space Fusion of LLMs for RNA Interaction Prediction](https://mlsanity.com/paper/2602.22236/) (2026)
- [ViroBench: Benchmarking Nucleotide Foundation Models on Viral Genomics Tasks](https://mlsanity.com/paper/2605.25388/) (2026)
- [TadA-Bench: A Million-Variant Benchmark for Future-Round Discovery Toward Agentic Protein Engineering](https://mlsanity.com/paper/2606.02624/) (2026)
- [MultiMolecule: a modular ecosystem for biomolecular sequence-model workflows](https://mlsanity.com/paper/2606.16540/) (2026)
- [RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling](https://mlsanity.com/paper/2608.22849/) (2026)
- [Sequence-Informed Geometric Evaluation of RNA 3D Structures](https://mlsanity.com/paper/2609.10644/) (2026)
- [Differentiable RNA Secondary Structure Extraction for Deep Learning](https://mlsanity.com/paper/2609.30752/) (2026)

## Abstract (arXiv)

While RNA has recently been recognized as an interesting small-molecule drug target, many challenges remain to be addressed before we take full advantage of it. This emphasizes the necessity to improve our understanding of its structures and functions. Over the years, sequencing technologies have produced an enormous amount of unlabeled RNA data, which hides a huge potential. Motivated by the successes of protein language models, we introduce RiboNucleic Acid Language Model (RiNALMo) to unveil the hidden code of RNA. RiNALMo is the largest RNA language model to date, with 650M parameters pre-trained on 36M non-coding RNA sequences from several databases. It can extract hidden knowledge and capture the underlying structure information implicitly embedded within the RNA sequences. RiNALMo achieves state-of-the-art results on several downstream tasks. Notably, we show that its generalization capabilities overcome the inability of other deep learning methods for secondary structure prediction to generalize on unseen RNA families.