# Comprehensive benchmarking of large language models for RNA secondary structure prediction

arXiv: https://arxiv.org/abs/2410.16212 · HTML: https://mlsanity.com/paper/2410.16212/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 81 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [Deep Learning in Bioinformatics](https://mlsanity.com/paper/1603.06430/) (2016)
- [RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks](https://mlsanity.com/paper/2403.00043/) (2024)
- [Designing RNA Secondary Structures is Hard](https://mlsanity.com/paper/1710.11513/) (2017)
- [Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information](https://mlsanity.com/paper/1912.05625/) (2019)
- [LinearPartition: Linear-Time Approximation of RNA Folding Partition Function and Base Pairing Probabilities](https://mlsanity.com/paper/1912.13190/) (2019)
- [Rethinking Performance Measures of RNA Secondary Structure Problems](https://mlsanity.com/paper/2401.05351/) (2023)

## Built on by (most cited first, 3 of 3)

- [Teaching Metric Distance to Discrete Autoregressive Language Models](https://mlsanity.com/paper/2503.02379/) (2025)
- [Designing RNAs with Language Models](https://mlsanity.com/paper/2602.12470/) (2026)
- [An Evolutionary Approach for Designing Stable and Highly Expressible Low-Immunogenicity Therapeutic mRNA Sequences](https://mlsanity.com/paper/2605.27986/) (2026)

## Abstract (arXiv)

Inspired by the success of large language models (LLM) for DNA and proteins, several LLM for RNA have been developed recently. RNA-LLM uses large datasets of RNA sequences to learn, in a self-supervised way, how to represent each RNA base with a semantically rich numerical vector. This is done under the hypothesis that obtaining high-quality RNA representations can enhance data-costly downstream tasks. Among them, predicting the secondary structure is a fundamental task for uncovering RNA functional mechanisms. In this work we present a comprehensive experimental analysis of several pre-trained RNA-LLM, comparing them for the RNA secondary structure prediction task in an unified deep learning framework. The RNA-LLM were assessed with increasing generalization difficulty on benchmark datasets. Results showed that two LLM clearly outperform the other models, and revealed significant challenges for generalization in low-homology scenarios.