# MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design

arXiv: https://arxiv.org/abs/2610.13305 · HTML: https://mlsanity.com/paper/2610.13305/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Preference optimization for molecular design, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference optimization for molecular design](https://mlsanity.com/alignment/preference-optimization-for-molecular-design/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 16 of 16)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design](https://mlsanity.com/paper/2410.13643/) (2024)
- [Learning to Design RNA](https://mlsanity.com/paper/1812.11951/) (2018)
- [gRNAde: Geometric Deep Learning for 3D RNA inverse design](https://mlsanity.com/paper/2305.14749/) (2023)
- [Accurate RNA 3D structure prediction using a language model-based deep learning approach](https://mlsanity.com/paper/2207.01586/) (2022)
- [RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design](https://mlsanity.com/paper/2301.10774/) (2023)
- [Reinforcement learning on structure-conditioned categorical diffusion for protein inverse folding](https://mlsanity.com/paper/2410.17173/) (2024)
- [ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning](https://mlsanity.com/paper/2506.07459/) (2025)
- [RIDER: 3D RNA Inverse Design with Reinforcement Learning-Guided Diffusion](https://mlsanity.com/paper/2602.16548/) (2026)
- [Structure-based RNA Design by Step-wise Optimization of Latent Diffusion Model](https://mlsanity.com/paper/2601.19232/) (2026)

## Abstract (arXiv)

We introduce MORAD, an online reinforcement learning framework for backbone-conditioned RNA diffusion models that trains a single shared policy across diverse target backbones. Structure-conditioned RNA inverse design seeks nucleotide sequences compatible with a target three-dimensional backbone, yet existing reinforcement learning approaches either optimize a separate policy for each target backbone or rely on offline preference optimization, leaving online post-training of a reusable backbone-conditioned model largely unexplored. MORAD combines backbone-conditioned group sampling, group-relative reward normalization, and a forward-process diffusion objective. Beyond existing RL rewards that primarily target tertiary structural quality, MORAD jointly optimizes six measurements spanning tertiary geometry, base pairing, ensemble behavior, and sequence composition. To aggregate these heterogeneous objectives despite differences in scale and optimization direction, MORAD maps each measurement to a bounded desirability score and combines them with a weighted geometric mean. Compared with a tertiary-structure-only reward, this multi-objective formulation improves 10 of the 11 evaluation metrics that have a preferred direction. Applied to pretrained RIDE, MORAD improves GDT-TS by 16.0% (from 0.3300 to 0.3827) and pairing MCC by 19.4% (from 0.6110 to 0.7295), while reducing normalized ensemble defect by 27.5% (from 0.3906 to 0.2830). On the test set, MORAD outperforms RiboDiffusion, gRNAde, and RDesign across tertiary structural accuracy, secondary-structure pairing, and thermodynamic quality, establishing a new state of the art in structure-conditioned RNA inverse design.