# Adaptive Multi-Fidelity Reinforcement Learning for Variance Reduction in Engineering Design Optimization

arXiv: https://arxiv.org/abs/2503.18229 · HTML: https://mlsanity.com/paper/2503.18229/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Survey of multifidelity methods in uncertainty propagation, inference, and optimization](https://mlsanity.com/paper/1806.10761/) (2018)
- [A Deep Reinforcement Learning Approach for Global Routing](https://mlsanity.com/paper/1906.08809/) (2019)
- [A Latent Variable Approach for Non-Hierarchical Multi-Fidelity Adaptive Sampling](https://mlsanity.com/paper/2310.03298/) (2023)
- [Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree search](https://mlsanity.com/paper/2211.15068/) (2022)
- [Fairness- and uncertainty-aware data generation for data-driven design](https://mlsanity.com/paper/2309.05842/) (2023)
- [On the Effects of Heterogeneous Errors on Multi-fidelity Bayesian Optimization](https://mlsanity.com/paper/2309.02771/) (2023)

## Abstract (arXiv)

Multi-fidelity Reinforcement Learning (RL) frameworks efficiently utilize computational resources by integrating analysis models of varying accuracy and costs. The prevailing methodologies, characterized by transfer learning, human-inspired strategies, control variate techniques, and adaptive sampling, predominantly depend on a structured hierarchy of models. However, this reliance on a model hierarchy can exacerbate variance in policy learning when the underlying models exhibit heterogeneous error distributions across the design space. To address this challenge, this work proposes a novel adaptive multi-fidelity RL framework, in which multiple heterogeneous, non-hierarchical low-fidelity models are dynamically leveraged alongside a high-fidelity model to efficiently learn a high-fidelity policy. Specifically, low-fidelity policies and their experience data are adaptively used for efficient targeted learning, guided by their alignment with the high-fidelity policy. The effectiveness of the approach is demonstrated in an octocopter design optimization problem, utilizing two low-fidelity models alongside a high-fidelity simulator. The results demonstrate that the proposed approach substantially reduces variance in policy learning, leading to improved convergence and consistent high-quality solutions relative to traditional hierarchical multi-fidelity RL methods. Moreover, the framework eliminates the need for manually tuning model usage schedules, which can otherwise introduce significant computational overhead. This positions the framework as an effective variance-reduction strategy for multi-fidelity RL, while also mitigating the computational and operational burden of manual fidelity scheduling.