# Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning

arXiv: https://arxiv.org/abs/2602.00540 · HTML: https://mlsanity.com/paper/2602.00540/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 65 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Deep Reinforcement Learning Based Parameter Control in Differential Evolution](https://mlsanity.com/paper/1905.08006/) (2019)
- [MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learning](https://mlsanity.com/paper/2310.08252/) (2023)
- [Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation](https://mlsanity.com/paper/2311.13184/) (2023)
- [B2Opt: Learning to Optimize Black-box Optimization with Little Budget](https://mlsanity.com/paper/2304.11787/) (2023)
- [ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning](https://mlsanity.com/paper/2412.07507/) (2024)
- [Reinforcement Learning-based Self-adaptive Differential Evolution through Automated Landscape Feature Learning](https://mlsanity.com/paper/2503.18061/) (2025)
- [MetaDE: Evolving Differential Evolution by Differential Evolution](https://mlsanity.com/paper/2502.10470/) (2025)
- [Automated Metaheuristic Algorithm Design with Autoregressive Learning](https://mlsanity.com/paper/2405.03419/) (2024)
- [Surrogate Learning in Meta-Black-Box Optimization: A Preliminary Study](https://mlsanity.com/paper/2503.18060/) (2025)

## Abstract (arXiv)

Surrogate-assisted Evolutionary Algorithms~(SAEAs) have shown promising robustness in solving expensive optimization problems. A key aspect that impacts SAEAs' effectiveness is surrogate model selection, which in existing works is predominantly decided by human developer. Such human-made design choice introduces strong bias into SAEAs and may hurt their expected performance on out-of-scope tasks. In this paper, we propose a reinforcement learning-assisted ensemble framework, termed as SEEMOO, which is capable of scheduling different surrogate models within a single optimization process, hence boosting the overall optimization performance in a cooperative paradigm. Specifically, we focus on expensive multi-objective optimization problems, where multiple objective functions shape a compositional landscape and hence challenge surrogate selection. SEEMOO comprises following core designs: 1) A pre-collected model pool that maintains different surrogate models; 2) An attention-based state-extractor supports universal optimization state representation of problems with varied objective numbers; 3) a deep Q-network serves as dynamic surrogate selector: Given the optimization state, it selects desired surrogate model for current-step evaluation. SEEMOO is trained to maximize the overall optimization performance under a training problem distribution. Extensive benchmark results demonstrate SEEMOO's surrogate ensemble paradigm boosts the optimization performance of single-surrogate baselines. Further ablation studies underscore the importance of SEEMOO's design components.