# RLAE: Reinforcement Learning-Assisted Ensemble for LLMs

arXiv: https://arxiv.org/abs/2506.00439 · HTML: https://mlsanity.com/paper/2506.00439/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration](https://mlsanity.com/paper/2404.12715/) (2024)
- [GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning](https://mlsanity.com/paper/2505.23399/) (2025)
- [Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization](https://mlsanity.com/paper/2404.11531/) (2024)
- [LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity](https://mlsanity.com/paper/2410.03953/) (2024)
- [Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation](https://mlsanity.com/paper/2503.12854/) (2025)
- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing](https://mlsanity.com/paper/2502.01976/) (2025)
- [Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling](https://mlsanity.com/paper/2410.03777/) (2024)
- [HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation](https://mlsanity.com/paper/2505.19866/) (2025)
- [Collab: Controlled Decoding using Mixture of Agents for LLM Alignment](https://mlsanity.com/paper/2503.21720/) (2025)
- [SpecFuse: Ensembling Large Language Models via Next-Segment Prediction](https://mlsanity.com/paper/2412.07380/) (2024)

## Built on by (most cited first, 7 of 7)

- [LLMBoost: Make Large Language Models Stronger with Boosting](https://mlsanity.com/paper/2512.22309/) (2025)
- [AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs](https://mlsanity.com/paper/2601.06022/) (2026)
- [WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agent](https://mlsanity.com/paper/2510.11221/) (2025)
- [MiRAGE: Misconception Detection with Retrieval-Guided Multi-Stage Reasoning and Ensemble Fusion](https://mlsanity.com/paper/2511.01182/) (2025)
- [AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning](https://mlsanity.com/paper/2604.17337/) (2026)
- [RV-HATE: Reinforced Multi-Module Voting for Implicit Hate Speech Detection](https://mlsanity.com/paper/2510.10971/) (2025)
- [Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning](https://mlsanity.com/paper/2610.17456/) (2026)

## Abstract (arXiv)

Ensembling large language models (LLMs) can effectively combine diverse strengths of different models, offering a promising approach to enhance performance across various tasks. However, existing methods typically rely on fixed weighting strategies that fail to adapt to the dynamic, context-dependent characteristics of LLM capabilities. In this work, we propose Reinforcement Learning-Assisted Ensemble for LLMs (RLAE), a novel framework that reformulates LLM ensemble through the lens of a Markov Decision Process (MDP). Our approach introduces a RL agent that dynamically adjusts ensemble weights by considering both input context and intermediate generation states, with the agent being trained using rewards that directly correspond to the quality of final outputs. We implement RLAE using both single-agent and multi-agent reinforcement learning algorithms (RLAEPPO and RLAEMAPPO ), demonstrating substantial improvements over conventional ensemble methods. Extensive evaluations on a diverse set of tasks show that RLAE outperforms existing approaches by up to 3.3% accuracy points, offering a more effective framework for LLM ensembling. Furthermore, our method exhibits superior generalization capabilities across different tasks without the need for retraining, while simultaneously achieving lower time latency.