# Heuristic Algorithm-based Action Masking Reinforcement Learning (HAAM-RL) with Ensemble Inference Method

arXiv: https://arxiv.org/abs/2403.14110 · HTML: https://mlsanity.com/paper/2403.14110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 18 of 18)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization](https://mlsanity.com/paper/2006.01610/) (2020)
- [Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity](https://mlsanity.com/paper/2210.09579/) (2022)
- [Heuristic-Guided Reinforcement Learning](https://mlsanity.com/paper/2106.02757/) (2021)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems](https://mlsanity.com/paper/2105.12899/) (2021)
- [On Effective Scheduling of Model-based Reinforcement Learning](https://mlsanity.com/paper/2111.08550/) (2021)
- [A new Potential-Based Reward Shaping for Reinforcement Learning Agent](https://mlsanity.com/paper/1902.06239/) (2019)
- [DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learning](https://mlsanity.com/paper/2201.13357/) (2022)
- [Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach](https://mlsanity.com/paper/2402.19265/) (2024)
- [Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft](https://mlsanity.com/paper/1803.08456/) (2018)
- [Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains](https://mlsanity.com/paper/2204.09603/) (2022)
- [Sample-efficient Model-based Reinforcement Learning for Quantum Control](https://mlsanity.com/paper/2304.09718/) (2023)
- [Paint shop vehicle sequencing based on quantum computing considering color changeover and painting quality](https://mlsanity.com/paper/2206.11204/) (2022)
- [Learning-Based Heuristic for Combinatorial Optimization of the Minimum Dominating Set Problem using Graph Convolutional Networks](https://mlsanity.com/paper/2306.03434/) (2023)

## Abstract (arXiv)

This paper presents a novel reinforcement learning (RL) approach called HAAM-RL (Heuristic Algorithm-based Action Masking Reinforcement Learning) for optimizing the color batching re-sequencing problem in automobile painting processes. The existing heuristic algorithms have limitations in adequately reflecting real-world constraints and accurately predicting logistics performance. Our methodology incorporates several key techniques including a tailored Markov Decision Process (MDP) formulation, reward setting including Potential-Based Reward Shaping, action masking using heuristic algorithms (HAAM-RL), and an ensemble inference method that combines multiple RL models. The RL agent is trained and evaluated using FlexSim, a commercial 3D simulation software, integrated with our RL MLOps platform BakingSoDA. Experimental results across 30 scenarios demonstrate that HAAM-RL with an ensemble inference method achieves a 16.25% performance improvement over the conventional heuristic algorithm, with stable and consistent results. The proposed approach exhibits superior performance and generalization capability, indicating its effectiveness in optimizing complex manufacturing processes. The study also discusses future research directions, including alternative state representations, incorporating model-based RL methods, and integrating additional real-world constraints.