# Reinforcement Learning with a Bilevel World-Model Architecture for Scan-Order Optimisation in Laser Directed Energy Deposition

arXiv: https://arxiv.org/abs/2605.25063 · HTML: https://mlsanity.com/paper/2605.25063/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 24 May 2026.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)
- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [Reinforcement Learning for Combinatorial Optimization: A Survey](https://mlsanity.com/paper/2003.03600/) (2020)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [Lexicographic Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2212.13769/) (2022)
- [Thresholded Lexicographic Ordered Multiobjective Reinforcement Learning](https://mlsanity.com/paper/2408.13493/) (2024)
- [Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription](https://mlsanity.com/paper/2002.05368/) (2020)
- [Deep Reinforcement Learning Based Toolpath Generation for Thermal Uniformity in Laser Powder Bed Fusion Process](https://mlsanity.com/paper/2404.07209/) (2024)

## Abstract (arXiv)

Scan-order design in laser directed energy deposition (LDED) is a delayed, path-dependent thermo-mechanical decision problem, because sequence quality becomes observable only after the complete deposition and cooling cycle. This work formulates LDED scan-order optimisation as a finite-horizon, permutation-constrained reinforcement-learning problem and develops a bilevel finite-element-teacher-labelled AI workflow. A surrogate-assisted teacher-guided optimisation loop learns the Abaqus-labelled response landscape and provides a tractable terminal-reward environment for policy training. A frozen Maskable Proximal Policy Optimization (MaskablePPO) policy is then used to generate legal scan-order candidates, which are independently validated through Abaqus thermo-mechanical simulations. The results show bounded, N-dependent policy-generation value rather than record-level dominance over the mature surrogate-assisted optimiser. The strongest scan orders are obtained by the teacher-guided surrogate loop, whereas PPO autonomously reaches competitive regions of the native response landscape, with stronger rank concentration at smaller track counts and a clear reliability boundary at longer horizons. The teacher-labelled landscape further supports a physically gated lexicographic reward hierarchy in which warpage admissibility is the primary constraint, plastic strain acts as a safety filter and residual-stress-related improvement is pursued conditionally within the admissible region. Validated sequences also reveal an interpretable scale-separated ordering tendency that combines global spatial dispersion with local structured grouping. This workflow provides a route from fixed scan-rule selection toward finite-element-teacher-validated policy generation, while preserving independent finite-element validation as the final physical gate.