# Mesh-RL: Coupled subgrid reinforcement learning

arXiv: https://arxiv.org/abs/2606.26333 · HTML: https://mlsanity.com/paper/2606.26333/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 15 of 15)

- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [Divide-and-Conquer Reinforcement Learning](https://mlsanity.com/paper/1711.09874/) (2017)
- [Dealing with Sparse Rewards in Reinforcement Learning](https://mlsanity.com/paper/1910.09281/) (2019)
- [Reinforcement Learning for Adaptive Mesh Refinement](https://mlsanity.com/paper/2103.01342/) (2021)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.16761/) (2025)
- [Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2506.14045/) (2025)
- [Swarm Reinforcement Learning For Adaptive Mesh Refinement](https://mlsanity.com/paper/2304.00818/) (2023)
- [State Space Decomposition and Subgoal Creation for Transfer in Deep Reinforcement Learning](https://mlsanity.com/paper/1705.08997/) (2017)
- [Enhancing data efficiency in reinforcement learning: a novel imagination mechanism based on mesh information propagation](https://mlsanity.com/paper/2309.14243/) (2023)
- [Multi-layer Abstraction for Nested Generation of Options (MANGO) in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2508.17751/) (2025)

## Abstract (arXiv)

Reinforcement learning in large or sparse-reward environments suffers from slow temporal-difference reward propagation, as value information spreads only locally across the state space. We propose Mesh-RL, a spatial domain-decomposition framework inspired by the finite element method and domain decomposition theory, which partitions the environment into overlapping subgrids and enforces boundary-consistent temporal-difference updates. Such an approach enables localized learning while ensuring globally coherent value propagation. Unlike hierarchical or model-based approaches, Mesh-RL accelerates long-range credit assignment without modifying the reward function, Bellman operator, or introducing explicit planning mechanisms. We evaluate Mesh-RL on hazard-dense grid-world environments with varying geometries and mesh resolutions. Across Q-learning, SARSA, and Dyna-Q, Mesh-RL consistently improves convergence speed, cumulative reward, and learning stability. Higher mesh resolutions sustain exploration, prevent premature convergence, and substantially accelerate value propagation to distant states. While Dyna-Q already benefits from internal planning, it still achieves additional gains under structured decomposition. Overall, Mesh-RL introduces a principled spatial domain-decomposition mechanism for accelerating temporal-difference learning. Our framework bridges finite element method-inspired boundary-consistency techniques from scientific computing with reinforcement learning to improve sample efficiency in sparse-reward environments. We will release source code of the study.