# CORE: Constraint-Aware One-Step Reinforcement Learning for Simulation-Guided Neural Network Accelerator Design

arXiv: https://arxiv.org/abs/2506.03474 · HTML: https://mlsanity.com/paper/2506.03474/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Revisiting Design Choices in Proximal Policy Optimization](https://mlsanity.com/paper/2009.10897/) (2020)
- [ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning](https://mlsanity.com/paper/2009.02010/) (2020)
- [Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows](https://mlsanity.com/paper/2006.02979/) (2020)
- [HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation](https://mlsanity.com/paper/2105.01585/) (2021)
- [Delving into Macro Placement with Reinforcement Learning](https://mlsanity.com/paper/2109.02587/) (2021)
- [DiGamma: Domain-aware Genetic Algorithm for HW-Mapping Co-optimization for DNN Accelerators](https://mlsanity.com/paper/2201.11220/) (2022)
- [ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Design](https://mlsanity.com/paper/2306.08888/) (2023)
- [Theta-Resonance: A Single-Step Reinforcement Learning Method for Design Space Exploration](https://mlsanity.com/paper/2211.02052/) (2022)

## Built on by (most cited first, 1 of 1)

- [ArchAgent: Agentic AI-driven Computer Architecture Discovery](https://mlsanity.com/paper/2602.22425/) (2026)

## Abstract (arXiv)

Simulation-based design space exploration (DSE) aims to efficiently optimize high-dimensional structured designs under complex constraints and expensive evaluation costs. Existing approaches, including heuristic and multi-step reinforcement learning (RL) methods, struggle to balance sampling efficiency and constraint satisfaction due to sparse, delayed feedback, and large hybrid action spaces. In this paper, we introduce CORE, a constraint-aware, one-step RL method for simulationguided DSE. In CORE, the policy agent learns to sample design configurations by defining a structured distribution over them, incorporating dependencies via a scaling-graph-based decoder, and by reward shaping to penalize invalid designs based on the feedback obtained from simulation. CORE updates the policy using a surrogate objective that compares the rewards of designs within a sampled batch, without learning a value function. This critic-free formulation enables efficient learning by encouraging the selection of higher-reward designs. We instantiate CORE for hardware-mapping co-design of neural network accelerators, demonstrating that it significantly improves sample efficiency and achieves better accelerator configurations compared to state-of-the-art baselines. Our approach is general and applicable to a broad class of discrete-continuous constrained design problems.