# Theta-Resonance: A Single-Step Reinforcement Learning Method for Design Space Exploration

arXiv: https://arxiv.org/abs/2211.02052 · HTML: https://mlsanity.com/paper/2211.02052/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chip Placement with Deep Reinforcement Learning](https://mlsanity.com/paper/2004.10746/) (2020)
- [FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edge](https://mlsanity.com/paper/1904.04421/) (2019)
- [Revisiting Design Choices in Proximal Policy Optimization](https://mlsanity.com/paper/2009.10897/) (2020)
- [Practical Design Space Exploration](https://mlsanity.com/paper/1810.05236/) (2018)
- [An Efficient Bandit Algorithm for Realtime Multivariate Optimization](https://mlsanity.com/paper/1810.09558/) (2018)
- [IronMan: GNN-assisted Design Space Exploration in High-Level Synthesis via Reinforcement Learning](https://mlsanity.com/paper/2102.08138/) (2021)
- [DNNExplorer: A Framework for Modeling and Exploring a Novel Paradigm of FPGA-based DNN Accelerator](https://mlsanity.com/paper/2008.12745/) (2020)
- [Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows](https://mlsanity.com/paper/2006.02979/) (2020)
- [Delving into Macro Placement with Reinforcement Learning](https://mlsanity.com/paper/2109.02587/) (2021)

## Built on by (most cited first, 4 of 4)

- [Challenges for Reinforcement Learning in Quantum Circuit Design](https://mlsanity.com/paper/2312.11337/) (2023)
- [Reinforcement Learning-Guided Semi-Supervised Learning](https://mlsanity.com/paper/2405.01760/) (2024)
- [CORE: Constraint-Aware One-Step Reinforcement Learning for Simulation-Guided Neural Network Accelerator Design](https://mlsanity.com/paper/2506.03474/) (2025)
- [Inverse Design in Distributed Circuits Using Single-Step Reinforcement Learning](https://mlsanity.com/paper/2506.08029/) (2025)

## Abstract (arXiv)

Given an environment (e.g., a simulator) for evaluating samples in a specified design space and a set of weighted evaluation metrics -- one can use Theta-Resonance, a single-step Markov Decision Process (MDP), to train an intelligent agent producing progressively more optimal samples. In Theta-Resonance, a neural network consumes a constant input tensor and produces a policy as a set of conditional probability density functions (PDFs) for sampling each design dimension. We specialize existing policy gradient algorithms in deep reinforcement learning (D-RL) in order to use evaluation feedback (in terms of cost, penalty or reward) to update our policy network with robust algorithmic stability and minimal design evaluations. We study multiple neural architectures (for our policy network) within the context of a simple SoC design space and propose a method of constructing synthetic space exploration problems to compare and improve design space exploration (DSE) algorithms. Although we only present categorical design spaces, we also outline how to use Theta-Resonance in order to explore continuous and mixed continuous-discrete design spaces.