# Automated Gadget Discovery in Science

arXiv: https://arxiv.org/abs/2212.12743 · HTML: https://mlsanity.com/paper/2212.12743/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 24 Dec 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Discovering physical concepts with neural networks](https://mlsanity.com/paper/1807.10300/) (2018)
- [Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/1703.01732/) (2017)
- [CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1810.06284/) (2018)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations](https://mlsanity.com/paper/1912.09007/) (2019)
- [Applications of Deep Learning and Reinforcement Learning to Biological Data](https://mlsanity.com/paper/1711.03985/) (2017)
- [Active learning machine learns to create new quantum experiments](https://mlsanity.com/paper/1706.00868/) (2017)
- [Deep reinforcement learning in medical imaging: A literature review](https://mlsanity.com/paper/2103.05115/) (2021)
- [Optimizing Quantum Error Correction Codes with Reinforcement Learning](https://mlsanity.com/paper/1812.08451/) (2018)
- [Operationally meaningful representations of physical systems in neural networks](https://mlsanity.com/paper/2001.00593/) (2020)
- [Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning](https://mlsanity.com/paper/2012.11293/) (2020)
- [Toward an AI Physicist for Unsupervised Learning](https://mlsanity.com/paper/1810.10525/) (2018)
- [How a minimal learning agent can infer the existence of unobserved variables in a complex environment](https://mlsanity.com/paper/1910.06985/) (2019)
- [Learning the Arrow of Time](https://mlsanity.com/paper/1907.01285/) (2019)
- [Accelerating Empowerment Computation with UCT Tree Search](https://mlsanity.com/paper/1803.09866/) (2018)

## Built on by (most cited first, 1 of 1)

- [Free Energy Projective Simulation (FEPS): Active inference with interpretability](https://mlsanity.com/paper/2411.14991/) (2024)

## Abstract (arXiv)

In recent years, reinforcement learning (RL) has become increasingly successful in its application to science and the process of scientific discovery in general. However, while RL algorithms learn to solve increasingly complex problems, interpreting the solutions they provide becomes ever more challenging. In this work, we gain insights into an RL agent's learned behavior through a post-hoc analysis based on sequence mining and clustering. Specifically, frequent and compact subroutines, used by the agent to solve a given task, are distilled as gadgets and then grouped by various metrics. This process of gadget discovery develops in three stages: First, we use an RL agent to generate data, then, we employ a mining algorithm to extract gadgets and finally, the obtained gadgets are grouped by a density-based clustering algorithm. We demonstrate our method by applying it to two quantum-inspired RL environments. First, we consider simulated quantum optics experiments for the design of high-dimensional multipartite entangled states where the algorithm finds gadgets that correspond to modern interferometer setups. Second, we consider a circuit-based quantum computing environment where the algorithm discovers various gadgets for quantum information processing, such as quantum teleportation. This approach for analyzing the policy of a learned agent is agent and environment agnostic and can yield interesting insights into any agent's policy.