# Self-evolving Autoencoder Embedded Q-Network

arXiv: https://arxiv.org/abs/2402.11604 · HTML: https://mlsanity.com/paper/2402.11604/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 17 of 17)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Deep Reinforcement Learning for De-Novo Drug Design](https://mlsanity.com/paper/1711.10907/) (2017)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Shaping Belief States with Generative Environment Models for RL](https://mlsanity.com/paper/1906.09237/) (2019)
- [CAQL: Continuous Action Q-Learning](https://mlsanity.com/paper/1909.12397/) (2019)
- [Deep Reinforcement Learning with Quantum-inspired Experience Replay](https://mlsanity.com/paper/2101.02034/) (2021)
- [LSTM-Autoencoder based Anomaly Detection for Indoor Air Quality Time Series Data](https://mlsanity.com/paper/2204.06701/) (2022)
- [DEVDAN: Deep Evolving Denoising Autoencoder](https://mlsanity.com/paper/1910.04062/) (2019)
- [DAQN: Deep Auto-encoder and Q-Network](https://mlsanity.com/paper/1806.00630/) (2018)
- [Empirical Policy Optimization for n-Player Markov Games](https://mlsanity.com/paper/2110.08979/) (2021)
- [DRBM-ClustNet: A Deep Restricted Boltzmann-Kohonen Architecture for Data Clustering](https://mlsanity.com/paper/2205.06697/) (2022)

## Built on by (most cited first, 2 of 2)

- [XMOL: Explainable Multi-property Optimization of Molecules](https://mlsanity.com/paper/2409.07786/) (2024)
- [Latent Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/2603.05440/) (2026)

## Abstract (arXiv)

In the realm of sequential decision-making tasks, the exploration capability of a reinforcement learning (RL) agent is paramount for achieving high rewards through interactions with the environment. To enhance this crucial ability, we propose SAQN, a novel approach wherein a self-evolving autoencoder (SA) is embedded with a Q-Network (QN). In SAQN, the self-evolving autoencoder architecture adapts and evolves as the agent explores the environment. This evolution enables the autoencoder to capture a diverse range of raw observations and represent them effectively in its latent space. By leveraging the disentangled states extracted from the encoder generated latent space, the QN is trained to determine optimal actions that improve rewards. During the evolution of the autoencoder architecture, a bias-variance regulatory strategy is employed to elicit the optimal response from the RL agent. This strategy involves two key components: (i) fostering the growth of nodes to retain previously acquired knowledge, ensuring a rich representation of the environment, and (ii) pruning the least contributing nodes to maintain a more manageable and tractable latent space. Extensive experimental evaluations conducted on three distinct benchmark environments and a real-world molecular environment demonstrate that the proposed SAQN significantly outperforms state-of-the-art counterparts. The results highlight the effectiveness of the self-evolving autoencoder and its collaboration with the Q-Network in tackling sequential decision-making tasks.