APF+: Boosting adaptive-potential function reinforcement learning methods with a W-shaped network for high-dimensional games
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Studies in reward shaping for reinforcement learning (RL) have flourished in recent years due to its ability to speed up training. Our previous work proposed an adaptive potential function (APF) and showed that APF can accelerate the Q-learning with a Multi-layer Perceptron algorithm in the low-dimensional domain. This paper proposes to extend APF with an encoder (APF+) for RL state representation, allowing applying APF to the pixel-based Atari games using a state-encoding method that projects high-dimensional game's pixel frames to low-dimensional embeddings. We approach by designing the state-representation encoder as a W-shaped network (W-Net), by using which we are able to encode both the background as well as the moving entities in the game frames. Specifically, the embeddings derived from the pre-trained W-Net consist of two latent vectors: One represents the input state, and the other represents the deviation of the input state's representation from itself. We then incorporate W-Net into APF to train a downstream Dueling Deep Q-Network (DDQN), obtain the APF-WNet-DDQN, and demonstrate its effectiveness in Atari game-playing tasks. To evaluate the APF+W-Net module in such high-dimensional tasks, we compare with two types of baseline methods: (i) the basic DDQN; and (ii) two encoder-replaced APF-DDQN methods where we replace W-Net by (a) an unsupervised state representation method called Spatiotemporal Deep Infomax (ST-DIM) and (b) a ground truth state representation provided by the Atari Annotated RAM Interface (ARI). The experiment results show that out of 20 Atari games, APF-WNet-DDQN outperforms DDQN (14/20 games) and APF-STDIM-DDQN (13/20 games) significantly. In comparison against the APF-ARI-DDQN which employs embeddings directly of the detailed game-internal state information, the APF-WNet-DDQN achieves a comparable performance.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 31 references that are in the map. First posted on arXiv 17 Mar 2025.
Builds on
13 of 31 references.
References · 13
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2016OpenAI Gymcited by 1,736
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2019Solving Rubik's Cube with a Robot Handcited by 734
- 2018Towards a Definition of Disentangled Representationscited by 296
- 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
- 2020First return, then explorecited by 266
- 2019Unsupervised State Representation Learning in Ataricited by 144
- 2018State Representation Learning for Control: An Overviewcited by 127
- 2018Playing hard exploration games by watching YouTubecited by 99
- 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
- 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning
- 2025Bootstrapped Reward Shaping
- 2024Boosting Reinforcement Learning Algorithms in Continuous Robotic Reaching Tasks using Adaptive Potential Functions
- 2026Automating Potential-based Reward Shaping with Vision Language Model Guidance
- 2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning
- 2018Inverse reinforcement learning for video games
- 2013Playing Atari with Deep Reinforcement Learning
- 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learning