# Investigating the Impact of Action Representations in Policy Gradient Algorithms

arXiv: https://arxiv.org/abs/2309.06921 · HTML: https://mlsanity.com/paper/2309.06921/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?](https://mlsanity.com/paper/1611.01055/) (2016)
- [Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning](https://mlsanity.com/paper/2010.04304/) (2020)
- [A Comparison of Action Spaces for Learning Manipulation Tasks](https://mlsanity.com/paper/1908.08659/) (2019)
- [Learning Variable Impedance Control for Contact Sensitive Tasks](https://mlsanity.com/paper/1907.07500/) (2019)
- [Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments](https://mlsanity.com/paper/2205.07015/) (2022)
- [Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks](https://mlsanity.com/paper/2207.03952/) (2022)
- [Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimization](https://mlsanity.com/paper/2009.02391/) (2020)

## Built on by (most cited first, 2 of 2)

- [Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2606.18594/) (2026)
- [Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms](https://mlsanity.com/paper/2603.14535/) (2026)

## Abstract (arXiv)

Reinforcement learning~(RL) is a versatile framework for learning to solve complex real-world tasks. However, influences on the learning performance of RL algorithms are often poorly understood in practice. We discuss different analysis techniques and assess their effectiveness for investigating the impact of action representations in RL. Our experiments demonstrate that the action representation can significantly influence the learning performance on popular RL benchmark tasks. The analysis results indicate that some of the performance differences can be attributed to changes in the complexity of the optimization landscape. Finally, we discuss open challenges of analysis techniques for RL algorithms.