# Vector Symbolic Policy Gradient

arXiv: https://arxiv.org/abs/2608.18404 · HTML: https://mlsanity.com/paper/2608.18404/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [On the Global Convergence Rates of Softmax Policy Gradient Methods](https://mlsanity.com/paper/2005.06392/) (2020)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations](https://mlsanity.com/paper/2111.06077/) (2021)
- [Blessing of dimensionality: mathematical foundations of the statistical physics of data](https://mlsanity.com/paper/1801.03421/) (2018)
- [A Theoretical Perspective on Hyperdimensional Computing](https://mlsanity.com/paper/2010.07426/) (2020)
- [EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM](https://mlsanity.com/paper/1910.05340/) (2019)
- [Understanding Hyperdimensional Computing for Parallel Single-Pass Learning](https://mlsanity.com/paper/2202.04805/) (2022)
- [HyperSense: Hyperdimensional Intelligent Sensing for Energy-Efficient Sparse Data Processing](https://mlsanity.com/paper/2401.10267/) (2024)
- [BERRY: Bit Error Robustness for Energy-Efficient Reinforcement Learning-Based Autonomous Systems](https://mlsanity.com/paper/2307.10041/) (2023)
- [Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach](https://mlsanity.com/paper/2410.10674/) (2024)
- [HPVM-HDC: A Heterogeneous Programming System for Accelerating Hyperdimensional Computing](https://mlsanity.com/paper/2410.15179/) (2024)
- [LogHD: Robust Compression of Hyperdimensional Classifiers via Logarithmic Class-Axis Reduction](https://mlsanity.com/paper/2511.03938/) (2025)
- [Bridging the Gap Between Hyperdimensional Computing and Kernel Methods via the Nystr"om Method](https://mlsanity.com/paper/2608.06860/) (2026)
- [ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computing](https://mlsanity.com/paper/2604.21280/) (2026)

## Abstract (arXiv)

We answer this question with Vector-Symbolic Policy Gradient (VSPG), a discrete-action actor that represents each action by a unit-norm hypervector and scores it by similarity to the encoded state. Under the standard softmax policy-gradient surrogate, we prove that its update is exactly advantage-weighted hypervector bundling followed by normalization, and therefore supports standard advantage estimators. We further show that each trained action hypervector is a fixed-size compressed kernel memory, storing an advantage-weighted kernel expansion over visited states and transferring evidence according to the encoder-induced similarity. This provides a concrete mechanism that can support sample-efficient learning without increasing inference-time memory. Finally, for bipolar action memories, we prove that greedy action selection is stable under random bit flips, with failure probability decaying exponentially in the hypervector dimension. VSPG thus connects VSA action memories, log-linear policy gradients, and kernel policy search while providing a quantitative robustness guarantee.