# Improving exploration in policy gradient search: Application to symbolic optimization

arXiv: https://arxiv.org/abs/2107.09158 · HTML: https://mlsanity.com/paper/2107.09158/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Neural Combinatorial Optimization with Reinforcement Learning](https://mlsanity.com/paper/1611.09940/) (2016)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Neural Optimizer Search with Reinforcement Learning](https://mlsanity.com/paper/1709.07417/) (2017)
- [MolecularRNN: Generating realistic molecular graphs with optimized properties](https://mlsanity.com/paper/1905.13372/) (2019)
- [Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing](https://mlsanity.com/paper/1807.02322/) (2018)
- [Neural Program Synthesis with Priority Queue Training](https://mlsanity.com/paper/1801.03526/) (2018)
- [PyGlove: Symbolic Programming for Automated Machine Learning](https://mlsanity.com/paper/2101.08809/) (2021)
- [Policy Gradient Search: Online Planning and Expert Iteration without Search Trees](https://mlsanity.com/paper/1904.03646/) (2019)

## Built on by (most cited first, 14 of 14)

- [Symbolic Regression via Neural-Guided Genetic Programming Population Seeding](https://mlsanity.com/paper/2111.00053/) (2021)
- [DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experience](https://mlsanity.com/paper/2506.04282/) (2025)
- [GSR: A Generalized Symbolic Regression Approach](https://mlsanity.com/paper/2205.15569/) (2022)
- [Deep Symbolic Optimization for Combinatorial Optimization: Accelerating Node Selection by Discovering Potential Heuristics](https://mlsanity.com/paper/2406.09740/) (2024)
- [CoNSoLe: Convex Neural Symbolic Learning](https://mlsanity.com/paper/2206.00257/) (2022)
- [LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimization](https://mlsanity.com/paper/2602.10576/) (2026)
- [Deep Symbolic Optimization: Reinforcement Learning for Symbolic Mathematics](https://mlsanity.com/paper/2505.10762/) (2025)
- [ExPoSe: Combining State-Based Exploration with Gradient-Based Online Search](https://mlsanity.com/paper/2202.01461/) (2022)
- [Generalizing the SINDy approach with nested neural networks](https://mlsanity.com/paper/2404.15742/) (2024)
- [Physical Symbolic Optimization](https://mlsanity.com/paper/2312.03612/) (2023)
- [Discovering Car-following Dynamics from Trajectory Data through Deep Learning](https://mlsanity.com/paper/2408.00251/) (2024)
- [FePySR: A Neural Feature Extraction Framework for Efficient and Scalable Symbolic Regression](https://mlsanity.com/paper/2605.12704/) (2026)
- [MOT-SR: Multi-Objective Tool-Augmented Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2607.29561/) (2026)
- [A machine learning framework for uncovering stochastic nonlinear dynamics from noisy data](https://mlsanity.com/paper/2604.06081/) (2026)

## Abstract (arXiv)

Many machine learning strategies designed to automate mathematical tasks leverage neural networks to search large combinatorial spaces of mathematical symbols. In contrast to traditional evolutionary approaches, using a neural network at the core of the search allows learning higher-level symbolic patterns, providing an informed direction to guide the search. When no labeled data is available, such networks can still be trained using reinforcement learning. However, we demonstrate that this approach can suffer from an early commitment phenomenon and from initialization bias, both of which limit exploration. We present two exploration methods to tackle these issues, building upon ideas of entropy regularization and distribution initialization. We show that these techniques can improve the performance, increase sample efficiency, and lower the complexity of solutions for the task of symbolic regression.