# QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE

arXiv: https://arxiv.org/abs/2409.05144 · HTML: https://mlsanity.com/paper/2409.05144/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 82 references that are in the map.
- First posted on arXiv 8 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance](https://mlsanity.com/paper/2011.09607/) (2020)
- [Sample Efficient Reinforcement Learning with REINFORCE](https://mlsanity.com/paper/2010.11364/) (2020)
- [The Optimal Reward Baseline for Gradient-Based Reinforcement Learning](https://mlsanity.com/paper/1301.2315/) (2013)
- [Qlib: An AI-oriented Quantitative Investment Platform](https://mlsanity.com/paper/2009.11189/) (2020)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [Leveraging Non-uniformity in First-order Non-convex Optimization](https://mlsanity.com/paper/2105.06072/) (2021)
- [Generating Synergistic Formulaic Alpha Collections via Reinforcement Learning](https://mlsanity.com/paper/2306.12964/) (2023)
- [AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investment](https://mlsanity.com/paper/2103.16196/) (2021)
- [The Role of Baselines in Policy Gradient Optimization](https://mlsanity.com/paper/2301.06276/) (2023)
- [TD Convergence: An Optimization Perspective](https://mlsanity.com/paper/2306.17750/) (2023)
- [Reducing Idleness in Financial Cloud Services via Multi-objective Evolutionary Reinforcement Learning based Load Balancer](https://mlsanity.com/paper/2305.03463/) (2023)
- [Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search](https://mlsanity.com/paper/2009.03603/) (2020)
- [Adversarial Deep Reinforcement Learning in Portfolio Management](https://mlsanity.com/paper/1808.09940/) (2018)
- [Parallel Exploration via Negatively Correlated Search](https://mlsanity.com/paper/1910.07151/) (2019)
- [SimLOB: Learning Representations of Limited Order Book for Financial Market Simulation](https://mlsanity.com/paper/2406.19396/) (2024)
- [Neural Augmented Kalman Filtering with Bollinger Bands for Pairs Trading](https://mlsanity.com/paper/2210.15448/) (2022)
- [Efficient and Scalable Parametric High-Order Portfolios Design via the Skew-t Distribution](https://mlsanity.com/paper/2206.02412/) (2022)
- [A Fast Successive QP Algorithm for General Mean-Variance Portfolio Optimization](https://mlsanity.com/paper/2212.06983/) (2022)
- [Combining Reinforcement Learning and Barrier Functions for Adaptive Risk Management in Portfolio Optimization](https://mlsanity.com/paper/2306.07013/) (2023)

## Built on by (most cited first, 14 of 14)

- [R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization](https://mlsanity.com/paper/2505.15155/) (2025)
- [AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining](https://mlsanity.com/paper/2508.13174/) (2025)
- [Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining](https://mlsanity.com/paper/2507.20263/) (2025)
- [AlphaPROBE: Alpha Mining via Principled Retrieval and On-graph biased evolution](https://mlsanity.com/paper/2602.11917/) (2026)
- [AlphaMemo: Structured Search-Process Memory for Self-Evolving Alpha Mining Agents](https://mlsanity.com/paper/2606.20625/) (2026)
- [AQuA: Recursively Self-Improving Quantitative Trading Research Agents](https://mlsanity.com/paper/2608.12841/) (2026)
- [Representation Learning of Limit Order Book: A Comprehensive Study and Benchmarking](https://mlsanity.com/paper/2505.02139/) (2025)
- [Deep Reinforcement Learning for Optimal Asset Allocation Using DDPG with TiDE](https://mlsanity.com/paper/2508.20103/) (2025)
- [From Observations to Parameters: Detecting Changepoint in Nonlinear Dynamics with Simulation-based Inference](https://mlsanity.com/paper/2510.17933/) (2025)
- [Online Regime-aware Calibration for Black-box Social Simulators via Posterior-assisted Evolutionary Dynamic Optimization](https://mlsanity.com/paper/2601.19481/) (2026)
- [EVOQUANT: Self-Evolving Verifier-Guided Strategy Optimization for Robust Quantitative Trading](https://mlsanity.com/paper/2607.12455/) (2026)
- [AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discovery](https://mlsanity.com/paper/2608.01303/) (2026)
- [AgonAlpha: Autonomous Alpha Discovery via Prompt Economy and Scalable Agentic Search](https://mlsanity.com/paper/2608.11250/) (2026)
- [AlphaRLM: Aligning Language Models for Literature-Grounded Symbolic Factor Discovery](https://mlsanity.com/paper/2610.14959/) (2026)

## Abstract (arXiv)

Alpha factor mining aims to discover investment signals from the historical financial market data, which can be used to predict asset returns and gain excess profits. Powerful deep learning methods for alpha factor mining lack interpretability, making them unacceptable in the risk-sensitive real markets. Formulaic alpha factors are preferred for their interpretability, while the search space is complex and powerful explorative methods are urged. Recently, a promising framework is proposed for generating formulaic alpha factors using deep reinforcement learning, and quickly gained research focuses from both academia and industries. This paper first argues that the originally employed policy training method, i.e., Proximal Policy Optimization (PPO), faces several important issues in the context of alpha factors mining. Herein, a novel reinforcement learning algorithm based on the well-known REINFORCE algorithm is proposed. REINFORCE employs Monte Carlo sampling to estimate the policy gradient-yielding unbiased but high variance estimates. The minimal environmental variability inherent in the underlying state transition function, which adheres to the Dirac distribution, can help alleviate this high variance issue, making REINFORCE algorithm more appropriate than PPO. A new dedicated baseline is designed to theoretically reduce the commonly suffered high variance of REINFORCE. Moreover, the information ratio is introduced as a reward shaping mechanism to encourage the generation of steady alpha factors that can better adapt to changes in market volatility. Evaluations on real assets data indicate the proposed algorithm boosts correlation with returns by 3.83%, and a stronger ability to obtain excess returns compared to the latest alpha factors mining methods, which meets the theoretical results well.