# Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining

arXiv: https://arxiv.org/abs/2507.20263 · HTML: https://mlsanity.com/paper/2507.20263/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 27 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Qlib: An AI-oriented Quantitative Investment Platform](https://mlsanity.com/paper/2009.11189/) (2020)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [Generating Synergistic Formulaic Alpha Collections via Reinforcement Learning](https://mlsanity.com/paper/2306.12964/) (2023)
- [Learning Continuous Semantic Representations of Symbolic Expressions](https://mlsanity.com/paper/1611.01423/) (2016)
- [Reward Centering](https://mlsanity.com/paper/2405.09999/) (2024)
- [QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE](https://mlsanity.com/paper/2409.05144/) (2024)

## Built on by (most cited first, 5 of 5)

- [AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining](https://mlsanity.com/paper/2508.13174/) (2025)
- [AlphaPROBE: Alpha Mining via Principled Retrieval and On-graph biased evolution](https://mlsanity.com/paper/2602.11917/) (2026)
- [AQuA: Recursively Self-Improving Quantitative Trading Research Agents](https://mlsanity.com/paper/2608.12841/) (2026)
- [From Observations to Parameters: Detecting Changepoint in Nonlinear Dynamics with Simulation-based Inference](https://mlsanity.com/paper/2510.17933/) (2025)
- [AlphaPilot: Multi-Turn Tool-Using LLM Agents for Adaptive Alpha Mining with Reinforcement Learning](https://mlsanity.com/paper/2610.15488/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has successfully automated the complex process of mining formulaic alpha factors, for creating interpretable and profitable investment strategies. However, existing methods are hampered by the sparse rewards given the underlying Markov Decision Process. This inefficiency limits the exploration of the vast symbolic search space and destabilizes the training process. To address this, Trajectory-level Reward Shaping (TLRS), a novel reward shaping method, is proposed. TLRS provides dense, intermediate rewards by measuring the subsequence-level similarity between partially generated expressions and a set of expert-designed formulas. Furthermore, a reward centering mechanism is introduced to reduce training variance. Extensive experiments on six major Chinese and U.S. stock indices show that TLRS significantly improves the predictive power of mined factors, boosting the Rank Information Coefficient by 9.29% over existing potential-based shaping algorithms. Notably, TLRS achieves a major leap in computational efficiency by reducing its time complexity with respect to the feature dimension from linear to constant, which is a significant improvement over distance-based baselines.