# DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading

arXiv: https://arxiv.org/abs/2605.25527 · HTML: https://mlsanity.com/paper/2605.25527/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 22 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 6 of 6)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Model-based Reinforcement Learning for Predictions and Control for Limit Order Books](https://mlsanity.com/paper/1910.03743/) (2019)
- [Time-Constrained Recommendations: Reinforcement Learning Strategies for E-Commerce](https://mlsanity.com/paper/2512.13726/) (2025)
- [Combining Deep Learning on Order Books with Reinforcement Learning for Profitable Trading](https://mlsanity.com/paper/2311.02088/) (2023)
- [Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity](https://mlsanity.com/paper/2605.03393/) (2026)

## Abstract (arXiv)

This paper studies reinforcement learning for high-frequency trading on limit order books by pairing an Order-Flow-based state model with policy-gradient methods. Instead of value-based RL techniques like tabular Q-learning, our approach deploys policy-based methods like vanilla PPO and DeepSeekMath-inspired variants like GRPO and GSPO, that use group-normalized updates and downside-aware shaping. On backtests with financial assets AMZN, AAPL, and GOOG under a simplified backtesting setup based on spread-scaled rewards, these new policies improve net average PnL, profitability, and drawdown over the Q-Learning baseline. Our results show that (1) Order-Flow signals are an adequate state for policy RL and (2) group-aware PPO surrogates are preferable over value-based baselines.