# DeepScalper: A Risk-Aware Reinforcement Learning Framework to Capture Fleeting Intraday Trading Opportunities

arXiv: https://arxiv.org/abs/2201.09058 · HTML: https://mlsanity.com/paper/2201.09058/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 16 of 16)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem](https://mlsanity.com/paper/1706.10059/) (2017)
- [Temporal Relational Ranking for Stock Prediction](https://mlsanity.com/paper/1809.09441/) (2018)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Deep Reinforcement Learning for Trading](https://mlsanity.com/paper/1911.10107/) (2019)
- [Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction](https://mlsanity.com/paper/1712.02136/) (2017)
- [Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States](https://mlsanity.com/paper/2002.05780/) (2020)
- [Risk-Averse Trust Region Optimization for Reward-Volatility Reduction](https://mlsanity.com/paper/1912.03193/) (2019)
- [Universal Trading for Order Execution with Oracle Policy Distillation](https://mlsanity.com/paper/2103.10860/) (2021)
- [Reinforcement Learning for Quantitative Trading](https://mlsanity.com/paper/2109.13851/) (2021)
- [Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization](https://mlsanity.com/paper/1901.08740/) (2019)
- [Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution](https://mlsanity.com/paper/2012.12620/) (2020)
- [CLVSA: A Convolutional LSTM Based Variational Sequence-to-Sequence Model with Attention for Predicting Trends of Financial Markets](https://mlsanity.com/paper/2104.04041/) (2021)

## Built on by (most cited first, 15 of 15)

- [Reinforcement Learning with Maskable Stock Representation for Portfolio Management in Customizable Stock Pools](https://mlsanity.com/paper/2311.10801/) (2023)
- [MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading](https://mlsanity.com/paper/2406.14537/) (2024)
- [MILLION: A General Multi-Objective Framework with Controllable Risk for Portfolio Management](https://mlsanity.com/paper/2412.03038/) (2024)
- [PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets](https://mlsanity.com/paper/2302.00586/) (2023)
- [DeepClair: Utilizing Market Forecasts for Effective Portfolio Selection](https://mlsanity.com/paper/2407.13427/) (2024)
- [STORM: A Spatio-Temporal Factor Model Based on Dual Vector Quantized Variational Autoencoders for Financial Trading](https://mlsanity.com/paper/2412.09468/) (2024)
- [Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning](https://mlsanity.com/paper/2407.13279/) (2024)
- [Robot See, Robot Do: Imitation Reward for Noisy Financial Environments](https://mlsanity.com/paper/2411.08637/) (2024)
- [Plan Before You Trade: Inference-Time Optimization for RL Trading Agents](https://mlsanity.com/paper/2605.12653/) (2026)
- [Large Investment Model](https://mlsanity.com/paper/2408.10255/) (2024)
- [Bayesian Robust Financial Trading with Adversarial Synthetic Market Data](https://mlsanity.com/paper/2601.17008/) (2026)
- [PandaAI: A Practical Agent CQ2 for Neuro-symbolic Data Analysis And Integrated Decision-Making in Quantitative Finance](https://mlsanity.com/paper/2606.06823/) (2026)
- [Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2608.15770/) (2026)
- [Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading](https://mlsanity.com/paper/2608.15841/) (2026)
- [Enhanced Trading Systems with Intelligent Market Analysis](https://mlsanity.com/paper/2610.13998/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) techniques have shown great success in many challenging quantitative trading tasks, such as portfolio management and algorithmic trading. Especially, intraday trading is one of the most profitable and risky tasks because of the intraday behaviors of the financial market that reflect billions of rapidly fluctuating capitals. However, a vast majority of existing RL methods focus on the relatively low frequency trading scenarios (e.g., day-level) and fail to capture the fleeting intraday investment opportunities due to two major challenges: 1) how to effectively train profitable RL agents for intraday investment decision-making, which involves high-dimensional fine-grained action space; 2) how to learn meaningful multi-modality market representation to understand the intraday behaviors of the financial market at tick-level. Motivated by the efficient workflow of professional human intraday traders, we propose DeepScalper, a deep reinforcement learning framework for intraday trading to tackle the above challenges. Specifically, DeepScalper includes four components: 1) a dueling Q-network with action branching to deal with the large action space of intraday trading for efficient RL optimization; 2) a novel reward function with a hindsight bonus to encourage RL agents making trading decisions with a long-term horizon of the entire trading day; 3) an encoder-decoder architecture to learn multi-modality temporal market embedding, which incorporates both macro-level and micro-level market information; 4) a risk-aware auxiliary task to maintain a striking balance between maximizing profit and minimizing risk. Through extensive experiments on real-world market data spanning over three years on six financial futures, we demonstrate that DeepScalper significantly outperforms many state-of-the-art baselines in terms of four financial criteria.