# Universal Trading for Order Execution with Oracle Policy Distillation

arXiv: https://arxiv.org/abs/2103.10860 · HTML: https://mlsanity.com/paper/2103.10860/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Suphx: Mastering Mahjong with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.13590/) (2020)
- [Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States](https://mlsanity.com/paper/2002.05780/) (2020)
- [Reinforcement Learning with Non-Markovian Rewards](https://mlsanity.com/paper/1912.02552/) (2019)
- [Double Deep Q-Learning for Optimal Execution](https://mlsanity.com/paper/1812.06600/) (2018)
- [Cryptocurrency Portfolio Management with Deep Reinforcement Learning](https://mlsanity.com/paper/1612.01277/) (2016)
- [A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Network](https://mlsanity.com/paper/1903.00714/) (2019)
- [Distillation Strategies for Proximal Policy Optimization](https://mlsanity.com/paper/1901.08128/) (2019)
- [Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting](https://mlsanity.com/paper/2002.07638/) (2020)
- [Trading algorithms with learning in latent alpha models](https://mlsanity.com/paper/1806.04472/) (2018)

## Built on by (most cited first, 30 of 30)

- [Bootstrapped Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.08569/) (2022)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Hindsight Learning for MDPs with Exogenous Inputs](https://mlsanity.com/paper/2207.06272/) (2022)
- [Reinforcement Learning for Quantitative Trading](https://mlsanity.com/paper/2109.13851/) (2021)
- [Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble](https://mlsanity.com/paper/2205.09284/) (2022)
- [DeepScalper: A Risk-Aware Reinforcement Learning Framework to Capture Fleeting Intraday Trading Opportunities](https://mlsanity.com/paper/2201.09058/) (2021)
- [PerfectDou: Dominating DouDizhu with Perfect Information Distillation](https://mlsanity.com/paper/2203.16406/) (2022)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective](https://mlsanity.com/paper/2305.18464/) (2023)
- [From Deep Learning to LLMs: A survey of AI in Quantitative Investment](https://mlsanity.com/paper/2503.21422/) (2025)
- [Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance](https://mlsanity.com/paper/2307.03119/) (2023)
- [Reinforcement Learning with Maskable Stock Representation for Portfolio Management in Customizable Stock Pools](https://mlsanity.com/paper/2311.10801/) (2023)
- [IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making](https://mlsanity.com/paper/2308.08918/) (2023)
- [PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets](https://mlsanity.com/paper/2302.00586/) (2023)
- [Towards Generalizable Reinforcement Learning for Trade Execution](https://mlsanity.com/paper/2307.11685/) (2023)
- [Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution](https://mlsanity.com/paper/2207.11152/) (2022)
- [Microstructure-Empowered Stock Factor Extraction and Utilization](https://mlsanity.com/paper/2308.08135/) (2023)
- [STORM: A Spatio-Temporal Factor Model Based on Dual Vector Quantized Variational Autoencoders for Financial Trading](https://mlsanity.com/paper/2412.09468/) (2024)
- [QuantBench: Benchmarking AI Methods for Quantitative Investment](https://mlsanity.com/paper/2504.18600/) (2025)
- [Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock Trading](https://mlsanity.com/paper/2206.05910/) (2022)
- [A Modular Framework for Reinforcement Learning Optimal Execution](https://mlsanity.com/paper/2208.06244/) (2022)
- [Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization](https://mlsanity.com/paper/2212.14670/) (2022)
- [Curriculum Learning and Imitation Learning for Model-free Control on Financial Time-series](https://mlsanity.com/paper/2311.13326/) (2023)
- [Large Investment Model](https://mlsanity.com/paper/2408.10255/) (2024)
- [Two Kinds of Learning Algorithms for Continuous-Time VWAP Targeting Execution](https://mlsanity.com/paper/2411.06645/) (2024)
- [Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixture](https://mlsanity.com/paper/2511.07110/) (2025)
- [State-Action Inpainting Diffuser for Continuous Control with Delay](https://mlsanity.com/paper/2603.01553/) (2026)
- [TT-DAC-PS: Twin-Target Deterministic Actor-Critic with Policy Smoothing for Optimal Trade Execution](https://mlsanity.com/paper/2606.08379/) (2026)
- [Can Large Language Models Execute Parent Orders?](https://mlsanity.com/paper/2607.28410/) (2026)

## Abstract (arXiv)

As a fundamental problem in algorithmic trading, order execution aims at fulfilling a specific trading order, either liquidation or acquirement, for a given instrument. Towards effective execution strategy, recent years have witnessed the shift from the analytical view with model-based market assumptions to model-free perspective, i.e., reinforcement learning, due to its nature of sequential decision optimization. However, the noisy and yet imperfect market information that can be leveraged by the policy has made it quite challenging to build up sample efficient reinforcement learning methods to achieve effective order execution. In this paper, we propose a novel universal trading policy optimization framework to bridge the gap between the noisy yet imperfect market states and the optimal action sequences for order execution. Particularly, this framework leverages a policy distillation method that can better guide the learning of the common policy towards practically optimal execution by an oracle teacher with perfect information to approximate the optimal trading strategy. The extensive experiments have shown significant improvements of our method over various strong baselines, with reasonable trading actions.