# IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making

arXiv: https://arxiv.org/abs/2308.08918 · HTML: https://mlsanity.com/paper/2308.08918/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Market Making via Reinforcement Learning](https://mlsanity.com/paper/1804.04216/) (2018)
- [Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning](https://mlsanity.com/paper/1805.11240/) (2018)
- [Universal Trading for Order Execution with Oracle Policy Distillation](https://mlsanity.com/paper/2103.10860/) (2021)
- [Guided Meta-Policy Search](https://mlsanity.com/paper/1904.00956/) (2019)
- [Optimizing Market Making using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1812.10252/) (2018)
- [MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization](https://mlsanity.com/paper/2210.01774/) (2022)
- [Market Making with Scaled Beta Policies](https://mlsanity.com/paper/2207.03352/) (2022)

## Built on by (most cited first, 6 of 6)

- [FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting](https://mlsanity.com/paper/2502.18834/) (2025)
- [A Survey of Financial AI: Architectures, Advances and Open Challenges](https://mlsanity.com/paper/2411.12747/) (2024)
- [FineFT: Efficient and Risk-Aware Ensemble Reinforcement Learning for Futures Trading](https://mlsanity.com/paper/2512.23773/) (2025)
- [An Imitative Reinforcement Learning Framework for Pursuit-Lock-Launch Missions](https://mlsanity.com/paper/2406.11562/) (2024)
- [Resolving Latency and Inventory Risk in Market Making with Reinforcement Learning](https://mlsanity.com/paper/2505.12465/) (2025)
- [Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixture](https://mlsanity.com/paper/2511.07110/) (2025)

## Abstract (arXiv)

Market making (MM) has attracted significant attention in financial trading owing to its essential function in ensuring market liquidity. With strong capabilities in sequential decision-making, Reinforcement Learning (RL) technology has achieved remarkable success in quantitative trading. Nonetheless, most existing RL-based MM methods focus on optimizing single-price level strategies which fail at frequent order cancellations and loss of queue priority. Strategies involving multiple price levels align better with actual trading scenarios. However, given the complexity that multi-price level strategies involves a comprehensive trading action space, the challenge of effectively training profitable RL agents for MM persists. Inspired by the efficient workflow of professional human market makers, we propose Imitative Market Maker (IMM), a novel RL framework leveraging both knowledge from suboptimal signal-based experts and direct policy interactions to develop multi-price level MM strategies efficiently. The framework start with introducing effective state and action representations adept at encoding information about multi-price level orders. Furthermore, IMM integrates a representation learning unit capable of capturing both short- and long-term market trends to mitigate adverse selection risk. Subsequently, IMM formulates an expert strategy based on signals and trains the agent through the integration of RL and imitation learning techniques, leading to efficient learning. Extensive experimental results on four real-world market datasets demonstrate that IMM outperforms current RL-based market making strategies in terms of several financial criteria. The findings of the ablation study substantiate the effectiveness of the model components.