# Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network

arXiv: https://arxiv.org/abs/2309.00638 · HTML: https://mlsanity.com/paper/2309.00638/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks](https://mlsanity.com/paper/2009.07769/) (2020)
- [Effectively Modeling Time Series with Simple Discrete State Spaces](https://mlsanity.com/paper/2303.09489/) (2023)
- [Autoregressive Quantile Networks for Generative Modeling](https://mlsanity.com/paper/1806.05575/) (2018)
- [ExGAN: Adversarial Generation of Extreme Samples](https://mlsanity.com/paper/2009.08454/) (2020)
- [Towards Realistic Market Simulations: a Generative Adversarial Networks Approach](https://mlsanity.com/paper/2110.13287/) (2021)
- [Learning to simulate realistic limit order book markets from data as a World Agent](https://mlsanity.com/paper/2210.09897/) (2022)
- [Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN](https://mlsanity.com/paper/1903.12549/) (2019)
- [Conditional Generators for Limit Order Book Environments: Explainability, Challenges, and Robustness](https://mlsanity.com/paper/2306.12806/) (2023)
- [Deep Learning for Market by Order Data](https://mlsanity.com/paper/2102.08811/) (2021)

## Built on by (most cited first, 20 of 20)

- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [Time-Causal VAE: Robust Financial Time Series Generator](https://mlsanity.com/paper/2411.02947/) (2024)
- [TradeFM: A Generative Foundation Model for Trade-flow and Market Microstructure](https://mlsanity.com/paper/2602.23784/) (2026)
- [TRADES: Generating Realistic Market Simulations with Diffusion Models](https://mlsanity.com/paper/2502.07071/) (2025)
- [LOB-Bench: Benchmarking Generative AI for Finance -- an Application to Limit Order Book Data](https://mlsanity.com/paper/2502.09172/) (2025)
- [Are Large Language Models Good In-context Learners for Financial Sentiment Analysis?](https://mlsanity.com/paper/2503.04873/) (2025)
- [TLOB: A Novel Transformer Model with Dual Attention for Price Trend Prediction with Limit Order Book Data](https://mlsanity.com/paper/2502.15757/) (2025)
- [ByteGen: A Tokenizer-Free Generative Model for Orderbook Events in Byte Space](https://mlsanity.com/paper/2508.02247/) (2025)
- [LOBERT: Generative AI Foundation Model for Limit Order Book Messages](https://mlsanity.com/paper/2511.12563/) (2025)
- [Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution](https://mlsanity.com/paper/2511.15262/) (2025)
- [DiffLOB: Diffusion Models for Counterfactual Generation in Limit Order Books](https://mlsanity.com/paper/2602.03776/) (2026)
- [DiffVolume: Diffusion Models for Volume Generation in Limit Order Books](https://mlsanity.com/paper/2508.08698/) (2025)
- [Large Investment Model](https://mlsanity.com/paper/2408.10255/) (2024)
- [Guided Learning: Lubricating End-to-End Modeling for Multi-stage Decision-making](https://mlsanity.com/paper/2411.10496/) (2024)
- [TABL-ABM: A Hybrid Framework for Synthetic LOB Generation](https://mlsanity.com/paper/2510.22685/) (2025)
- [M3: A State-Event Generative Foundation Model for Market Microstructure Dynamics](https://mlsanity.com/paper/2608.19227/) (2026)
- [UQ-LOB: Uncertainty-Aware Limit Order Book Mid-Price Forecasting](https://mlsanity.com/paper/2609.31491/) (2026)
- [ReLOBGen: Replayable Limit Order Book Message Generation](https://mlsanity.com/paper/2609.35867/) (2026)
- [What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books](https://mlsanity.com/paper/2610.20104/) (2026)

## Abstract (arXiv)

Developing a generative model of realistic order flow in financial markets is a challenging open problem, with numerous applications for market participants. Addressing this, we propose the first end-to-end autoregressive generative model that generates tokenized limit order book (LOB) messages. These messages are interpreted by a Jax-LOB simulator, which updates the LOB state. To handle long sequences efficiently, the model employs simplified structured state-space layers to process sequences of order book states and tokenized messages. Using LOBSTER data of NASDAQ equity LOBs, we develop a custom tokenizer for message data, converting groups of successive digits to tokens, similar to tokenization in large language models. Out-of-sample results show promising performance in approximating the data distribution, as evidenced by low model perplexity. Furthermore, the mid-price returns calculated from the generated order flow exhibit a significant correlation with the data, indicating impressive conditional forecast performance. Due to the granularity of generated data, and the accuracy of the model, it offers new application areas for future work beyond forecasting, e.g. acting as a world model in high-frequency financial reinforcement learning applications. Overall, our results invite the use and extension of the model in the direction of autoregressive large financial models for the generation of high-frequency financial data and we commit to open-sourcing our code to facilitate future research.