# LOBERT: Generative AI Foundation Model for Limit Order Book Messages

arXiv: https://arxiv.org/abs/2511.12563 · HTML: https://mlsanity.com/paper/2511.12563/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Universal features of price formation in financial markets: perspectives from Deep Learning](https://mlsanity.com/paper/1803.06917/) (2018)
- [Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis](https://mlsanity.com/paper/1712.00975/) (2017)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead](https://mlsanity.com/paper/2105.09121/) (2021)
- [Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network](https://mlsanity.com/paper/2309.00638/) (2023)
- [Continual Transformers: Redundancy-Free Attention for Online Inference](https://mlsanity.com/paper/2201.06268/) (2022)
- [Structured Pruning Adapters](https://mlsanity.com/paper/2211.10155/) (2022)
- [Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch](https://mlsanity.com/paper/2204.03418/) (2022)
- [TRADES: Generating Realistic Market Simulations with Diffusion Models](https://mlsanity.com/paper/2502.07071/) (2025)
- [TLOB: A Novel Transformer Model with Dual Attention for Price Trend Prediction with Limit Order Book Data](https://mlsanity.com/paper/2502.15757/) (2025)

## Built on by (most cited first, 2 of 2)

- [Repurposing Deep Limit Order Book Forecasting for Scenario-Conditioned Market Impact Modeling](https://mlsanity.com/paper/2609.16930/) (2026)
- [UQ-LOB: Uncertainty-Aware Limit Order Book Mid-Price Forecasting](https://mlsanity.com/paper/2609.31491/) (2026)

## Abstract (arXiv)

Modeling the dynamics of financial Limit Order Books (LOB) at the message level is challenging due to irregular event timing, rapid regime shifts, and the reactions of high-frequency traders to visible order flow. Previous LOB models require cumbersome data representations and lack adaptability outside their original tasks, leading us to introduce LOBERT, a general-purpose encoder-only foundation model for LOB data suitable for downstream fine-tuning. LOBERT adapts the original BERT architecture for LOB data by using a novel tokenization scheme that treats complete multi-dimensional messages as single tokens while retaining continuous representations of price, volume, and time. With these methods, LOBERT achieves leading performance in tasks such as predicting mid-price movements and next messages, while reducing the required context length compared to previous methods.