# Transformers versus LSTMs for electronic trading

arXiv: https://arxiv.org/abs/2309.11400 · HTML: https://mlsanity.com/paper/2309.11400/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 64 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Universal features of price formation in financial markets: perspectives from Deep Learning](https://mlsanity.com/paper/1803.06917/) (2018)
- [Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1610.09975/) (2016)
- [Using Deep Learning for price prediction by exploiting stationary limit order book features](https://mlsanity.com/paper/1810.09965/) (2018)
- [A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation](https://mlsanity.com/paper/1806.06957/) (2018)
- [Fast Multi-language LSTM-based Online Handwriting Recognition](https://mlsanity.com/paper/1902.10525/) (2019)
- [A Comparative Analysis of Forecasting Financial Time Series Using ARIMA, LSTM, and BiLSTM](https://mlsanity.com/paper/1911.09512/) (2019)
- [Stock price prediction using BERT and GAN](https://mlsanity.com/paper/2107.09055/) (2021)
- [Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units](https://mlsanity.com/paper/2105.10430/) (2021)
- [Financial series prediction using Attention LSTM](https://mlsanity.com/paper/1902.10877/) (2019)
- [Exploring the Advantages of Transformers for High-Frequency Trading](https://mlsanity.com/paper/2302.13850/) (2023)

## Built on by (most cited first, 4 of 4)

- [COTN: A Chaotic Oscillatory Transformer Network for Complex Volatile Systems under Extreme Conditions](https://mlsanity.com/paper/2511.06273/) (2025)
- [HieroLM: Egyptian Hieroglyph Recovery with Next Word Prediction Language Model](https://mlsanity.com/paper/2503.04996/) (2025)
- [Forecast-Then-Optimize Deep Learning Methods](https://mlsanity.com/paper/2506.13036/) (2025)
- [Multivariate Spatio-Temporal Neural Hawkes Processes](https://mlsanity.com/paper/2602.23629/) (2026)

## Abstract (arXiv)

With the rapid development of artificial intelligence, long short term memory (LSTM), one kind of recurrent neural network (RNN), has been widely applied in time series prediction. Like RNN, Transformer is designed to handle the sequential data. As Transformer achieved great success in Natural Language Processing (NLP), researchers got interested in Transformer's performance on time series prediction, and plenty of Transformer-based solutions on long time series forecasting have come out recently. However, when it comes to financial time series prediction, LSTM is still a dominant architecture. Therefore, the question this study wants to answer is: whether the Transformer-based model can be applied in financial time series prediction and beat LSTM. To answer this question, various LSTM-based and Transformer-based models are compared on multiple financial prediction tasks based on high-frequency limit order book data. A new LSTM-based model called DLSTM is built and new architecture for the Transformer-based model is designed to adapt for financial prediction. The experiment result reflects that the Transformer-based model only has the limited advantage in absolute price sequence prediction. The LSTM-based models show better and more robust performance on difference sequence prediction, such as price difference and price movement.