# Trading with the Momentum Transformer: An Intelligent and Interpretable Architecture

arXiv: https://arxiv.org/abs/2112.08534 · HTML: https://mlsanity.com/paper/2112.08534/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [Deep Reinforcement Learning for Trading](https://mlsanity.com/paper/1911.10107/) (2019)
- [Universal features of price formation in financial markets: perspectives from Deep Learning](https://mlsanity.com/paper/1803.06917/) (2018)
- [Enhancing Time Series Momentum Strategies Using Deep Neural Networks](https://mlsanity.com/paper/1904.04912/) (2019)
- [Slow Momentum with Fast Reversion: A Trading Strategy Using Deep Learning and Changepoint Detection](https://mlsanity.com/paper/2105.13727/) (2021)

## Built on by (most cited first, 16 of 16)

- [FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading](https://mlsanity.com/paper/2502.11433/) (2025)
- [Deep Limit Order Book Forecasting](https://mlsanity.com/paper/2403.09267/) (2024)
- [Spatio-Temporal Momentum: Jointly Learning Time-Series and Cross-Sectional Strategies](https://mlsanity.com/paper/2302.10175/) (2023)
- [Constructing Time-Series Momentum Portfolios with Deep Multi-Task Learning](https://mlsanity.com/paper/2306.13661/) (2023)
- [Few-Shot Learning Patterns in Financial Time-Series for Trend-Following Strategies](https://mlsanity.com/paper/2310.10500/) (2023)
- [Neuro-symbolic Meta Reinforcement Learning for Trading](https://mlsanity.com/paper/2302.08996/) (2023)
- [Deep Learning for Options Trading: An End-To-End Approach](https://mlsanity.com/paper/2407.21791/) (2024)
- [Robust Detection of Lead-Lag Relationships in Lagged Multi-Factor Models](https://mlsanity.com/paper/2305.06704/) (2023)
- [Network Momentum across Asset Classes](https://mlsanity.com/paper/2308.11294/) (2023)
- [Conservative Predictions on Noisy Financial Data](https://mlsanity.com/paper/2310.11815/) (2023)
- [Learning to Learn Financial Networks for Optimising Momentum Strategies](https://mlsanity.com/paper/2308.12212/) (2023)
- [Cross-Modal Temporal Fusion for Financial Market Forecasting](https://mlsanity.com/paper/2504.13522/) (2025)
- [DeePM: Regime-Robust Deep Learning for Systematic Macro Portfolio Management](https://mlsanity.com/paper/2601.05975/) (2026)
- [Enhancing Portfolio Optimization with Deep Learning Insights](https://mlsanity.com/paper/2601.07942/) (2026)
- [Deep Learning for Financial Time Series: A Large-Scale Benchmark of Risk-Adjusted Performance](https://mlsanity.com/paper/2603.01820/) (2026)
- [Taming the Greeks: Option Portfolios with Inductive Biases](https://mlsanity.com/paper/2609.33767/) (2026)

## Abstract (arXiv)

We introduce the Momentum Transformer, an attention-based deep-learning architecture, which outperforms benchmark time-series momentum and mean-reversion trading strategies. Unlike state-of-the-art Long Short-Term Memory (LSTM) architectures, which are sequential in nature and tailored to local processing, an attention mechanism provides our architecture with a direct connection to all previous time-steps. Our architecture, an attention-LSTM hybrid, enables us to learn longer-term dependencies, improves performance when considering returns net of transaction costs and naturally adapts to new market regimes, such as during the SARS-CoV-2 crisis. Via the introduction of multiple attention heads, we can capture concurrent regimes, or temporal dynamics, which are occurring at different timescales. The Momentum Transformer is inherently interpretable, providing us with greater insights into our deep-learning momentum trading strategy, including the importance of different factors over time and the past time-steps which are of the greatest significance to the model.