# Rough Transformers for Continuous and Efficient Time-Series Modelling

arXiv: https://arxiv.org/abs/2403.10288 · HTML: https://mlsanity.com/paper/2403.10288/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Neural Rough Differential Equations for Long Time Series](https://mlsanity.com/paper/2009.08295/) (2020)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations](https://mlsanity.com/paper/2206.08311/) (2022)
- [Neural ODE Processes](https://mlsanity.com/paper/2103.12413/) (2021)
- [Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences](https://mlsanity.com/paper/2210.11794/) (2022)
- [Deep Autoregressive Models with Spectral Attention](https://mlsanity.com/paper/2107.05984/) (2021)
- [DeepVol: Volatility Forecasting from High-Frequency Data with Dilated Causal Convolutions](https://mlsanity.com/paper/2210.04797/) (2022)
- [Nonstationary Portfolios: Diversification in the Spectral Domain](https://mlsanity.com/paper/2102.00477/) (2021)
- [Dynamic Portfolio Cuts: A Spectral Approach to Graph-Theoretic Diversification](https://mlsanity.com/paper/2106.03417/) (2021)
- [Heterogeneous Hidden Markov Models for Sleep Activity Recognition from Multi-Source Passively Sensed Data](https://mlsanity.com/paper/2211.10371/) (2022)
- [Sleep Activity Recognition and Characterization from Multi-Source Passively Sensed Data](https://mlsanity.com/paper/2301.10156/) (2023)

## Built on by (most cited first, 6 of 6)

- [On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning](https://mlsanity.com/paper/2502.10818/) (2025)
- [Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching](https://mlsanity.com/paper/2405.20799/) (2024)
- [SigGate: Enhancing Recurrent Neural Networks with Signature-Based Gating Mechanisms](https://mlsanity.com/paper/2502.09318/) (2025)
- [SigMA: Path Signatures and Multi-head Attention for Learning Parameters in fBm-driven SDEs](https://mlsanity.com/paper/2512.15088/) (2025)
- [TIDES: Implicit Time-Awareness in Selective State Space Models](https://mlsanity.com/paper/2605.09742/) (2026)
- [LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Models](https://mlsanity.com/paper/2610.05051/) (2026)

## Abstract (arXiv)

Time-series data in real-world medical settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In such contexts, traditional sequence-based recurrent models struggle. To overcome this, researchers replace recurrent architectures with Neural ODE-based models to model irregularly sampled data and use Transformer-based architectures to account for long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of moderate lengths and greater. To mitigate this, we introduce the Rough Transformer, a variation of the Transformer model which operates on continuous-time representations of input sequences and incurs significantly reduced computational costs, critical for addressing long-range dependencies common in medical contexts. In particular, we propose multi-view signature attention, which uses path signatures to augment vanilla attention and to capture both local and global dependencies in input data, while remaining robust to changes in the sequence length and sampling frequency. We find that Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the benefits of Neural ODE-based models using a fraction of the computational time and memory resources on synthetic and real-world time-series tasks.