# Long-term series forecasting with Query Selector -- efficient model of sparse attention

arXiv: https://arxiv.org/abs/2107.08687 · HTML: https://mlsanity.com/paper/2107.08687/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [A Multi-Horizon Quantile Recurrent Forecaster](https://mlsanity.com/paper/1711.11053/) (2017)
- [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016)
- [Do We Really Need Deep Learning Models for Time Series Forecasting?](https://mlsanity.com/paper/2101.02118/) (2021)
- [Time and Activity Sequence Prediction of Business Process Instances](https://mlsanity.com/paper/1602.07566/) (2016)
- [Sequence to Sequence with Attention for Influenza Prevalence Prediction using Google Trends](https://mlsanity.com/paper/1907.02786/) (2019)

## Built on by (most cited first, 4 of 4)

- [Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting](https://mlsanity.com/paper/2110.08255/) (2021)
- [CREATED: Generating Viable Counterfactual Sequences for Predictive Process Analytics](https://mlsanity.com/paper/2303.15844/) (2023)
- [Temporal Saliency Detection Towards Explainable Transformer-based Timeseries Forecasting](https://mlsanity.com/paper/2212.07771/) (2022)
- [Naga: Vedic Encoding for Deep State Space Models](https://mlsanity.com/paper/2511.13510/) (2025)

## Abstract (arXiv)

Various modifications of TRANSFORMER were recently used to solve time-series forecasting problem. We propose Query Selector - an efficient, deterministic algorithm for sparse attention matrix. Experiments show it achieves state-of-the art results on ETT, Helpdesk and BPI'12 datasets.