# ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation

arXiv: https://arxiv.org/abs/2312.01728 · HTML: https://mlsanity.com/paper/2312.01728/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 71 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [GAIN: Missing Data Imputation using Generative Adversarial Nets](https://mlsanity.com/paper/1806.02920/) (2018)
- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [SAITS: Self-Attention-based Imputation for Time Series](https://mlsanity.com/paper/2202.08516/) (2022)
- [TimeGPT-1](https://mlsanity.com/paper/2310.03589/) (2023)
- [Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2208.05233/) (2022)
- [A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection](https://mlsanity.com/paper/2307.03759/) (2023)
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.09113/) (2022)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks](https://mlsanity.com/paper/2108.00298/) (2021)
- [Inductive Graph Neural Networks for Spatiotemporal Kriging](https://mlsanity.com/paper/2006.07527/) (2020)
- [Low-Rank Bottleneck in Multi-head Attention Models](https://mlsanity.com/paper/2002.07028/) (2020)
- [PriSTI: A Conditional Diffusion Framework for Spatiotemporal Imputation](https://mlsanity.com/paper/2302.09746/) (2023)
- [Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations](https://mlsanity.com/paper/2205.13479/) (2022)
- [Ti-MAE: Self-Supervised Masked Time Series Autoencoders](https://mlsanity.com/paper/2301.08871/) (2023)
- [NAOMI: Non-Autoregressive Multiresolution Sequence Imputation](https://mlsanity.com/paper/1901.10946/) (2019)
- [Bayesian Temporal Factorization for Multidimensional Time Series Prediction](https://mlsanity.com/paper/1910.06366/) (2019)
- [Physics-guided Convolutional Neural Network (PhyCNN) for Data-driven Seismic Response Modeling](https://mlsanity.com/paper/1909.08118/) (2019)
- [Taming Local Effects in Graph-based Spatiotemporal Forecasting](https://mlsanity.com/paper/2302.04071/) (2023)

## Built on by (most cited first, 30 of 50)

- [Deep Learning for Multivariate Time Series Imputation: A Survey](https://mlsanity.com/paper/2402.04059/) (2024)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [SSD-TS: Exploring the Potential of Linear State Space Models for Diffusion Models in Time Series Imputation](https://mlsanity.com/paper/2410.13338/) (2024)
- [Geolocation Representation from Large Language Models are Generic Enhancers for Spatio-Temporal Learning](https://mlsanity.com/paper/2408.12116/) (2024)
- [Channel-Aware Low-Rank Adaptation in Time Series Forecasting](https://mlsanity.com/paper/2407.17246/) (2024)
- [Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecasting](https://mlsanity.com/paper/2410.12593/) (2024)
- [Diffusion Transformers as Open-World Spatiotemporal Foundation Models](https://mlsanity.com/paper/2411.12164/) (2024)
- [Glocal Information Bottleneck for Time Series Imputation](https://mlsanity.com/paper/2510.04910/) (2025)
- [Spatiotemporal Implicit Neural Representation as a Generalized Traffic Data Learner](https://mlsanity.com/paper/2405.03185/) (2024)
- [Revisiting Multivariate Time Series Forecasting with Missing Values](https://mlsanity.com/paper/2509.23494/) (2025)
- [Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approach](https://mlsanity.com/paper/2408.17258/) (2024)
- [Task-oriented Time Series Imputation Evaluation via Generalized Representers](https://mlsanity.com/paper/2410.06652/) (2024)
- [FreshRetailNet-50K: A Stockout-Annotated Censored Demand Dataset for Latent Demand Recovery and Forecasting in Fresh Retail](https://mlsanity.com/paper/2505.16319/) (2025)
- [Decoupling Spatio-Temporal Prediction: When Lightweight Large Models Meet Adaptive Hypergraphs](https://mlsanity.com/paper/2505.19620/) (2025)
- [Trajectory Imputation in Multi-Agent Sports with Derivative-Accumulating Self-Ensemble](https://mlsanity.com/paper/2408.10878/) (2024)
- [ImputeINR: Time Series Imputation via Implicit Neural Representations for Disease Diagnosis with Missing Data](https://mlsanity.com/paper/2505.10856/) (2025)
- [MH-GIN: Multi-scale Heterogeneous Graph-based Imputation Network for AIS Data (Extended Version)](https://mlsanity.com/paper/2507.20362/) (2025)
- [Latent Space Score-based Diffusion Model for Probabilistic Multivariate Time Series Imputation](https://mlsanity.com/paper/2409.08917/) (2024)
- [STAMImputer: Spatio-Temporal Attention MoE for Traffic Data Imputation](https://mlsanity.com/paper/2506.08054/) (2025)
- [FADTI: Fourier and Attention Driven Diffusion for Multivariate Time Series Imputation](https://mlsanity.com/paper/2512.15116/) (2025)
- [T1: One-to-One Channel-Head Binding for Multivariate Time-Series Imputation](https://mlsanity.com/paper/2602.21043/) (2026)
- [High-Dimensional Fault Tolerance Testing of Highly Automated Vehicles Based on Low-Rank Models](https://mlsanity.com/paper/2407.21069/) (2024)
- [Causal View of Time Series Imputation: Some Identification Results on Missing Mechanism](https://mlsanity.com/paper/2505.07180/) (2025)
- [Predicting Large-scale Urban Network Dynamics with Energy-informed Graph Neural Diffusion](https://mlsanity.com/paper/2508.00037/) (2025)
- [FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networks](https://mlsanity.com/paper/2510.03589/) (2025)
- [RINS-T: Robust Implicit Neural Solvers for Time Series Linear Inverse Problems](https://mlsanity.com/paper/2510.17396/) (2025)
- [Are Time-Indexed Foundation Models the Future of Time Series Imputation?](https://mlsanity.com/paper/2511.05980/) (2025)
- [Spatial-Temporal Feedback Diffusion Guidance for Controlled Traffic Imputation](https://mlsanity.com/paper/2601.04572/) (2026)
- [UrbanFM: Scaling Urban Spatio-Temporal Foundation Models](https://mlsanity.com/paper/2602.20677/) (2026)
- [SPLICE: Latent Diffusion over JEPA Embeddings for Conformal Time-Series Inpainting](https://mlsanity.com/paper/2605.00126/) (2026)

## Abstract (arXiv)

Missing data is a pervasive issue in both scientific and engineering tasks, especially for the modeling of spatiotemporal data. This problem attracts many studies to contribute to data-driven solutions. Existing imputation solutions mainly include low-rank models and deep learning models. The former assumes general structural priors but has limited model capacity. The latter possesses salient features of expressivity but lacks prior knowledge of the underlying spatiotemporal structures. Leveraging the strengths of both two paradigms, we demonstrate a low rankness-induced Transformer to achieve a balance between strong inductive bias and high model expressivity. The exploitation of the inherent structures of spatiotemporal data enables our model to learn balanced signal-noise representations, making it generalizable for a variety of imputation problems. We demonstrate its superiority in terms of accuracy, efficiency, and versatility in heterogeneous datasets, including traffic flow, solar energy, smart meters, and air quality. Promising empirical results provide strong conviction that incorporating time series primitives, such as low-rankness, can substantially facilitate the development of a generalizable model to approach a wide range of spatiotemporal imputation problems.