# Tabular Transformers for Modeling Multivariate Time Series

arXiv: https://arxiv.org/abs/2011.01843 · HTML: https://mlsanity.com/paper/2011.01843/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2020.

## Built on by (most cited first, 30 of 77)

- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [TabuLa: Harnessing Language Models for Tabular Data Synthesis](https://mlsanity.com/paper/2310.12746/) (2023)
- [TabMT: Generating tabular data with masked transformers](https://mlsanity.com/paper/2312.06089/) (2023)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition](https://mlsanity.com/paper/2106.10598/) (2021)
- [Large Language Models are Good Relational Learners](https://mlsanity.com/paper/2506.05725/) (2025)
- [BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model](https://mlsanity.com/paper/2404.03830/) (2024)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data](https://mlsanity.com/paper/2310.13818/) (2023)
- [One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Data](https://mlsanity.com/paper/2302.06375/) (2023)
- [Auditing and Generating Synthetic Data with Controllable Trust Trade-offs](https://mlsanity.com/paper/2304.10819/) (2023)
- [HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?](https://mlsanity.com/paper/2406.14341/) (2024)
- [MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns](https://mlsanity.com/paper/2511.10390/) (2025)
- [Multimodal Banking Dataset: Understanding Client Needs through Event Sequences](https://mlsanity.com/paper/2409.17587/) (2024)
- [LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training](https://mlsanity.com/paper/2212.02691/) (2022)
- [LaTable: Towards Large Tabular Models](https://mlsanity.com/paper/2406.17673/) (2024)
- [Forging Time Series with Language: A Large Language Model Approach to Synthetic Data Generation](https://mlsanity.com/paper/2505.17103/) (2025)
- [Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems](https://mlsanity.com/paper/2211.07455/) (2022)
- [AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Framework](https://mlsanity.com/paper/2401.00546/) (2023)
- [Learning Transactions Representations for Information Management in Banks: Mastering Local, Global, and External Knowledge](https://mlsanity.com/paper/2404.02047/) (2024)
- [Detecting the Future: All-at-Once Event Sequence Forecasting with Horizon Matching](https://mlsanity.com/paper/2408.13131/) (2024)

## Abstract (arXiv)

Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-of-the-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular sequences. We demonstrate our models on two datasets: a synthetic credit card transaction dataset, where the learned representations are used for fraud detection and synthetic data generation, and on a real pollution dataset, where the learned encodings are used to predict atmospheric pollutant concentrations. Code and data are available at https://github.com/IBM/TabFormer.