# One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Data

arXiv: https://arxiv.org/abs/2302.06375 · HTML: https://mlsanity.com/paper/2302.06375/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [Tabular Transformers for Modeling Multivariate Time Series](https://mlsanity.com/paper/2011.01843/) (2020)
- [Adversarial Attacks on Deep Models for Financial Transaction Records](https://mlsanity.com/paper/2106.08361/) (2021)
- [Hopular: Modern Hopfield Networks for Tabular Data](https://mlsanity.com/paper/2206.00664/) (2022)
- [Google Dataset Search by the Numbers](https://mlsanity.com/paper/2006.06894/) (2020)
- [OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction](https://mlsanity.com/paper/2208.04482/) (2022)
- [LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training](https://mlsanity.com/paper/2212.02691/) (2022)
- [Attention Augmented Convolutional Transformer for Tabular Time-series](https://mlsanity.com/paper/2110.01825/) (2021)

## Built on by (most cited first, 13 of 13)

- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data](https://mlsanity.com/paper/2310.13818/) (2023)
- [Diffusion Transformers for Tabular Data Time Series Generation](https://mlsanity.com/paper/2504.07566/) (2025)
- [FinTRec: Transformer Based Unified Contextual Ads Targeting and Personalization for Financial Applications](https://mlsanity.com/paper/2511.14865/) (2025)
- [TIMeSynC: Temporal Intent Modelling with Synchronized Context Encodings for Financial Service Applications](https://mlsanity.com/paper/2410.12825/) (2024)
- [LATTE: Learning Aligned Transactions and Textual Embeddings for Bank Clients](https://mlsanity.com/paper/2508.10021/) (2025)
- [Fine-grained Attention in Hierarchical Transformers for Tabular Time-series](https://mlsanity.com/paper/2406.15327/) (2024)
- [A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers](https://mlsanity.com/paper/2410.10648/) (2024)
- [Comparative Analysis of Transformers for Modeling Tabular Data: A Casestudy using Industry Scale Dataset](https://mlsanity.com/paper/2311.14335/) (2023)
- [Basis Transformers for Multi-Task Tabular Regression](https://mlsanity.com/paper/2506.06926/) (2025)
- [SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training](https://mlsanity.com/paper/2606.21356/) (2026)
- [Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective](https://mlsanity.com/paper/2610.12586/) (2026)

## Abstract (arXiv)

There is a recent growing interest in applying Deep Learning techniques to tabular data, in order to replicate the success of other Artificial Intelligence areas in this structured domain. Specifically interesting is the case in which tabular data have a time dependence, such as, for instance financial transactions. However, the heterogeneity of the tabular values, in which categorical elements are mixed with numerical items, makes this adaptation difficult. In this paper we propose a Transformer architecture to represent heterogeneous time-dependent tabular data, in which numerical features are represented using a set of frequency functions and the whole network is uniformly trained with a unique loss function.