# FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data

arXiv: https://arxiv.org/abs/2310.13818 · HTML: https://mlsanity.com/paper/2310.13818/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [Tabular Transformers for Modeling Multivariate Time Series](https://mlsanity.com/paper/2011.01843/) (2020)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks](https://mlsanity.com/paper/1404.5772/) (2014)
- [Denoising Self-attentive Sequential Recommendation](https://mlsanity.com/paper/2212.04120/) (2022)
- [Interleaved Sequence RNNs for Fraud Detection](https://mlsanity.com/paper/2002.05988/) (2020)
- [Simple Modifications to Improve Tabular Neural Networks](https://mlsanity.com/paper/2108.03214/) (2021)
- [One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Data](https://mlsanity.com/paper/2302.06375/) (2023)
- [Synthesizing Credit Card Transactions](https://mlsanity.com/paper/1910.03033/) (2019)
- [LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training](https://mlsanity.com/paper/2212.02691/) (2022)
- [Attributed Sequence Embedding](https://mlsanity.com/paper/1911.00949/) (2019)

## Built on by (most cited first, 14 of 14)

- [Multimodal Banking Dataset: Understanding Client Needs through Event Sequences](https://mlsanity.com/paper/2409.17587/) (2024)
- [VecAug: Unveiling Camouflaged Frauds with Cohort Augmentation for Enhanced Detection](https://mlsanity.com/paper/2408.00513/) (2024)
- [FinTRec: Transformer Based Unified Contextual Ads Targeting and Personalization for Financial Applications](https://mlsanity.com/paper/2511.14865/) (2025)
- [TREASURE: The Visa Payment Foundation Model for High-Volume Transaction Understanding](https://mlsanity.com/paper/2511.19693/) (2025)
- [TIMeSynC: Temporal Intent Modelling with Synchronized Context Encodings for Financial Service Applications](https://mlsanity.com/paper/2410.12825/) (2024)
- [LATTE: Learning Aligned Transactions and Textual Embeddings for Bank Clients](https://mlsanity.com/paper/2508.10021/) (2025)
- [Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings](https://mlsanity.com/paper/2601.05271/) (2025)
- [Fine-grained Attention in Hierarchical Transformers for Tabular Time-series](https://mlsanity.com/paper/2406.15327/) (2024)
- [A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers](https://mlsanity.com/paper/2410.10648/) (2024)
- [PersonaLedger: Generating Realistic Financial Transactions with Persona Conditioned LLMs and Rule Grounded Feedback](https://mlsanity.com/paper/2601.03149/) (2026)
- [A-THENA: Early Intrusion Detection for IoT with Time-Aware Hybrid Encoding and Network-Specific Augmentation](https://mlsanity.com/paper/2604.21623/) (2026)
- [CoLafier: Collaborative Noisy Label Purifier With Local Intrinsic Dimensionality Guidance](https://mlsanity.com/paper/2401.05458/) (2024)
- [TransactionGPT](https://mlsanity.com/paper/2511.08939/) (2025)
- [SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training](https://mlsanity.com/paper/2606.21356/) (2026)

## Abstract (arXiv)

Sequential tabular data is one of the most commonly used data types in real-world applications. Different from conventional tabular data, where rows in a table are independent, sequential tabular data contains rich contextual and sequential information, where some fields are dynamically changing over time and others are static. Existing transformer-based approaches analyzing sequential tabular data overlook the differences between dynamic and static fields by replicating and filling static fields into each transformer, and ignore temporal information between rows, which leads to three major disadvantages: (1) computational overhead, (2) artificially simplified data for masked language modeling pre-training task that may yield less meaningful representations, and (3) disregarding the temporal behavioral patterns implied by time intervals. In this work, we propose FATA-Trans, a model with two field transformers for modeling sequential tabular data, where each processes static and dynamic field information separately. FATA-Trans is field- and time-aware for sequential tabular data. The field-type embedding in the method enables FATA-Trans to capture differences between static and dynamic fields. The time-aware position embedding exploits both order and time interval information between rows, which helps the model detect underlying temporal behavior in a sequence. Our experiments on three benchmark datasets demonstrate that the learned representations from FATA-Trans consistently outperform state-of-the-art solutions in the downstream tasks. We also present visualization studies to highlight the insights captured by the learned representations, enhancing our understanding of the underlying data. Our codes are available at https://github.com/zdy93/FATA-Trans.