# Scalable Representation Learning for Multimodal Tabular Transactions

arXiv: https://arxiv.org/abs/2410.07851 · HTML: https://mlsanity.com/paper/2410.07851/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Hierarchical Clustering with Hard-batch Triplet Loss for Person Re-identification](https://mlsanity.com/paper/1910.12278/) (2019)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [PTaRL: Prototype-based Tabular Representation Learning via Space Calibration](https://mlsanity.com/paper/2407.05364/) (2024)
- [Learning Multi-granular Quantized Embeddings for Large-Vocab Categorical Features in Recommender Systems](https://mlsanity.com/paper/2002.08530/) (2020)
- [A Survey on Self-Supervised Learning for Non-Sequential Tabular Data](https://mlsanity.com/paper/2402.01204/) (2024)
- [Embedding Compression in Recommender Systems: A Survey](https://mlsanity.com/paper/2408.02304/) (2024)

## Built on by (most cited first, 5 of 5)

- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [TradeFM: A Generative Foundation Model for Trade-flow and Market Microstructure](https://mlsanity.com/paper/2602.23784/) (2026)
- [Innovative tokenisation of structured data for LLM training](https://mlsanity.com/paper/2508.01685/) (2025)
- [A Foundation Model for Multimodal Event Sequences in Financial Applications](https://mlsanity.com/paper/2607.09955/) (2026)
- [MINT: A Universal Zero-Shot Predictor for Transaction Data](https://mlsanity.com/paper/2608.14198/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are primarily designed to understand unstructured text. When directly applied to structured formats such as tabular data, they may struggle to discern inherent relationships and overlook critical patterns. While tabular representation learning methods can address some of these limitations, existing efforts still face challenges with sparse high-cardinality fields, precise numerical reasoning, and column-heavy tables. Furthermore, leveraging these learned representations for downstream tasks through a language based interface is not apparent. In this paper, we present an innovative and scalable solution to these challenges. Concretely, our approach introduces a multi-tier partitioning mechanism that utilizes power-law dynamics to handle large vocabularies, an adaptive quantization mechanism to impose priors on numerical continuity, and a distinct treatment of core-columns and meta-information columns. To facilitate instruction tuning on LLMs, we propose a parameter efficient decoder that interleaves transaction and text modalities using a series of adapter layers, thereby exploiting rich cross-task knowledge. We validate the efficacy of our solution on a large-scale dataset of synthetic payments transactions.