# LLM Embeddings for Deep Learning on Tabular Data

arXiv: https://arxiv.org/abs/2502.11596 · HTML: https://mlsanity.com/paper/2502.11596/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 48 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models](https://mlsanity.com/paper/2404.06209/) (2024)
- [Mambular: A Sequential Model for Tabular Deep Learning](https://mlsanity.com/paper/2408.06291/) (2024)
- [BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models](https://mlsanity.com/paper/2402.11573/) (2024)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)
- [Propositionalization and Embeddings: Two Sides of the Same Coin](https://mlsanity.com/paper/2006.04410/) (2020)

## Built on by (most cited first, 12 of 12)

- [Eliciting Numerical Predictive Distributions of LLMs Without Autoregression](https://mlsanity.com/paper/2603.02913/) (2026)
- [AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs](https://mlsanity.com/paper/2509.08000/) (2025)
- [Language Model Representations for Efficient Few-Shot Tabular Classification](https://mlsanity.com/paper/2602.15844/) (2026)
- [Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Design](https://mlsanity.com/paper/2603.17737/) (2026)
- [Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays](https://mlsanity.com/paper/2509.15234/) (2025)
- [Schema-Adaptive Tabular Representation Learning with LLMs for Generalizable Multimodal Clinical Reasoning](https://mlsanity.com/paper/2604.11835/) (2026)
- [Universal Laboratory Model: prognosis of abnormal clinical outcomes based on routine tests](https://mlsanity.com/paper/2506.15330/) (2025)
- [Innovative tokenisation of structured data for LLM training](https://mlsanity.com/paper/2508.01685/) (2025)
- [MAPLE: Multi-Path Adaptive Propagation with Level-Aware Embeddings for Hierarchical Multi-Label Image Classification](https://mlsanity.com/paper/2603.29784/) (2026)
- [Predicting Decisions of AI Agents from Limited Interaction through Text-Tabular Modeling](https://mlsanity.com/paper/2605.12411/) (2026)
- [TabNSM: Neural Sparse Mixer for Tabular Regression](https://mlsanity.com/paper/2608.18026/) (2026)
- [TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings](https://mlsanity.com/paper/2610.17065/) (2026)

## Abstract (arXiv)

Tabular deep-learning methods require embedding numerical and categorical input features into high-dimensional spaces before processing them. Existing methods deal with this heterogeneous nature of tabular data by employing separate type-specific encoding approaches. This limits the cross-table transfer potential and the exploitation of pre-trained knowledge. We propose a novel approach that first transforms tabular data into text, and then leverages pre-trained representations from LLMs to encode this data, resulting in a plug-and-play solution to improv ing deep-learning tabular methods. We demonstrate that our approach improves accuracy over competitive models, such as MLP, ResNet and FT-Transformer, by validating on seven classification datasets.