# Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learning

arXiv: https://arxiv.org/abs/2406.13846 · HTML: https://mlsanity.com/paper/2406.13846/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 96 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)

## Built on by (most cited first, 17 of 17)

- [Clinical ModernBERT: An efficient and long context encoder for biomedical text](https://mlsanity.com/paper/2504.03964/) (2025)
- [HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series](https://mlsanity.com/paper/2510.25785/) (2025)
- [A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs](https://mlsanity.com/paper/2504.14657/) (2025)
- [ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling](https://mlsanity.com/paper/2504.07373/) (2025)
- [FEET: A Framework for Evaluating Embedding Techniques](https://mlsanity.com/paper/2411.01322/) (2024)
- [Serialized EHR make for good text representations](https://mlsanity.com/paper/2510.13843/) (2025)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Exploring Neural Ordinary Differential Equations as Interpretable Healthcare classifiers](https://mlsanity.com/paper/2503.03129/) (2025)
- [Temporal Entailment Pretraining for Clinical Language Models over EHR Data](https://mlsanity.com/paper/2504.18128/) (2025)
- [Learning Longitudinal Health Representations from EHR and Wearable Data](https://mlsanity.com/paper/2601.12227/) (2026)
- [Uncertainty-Aware Foundation Models for Clinical Data](https://mlsanity.com/paper/2604.04175/) (2026)
- [An Optimization Algorithm for Multimodal Data Alignment](https://mlsanity.com/paper/2503.07636/) (2025)
- [Dense Feature Learning via Linear Structure Preservation in Medical Data](https://mlsanity.com/paper/2602.07706/) (2026)
- [Event Fields: Learning Latent Event Structure for Waveform Foundation Models](https://mlsanity.com/paper/2605.08685/) (2026)
- [WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Records](https://mlsanity.com/paper/2605.09765/) (2026)
- [AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Models](https://mlsanity.com/paper/2605.17765/) (2026)

## Abstract (arXiv)

Recent research has explored how Language Models (LMs) can be used for feature representation and prediction in tabular machine learning tasks. This involves employing text serialization and supervised fine-tuning (SFT) techniques. Despite the simplicity of these techniques, significant gaps remain in our understanding of the applicability and reliability of LMs in this context. Our study assesses how emerging LM technologies compare with traditional paradigms in tabular machine learning and evaluates the feasibility of adopting similar approaches with these advanced technologies. At the data level, we investigate various methods of data representation and curation of serialized tabular data, exploring their impact on prediction performance. At the classification level, we examine whether text serialization combined with LMs enhances performance on tabular datasets (e.g. class imbalance, distribution shift, biases, and high dimensionality), and assess whether this method represents a state-of-the-art (SOTA) approach for addressing tabular machine learning challenges. Our findings reveal current pre-trained models should not replace conventional approaches.