# A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models

arXiv: https://arxiv.org/abs/2405.17712 · HTML: https://mlsanity.com/paper/2405.17712/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 77 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [GAIN: Missing Data Imputation using Generative Adversarial Nets](https://mlsanity.com/paper/1806.02920/) (2018)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Handling Missing Data with Graph Representation Learning](https://mlsanity.com/paper/2010.16418/) (2020)
- [Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares](https://mlsanity.com/paper/1410.2596/) (2014)
- [Missing Data Imputation using Optimal Transport](https://mlsanity.com/paper/2002.03860/) (2020)
- [HyperImpute: Generalized Iterative Imputation with Automatic Model Selection](https://mlsanity.com/paper/2206.07769/) (2022)
- [MisGAN: Learning from Incomplete Data with Generative Adversarial Networks](https://mlsanity.com/paper/1902.09599/) (2019)
- [Variational Autoencoder with Arbitrary Conditioning](https://mlsanity.com/paper/1806.02382/) (2018)
- [Graphical Models for Processing Missing Data](https://mlsanity.com/paper/1801.03583/) (2018)
- [MCFlow: Monte Carlo Flow Models for Data Imputation](https://mlsanity.com/paper/2003.12628/) (2020)
- [TabMT: Generating tabular data with masked transformers](https://mlsanity.com/paper/2312.06089/) (2023)
- [Transformed Distribution Matching for Missing Value Imputation](https://mlsanity.com/paper/2302.10363/) (2023)
- [Personalized Risk Scoring for Critical Care Prognosis using Mixtures of Gaussian Processes](https://mlsanity.com/paper/1610.08853/) (2016)
- [Improving Missing Data Imputation with Deep Generative Models](https://mlsanity.com/paper/1902.10666/) (2019)
- [Simultaneous Missing Value Imputation and Structure Learning with Groups](https://mlsanity.com/paper/2110.08223/) (2021)
- [Multiple Imputation via Generative Adversarial Network for High-dimensional Blockwise Missing Value Problems](https://mlsanity.com/paper/2112.11507/) (2021)
- [Handling Missing Data via Max-Entropy Regularized Graph Autoencoder](https://mlsanity.com/paper/2211.16771/) (2022)
- [Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretraining](https://mlsanity.com/paper/2403.20208/) (2024)
- [EMFlow: Data Imputation in Latent Space via EM and Deep Flow Models](https://mlsanity.com/paper/2106.04804/) (2021)
- [FragmGAN: Generative Adversarial Nets for Fragmentary Data Imputation and Prediction](https://mlsanity.com/paper/2203.04692/) (2022)

## Built on by (most cited first, 5 of 5)

- [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025)
- [Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs](https://mlsanity.com/paper/2601.17058/) (2026)
- [Large Language Models for Missing Data Imputation: Understanding Behavior, Hallucination Effects, and Control Mechanisms](https://mlsanity.com/paper/2603.22332/) (2026)
- [Precision Adaptive Imputation Network : An Unified Technique for Mixed Datasets](https://mlsanity.com/paper/2501.10667/) (2025)
- [An Interdisciplinary and Cross-Task Review on Missing Data Imputation](https://mlsanity.com/paper/2511.01196/) (2025)

## Abstract (arXiv)

This paper presents a novel approach named Contextually Relevant Imputation leveraging pre-trained Language Models (CRILM) for handling missing data in tabular datasets. Instead of relying on traditional numerical estimations, CRILM uses pre-trained language models (LMs) to create contextually relevant descriptors for missing values. This method aligns datasets with LMs' strengths, allowing large LMs to generate these descriptors and small LMs to be fine-tuned on the enriched datasets for enhanced downstream task performance. Our evaluations demonstrate CRILM's superior performance and robustness across MCAR, MAR, and challenging MNAR scenarios, with up to a 10% improvement over the best-performing baselines. By mitigating biases, particularly in MNAR settings, CRILM improves downstream task performance and offers a cost-effective solution for resource-constrained environments.