# Synthetic Oversampling: Theory and A Practical Approach Using LLMs to Address Data Imbalance

arXiv: https://arxiv.org/abs/2406.03628 · HTML: https://mlsanity.com/paper/2406.03628/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2024.

## Built on by (most cited first, 17 of 17)

- [Dual Active Learning for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2410.02504/) (2024)
- [A Survey on Efficient Large Language Model Training: From Data-centric Perspectives](https://mlsanity.com/paper/2510.25817/) (2025)
- [Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback](https://mlsanity.com/paper/2512.03208/) (2025)
- [Shortcut Learning in In-Context Learning: A Survey](https://mlsanity.com/paper/2411.02018/) (2024)
- [A text-to-tabular approach to generate synthetic patient data using LLMs](https://mlsanity.com/paper/2412.05153/) (2024)
- [MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platform](https://mlsanity.com/paper/2506.00308/) (2025)
- [Filtering with Confidence: When Data Augmentation Meets Conformal Prediction](https://mlsanity.com/paper/2509.21479/) (2025)
- [A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellness](https://mlsanity.com/paper/2512.08936/) (2025)
- [Synthetic Augmentation in Imbalanced Learning: When It Helps, When It Hurts, and How Much to Add](https://mlsanity.com/paper/2601.16120/) (2026)
- [Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities](https://mlsanity.com/paper/2604.23904/) (2026)
- [A Technical Exploration of Causal Inference with Hybrid LLM Synthetic Data](https://mlsanity.com/paper/2511.00318/) (2025)
- [Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation](https://mlsanity.com/paper/2603.15134/) (2026)
- [Extrapolated Markov Chain Oversampling Method for Imbalanced Text Classification](https://mlsanity.com/paper/2509.02332/) (2025)
- [High-quality data augmentation for code comment classification](https://mlsanity.com/paper/2601.19383/) (2026)
- [Stance Detection in Prediction Markets: Addressing Imbalanced Trader Commentary via Counterfactual Augmentation and Market Context](https://mlsanity.com/paper/2605.28745/) (2026)
- [When Does Synthetic Data Augmentation Improve Score-Based Imbalanced Classification?](https://mlsanity.com/paper/2606.26053/) (2026)
- [Learning a Size-Weight Frontier for Synthetic-Augmented Inference](https://mlsanity.com/paper/2608.28576/) (2026)

## Abstract (arXiv)

Imbalanced classification and spurious correlation are common challenges in data science and machine learning. Both issues are linked to data imbalance, with certain groups of data samples significantly underrepresented, which in turn would compromise the accuracy, robustness and generalizability of the learned models. Recent advances have proposed leveraging the flexibility and generative capabilities of large language models (LLMs), typically built on transformer architectures, to generate synthetic samples and to augment the observed data. In the context of imbalanced data, LLMs are used to oversample underrepresented groups and have shown promising improvements. However, there is a clear lack of theoretical understanding of such synthetic data approaches. In this article, we develop novel theoretical foundations to systematically study the roles of synthetic samples in addressing imbalanced classification and spurious correlation. Specifically, we first explicitly quantify the benefits of synthetic oversampling. Next, we analyze the scaling dynamics in synthetic data augmentation, and derive the corresponding scaling law. Finally, we demonstrate the capacity of transformer models to generate high-quality synthetic samples. We further conduct extensive numerical experiments to validate the efficacy of the LLM-based synthetic oversampling and augmentation.