mlsanityData to 14 Oct 2026

Fill In The Gaps: Model Calibration and Generalization with Synthetic Data

arXiv 2410.10864 ↗First posted 7 Oct 2024cs.CL cs.AI cs.LGcited by 8

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

As machine learning models continue to swiftly advance, calibrating their performance has become a major concern prior to practical and widespread implementation. Most existing calibration methods often negatively impact model accuracy due to the lack of diversity of validation data, resulting in reduced generalizability. To address this, we propose a calibration method that incorporates synthetic data without compromising accuracy. We derive the expected calibration error (ECE) bound using the Probably Approximately Correct (PAC) learning framework. Large language models (LLMs), known for their ability to mimic real data and generate text with mixed class labels, are utilized as a synthetic data generation strategy to lower the ECE bound and improve model accuracy on real test data. Additionally, we propose data generation mechanisms for efficient calibration. Testing our method on four different natural language processing tasks, we observed an average up to 34% increase in accuracy and 33% decrease in ECE.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 23 of its 35 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

23 of 35 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2019Measuring Calibration in Deep Learningcited by 446
  10. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  12. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
11 more
  1. 2020Deep Entity Matching with Pre-Trained Language Modelscited by 144
  2. 2022DDXPlus: A New Dataset For Automatic Medical Diagnosiscited by 88
  3. 2019Pretrained Language Models for Sequential Sentence Classificationcited by 53
  4. 2022Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learningcited by 48
  5. 2020Combining Ensembles and Data Augmentation can Harm your Calibrationcited by 44
  6. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 32
  7. 2023Synthetic data, real errors: how (not) to publish and use synthetic datacited by 29
  8. 2019Automatically Identifying Complaints in Social Mediacited by 25
  9. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25
  10. 2021FairCal: Fairness Calibration for Face Verificationcited by 17
  11. 2022Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcarecited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labelingcited by 8
  2. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  3. 2025Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMscited by 2
  4. 2025Modeling Contextual Passage Utility for Multihop Question Answeringcited by 2
  5. 2025A Hybrid Machine Learning Approach for Synthetic Data Generation with Post Hoc Calibration for Clinical Tabular Datasetscited by 0
  6. 2025Predict Training Data Quality via Its Geometry in Metric Spacecited by 0
  7. 2026An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibrationcited by 0
  8. 2026Training-Aware Target Coverage for Synthetic Data Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  2. 2024Calibrating Long-form Generations from Large Language Models
  3. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  4. 2023On the Calibration of Large Language Models and Alignment
  5. 2024Efficacy of Synthetic Data as a Benchmark
  6. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  7. 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
  8. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code