# A synthetic data approach for domain generalization of NLI models

arXiv: https://arxiv.org/abs/2402.12368 · HTML: https://mlsanity.com/paper/2402.12368/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Learning to Generate Novel Domains for Domain Generalization](https://mlsanity.com/paper/2007.03304/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [The Fact Extraction and VERification (FEVER) Shared Task](https://mlsanity.com/paper/1811.10971/) (2018)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models](https://mlsanity.com/paper/2305.11171/) (2023)
- [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://mlsanity.com/paper/2105.00071/) (2021)
- [DialFact: A Benchmark for Fact-Checking in Dialogue](https://mlsanity.com/paper/2110.08222/) (2021)
- [Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference](https://mlsanity.com/paper/1907.04380/) (2019)
- [Multi-component Image Translation for Deep Domain Generalization](https://mlsanity.com/paper/1812.08974/) (2018)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark](https://mlsanity.com/paper/1905.10425/) (2019)
- [QAmeleon: Multilingual QA with Only 5 Examples](https://mlsanity.com/paper/2211.08264/) (2022)
- [An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference](https://mlsanity.com/paper/2010.03777/) (2020)

## Built on by (most cited first, 15 of 15)

- [Entailed Between the Lines: Incorporating Implication into NLI](https://mlsanity.com/paper/2501.07719/) (2025)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [Scalable and Domain-General Abstractive Proposition Segmentation](https://mlsanity.com/paper/2406.19803/) (2024)
- [A Little Human Data Goes A Long Way](https://mlsanity.com/paper/2410.13098/) (2024)
- [VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI](https://mlsanity.com/paper/2508.00965/) (2025)
- [Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection](https://mlsanity.com/paper/2505.20209/) (2025)
- [FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking](https://mlsanity.com/paper/2504.16188/) (2025)
- [Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.03461/) (2024)
- [Bonafide at LegalLens 2024 Shared Task: Using Lightweight DeBERTa Based Encoder For Legal Violation Detection and Resolution](https://mlsanity.com/paper/2410.22977/) (2024)
- [Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis](https://mlsanity.com/paper/2504.03877/) (2025)
- [Explainable Semantic Text Relations: A Question-Answering Framework for Comparing Document Content](https://mlsanity.com/paper/2509.08304/) (2025)
- [Disambiguation of Emotion Annotations by Contextualizing Events in Plausible Narratives](https://mlsanity.com/paper/2508.09954/) (2025)
- [SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding](https://mlsanity.com/paper/2606.03284/) (2026)
- [Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation](https://mlsanity.com/paper/2608.18681/) (2026)

## Abstract (arXiv)

Natural Language Inference (NLI) remains an important benchmark task for LLMs. NLI datasets are a springboard for transfer learning to other semantic tasks, and NLI models are standard tools for identifying the faithfulness of model-generated text. There are several large scale NLI datasets today, and models have improved greatly by hill-climbing on these collections. Yet their realistic performance on out-of-distribution/domain data is less well-understood. We explore the opportunity for synthetic high-quality datasets to adapt NLI models for zero-shot use in downstream applications across new and unseen text domains. We demonstrate a new approach for generating NLI data in diverse domains and lengths, so far not covered by existing training sets. The resulting examples have meaningful premises, the hypotheses are formed in creative ways rather than simple edits to a few premise tokens, and the labels have high accuracy. We show that models trained on this data (685K synthetic examples) have the best generalization to completely new downstream test settings. On the TRUE benchmark, a T5-small model trained with our data improves around 7% on average compared to training on the best alternative dataset. The improvements are more pronounced for smaller models, while still meaningful on a T5 XXL model. We also demonstrate gains on test sets when in-domain training data is augmented with our domain-general synthetic data.