# Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion

arXiv: https://arxiv.org/abs/2502.00245 · HTML: https://mlsanity.com/paper/2502.00245/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 16 of its 55 references that are in the map.
- First posted on arXiv 1 Feb 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 16 of 16)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [More Agents Is All You Need](https://mlsanity.com/paper/2402.05120/) (2024)
- [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024)
- [Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization](https://mlsanity.com/paper/2404.11531/) (2024)
- [Composition of Differential Privacy & Privacy Amplification by Subsampling](https://mlsanity.com/paper/2210.00597/) (2022)
- [C-ICL: Contrastive In-context Learning for Information Extraction](https://mlsanity.com/paper/2402.11254/) (2024)
- [Differentially Private Knowledge Distillation via Synthetic Text Generation](https://mlsanity.com/paper/2403.00932/) (2024)
- [Knowledge Fusion of Chat LLMs: A Preliminary Technical Report](https://mlsanity.com/paper/2402.16107/) (2024)
- [In-context Contrastive Learning for Event Causality Identification](https://mlsanity.com/paper/2405.10512/) (2024)
- [Multimodal Contrastive In-Context Learning](https://mlsanity.com/paper/2408.12959/) (2024)
- [Differentially Private Synthetic Data with Private Density Estimation](https://mlsanity.com/paper/2405.04554/) (2024)

## Built on by (most cited first, 11 of 11)

- [Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs](https://mlsanity.com/paper/2503.12347/) (2025)
- [POPri: Private Federated Learning using Preference-Optimized Synthetic Data](https://mlsanity.com/paper/2504.16438/) (2025)
- [Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Model](https://mlsanity.com/paper/2502.05505/) (2025)
- [Struct-Bench: A Benchmark for Differentially Private Structured Text Generation](https://mlsanity.com/paper/2509.10696/) (2025)
- [RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis](https://mlsanity.com/paper/2502.18517/) (2025)
- [Private Evolution Converges](https://mlsanity.com/paper/2506.08312/) (2025)
- [Differentially Private Synthetic Data via APIs 4: Tabular Data](https://mlsanity.com/paper/2606.08259/) (2026)
- [Secret-Protected Evolution for Differentially Private Synthetic Text Generation](https://mlsanity.com/paper/2510.10990/) (2025)
- [Federated Reasoning Distillation Framework with Model Learnability-Aware Data Allocation](https://mlsanity.com/paper/2602.18749/) (2026)
- [MAPLE: Metadata Augmented Private Language Evolution](https://mlsanity.com/paper/2603.19258/) (2026)
- [ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?](https://mlsanity.com/paper/2606.01849/) (2026)

## Abstract (arXiv)

Substantial quantity and high quality are the golden rules of making a good training dataset with sample privacy protection equally important. Generating synthetic samples that resemble high-quality private data while ensuring Differential Privacy (DP), a formal privacy guarantee, promises scalability and practicality. However, existing methods relying on pre-trained models for data synthesis %that avoid fine-tuning large pre-trained generative models often struggle in data-deficient scenarios, suffering from limited sample size, inevitable generation noise and existing pre-trained model bias. To address these challenges, we propose a novel contrAstive private data Synthesis via Weighted multiple Pre-trained language models (PLM) framework, named as WASP. WASP utilizes limited private samples for more accurate private data distribution estimation via a Top-Q voting mechanism, and leverages low-quality synthetic samples for contrastive generation via collaboration among dynamically weighted multiple pre-trained models.Extensive experiments on 6 well-developed datasets with 6 open-source and 3 closed-source PLMs demonstrate the superiority of WASP in improving model performance over diverse downstream tasks. Code is available at https://anonymous.4open.science/r/WASP.