mlsanityData to 14 Oct 2026

Struct-Bench: A Benchmark for Differentially Private Structured Text Generation

arXiv 2509.10696 ↗First posted 12 Sep 2025cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Differentially private (DP) synthetic data generation is a promising technique for utilizing private datasets that otherwise cannot be exposed for model training or other analytics. While much research literature has focused on generating private unstructured text and image data, in enterprise settings, structured data (e.g., tabular) is more common, often including natural language fields or components. Existing synthetic data evaluation techniques (e.g., FID) struggle to capture the structural properties and correlations of such datasets. In this work, we propose Struct-Bench, a framework and benchmark for evaluating synthetic datasets derived from structured datasets that contain natural language data. The Struct-Bench framework requires users to provide a representation of their dataset structure as a Context-Free Grammar (CFG). Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with CFGs. We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods. Struct-Bench also includes reference implementations of different metrics and a leaderboard, thereby providing researchers a standardized evaluation platform to benchmark and investigate privacy-preserving synthetic data generation methods. Further, we also present a case study showing how to use Struct-Bench to improve the synthetic data quality of Private Evolution (PE) on structured data. The benchmark and the leaderboard have been publicly made available at https://struct-bench.github.io.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

41 of 64 references.

References · 41

  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  4. 2018Assessing Generative Models via Precision and Recallcited by 406
  5. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  6. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  7. 2020DP-CGAN: Differentially Private Synthetic Data and Label Generationcited by 118
  8. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  9. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  10. 2022AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Datacited by 90
  11. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 86
  12. 2021Benchmarking Differentially Private Synthetic Data Generation Algorithmscited by 81
29 more
  1. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76
  2. 2020Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learningcited by 67
  3. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 65
  4. 2022A Unified Framework for Quantifying Privacy Risk in Synthetic Datacited by 46
  5. 2021Formalizing and Estimating Distribution Inference Riskscited by 43
  6. 2023SoK: Privacy-Preserving Data Synthesiscited by 38
  7. 2020Differentially Private Synthetic Data: Applied Evaluations and Enhancementscited by 36
  8. 2023Synthetic data, real errors: how (not) to publish and use synthetic datacited by 29
  9. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  10. 2022TabSynDex: A Universal Metric for Robust Evaluation of Synthetic Tabular Datacited by 27
  11. 2021Conditional Synthetic Data Generation for Robust Machine Learning Applications with Limited Pandemic Datacited by 18
  12. 2024Differentially Private Tabular Data Synthesis using Large Language Modelscited by 16
  13. 2025Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMscited by 16
  14. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  15. 2025Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Modelcited by 13
  16. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domainscited by 12
  17. 2020Really Useful Synthetic Data -- A Framework to Evaluate the Quality of Differentially Private Synthetic Datacited by 11
  18. 2025Contrastive Private Data Synthesis via Weighted Multi-PLM Fusioncited by 11
  19. 2025DPImageBench: A Unified Benchmark for Differentially Private Image Synthesiscited by 11
  20. 2024A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Modelscited by 8
  21. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  22. 2024Structured Evaluation of Synthetic Tabular Datacited by 7
  23. 2023Summary Statistic Privacy in Data Sharingcited by 6
  24. 2024An evaluation framework for synthetic data generation modelscited by 5
  25. 2024DP-2Stage: Adapting Language Models as Differentially Private Tabular Data Generatorscited by 5
  26. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacycited by 5
  27. 2025Generating Synthetic Data with Formal Privacy Guarantees: State of the Art and the Road Aheadcited by 4
  28. 2025Evaluating Inter-Column Logical Relationships in Synthetic Tabular Data Generationcited by 3
  29. 2025Benchmarking Fraud Detectors on Private Graph Datacited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 2
  2. 2026Differentially Private Synthetic Data via APIs 4: Tabular Datacited by 2
  3. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuningcited by 0
  4. 2026MAPLE: Metadata Augmented Private Language Evolutioncited by 0
  5. 2026ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?cited by 0
  6. 2026QuanText: Protecting Dataset-Level Secrets in Textual Data Sharingcited by 0
  7. 2026PrivCert: Certifying Statement Support under Differential Privacycited by 0