# Faking feature importance: A cautionary tale on the use of differentially-private synthetic data

arXiv: https://arxiv.org/abs/2203.01363 · HTML: https://mlsanity.com/paper/2203.01363/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 51 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Reconstruction and Membership Inference Attacks against Generative Models](https://mlsanity.com/paper/1906.03006/) (2019)
- [Differentially Private Synthetic Data: Applied Evaluations and Enhancements](https://mlsanity.com/paper/2011.05537/) (2020)
- [One button machine for automating feature engineering in relational databases](https://mlsanity.com/paper/1706.00327/) (2017)
- [Airline Passenger Name Record Generation using Generative Adversarial Networks](https://mlsanity.com/paper/1807.06657/) (2018)
- [Data Readiness Levels](https://mlsanity.com/paper/1705.02245/) (2017)
- [Foundations of Bayesian Learning from Synthetic Data](https://mlsanity.com/paper/2011.08299/) (2020)

## Built on by (most cited first, 8 of 8)

- [SoK: Privacy-Preserving Data Synthesis](https://mlsanity.com/paper/2307.02106/) (2023)
- [Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective](https://mlsanity.com/paper/2410.01720/) (2024)
- [On the Utility Recovery Incapability of Neural Net-based Differential Private Tabular Training Data Synthesizer under Privacy Deregulation](https://mlsanity.com/paper/2211.15809/) (2022)
- [Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Data](https://mlsanity.com/paper/2310.19250/) (2023)
- [Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?](https://mlsanity.com/paper/2403.13612/) (2024)
- [The Berkelmans-Pries Feature Importance Method: A Generic Measure of Informativeness of Features](https://mlsanity.com/paper/2301.04740/) (2023)
- [Generating High-quality Privacy-preserving Synthetic Data](https://mlsanity.com/paper/2602.06390/) (2026)
- [K-IPO: Kendall-constrained Importance Preserving Oversampling for Imbalanced Tabular Data](https://mlsanity.com/paper/2607.16478/) (2026)

## Abstract (arXiv)

Synthetic datasets are often presented as a silver-bullet solution to the problem of privacy-preserving data publishing. However, for many applications, synthetic data has been shown to have limited utility when used to train predictive models. One promising potential application of these data is in the exploratory phase of the machine learning workflow, which involves understanding, engineering and selecting features. This phase often involves considerable time, and depends on the availability of data. There would be substantial value in synthetic data that permitted these steps to be carried out while, for example, data access was being negotiated, or with fewer information governance restrictions. This paper presents an empirical analysis of the agreement between the feature importance obtained from raw and from synthetic data, on a range of artificially generated and real-world datasets (where feature importance represents how useful each feature is when predicting a the outcome). We employ two differentially-private methods to produce synthetic data, and apply various utility measures to quantify the agreement in feature importance as this varies with the level of privacy. Our results indicate that synthetic data can sometimes preserve several representations of the ranking of feature importance in simple settings but their performance is not consistent and depends upon a number of factors. Particular caution should be exercised in more nuanced real-world settings, where synthetic data can lead to differences in ranked feature importance that could alter key modelling decisions. This work has important implications for developing synthetic versions of highly sensitive data sets in fields such as finance and healthcare.