# Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data arXiv: https://arxiv.org/abs/2608.27996 ยท HTML: https://mlsanity.com/paper/2608.27996/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in Embodied and robot world models, one of 11 directions in World models. - Builds on 34 of its 42 references that are in the map. - First posted on arXiv 28 Aug 2026. ## Placed in - World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/) ## Builds on (most cited first, 30 of 34) - [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017) - [World Models](https://mlsanity.com/paper/1803.10122/) (2018) - [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017) - [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023) - [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024) - [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024) - [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019) - [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021) - [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022) - [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023) - [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020) - [Sionna: An Open-Source Library for Next-Generation Physical Layer Research](https://mlsanity.com/paper/2203.11854/) (2022) - [Hypothesis testing with e-values](https://mlsanity.com/paper/2410.23614/) (2024) - [Benchmarking Reinforcement Learning Algorithms on Real-World Robots](https://mlsanity.com/paper/1809.07731/) (2018) - [Sionna RT: Differentiable Ray Tracing for Radio Propagation Modeling](https://mlsanity.com/paper/2303.11103/) (2023) - [Learning Radio Environments by Differentiable Ray Tracing](https://mlsanity.com/paper/2311.18558/) (2023) - [Wireless Network Digital Twin for 6G: Generative AI as A Key Enabler](https://mlsanity.com/paper/2311.17451/) (2023) - [A Bayesian Framework for Digital Twin-Based Control, Monitoring, and Data Collection in Wireless Systems](https://mlsanity.com/paper/2212.01351/) (2022) - [A Scalable and Generalizable Pathloss Map Prediction](https://mlsanity.com/paper/2312.03950/) (2023) - [Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection](https://mlsanity.com/paper/2409.15844/) (2024) - [Calibrating Wireless Ray Tracing for Digital Twinning using Local Phase Error Estimates](https://mlsanity.com/paper/2312.12625/) (2023) - [Semi-Supervised Learning via Cross-Prediction-Powered Inference for Wireless Systems](https://mlsanity.com/paper/2405.15415/) (2024) - [Colosseum: The Open RAN Digital Twin](https://mlsanity.com/paper/2404.17317/) (2024) - [Optimal and Fast Real-time Resources Slicing with Deep Dueling Neural Networks](https://mlsanity.com/paper/1902.09696/) (2019) - [How to Bridge the Sim-to-Real Gap in Digital Twin-Aided Telecommunication Networks](https://mlsanity.com/paper/2507.07067/) (2025) - [Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network](https://mlsanity.com/paper/2310.09299/) (2023) - [Sequential Monte-Carlo testing by betting](https://mlsanity.com/paper/2401.07365/) (2024) - [Large Vision Model-Enhanced Digital Twin with Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks](https://mlsanity.com/paper/2410.07611/) (2024) - [Sub-channel Assignment, Power Allocation and User Scheduling for Non-Orthogonal Multiple Access Networks](https://mlsanity.com/paper/1608.08313/) (2016) - [Context-Aware Doubly-Robust Semi-Supervised Learning](https://mlsanity.com/paper/2502.15577/) (2025) ## Abstract (arXiv) Digital twins (DTs) and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems. Owing to the inevitable simulation-to-reality (sim-to-real) gap, however, augmentation may fail to improve the performance of the trained model on the real data distribution. This paper addresses the resulting decision problem: Given a real dataset, a candidate synthetic dataset, and a fixed learning algorithm, decide whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible. Two formulations are considered: a direct test on the mean loss difference between the two trained models, and a symmetry-based test on the paired loss difference, which trades a stronger null assumption for faster evidence accumulation. For the latter, we introduce the adaptive e-process sign-flip test (aeSFT), a doubly adaptive procedure that adapts both the number of Monte Carlo sign-flip rounds, and hence the computational cost, and the amount of real test data consumed. aeSFT yields anytime-valid Type-I error control, with no need to pre-specify the test-set size. Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing, matches the power of fixed-sample sign-flip testing and the paired t-test, while keeping the false-positive rate below the target level.