# An "On The Fly" Framework for Efficiently Generating Synthetic Big Data Sets

arXiv: https://arxiv.org/abs/1903.06798 · HTML: https://mlsanity.com/paper/1903.06798/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2019.

## Built on by (most cited first, 2 of 2)

- [Reducing the Amount of Real World Data for Object Detector Training with Synthetic Data](https://mlsanity.com/paper/2202.00632/) (2022)
- [PGCS: Physical Law embedded Generative Cloud Synthesis in Remote Sensing Images](https://mlsanity.com/paper/2410.16955/) (2024)

## Abstract (arXiv)

Collecting, analyzing and gaining insight from large volumes of data is now the norm in an ever increasing number of industries. Data analytics techniques, such as machine learning, are powerful tools used to analyze these large volumes of data. Synthetic data sets are routinely relied upon to train and develop such data analytics methods for several reasons: to generate larger data sets than are available, to generate diverse data sets, to preserve anonymity in data sets with sensitive information, etc. Processing, transmitting and storing data is a key issue faced when handling large data sets. This paper presents an "On the fly" framework for generating big synthetic data sets, suitable for these data analytics methods, that is both computationally efficient and applicable to a diverse set of problems. An example application of the proposed framework is presented along with a mathematical analysis of its computational efficiency, demonstrating its effectiveness.