# Stylized Structural Patterns for Improved Neural Network Pre-training

arXiv: https://arxiv.org/abs/2506.19465 · HTML: https://mlsanity.com/paper/2506.19465/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [Neural Style Transfer: A Review](https://mlsanity.com/paper/1705.04058/) (2017)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Pre-training without Natural Images](https://mlsanity.com/paper/2101.08515/) (2021)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models](https://mlsanity.com/paper/2407.02687/) (2024)
- [Can Vision Transformers Learn without Natural Images?](https://mlsanity.com/paper/2103.13023/) (2021)
- [Procedural Image Programs for Representation Learning](https://mlsanity.com/paper/2211.16412/) (2022)
- [Improving Fractal Pre-training](https://mlsanity.com/paper/2110.03091/) (2021)
- [Neural Neighbor Style Transfer](https://mlsanity.com/paper/2203.13215/) (2022)
- [Pre-training Vision Transformers with Formula-driven Supervised Learning](https://mlsanity.com/paper/2206.09132/) (2022)
- [SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning](https://mlsanity.com/paper/2309.17083/) (2023)
- [DIFF-NST: Diffusion Interleaving For deFormable Neural Style Transfer](https://mlsanity.com/paper/2307.04157/) (2023)

## Built on by (most cited first, 1 of 1)

- [MoireMix: A Formula-Based Data Augmentation for Improving Image Classification Robustness](https://mlsanity.com/paper/2603.25109/) (2026)

## Abstract (arXiv)

Modern deep learning models in computer vision require large datasets of real images, which are difficult to curate and pose privacy and legal concerns, limiting their commercial use. Recent works suggest synthetic data as an alternative, yet models trained with it often underperform. This paper proposes a two-step approach to bridge this gap. First, we propose an improved neural fractal formulation through which we introduce a new class of synthetic data. Second, we propose reverse stylization, a technique that transfers visual features from a small, license-free set of real images onto synthetic datasets, enhancing their effectiveness. We analyze the domain gap between our synthetic datasets and real images using Kernel Inception Distance (KID) and show that our method achieves a significantly lower distributional gap compared to existing synthetic datasets. Furthermore, our experiments across different tasks demonstrate the practical impact of this reduced gap. We show that pretraining the EDM2 diffusion model on our synthetic dataset leads to an 11% reduction in FID during image generation, compared to models trained on existing synthetic datasets, and a 20% decrease in autoencoder reconstruction error, indicating improved performance in data representation. Furthermore, a ViT-S model trained for classification on this synthetic data achieves over a 10% improvement in ImageNet-100 accuracy. Our work opens up exciting possibilities for training practical models when sufficiently large real training sets are not available.