# Generative Kaleidoscopic Networks

arXiv: https://arxiv.org/abs/2402.11793 · HTML: https://mlsanity.com/paper/2402.11793/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Adversarial Attacks and Defenses in Images, Graphs and Text: A Review](https://mlsanity.com/paper/1909.08072/) (2019)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [A Review on Deep Learning Techniques for Video Prediction](https://mlsanity.com/paper/2004.05214/) (2020)
- [The Kolmogorov-Arnold representation theorem revisited](https://mlsanity.com/paper/2007.15884/) (2020)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [The Dimpled Manifold Model of Adversarial Examples in Machine Learning](https://mlsanity.com/paper/2106.10151/) (2021)
- [Diverse Video Generation using a Gaussian Process Trigger](https://mlsanity.com/paper/2107.04619/) (2021)
- [Neural Graphical Models](https://mlsanity.com/paper/2210.00453/) (2022)
- [Methods for Recovering Conditional Independence Graphs: A Survey](https://mlsanity.com/paper/2211.06829/) (2022)
- [Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements](https://mlsanity.com/paper/2312.07835/) (2023)
- [Federated Learning with Neural Graphical Models](https://mlsanity.com/paper/2309.11680/) (2023)

## Abstract (arXiv)

We discovered that the neural networks, especially the deep ReLU networks, demonstrate an `over-generalization' phenomenon. That is, the output values for the inputs that were not seen during training are mapped close to the output range that were observed during the learning process. In other words, the neural networks learn a many-to-one mapping and this effect is more prominent as we increase the number of layers or the depth of the neural network. We utilize this property of neural networks to design a dataset kaleidoscope, termed as `Generative Kaleidoscopic Networks'. Succinctly, if we learn a model to map from input x∈ℝD to itself fN(x)rightarrow x, the proposed `Kaleidoscopic sampling' procedure starts with a random input noise z∈ℝD and recursively applies fN(cdots fN(z)cdots ). After a burn-in period duration, we start observing samples from the input distribution and the quality of samples recovered improves as we increase the depth of the model. Scope: We observed this phenomenon to various degrees for the other deep learning architectures like CNNs, Transformers & U-Nets and we are currently investigating them further.