# Separating Knowledge and Perception with Procedural Data

arXiv: https://arxiv.org/abs/2508.11697 · HTML: https://mlsanity.com/paper/2508.11697/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 30 of 36)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning](https://mlsanity.com/paper/1803.04765/) (2018)
- [SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning](https://mlsanity.com/paper/1911.04623/) (2019)
- [Deep Nearest Neighbor Anomaly Detection](https://mlsanity.com/paper/2002.10445/) (2020)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [Pre-training without Natural Images](https://mlsanity.com/paper/2101.08515/) (2021)
- [Neural Nearest Neighbors Networks](https://mlsanity.com/paper/1810.12575/) (2018)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore](https://mlsanity.com/paper/2308.04430/) (2023)
- [Retrieval-Enhanced Contrastive Vision-Text Models](https://mlsanity.com/paper/2306.07196/) (2023)
- [Can Vision Transformers Learn without Natural Images?](https://mlsanity.com/paper/2103.13023/) (2021)
- [Procedural Image Programs for Representation Learning](https://mlsanity.com/paper/2211.16412/) (2022)
- [Explaining and Improving Model Behavior with k Nearest Neighbor Representations](https://mlsanity.com/paper/2010.09030/) (2020)
- [Improving Fractal Pre-training](https://mlsanity.com/paper/2110.03091/) (2021)
- [Revisiting a kNN-based Image Classification System with High-capacity Storage](https://mlsanity.com/paper/2204.01186/) (2022)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [Differentially Private Image Classification by Learning Priors from Random Processes](https://mlsanity.com/paper/2306.06076/) (2023)
- [Interpreting Neural Networks With Nearest Neighbors](https://mlsanity.com/paper/1809.02847/) (2018)

## Abstract (arXiv)

We train representation models with procedural data only, and apply them on visual similarity, classification, and semantic segmentation tasks without further training by using visual memory -- an explicit database of reference image embeddings. Unlike prior work on visual memory, our approach achieves full compartmentalization with respect to all real-world images while retaining strong performance. Compared to a model trained on Places, our procedural model performs within 1% on NIGHTS visual similarity, outperforms by 8% and 15% on CUB200 and Flowers102 fine-grained classification, and is within 10% on ImageNet-1K classification. It also demonstrates strong zero-shot segmentation, achieving an R2 on COCO within 10% of the models trained on real data. Finally, we analyze procedural versus real data models, showing that parts of the same object have dissimilar representations in procedural models, resulting in incorrect searches in memory and explaining the remaining performance gap.