# Designing Perceptual Puzzles by Differentiating Probabilistic Programs

arXiv: https://arxiv.org/abs/2204.12301 · HTML: https://mlsanity.com/paper/2204.12301/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 65 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [Improved Contrastive Divergence Training of Energy Based Models](https://mlsanity.com/paper/2012.01316/) (2020)
- [Adversarial Examples that Fool both Computer Vision and Time-Limited Humans](https://mlsanity.com/paper/1802.08195/) (2018)
- [An Introduction to Probabilistic Programming](https://mlsanity.com/paper/1809.10756/) (2018)
- [Differentiable Annealed Importance Sampling and the Perils of Gradient Noise](https://mlsanity.com/paper/2107.10211/) (2021)
- [Humans can decipher adversarial images](https://mlsanity.com/paper/1809.04120/) (2018)
- [Exponential Family Estimation via Adversarial Dynamics Embedding](https://mlsanity.com/paper/1904.12083/) (2019)
- [Why Do Line Drawings Work? A Realism Hypothesis](https://mlsanity.com/paper/2002.06260/) (2020)
- [3DP3: 3D Scene Perception via Probabilistic Programming](https://mlsanity.com/paper/2111.00312/) (2021)
- [A New Approach to Probabilistic Programming Inference](https://mlsanity.com/paper/1507.00996/) (2015)
- [Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs](https://mlsanity.com/paper/1307.0060/) (2013)
- [LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models](https://mlsanity.com/paper/1903.02482/) (2019)
- [Reducing Lateral Visual Biases in Displays](https://mlsanity.com/paper/1904.05614/) (2019)
- [Disagreeing about Crocs and socks: Creating profoundly ambiguous color displays](https://mlsanity.com/paper/1908.05736/) (2019)

## Built on by (most cited first, 12 of 12)

- [Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models](https://mlsanity.com/paper/2311.17919/) (2023)
- [Factorized Diffusion: Perceptual Illusions by Noise Decomposition](https://mlsanity.com/paper/2404.11615/) (2024)
- [LookingGlass: Generative Anamorphoses via Laplacian Pyramid Warping](https://mlsanity.com/paper/2504.08902/) (2025)
- [Differentiating Metropolis-Hastings to Optimize Intractable Densities](https://mlsanity.com/paper/2306.07961/) (2023)
- [Diffusion-based Visual Anagram as Multi-task Learning](https://mlsanity.com/paper/2412.02693/) (2024)
- [Acting as Inverse Inverse Planning](https://mlsanity.com/paper/2305.16913/) (2023)
- [Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics](https://mlsanity.com/paper/2308.05739/) (2023)
- [Illusion3D: 3D Multiview Illusion with 2D Diffusion Priors](https://mlsanity.com/paper/2412.09625/) (2024)
- [Inferring the Future by Imagining the Past](https://mlsanity.com/paper/2305.17195/) (2023)
- [Hand-Shadow Poser](https://mlsanity.com/paper/2505.07012/) (2025)
- [Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitation](https://mlsanity.com/paper/2409.13507/) (2024)
- [Meschers: Geometry Processing of Impossible Objects](https://mlsanity.com/paper/2605.14960/) (2026)

## Abstract (arXiv)

We design new visual illusions by finding "adversarial examples" for principled models of human perception -- specifically, for probabilistic models, which treat vision as Bayesian inference. To perform this search efficiently, we design a differentiable probabilistic programming language, whose API exposes MCMC inference as a first-class differentiable function. We demonstrate our method by automatically creating illusions for three features of human vision: color constancy, size constancy, and face perception.