# Learning to Edit Visual Programs with Self-Supervision

arXiv: https://arxiv.org/abs/2406.02383 · HTML: https://mlsanity.com/paper/2406.02383/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023)
- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks](https://mlsanity.com/paper/2207.04632/) (2022)
- [CSGNet: Neural Shape Parser for Constructive Solid Geometry](https://mlsanity.com/paper/1712.08290/) (2017)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [Write, Execute, Assess: Program Synthesis with a REPL](https://mlsanity.com/paper/1906.04604/) (2019)
- [UCSG-Net -- Unsupervised Discovering of Constructive Solid Geometry Tree](https://mlsanity.com/paper/2006.09102/) (2020)
- [Learning to Infer and Execute 3D Shape Programs](https://mlsanity.com/paper/1901.02875/) (2019)
- [CAPRI-Net: Learning Compact CAD Shapes with Adaptive Primitive Assembly](https://mlsanity.com/paper/2104.05652/) (2021)
- [CSG-Stump: A Learning Friendly CSG-Like Representation for Interpretable Shape Parsing](https://mlsanity.com/paper/2108.11305/) (2021)
- [Inferring CAD Modeling Sequences Using Zone Graphs](https://mlsanity.com/paper/2104.03900/) (2021)
- [D2CSG: Unsupervised Learning of Compact CSG Trees with Dual Complements and Dropouts](https://mlsanity.com/paper/2301.11497/) (2023)
- [Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis](https://mlsanity.com/paper/2007.08095/) (2020)
- [Neurosymbolic Models for Computer Graphics](https://mlsanity.com/paper/2304.10320/) (2023)
- [Automatic Program Synthesis of Long Programs with a Learned Garbage Collector](https://mlsanity.com/paper/1809.04682/) (2018)
- [PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributions](https://mlsanity.com/paper/2011.13045/) (2020)
- [Learning to learn generative programs with Memoised Wake-Sleep](https://mlsanity.com/paper/2007.03132/) (2020)
- [Representing Partial Programs with Blended Abstract Semantics](https://mlsanity.com/paper/2012.12964/) (2020)
- [From Perception to Programs: Regularize, Overparameterize, and Amortize](https://mlsanity.com/paper/2206.05922/) (2022)
- [Improving Unsupervised Visual Program Inference with Code Rewriting Families](https://mlsanity.com/paper/2309.14972/) (2023)
- [Neural Program Synthesis with a Differentiable Fixer](https://mlsanity.com/paper/2006.10924/) (2020)
- [Learning to Infer Generative Template Programs for Visual Concepts](https://mlsanity.com/paper/2403.15476/) (2024)

## Built on by (most cited first, 7 of 7)

- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [CADReasoner: Iterative Program Editing for CAD Reverse Engineering](https://mlsanity.com/paper/2603.29847/) (2026)
- [ShapeLib: Designing a library of programmatic 3D shape abstractions with Large Language Models](https://mlsanity.com/paper/2502.08884/) (2025)
- [Residual Primitive Fitting of 3D Shapes with SuperFrusta](https://mlsanity.com/paper/2512.09201/) (2025)
- [Learning to Place Objects with Programs and Iterative Self Training](https://mlsanity.com/paper/2503.04496/) (2025)
- [PLLM: Pseudo-Labeling Large Language Models for CAD Program Synthesis](https://mlsanity.com/paper/2602.12561/) (2026)
- [Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy](https://mlsanity.com/paper/2412.12463/) (2024)

## Abstract (arXiv)

We design a system that learns how to edit visual programs. Our edit network consumes a complete input program and a visual target. From this input, we task our network with predicting a local edit operation that could be applied to the input program to improve its similarity to the target. In order to apply this scheme for domains that lack program annotations, we develop a self-supervised learning approach that integrates this edit network into a bootstrapped finetuning loop along with a network that predicts entire programs in one-shot. Our joint finetuning scheme, when coupled with an inference procedure that initializes a population from the one-shot model and evolves members of this population with the edit network, helps to infer more accurate visual programs. Over multiple domains, we experimentally compare our method against the alternative of using only the one-shot model, and find that even under equal search-time budgets, our editing-based paradigm provides significant advantages.