# Learning to Infer Graphics Programs from Hand-Drawn Images

arXiv: https://arxiv.org/abs/1707.09627 · HTML: https://mlsanity.com/paper/1707.09627/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2017.

## Built on by (most cited first, 30 of 74)

- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [SketchGen: Generating Constrained CAD Sketches](https://mlsanity.com/paper/2106.02711/) (2021)
- [Vitruvion: A Generative Model of Parametric CAD Sketches](https://mlsanity.com/paper/2109.14124/) (2021)
- [Engineering Sketch Generation for Computer-Aided Design](https://mlsanity.com/paper/2104.09621/) (2021)
- [LILO: Learning Interpretable Libraries by Compressing and Documenting Code](https://mlsanity.com/paper/2310.19791/) (2023)
- [Leveraging Language to Learn Program Abstractions and Search Heuristics](https://mlsanity.com/paper/2106.11053/) (2021)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [Can Large Language Models Understand Symbolic Graphics Programs?](https://mlsanity.com/paper/2408.08313/) (2024)
- [Few-Shot Bayesian Imitation Learning with Logical Program Policies](https://mlsanity.com/paper/1904.06317/) (2019)
- [Backpropagation through Combinatorial Algorithms: Identity with Projection Works](https://mlsanity.com/paper/2205.15213/) (2022)
- [TikZero: Zero-Shot Text-Guided Graphics Program Synthesis](https://mlsanity.com/paper/2503.11509/) (2025)
- [ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives](https://mlsanity.com/paper/2305.05661/) (2023)
- [AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations](https://mlsanity.com/paper/2602.03828/) (2026)
- [ProTo: Program-Guided Transformer for Program-Guided Tasks](https://mlsanity.com/paper/2110.00804/) (2021)
- [Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines](https://mlsanity.com/paper/2205.11558/) (2022)
- [IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering](https://mlsanity.com/paper/2506.23329/) (2025)
- [PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers](https://mlsanity.com/paper/2407.13677/) (2024)
- [Hierarchical Motion Understanding via Motion Programs](https://mlsanity.com/paper/2104.11216/) (2021)
- [Neurosymbolic Grounding for Compositional World Models](https://mlsanity.com/paper/2310.12690/) (2023)
- [AutoFigure-Edit: Generating Editable Scientific Illustration](https://mlsanity.com/paper/2603.06674/) (2026)
- [Neurosymbolic Transformers for Multi-Agent Communication](https://mlsanity.com/paper/2101.03238/) (2021)
- [From Perception to Programs: Regularize, Overparameterize, and Amortize](https://mlsanity.com/paper/2206.05922/) (2022)
- [Self-explaining deep models with logic rule reasoning](https://mlsanity.com/paper/2210.07024/) (2022)
- [GRS: Generating Robotic Simulation Tasks from Real-World Images](https://mlsanity.com/paper/2410.15536/) (2024)
- [Neurosymbolic Repair for Low-Code Formula Languages](https://mlsanity.com/paper/2207.11765/) (2022)
- [Neural-Symbolic Recursive Machine for Systematic Generalization](https://mlsanity.com/paper/2210.01603/) (2022)

## Abstract (arXiv)

We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LaTeX. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network, measure similarity between drawings by use of similar high-level geometric structures, and extrapolate drawings. Taken together these results are a step towards agents that induce useful, human-readable programs from perceptual input.