mlsanityData to 7 Oct 2026

Learning to Infer Graphics Programs from Hand-Drawn Images

arXiv 1707.09627 ↗First posted 30 Jul 2017cs.AIcited by 73

Not on the map. Why?

Abstract (arXiv)

We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LaTeX. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network, measure similarity between drawings by use of similar high-level geometric structures, and extrapolate drawings. Taken together these results are a step towards agents that induce useful, human-readable programs from perceptual input.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 26; 2022: 13; 2023: 6; 2024: 11; 2025: 7; 2026: 9.

73 papers.

Sort:

By area:

  1. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  2. 2021Im2Vec: Synthesizing Vector Graphics without Vector Supervisioncited by 114
  3. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  4. 2021Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networkscited by 101
  5. 2021SketchGen: Generating Constrained CAD Sketchescited by 68
  6. 2021Vitruvion: A Generative Model of Parametric CAD Sketchescited by 66
  7. 2021Engineering Sketch Generation for Computer-Aided Designcited by 49
  8. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  9. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 44
  10. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  11. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 34
  12. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 32
38 more
  1. 2019Few-Shot Bayesian Imitation Learning with Logical Program Policiescited by 30
  2. 2022Backpropagation through Combinatorial Algorithms: Identity with Projection Workscited by 29
  3. 2023ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitivescited by 22
  4. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  5. 2022Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machinescited by 20
  6. 2026AutoFigure: Generating and Refining Publication-Ready Scientific Illustrationscited by 20
  7. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19
  8. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  9. 2021Hierarchical Motion Understanding via Motion Programscited by 14
  10. 2023Neurosymbolic Grounding for Compositional World Modelscited by 14
  11. 2024PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformerscited by 14
  12. 2021Neurosymbolic Transformers for Multi-Agent Communicationcited by 13
  13. 2022From Perception to Programs: Regularize, Overparameterize, and Amortizecited by 13
  14. 2022Self-explaining deep models with logic rule reasoningcited by 13
  15. 2024GRS: Generating Robotic Simulation Tasks from Real-World Imagescited by 13
  16. 2022Neurosymbolic Repair for Low-Code Formula Languagescited by 12
  17. 2022Neural-Symbolic Recursive Machine for Systematic Generalizationcited by 12
  18. 2026AutoFigure-Edit: Generating Editable Scientific Illustrationcited by 12
  19. 2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
  20. 2022Discovering Design Concepts for CAD Sketchescited by 10
  21. 2024Diffusion On Syntax Trees For Program Synthesiscited by 10
  22. 2025VLMaterial: Procedural Material Generation with Large Vision-Language Modelscited by 9
  23. 2021Unsupervised Learning of Neurosymbolic Encoderscited by 8
  24. 2021Program Synthesis Guided Reinforcement Learning for Partially Observed Environmentscited by 7
  25. 2021Transcribing Natural Languages for The Deaf via Neural Editing Programscited by 7
  26. 2023Compositional diversity in visual concept learningcited by 7
  27. 2025Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Modelscited by 7
  28. 2021A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semanticscited by 6
  29. 2021Using Program Synthesis and Inductive Logic Programming to solve Bongard Problemscited by 6
  30. 2021Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysiscited by 6
  31. 2022Geoclidean: Few-Shot Generalization in Euclidean Geometrycited by 6
  32. 2022Programmatic Concept Learning for Human Motion Description and Synthesiscited by 5
  33. 2024Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Imagescited by 5
  34. 2021ShapeMOD: Macro Operation Discovery for 3D Shape Programscited by 4
  35. 2021Extrapolation Frameworks in Cognitive Psychology Suitable for Study of Image Classification Modelscited by 4
  36. 2023ImageEye: Batch Image Processing Using Program Synthesiscited by 4
  37. 2024An Exploratory Study of ML Sketches and Visual Code Assistantscited by 3
  38. 2021Towards A Measure Of General Machine Intelligencecited by 2

50 most cited of 73 All 73 (JSON)