# Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs

arXiv: https://arxiv.org/abs/1812.02788 · HTML: https://mlsanity.com/paper/1812.02788/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2018.

## Built on by (most cited first, 21 of 21)

- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [Learning to Synthesize Programs as Interpretable and Generalizable Policies](https://mlsanity.com/paper/2108.13643/) (2021)
- [Top-Down Synthesis for Library Learning](https://mlsanity.com/paper/2211.16605/) (2022)
- [LILO: Learning Interpretable Libraries by Compressing and Documenting Code](https://mlsanity.com/paper/2310.19791/) (2023)
- [Leveraging Language to Learn Program Abstractions and Search Heuristics](https://mlsanity.com/paper/2106.11053/) (2021)
- [Few-Shot Bayesian Imitation Learning with Logical Program Policies](https://mlsanity.com/paper/1904.06317/) (2019)
- [Abstraction for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.05839/) (2022)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [Learning Neuro-symbolic Programs for Language Guided Robot Manipulation](https://mlsanity.com/paper/2211.06652/) (2022)
- [From explanation to synthesis: Compositional program induction for learning from demonstration](https://mlsanity.com/paper/1902.10657/) (2019)
- [Hierarchical Neural Program Synthesis](https://mlsanity.com/paper/2303.06018/) (2023)
- [Motion Reasoning for Goal-Based Imitation Learning](https://mlsanity.com/paper/1911.05864/) (2019)
- [InfoCon: Concept Discovery with Generative and Discriminative Informativeness](https://mlsanity.com/paper/2404.10606/) (2024)
- [Bayesian Program Learning by Decompiling Amortized Knowledge](https://mlsanity.com/paper/2306.07856/) (2023)
- [On the Control of Attentional Processes in Vision](https://mlsanity.com/paper/2101.01533/) (2021)
- [The Construction of Reality in an AI: A Review](https://mlsanity.com/paper/2302.05448/) (2023)
- [Active Observer Visual Problem-Solving Methods are Dynamically Hypothesized, Deployed and Tested](https://mlsanity.com/paper/2108.08145/) (2021)
- [Socially Cognizant Robotics for a Technology Enhanced Society](https://mlsanity.com/paper/2310.18303/) (2023)
- [Guided Sketch-Based Program Induction by Search Gradients](https://mlsanity.com/paper/2402.06990/) (2024)
- [Disassociating performance from compositional feature learning](https://mlsanity.com/paper/2505.09716/) (2025)
- [Towards a universal language of concepts: A survey](https://mlsanity.com/paper/2609.04528/) (2026)

## Abstract (arXiv)

Humans can infer concepts from image pairs and apply those in the physical world in a completely different setting, enabling tasks like IKEA assembly from diagrams. If robots could represent and infer high-level concepts, it would significantly improve their ability to understand our intent and to transfer tasks between different environments. To that end, we introduce a computational framework that replicates aspects of human concept learning. Concepts are represented as programs on a novel computer architecture consisting of a visual perception system, working memory, and action controller. The instruction set of this "cognitive computer" has commands for parsing a visual scene, directing gaze and attention, imagining new objects, manipulating the contents of a visual working memory, and controlling arm movement. Inferring a concept corresponds to inducing a program that can transform the input to the output. Some concepts require the use of imagination and recursion. Previously learned concepts simplify the learning of subsequent more elaborate concepts, and create a hierarchy of abstractions. We demonstrate how a robot can use these abstractions to interpret novel concepts presented to it as schematic images, and then apply those concepts in dramatically different situations. By bringing cognitive science ideas on mental imagery, perceptual symbols, embodied cognition, and deictic mechanisms into the realm of machine learning, our work brings us closer to the goal of building robots that have interpretable representations and commonsense.