# Robot Program Parameter Inference via Differentiable Shadow Program Inversion

arXiv: https://arxiv.org/abs/2103.14452 · HTML: https://mlsanity.com/paper/2103.14452/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Robotics](https://mlsanity.com/paper/1602.04450/) (2016)
- [Neural Programmer-Interpreters](https://mlsanity.com/paper/1511.06279/) (2015)
- [Abstract Syntax Networks for Code Generation and Semantic Parsing](https://mlsanity.com/paper/1704.07535/) (2017)
- [Neuro-Symbolic Program Synthesis](https://mlsanity.com/paper/1611.01855/) (2016)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Neural Random-Access Machines](https://mlsanity.com/paper/1511.06392/) (2015)
- [Programming with a Differentiable Forth Interpreter](https://mlsanity.com/paper/1605.06640/) (2016)
- [Differentiable Programs with Neural Libraries](https://mlsanity.com/paper/1611.02109/) (2016)
- [Differentiable Gaussian Process Motion Planning](https://mlsanity.com/paper/1907.09591/) (2019)
- [Differentiable Algorithm Networks for Composable Robot Learning](https://mlsanity.com/paper/1905.11602/) (2019)
- [Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video](https://mlsanity.com/paper/1910.09430/) (2019)
- [Time Reversal as Self-Supervision](https://mlsanity.com/paper/1810.01128/) (2018)

## Built on by (most cited first, 5 of 5)

- [Heuristic-free Optimization of Force-Controlled Robot Search Strategies in Stochastic Environments](https://mlsanity.com/paper/2207.07524/) (2022)
- [MuTT: A Multimodal Trajectory Transformer for Robot Skills](https://mlsanity.com/paper/2407.15660/) (2024)
- [Human-AI Interaction in Industrial Robotics: Design and Empirical Evaluation of a User Interface for Explainable AI-Based Robot Program Optimization](https://mlsanity.com/paper/2404.19349/) (2024)
- [Shadow Program Inversion with Differentiable Planning: A Framework for Unified Robot Program Parameter and Trajectory Optimization](https://mlsanity.com/paper/2409.08678/) (2024)
- [AI-based Framework for Robust Model-Based Connector Mating in Robotic Wire Harness Installation](https://mlsanity.com/paper/2503.09409/) (2025)

## Abstract (arXiv)

Challenging manipulation tasks can be solved effectively by combining individual robot skills, which must be parameterized for the concrete physical environment and task at hand. This is time-consuming and difficult for human programmers, particularly for force-controlled skills. To this end, we present Shadow Program Inversion (SPI), a novel approach to infer optimal skill parameters directly from data. SPI leverages unsupervised learning to train an auxiliary differentiable program representation ("shadow program") and realizes parameter inference via gradient-based model inversion. Our method enables the use of efficient first-order optimizers to infer optimal parameters for originally non-differentiable skills, including many skill variants currently used in production. SPI zero-shot generalizes across task objectives, meaning that shadow programs do not need to be retrained to infer parameters for different task variants. We evaluate our methods on three different robots and skill frameworks in industrial and household scenarios. Code and examples are available at https://innolab.artiminds.com/icra2021.