# Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks

arXiv: https://arxiv.org/abs/1911.06636 · HTML: https://mlsanity.com/paper/1911.06636/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2019.

## Built on by (most cited first, 4 of 4)

- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [DASH: Modularized Human Manipulation Simulation with Vision and Language for Embodied AI](https://mlsanity.com/paper/2108.12536/) (2021)
- [A Simple Approach to Continual Learning by Transferring Skill Parameters](https://mlsanity.com/paper/2110.10255/) (2021)
- [Deep Dive into Model-free Reinforcement Learning for Biological and Robotic Systems: Theory and Practice](https://mlsanity.com/paper/2405.11457/) (2024)

## Abstract (arXiv)

We address the longstanding challenge of producing flexible, realistic humanoid character controllers that can perform diverse whole-body tasks involving object interactions. This challenge is central to a variety of fields, from graphics and animation to robotics and motor neuroscience. Our physics-based environment uses realistic actuation and first-person perception -- including touch sensors and egocentric vision -- with a view to producing active-sensing behaviors (e.g. gaze direction), transferability to real robots, and comparisons to the biology. We develop an integrated neural-network based approach consisting of a motor primitive module, human demonstrations, and an instructed reinforcement learning regime with curricula and task variations. We demonstrate the utility of our approach for several tasks, including goal-conditioned box carrying and ball catching, and we characterize its behavioral robustness. The resulting controllers can be deployed in real-time on a standard PC. See overview video, https://youtu.be/2rQAW-8gQQk .