# Visual Reaction: Learning to Play Catch with Your Drone

arXiv: https://arxiv.org/abs/1912.02155 · HTML: https://mlsanity.com/paper/1912.02155/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2019.

## Built on by (most cited first, 6 of 6)

- [Pushing it out of the Way: Interactive Visual Navigation](https://mlsanity.com/paper/2104.14040/) (2021)
- [Deep KKL: Data-driven Output Prediction for Non-Linear Systems](https://mlsanity.com/paper/2103.12443/) (2021)
- [Visual Forecasting as a Mid-level Representation for Avoidance](https://mlsanity.com/paper/2310.07724/) (2023)
- [Modular Neural Network Policies for Learning In-Flight Object Catching with a Robot Hand-Arm System](https://mlsanity.com/paper/2312.13987/) (2023)
- [Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences](https://mlsanity.com/paper/2211.07459/) (2022)
- [Learning Socio-Temporal Graphs for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2312.14373/) (2023)

## Abstract (arXiv)

In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself. Visual reaction entails predicting the future changes in a visual environment and planning accordingly. We study the problem of visual reaction in the context of playing catch with a drone in visually rich synthetic environments. This is a challenging problem since the agent is required to learn (1) how objects with different physical properties and shapes move, (2) what sequence of actions should be taken according to the prediction, (3) how to adjust the actions based on the visual feedback from the dynamic environment (e.g., when objects bouncing off a wall), and (4) how to reason and act with an unexpected state change in a timely manner. We propose a new dataset for this task, which includes 30K throws of 20 types of objects in different directions with different forces. Our results show that our model that integrates a forecaster with a planner outperforms a set of strong baselines that are based on tracking as well as pure model-based and model-free RL baselines. The code and dataset are available at github.com/KuoHaoZeng/Visual_Reaction.