# Towards Active Vision for Action Localization with Reactive Control and Predictive Learning

arXiv: https://arxiv.org/abs/2111.05448 · HTML: https://mlsanity.com/paper/2111.05448/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 9 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [An Exploration of Embodied Visual Exploration](https://mlsanity.com/paper/2001.02192/) (2020)
- [Finding Action Tubes](https://mlsanity.com/paper/1411.6031/) (2014)
- [Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos](https://mlsanity.com/paper/1703.10664/) (2017)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [VideoCapsuleNet: A Simplified Network for Action Detection](https://mlsanity.com/paper/1805.08162/) (2018)
- [End-to-end Active Object Tracking via Reinforcement Learning](https://mlsanity.com/paper/1705.10561/) (2017)
- [Pose-Assisted Multi-Camera Collaboration for Active Object Tracking](https://mlsanity.com/paper/2001.05161/) (2020)
- [Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks](https://mlsanity.com/paper/1612.09401/) (2016)
- [Action Localization through Continual Predictive Learning](https://mlsanity.com/paper/2003.12185/) (2020)
- [Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling](https://mlsanity.com/paper/2001.11580/) (2020)
- [Tubelets: Unsupervised action proposals from spatiotemporal super-voxels](https://mlsanity.com/paper/1607.02003/) (2016)
- [Tracking Randomly Moving Objects on Edge Box Proposals](https://mlsanity.com/paper/1507.08085/) (2015)
- [Correlation Filter Selection for Visual Tracking Using Reinforcement Learning](https://mlsanity.com/paper/1811.03196/) (2018)
- [Imitation-Based Active Camera Control with Deep Convolutional Neural Network](https://mlsanity.com/paper/2012.06428/) (2020)

## Built on by (most cited first, 5 of 5)

- [Capturing Temporal Components for Time Series Classification](https://mlsanity.com/paper/2406.14456/) (2024)
- [Self-supervised Multi-actor Social Activity Understanding in Streaming Videos](https://mlsanity.com/paper/2406.14472/) (2024)
- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [Leveraging Event Streams with Deep Reinforcement Learning for End-to-End UAV Tracking](https://mlsanity.com/paper/2410.14685/) (2024)
- [Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learning](https://mlsanity.com/paper/2512.04219/) (2025)

## Abstract (arXiv)

Visual event perception tasks such as action localization have primarily focused on supervised learning settings under a static observer, i.e., the camera is static and cannot be controlled by an algorithm. They are often restricted by the quality, quantity, and diversity of textitannotated training data and do not often generalize to out-of-domain samples. In this work, we tackle the problem of active action localization where the goal is to localize an action while controlling the geometric and physical parameters of an active camera to keep the action in the field of view without training data. We formulate an energy-based mechanism that combines predictive learning and reactive control to perform active action localization without rewards, which can be sparse or non-existent in real-world environments. We perform extensive experiments in both simulated and real-world environments on two tasks - active object tracking and active action localization. We demonstrate that the proposed approach can generalize to different tasks and environments in a streaming fashion, without explicit rewards or training. We show that the proposed approach outperforms unsupervised baselines and obtains competitive performance compared to those trained with reinforcement learning.