# SENSOR: Imitate Third-Person Expert's Behaviors via Active Sensoring

arXiv: https://arxiv.org/abs/2404.03386 · HTML: https://mlsanity.com/paper/2404.03386/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 32)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller](https://mlsanity.com/paper/1911.09676/) (2019)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Cross-domain Imitation from Observations](https://mlsanity.com/paper/2105.10037/) (2021)
- [Visual Adversarial Imitation Learning using Variational Models](https://mlsanity.com/paper/2107.08829/) (2021)
- [Model-based Adversarial Imitation Learning](https://mlsanity.com/paper/1612.02179/) (2016)
- [Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions](https://mlsanity.com/paper/1811.08067/) (2018)
- [Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning](https://mlsanity.com/paper/2108.01069/) (2021)
- [Domain-Robust Visual Imitation Learning with Mutual Information Constraints](https://mlsanity.com/paper/2103.05079/) (2021)
- [Adversarial Imitation Learning from Incomplete Demonstrations](https://mlsanity.com/paper/1905.12310/) (2019)
- [Learn what matters: cross-domain imitation learning with task-relevant embeddings](https://mlsanity.com/paper/2209.12093/) (2022)
- [SeMAIL: Eliminating Distractors in Visual Imitation via Separated Models](https://mlsanity.com/paper/2306.10695/) (2023)
- [Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators](https://mlsanity.com/paper/2007.08082/) (2020)

## Built on by (most cited first, 1 of 1)

- [FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making](https://mlsanity.com/paper/2507.12496/) (2025)

## Abstract (arXiv)

In many real-world visual Imitation Learning (IL) scenarios, there is a misalignment between the agent's and the expert's perspectives, which might lead to the failure of imitation. Previous methods have generally solved this problem by domain alignment, which incurs extra computation and storage costs, and these methods fail to handle the textithard cases where the viewpoint gap is too large. To alleviate the above problems, we introduce active sensoring in the visual IL setting and propose a model-based SENSory imitatOR (SENSOR) to automatically change the agent's perspective to match the expert's. SENSOR jointly learns a world model to capture the dynamics of latent states, a sensor policy to control the camera, and a motor policy to control the agent. Experiments on visual locomotion tasks show that SENSOR can efficiently simulate the expert's perspective and strategy, and outperforms most baseline methods.