mlsanityData to 7 Oct 2026

Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning

arXiv 2108.01069 ↗First posted 2 Aug 2021cs.RO cs.CV cs.LGcited by 23

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Humans learn to imitate by observing others. However, robot imitation learning generally requires expert demonstrations in the first-person view (FPV). Collecting such FPV videos for every robot could be very expensive. Third-person imitation learning (TPIL) is the concept of learning action policies by observing other agents in a third-person view (TPV), similar to what humans do. This ultimately allows utilizing human and robot demonstration videos in TPV from many different data sources, for the policy learning. In this paper, we present a TPIL approach for robot tasks with egomotion. Although many robot tasks with ground/aerial mobility often involve actions with camera egomotion, study on TPIL for such tasks has been limited. Here, FPV and TPV observations are visually very different; FPV shows egomotion while the agent appearance is only observable in TPV. To enable better state learning for TPIL, we propose our disentangled representation learning method. We use a dual auto-encoder structure plus representation permutation loss and time-contrastive loss to ensure the state and viewpoint representations are well disentangled. Our experiments show the effectiveness of our approach.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 41 references that are in the map. First posted on arXiv 2 Aug 2021.

Builds on

32 of 41 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2019Contrastive Multiview Codingcited by 1,297
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
20 more
  1. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  2. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 784
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  5. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  6. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  7. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  8. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  9. 2019Domain Agnostic Learning with Disentangled Representationscited by 107
  10. 2017Third-Person Imitation Learningcited by 103
  11. 2018Imitating Latent Policies from Observationcited by 101
  12. 2018Playing hard exploration games by watching YouTubecited by 98
  13. 2017Neural Face Editing with Intrinsic Image Disentanglingcited by 70
  14. 2019Gait Recognition via Disentangled Representation Learningcited by 70
  15. 2019Disentangled Representation Learning for 3D Face Shapecited by 48
  16. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  17. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  18. 2018Learning Actionable Representations from Visual Observationscited by 35
  19. 2019Adversarial Skill Networks: Unsupervised Robot Skill Learning from Videocited by 17
  20. 2016Cross-Modal Scene Networkscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 7; 2024: 5; 2025: 2; 2026: 1.

23 papers.

Sort:

By area:

  1. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  2. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  3. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  4. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  5. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
  6. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  7. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  8. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
  9. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  10. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  11. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13
  12. 2023A Survey of Demonstration Learningcited by 12
11 more
  1. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  2. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  3. 2023Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Taskscited by 7
  4. 2022ProtoX: Explaining a Reinforcement Learning Agent via Prototypingcited by 4
  5. 2022Contrastive Learning from Demonstrationscited by 3
  6. 2022Neural Neural Textures Make Sim2Real Consistentcited by 3
  7. 2023Active Third-Person Imitation Learningcited by 1
  8. 2024SENSOR: Imitate Third-Person Expert's Behaviors via Active Sensoringcited by 1
  9. 2023Active Vision Reinforcement Learning under Limited Visual Observabilitycited by 0
  10. 2023Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformerscited by 0
  11. 2026Mirror Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Manipulator-Independent Representations for Visual Imitation
  2. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  3. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning
  4. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
  5. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller
  6. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  7. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  8. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames