mlsanityData to 14 Oct 2026

Video Autoencoder: self-supervised disentanglement of static 3D structure and motion

arXiv 2110.02951 ↗First posted 6 Oct 2021cs.CV cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

A video autoencoder is proposed for learning disentan- gled representations of 3D structure and camera pose from videos in a self-supervised manner. Relying on temporal continuity in videos, our work assumes that the 3D scene structure in nearby video frames remains static. Given a sequence of video frames as input, the video autoencoder extracts a disentangled representation of the scene includ- ing: (i) a temporally-consistent deep voxel feature to represent the 3D structure and (ii) a 3D trajectory of camera pose for each frame. These two representations will then be re-entangled for rendering the input video frames. This video autoencoder can be trained directly using a pixel reconstruction loss, without any ground truth 3D or camera pose annotations. The disentangled representation can be applied to a range of tasks, including novel view synthesis, camera pose estimation, and video generation by motion following. We evaluate our method on several large- scale natural video datasets, and show generalization results on out-of-domain images.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 63 of its 81 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

63 of 81 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  8. 2017Improved Training of Wasserstein GANscited by 2,235
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  11. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  12. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
38 more
  1. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  2. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  3. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 782
  4. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  5. 2018Disentangling by Factorisingcited by 616
  6. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  7. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 405
  8. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  9. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  10. 2020State of the Art on Neural Renderingcited by 278
  11. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  12. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  13. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  14. 2015DeepStereo: Learning to Predict New Views from the World's Imagerycited by 185
  15. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  16. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  17. 2016View Synthesis by Appearance Flowcited by 172
  18. 2015Inverting Visual Representations with Convolutional Networkscited by 170
  19. 2020Adversarial Latent Autoencoderscited by 155
  20. 2017Unsupervised Learning of Disentangled Representations from Videocited by 139
  21. 20193D Ken Burns Effect from a Single Imagecited by 128
  22. 2018Visual Object Networks: Image Generation with Disentangled 3D Representationcited by 124
  23. 2015Deep Convolutional Inverse Graphics Networkcited by 121
  24. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 121
  25. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 114
  26. 2015Learning to See by Movingcited by 102
  27. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  28. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 92
  29. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  30. 2020Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Imagecited by 77
  31. 2016Unsupervised Learning of 3D Structure from Imagescited by 67
  32. 20204D Visualization of Dynamic Events from Unconstrained Multi-View Videoscited by 66
  33. 2015Multi-view 3D Models from Single Images with a Convolutional Networkcited by 58
  34. 2016Generative Image Modeling using Style and Structure Adversarial Networkscited by 50
  35. 2018Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoderscited by 49
  36. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  37. 2018Self-supervised learning of a facial attribute embedding from videocited by 45
  38. 2015Learning image representations tied to ego-motioncited by 42

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 8; 2024: 6; 2025: 5; 2026: 5.

34 papers.

Sort:

By area:

  1. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  2. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 274
  3. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  4. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  5. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 77
  6. 2022E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Contextcited by 76
  7. 2024FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descentcited by 61
  8. 2022Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Imagescited by 59
  9. 2022CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphismcited by 57
  10. 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
  11. 2022Few-View Object Reconstruction with Unknown Categories and Camera Posescited by 53
  12. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
22 more
  1. 2024SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splattingcited by 42
  2. 2023FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flowcited by 39
  3. 2023SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Imagescited by 35
  4. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  5. 2025E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-trainingcited by 22
  6. 2024GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-timecited by 20
  7. 2024FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimationcited by 12
  8. 2022MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Posecited by 10
  9. 2023Learning 3D Photography Videos via Self-supervised Diffusion on Single Imagescited by 8
  10. 2023VoxDet: Voxel Learning for Novel Instance Detectioncited by 8
  11. 2024Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVScited by 8
  12. 2023DEVIAS: Learning Disentangled Video Representations of Action and Scenecited by 7
  13. 2026Self-Improving 4D Perception via Self-Distillationcited by 4
  14. 2025MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidancecited by 3
  15. 2025Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representationscited by 3
  16. 20262Xplat: Decoupling Geometry and Appearance Modeling for Feed-Forward 3D Gaussian Splattingcited by 2
  17. 2026Latent-Compressed Variational Autoencoder for Video Diffusion Modelscited by 2
  18. 2023Fluid Viscosity Prediction Leveraging Computer Vision and Robot Interactioncited by 1
  19. 2026Pixel-to-4D: Camera-Controlled Image-to-Video Generation with Dynamic 3D Gaussianscited by 1
  20. 2022Linear Variational State-Space Filteringcited by 0
  21. 2022Video Extrapolation in Space and Timecited by 0
  22. 2026RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Videocited by 0