mlsanityData to 7 Oct 2026

Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image

arXiv 2203.09457 ↗First posted 17 Mar 2022cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

Novel view synthesis from a single image has recently attracted a lot of attention, and it has been primarily advanced by 3D deep learning and rendering techniques. However, most work is still limited by synthesizing new views within relatively small camera motions. In this paper, we propose a novel approach to synthesize a consistent long-term video given a single scene image and a trajectory of large camera motions. Our approach utilizes an autoregressive Transformer to perform sequential modeling of multiple frames, which reasons the relations between multiple frames and the corresponding cameras to predict the next frame. To facilitate learning and ensure consistency among generated frames, we introduce a locality constraint based on the input cameras to guide self-attention among a large number of patches across space and time. Our method outperforms state-of-the-art view synthesis approaches by a large margin, especially when synthesizing long-term future in indoor 3D scenes. Project page at https://xrenaa.github.io/look-outside-room/.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 56 of its 80 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

56 of 80 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
38 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2020Neural Sparse Voxel Fieldscited by 959
  6. 2021Multiscale Vision Transformerscited by 896
  7. 2020Object-Centric Learning with Slot Attentioncited by 865
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  9. 2016Pixel Recurrent Neural Networkscited by 814
  10. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  11. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  13. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  14. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  15. 2018Video-to-Video Synthesiscited by 388
  16. 2016Generating Videos with Scene Dynamicscited by 376
  17. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  18. 2018Everybody Dance Nowcited by 313
  19. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  20. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  21. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  22. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  23. 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223
  24. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  25. 2019Few-shot Video-to-Video Synthesiscited by 196
  26. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  27. 2016View Synthesis by Appearance Flowcited by 172
  28. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  29. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  30. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  31. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 99
  32. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86
  33. 2018Layer-structured 3D Scene Inference via View Synthesiscited by 82
  34. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  35. 2020Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Imagecited by 77
  36. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  37. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  38. 2019Boundless: Generative Adversarial Networks for Image Extensioncited by 62

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 19; 2024: 14; 2025: 9; 2026: 4.

55 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  3. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  4. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  5. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  6. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  7. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  8. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  9. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  10. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  11. 2022Generating Long Videos of Dynamic Scenescited by 113
  12. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
38 more
  1. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
  2. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  3. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  4. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  5. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  6. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  7. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 56
  8. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  9. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  10. 2022Implicit Warping for Animation with Image Setscited by 45
  11. 2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
  12. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  13. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  14. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  15. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  16. 2023Blind Video Deflickering by Neural Filtering with a Flawed Atlascited by 27
  17. 2023Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Imagecited by 19
  18. 2024Denoising Diffusion via Image-Based Renderingcited by 15
  19. 2022Towards Smooth Video Compositioncited by 12
  20. 2022Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesiscited by 11
  21. 2024PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesiscited by 10
  22. 2023ENTL: Embodied Navigation Trajectory Learnercited by 9
  23. 2022SSGVS: Semantic Scene Graph-to-Video Synthesiscited by 8
  24. 2024CarFormer: Self-Driving with Learned Object-Centric Representationscited by 8
  25. 2023ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priorscited by 7
  26. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  27. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpaintingcited by 6
  28. 2025WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusioncited by 6
  29. 2024Believing is Seeing: Unobserved Object Detection using Generative Modelscited by 4
  30. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesiscited by 4
  31. 2025Look Beyond: Two-Stage Scene View Generation via Panorama and Video Diffusioncited by 3
  32. 2023SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Viewscited by 2
  33. 2023Novel View Synthesis with View-Dependent Effects from a Single Imagecited by 2
  34. 2024iControl3D: An Interactive System for Controllable 3D Scene Generationcited by 2
  35. 2025DOMR: Establishing Cross-View Segmentation via Dense Object Matchingcited by 2
  36. 2023Single-view Neural Radiance Fields with Depth Teachercited by 1
  37. 2024Answerability Fields: Answerable Location Estimation via Diffusion Modelscited by 1
  38. 2025DreamJourney: Perpetual View Generation with Video Diffusion Modelscited by 1

50 most cited of 55 All 55 (JSON)