mlsanityData to 7 Oct 2026

Learning to Render Novel Views from Wide-Baseline Stereo Pairs

arXiv 2304.08463 ↗First posted 17 Apr 2023cs.CV cs.AIcited by 103

Not on the map. Why?

Abstract (arXiv)

We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. In this challenging regime, 3D scene points are regularly observed only once, requiring prior-based reconstruction of scene geometry and appearance. We find that existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry and due to the high cost of differentiable rendering that precludes their scaling to large-scale training. We take a step towards resolving these shortcomings by formulating a multi-view transformer encoder, proposing an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray, and a lightweight cross-attention-based renderer. Our contributions enable training of our method on a large-scale real-world dataset of indoor and outdoor scenes. We demonstrate that our method learns powerful multi-view geometry priors while reducing the rendering time. We conduct extensive comparisons on held-out test scenes across two real-world datasets, significantly outperforming prior work on novel view synthesis from sparse image observations and achieving multi-view-consistent novel view synthesis.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 42 of its 64 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

42 of 64 references.

References · 42

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2021Vision Transformers for Dense Predictioncited by 1,789
  6. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  8. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  9. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  10. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  11. 2020Neural Sparse Voxel Fieldscited by 959
  12. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
30 more
  1. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  2. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  3. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  4. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  5. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  6. 2021Neural Fields in Visual Computing and Beyondcited by 539
  7. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  8. 2021Advances in Neural Renderingcited by 365
  9. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  10. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  11. 2019Neural Point-Based Graphicscited by 326
  12. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  13. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  14. 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
  15. 2020Free View Synthesiscited by 231
  16. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  17. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  18. 2021GeoNeRF: Generalizing NeRF with Geometry Priorscited by 195
  19. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  20. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  21. 2021Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenescited by 190
  22. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 188
  23. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  24. 2019Neural Volumes: Learning Dynamic Renderable Volumes from Imagescited by 173
  25. 2022Generalizable Patch-Based Neural Renderingcited by 121
  26. 2021Light Field Neural Renderingcited by 108
  27. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  28. 2018RenderNet: A deep convolutional network for differentiable rendering from 3D shapescited by 42
  29. 2021Input-level Inductive Biases for 3D Reconstructioncited by 31
  30. 2022Volume Rendering Digest (for NeRF)cited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 40; 2025: 31; 2026: 17.

103 papers.

Sort:

By area:

  1. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  2. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
  3. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  4. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  5. 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 275
  6. 2024No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Imagescited by 241
  7. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  8. 2024Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairscited by 171
  9. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 139
  10. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 136
  11. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  12. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
38 more
  1. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  2. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 111
  3. 2024GES: Generalized Exponential Splatting for Efficient Radiance Field Renderingcited by 92
  4. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 75
  5. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  6. 2023MuRF: Multi-Baseline Radiance Fieldscited by 70
  7. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  8. 2024FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descentcited by 61
  9. 2024PF3plat: Pose-Free Feed-Forward 3D Gaussian Splattingcited by 60
  10. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  11. 2024HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstructioncited by 55
  12. 2024LaRa: Efficient Large-Baseline Radiance Fieldscited by 54
  13. 2024DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Inputcited by 53
  14. 2024STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenescited by 53
  15. 20243D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexescited by 50
  16. 2025Triangle Splatting for Real-Time Radiance Field Renderingcited by 50
  17. 2024Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videoscited by 48
  18. 2024SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splattingcited by 40
  19. 2023Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstructioncited by 39
  20. 2023FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flowcited by 38
  21. 2023Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairscited by 37
  22. 2024Relaxing Accurate Initialization Constraint for 3D Gaussian Splattingcited by 33
  23. 2024Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstructioncited by 32
  24. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  25. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27
  26. 20233DiffTection: 3D Object Detection with Geometry-Aware Diffusion Featurescited by 26
  27. 2025StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generationcited by 26
  28. 2024Splatter-360: Generalizable 360circ Gaussian Splatting for Wide-baseline Panoramic Imagescited by 23
  29. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  30. 2023PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramascited by 22
  31. 2024Explorative Inbetweening of Time and Spacecited by 21
  32. 2024MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Datacited by 21
  33. 2025EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesiscited by 20
  34. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priorscited by 19
  35. 2024PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Viewscited by 18
  36. 2024Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesiscited by 18
  37. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 18
  38. 2025SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observationscited by 16

50 most cited of 103 All 103 (JSON)