mlsanityData to 7 Oct 2026

Neural Rays for Occlusion-aware Image-based Rendering

arXiv 2107.13421 ↗First posted 28 Jul 2021cs.CV cs.GRcited by 192

Not on the map. Why?

Abstract (arXiv)

We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis task. Recent works construct radiance fields from image features of input views to render novel view images, which enables the generalization to new scenes. However, due to occlusions, a 3D point may be invisible to some input views. On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction. To solve this problem, we predict the visibility of 3D points to input views within our NeuRay representation. This visibility enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality. Meanwhile, a novel consistency loss is proposed to refine the visibility in NeuRay when finetuning on a specific scene. Experiments demonstrate that our approach achieves state-of-the-art performance on the novel view synthesis task when generalizing to unseen scenes and outperforms per-scene optimization methods after finetuning.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Builds on 42 of its 71 references that are in the map. First posted on arXiv 28 Jul 2021.

Builds on

42 of 71 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  6. 2020Neural Sparse Voxel Fieldscited by 959
  7. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  11. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
  12. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
30 more
  1. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  2. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  3. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  4. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  5. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  6. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  7. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  8. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  9. 2019Neural Point-Based Graphicscited by 326
  10. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  11. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  12. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  13. 2020Free View Synthesiscited by 231
  14. 2021NeX: Real-time View Synthesis with Neural Basis Expansioncited by 215
  15. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  16. 2021Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenescited by 190
  17. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 179
  18. 2020Stable View Synthesiscited by 165
  19. 2016Learning-Based View Synthesis for Light Field Camerascited by 145
  20. 2021Real-time Deep Dynamic Characterscited by 133
  21. 2020Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstructioncited by 112
  22. 2021Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Controlcited by 106
  23. 2021Neural Lumigraph Renderingcited by 105
  24. 2018Extreme View Synthesiscited by 94
  25. 2021ShaRF: Shape-conditioned Radiance Fields from a Single Viewcited by 92
  26. 2021PVA: Pixel-aligned Volumetric Avatarscited by 40
  27. 2021StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Visioncited by 38
  28. 2021Fast Training of Neural Lumigraph Representations using Meta Learningcited by 29
  29. 2020Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translationcited by 13
  30. 2018IGNOR: Image-guided Neural Object Renderingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 27; 2023: 69; 2024: 60; 2025: 23; 2026: 7.

192 papers.

Sort:

By area:

  1. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  2. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  3. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  4. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  5. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  6. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  7. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
  8. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 344
  9. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  10. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  11. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 283
  12. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
38 more
  1. 2021GeoNeRF: Generalizing NeRF with Geometry Priorscited by 195
  2. 2022SPARF: Neural Radiance Fields from Sparse and Noisy Posescited by 195
  3. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  4. 2022SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Viewscited by 175
  5. 2024Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairscited by 171
  6. 2023GauHuman: Articulated Gaussian Splatting from Monocular Human Videoscited by 168
  7. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  8. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  9. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priorscited by 156
  10. 2021Efficient Neural Radiance Fields for Interactive Free-viewpoint Videocited by 150
  11. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Viewscited by 148
  12. 2022Generalizable Patch-Based Neural Renderingcited by 121
  13. 2023F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectoriescited by 118
  14. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  15. 2022Is Attention All That NeRF Needs?cited by 113
  16. 2024RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videoscited by 113
  17. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  18. 2024Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splatscited by 111
  19. 2021Light Field Neural Renderingcited by 108
  20. 2023Learning to Render Novel Views from Wide-Baseline Stereo Pairscited by 103
  21. 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 92
  22. 2023SHERF: Generalizable Human NeRF from a Single Imagecited by 88
  23. 2022Differentiable Point-Based Radiance Fields for Efficient View Synthesiscited by 86
  24. 2024FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenescited by 86
  25. 2023MonoHuman: Animatable Human Neural Field from Monocular Videocited by 85
  26. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  27. 2023MuRF: Multi-Baseline Radiance Fieldscited by 70
  28. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  29. 2022GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRFcited by 64
  30. 2022MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videoscited by 63
  31. 2023DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fieldscited by 60
  32. 2023Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenescited by 55
  33. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  34. 2023Representing Volumetric Videos as Dynamic MLP Mapscited by 51
  35. 2024DOGS: Distributed-Oriented Gaussian Splatting for Large-Scale 3D Reconstruction Via Gaussian Consensuscited by 51
  36. 2023Explicit Correspondence Matching for Generalizable Neural Radiance Fieldscited by 46
  37. 2023ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fieldscited by 43
  38. 2022V4d: voxel for 4d novel view synthesiscited by 42

50 most cited of 192 All 192 (JSON)