mlsanityData to 7 Oct 2026

NeX: Real-time View Synthesis with Neural Basis Expansion

arXiv 2103.05606 ↗First posted 9 Mar 2021cs.CV cs.GR cs.LGcited by 215

Not on the map. Why?

Abstract (arXiv)

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. Unlike traditional MPI that uses a set of simple RGBα planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Builds on 25 of its 56 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

25 of 56 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  6. 2017Deep Image Priorcited by 1,097
  7. 2020Neural Sparse Voxel Fieldscited by 959
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  10. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  11. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  12. 2020Single-View View Synthesis with Multiplane Imagescited by 263
13 more
  1. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  2. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  3. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  4. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  5. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  6. 2015DeepStereo: Learning to Predict New Views from the World's Imagerycited by 185
  7. 20193D Ken Burns Effect from a Single Imagecited by 128
  8. 2018Extreme View Synthesiscited by 94
  9. 2018Layer-structured 3D Scene Inference via View Synthesiscited by 82
  10. 2020Neural Light Transport for Relighting and View Synthesiscited by 73
  11. 2020Crowdsampling the Plenoptic Functioncited by 62
  12. 2017Learning to Synthesize a 4D RGBD Light Field from a Single Imagecited by 60
  13. 2018Peeking Behind Objects: Layered Depth Prediction from a Single Imagecited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 58; 2023: 73; 2024: 29; 2025: 21; 2026: 4.

215 papers.

Sort:

By area:

  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  2. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  3. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  4. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  5. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  6. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  7. 2021Neural Fields in Visual Computing and Beyondcited by 539
  8. 2021Advances in Neural Renderingcited by 365
  9. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  10. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 344
  11. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  12. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
38 more
  1. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  2. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  3. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  4. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 220
  5. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 216
  6. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  7. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  8. 2021InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Renderingcited by 180
  9. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 173
  10. 2022Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-timecited by 164
  11. 2022Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoisingcited by 163
  12. 2023Segment Any 3D Gaussianscited by 158
  13. 20243DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videoscited by 156
  14. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 151
  15. 2021Efficient Neural Radiance Fields for Interactive Free-viewpoint Videocited by 150
  16. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  17. 2022StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learningcited by 140
  18. 2023Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Camerascited by 139
  19. 2022D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Videocited by 136
  20. 2022RigNeRF: Fully Controllable Neural 3D Portraitscited by 126
  21. 2021NeRFReN: Neural Radiance Fields with Reflectionscited by 121
  22. 2022Generalizable Patch-Based Neural Renderingcited by 121
  23. 2022Is Attention All That NeRF Needs?cited by 113
  24. 2022EfficientNeRF: Efficient Neural Radiance Fieldscited by 110
  25. 2021Light Field Neural Renderingcited by 108
  26. 2022NeILF: Neural Incident Light Field for Physically-based Material Estimationcited by 105
  27. 2022Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstructioncited by 92
  28. 20234K4D: Real-Time 4D View Synthesis at 4K Resolutioncited by 91
  29. 2022ReLU Fields: The Little Non-linearity That Couldcited by 87
  30. 2022Differentiable Point-Based Radiance Fields for Efficient View Synthesiscited by 86
  31. 2023Neural Point Catacaustics for Novel-View Synthesis of Reflectionscited by 81
  32. 2022BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fieldscited by 80
  33. 2021FoV-NeRF: Foveated Neural Radiance Fields for Virtual Realitycited by 79
  34. 2022Neural 3D Reconstruction in the Wildcited by 79
  35. 2023Neural Residual Radiance Fields for Streamably Free-Viewpoint Videoscited by 77
  36. 2021Learning to Stylize Novel Viewscited by 76
  37. 2024Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splattingcited by 73
  38. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69

50 most cited of 215 All 215 (JSON)