# Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts

arXiv: https://arxiv.org/abs/2308.11793 · HTML: https://mlsanity.com/paper/2308.11793/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 61 of its 74 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 61)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)

## Built on by (most cited first, 27 of 27)

- [DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization](https://mlsanity.com/paper/2403.06912/) (2024)
- [CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization](https://mlsanity.com/paper/2405.12110/) (2024)
- [Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos](https://mlsanity.com/paper/2412.03526/) (2024)
- [G3R: Gradient Guided Generalizable Reconstruction](https://mlsanity.com/paper/2409.19405/) (2024)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D](https://mlsanity.com/paper/2403.18922/) (2024)
- [SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steering](https://mlsanity.com/paper/2503.12024/) (2025)
- [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025)
- [4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation](https://mlsanity.com/paper/2506.18839/) (2025)
- [SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fields](https://mlsanity.com/paper/2408.06697/) (2024)
- [CaesarNeRF: Calibrated Semantic Representation for Few-shot Generalizable Neural Rendering](https://mlsanity.com/paper/2311.15510/) (2023)
- [CoMapGS: Covisibility Map-based Gaussian Splatting for Sparse Novel View Synthesis](https://mlsanity.com/paper/2503.20998/) (2025)
- [DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Robotics](https://mlsanity.com/paper/2507.04661/) (2025)
- [Uncertainty-guided Optimal Transport in Depth Supervised Sparse-View 3D Gaussian](https://mlsanity.com/paper/2405.19657/) (2024)
- [AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark](https://mlsanity.com/paper/2409.15041/) (2024)
- [Entangled View-Epipolar Information Aggregation for Generalizable Neural Radiance Fields](https://mlsanity.com/paper/2311.11845/) (2023)
- [H3R: Hybrid Multi-view Correspondence for Generalizable 3D Reconstruction](https://mlsanity.com/paper/2508.03118/) (2025)
- [GoLF-NRT: Integrating Global Context and Local Geometry for Few-Shot View Synthesis](https://mlsanity.com/paper/2505.19813/) (2025)
- [Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification](https://mlsanity.com/paper/2511.11460/) (2025)
- [MSI-NeRF: Linking Omni-Depth with View Synthesis through Multi-Sphere Image aided Generalizable Neural Radiance Field](https://mlsanity.com/paper/2403.10840/) (2024)
- [RDG-GS: Relative Depth Guidance with Gaussian Splatting for Real-time Sparse-View 3D Rendering](https://mlsanity.com/paper/2501.11102/) (2025)
- [Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields](https://mlsanity.com/paper/2403.11131/) (2024)
- [psPRF:Pansharpening Planar Neural Radiance Field for Generalized 3D Reconstruction Satellite Imagery](https://mlsanity.com/paper/2406.15707/) (2024)
- [MU-GeNeRF: Multi-view Uncertainty-guided Generalizable Neural Radiance Fields for Distractor-aware Scene](https://mlsanity.com/paper/2604.17965/) (2026)
- [Teacher-Guided Routing for Sparse Vision Mixture-of-Experts](https://mlsanity.com/paper/2604.21330/) (2026)
- [Gaussian Volumetric Representation for Efficient Shear-Warp Visualization](https://mlsanity.com/paper/2607.25377/) (2026)
- [IRIS: Implicit Rendering Matters for Pose-Free Novel View Synthesis](https://mlsanity.com/paper/2609.18034/) (2026)

## Abstract (arXiv)

Cross-scene generalizable NeRF models, which can directly synthesize novel views of unseen scenes, have become a new spotlight of the NeRF field. Several existing attempts rely on increasingly end-to-end "neuralized" architectures, i.e., replacing scene representation and/or rendering modules with performant neural networks such as transformers, and turning novel view synthesis into a feed-forward inference pipeline. While those feedforward "neuralized" architectures still do not fit diverse scenes well out of the box, we propose to bridge them with the powerful Mixture-of-Experts (MoE) idea from large language models (LLMs), which has demonstrated superior generalization ability by balancing between larger overall model capacity and flexible per-instance specialization. Starting from a recent generalizable NeRF architecture called GNT, we first demonstrate that MoE can be neatly plugged in to enhance the model. We further customize a shared permanent expert and a geometry-aware consistency loss to enforce cross-scene consistency and spatial smoothness respectively, which are essential for generalizable view synthesis. Our proposed model, dubbed GNT with Mixture-of-View-Experts (GNT-MOVE), has experimentally shown state-of-the-art results when transferring to unseen scenes, indicating remarkably better cross-scene generalization in both zero-shot and few-shot settings. Our codes are available at https://github.com/VITA-Group/GNT-MOVE.