# Pointersect: Neural Rendering with Cloud-Ray Intersection

arXiv: https://arxiv.org/abs/2304.12390 · HTML: https://mlsanity.com/paper/2304.12390/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 98 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)

## Built on by (most cited first, 23 of 23)

- [GVGEN: Text-to-3D Generation with Volumetric Representation](https://mlsanity.com/paper/2403.12957/) (2024)
- [VRP-UDF: Towards Unbiased Learning of Unsigned Distance Functions from Multi-view Images with Volume Rendering Priors](https://mlsanity.com/paper/2407.16396/) (2024)
- [PAPR: Proximity Attention Point Rendering](https://mlsanity.com/paper/2307.11086/) (2023)
- [LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes](https://mlsanity.com/paper/2405.00900/) (2024)
- [PointNeRF++: A multi-scale, point-based Neural Radiance Field](https://mlsanity.com/paper/2312.02362/) (2023)
- [3D Shape Tokenization via Latent Flow Matching](https://mlsanity.com/paper/2412.15618/) (2024)
- [Novel-View Acoustic Synthesis from 3D Reconstructed Rooms](https://mlsanity.com/paper/2310.15130/) (2023)
- [Arbitrary-Scale Point Cloud Upsampling by Voxel-Based Network with Latent Geometric-Consistent Learning](https://mlsanity.com/paper/2403.05117/) (2024)
- [Low Latency Point Cloud Rendering with Learned Splatting](https://mlsanity.com/paper/2409.16504/) (2024)
- [HashPoint: Accelerated Point Searching and Sampling for Neural Rendering](https://mlsanity.com/paper/2404.14044/) (2024)
- [Rendering-Oriented 3D Point Cloud Attribute Compression using Sparse Tensor-based Transformer](https://mlsanity.com/paper/2411.07899/) (2024)
- [UV-free Texture Generation with Denoising and Geodesic Heat Diffusions](https://mlsanity.com/paper/2408.16762/) (2024)
- [Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering](https://mlsanity.com/paper/2406.05915/) (2024)
- [Real-time Neural Rendering of LiDAR Point Clouds](https://mlsanity.com/paper/2502.11618/) (2025)
- [TeSO: Representing and Compressing 3D Point Cloud Scenes with Textured Surfel Octree](https://mlsanity.com/paper/2508.07083/) (2025)
- [RayletDF: Raylet Distance Fields for Generalizable 3D Surface Reconstruction from Point Clouds or Gaussians](https://mlsanity.com/paper/2508.09830/) (2025)
- [VioLA: Aligning Videos to 2D LiDAR Scans](https://mlsanity.com/paper/2311.04783/) (2023)
- [Learning Scene-Level Signed Directional Distance Function with Ellipsoidal Priors and Neural Residuals](https://mlsanity.com/paper/2503.20066/) (2025)
- [K-Buffers: A Plug-in Method for Enhancing Neural Fields with Multiple Buffers](https://mlsanity.com/paper/2505.19564/) (2025)
- [Conceptualizing Multi-scale Wavelet Attention and Ray-based Encoding for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.10977/) (2025)
- [CRISTAL: Real-time Camera Registration in Static LiDAR Scans using Neural Rendering](https://mlsanity.com/paper/2511.16349/) (2025)
- [PointGT: Simultaneous Geometry and Texture Editing for Point-Based Representations](https://mlsanity.com/paper/2609.03341/) (2026)
- [P-CORE: Self-Supervised Surface Consistency for Point-Based Neural Editing](https://mlsanity.com/paper/2609.03349/) (2026)

## Abstract (arXiv)

We propose a novel method that renders point clouds as if they are surfaces. The proposed method is differentiable and requires no scene-specific optimization. This unique capability enables, out-of-the-box, surface normal estimation, rendering room-scale point clouds, inverse rendering, and ray tracing with global illumination. Unlike existing work that focuses on converting point clouds to other representations--e.g., surfaces or implicit functions--our key idea is to directly infer the intersection of a light ray with the underlying surface represented by the given point cloud. Specifically, we train a set transformer that, given a small number of local neighbor points along a light ray, provides the intersection point, the surface normal, and the material blending weights, which are used to render the outcome of this light ray. Localizing the problem into small neighborhoods enables us to train a model with only 48 meshes and apply it to unseen point clouds. Our model achieves higher estimation accuracy than state-of-the-art surface reconstruction and point-cloud rendering methods on three test sets. When applied to room-scale point clouds, without any scene-specific optimization, the model achieves competitive quality with the state-of-the-art novel-view rendering methods. Moreover, we demonstrate ability to render and manipulate Lidar-scanned point clouds such as lighting control and object insertion.