# Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering

arXiv: https://arxiv.org/abs/2406.05915 · HTML: https://mlsanity.com/paper/2406.05915/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [OctSqueeze: Octree-Structured Entropy Model for LiDAR Compression](https://mlsanity.com/paper/2005.07178/) (2020)
- [Sparse Tensor-based Multiscale Representation for Point Cloud Geometry Compression](https://mlsanity.com/paper/2111.10633/) (2021)
- [VoxelContext-Net: An Octree based Framework for Point Cloud Compression](https://mlsanity.com/paper/2105.02158/) (2021)
- [Learned Point Cloud Geometry Compression](https://mlsanity.com/paper/1909.12037/) (2019)
- [OctAttention: Octree-Based Large-Scale Contexts Model for Point Cloud Compression](https://mlsanity.com/paper/2202.06028/) (2022)
- [AGG: Amortized Generative 3D Gaussians for Single Image to 3D](https://mlsanity.com/paper/2401.04099/) (2024)
- [Density-preserving Deep Point Cloud Compression](https://mlsanity.com/paper/2204.12684/) (2022)
- [Sparse Tensor-based Point Cloud Attribute Compression](https://mlsanity.com/paper/2204.01023/) (2022)
- [Pointersect: Neural Rendering with Cloud-Ray Intersection](https://mlsanity.com/paper/2304.12390/) (2023)
- [3DAC: Learning Attribute Compression for Point Clouds](https://mlsanity.com/paper/2203.09931/) (2022)
- [Learning to Predict on Octree for Scalable Point Cloud Geometry Coding](https://mlsanity.com/paper/2209.02226/) (2022)

## Built on by (most cited first, 2 of 2)

- [TeSO: Representing and Compressing 3D Point Cloud Scenes with Textured Surfel Octree](https://mlsanity.com/paper/2508.07083/) (2025)
- [TAFA-GSGC: Group-wise Scalable Point Cloud Geometry Compression with Progressive Residual Refinement](https://mlsanity.com/paper/2604.28045/) (2026)

## Abstract (arXiv)

Point cloud is a promising 3D representation for volumetric streaming in emerging AR/VR applications. Despite recent advances in point cloud compression, decoding and rendering high-quality images from lossy compressed point clouds is still challenging in terms of quality and complexity, making it a major roadblock to achieve real-time 6-Degree-of-Freedom video streaming. In this paper, we address this problem by developing a point cloud compression scheme that generates a bit stream that can be directly decoded to renderable 3D Gaussians. The encoder and decoder are jointly optimized to consider both bit-rates and rendering quality. It significantly improves the rendering quality while substantially reducing decoding and rendering time, compared to existing point cloud compression methods. Furthermore, the proposed scheme generates a scalable bit stream, allowing multiple levels of details at different bit-rate ranges. Our method supports real-time color decoding and rendering of high quality point clouds, thus paving the way for interactive 3D streaming applications with free view points.