# Normal Transformer: Extracting Surface Geometry from LiDAR Points Enhanced by Visual Semantics

arXiv: https://arxiv.org/abs/2211.10580 · HTML: https://mlsanity.com/paper/2211.10580/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 19 Nov 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs](https://mlsanity.com/paper/1902.10191/) (2019)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis](https://mlsanity.com/paper/2105.01288/) (2021)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [PCPNET: Learning Local Shape Properties from Raw Point Clouds](https://mlsanity.com/paper/1710.04954/) (2017)
- [Nesti-Net: Normal Estimation for Unstructured 3D Point Clouds using Convolutional Neural Networks](https://mlsanity.com/paper/1812.00709/) (2018)
- [DeepFit: 3D Surface Fitting via Neural Network Weighted Least Squares](https://mlsanity.com/paper/2003.10826/) (2020)
- [GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation](https://mlsanity.com/paper/2012.06980/) (2020)
- [Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks](https://mlsanity.com/paper/1612.07429/) (2016)
- [Marr Revisited: 2D-3D Alignment via Surface Normal Prediction](https://mlsanity.com/paper/1604.01347/) (2016)
- [FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data](https://mlsanity.com/paper/1912.08487/) (2019)
- [Refine-Net: Normal Refinement Neural Network for Noisy Point Clouds](https://mlsanity.com/paper/2203.12514/) (2022)
- [HSurf-Net: Normal Estimation for 3D Point Clouds by Learning Hyper Surfaces](https://mlsanity.com/paper/2210.07158/) (2022)
- [Deep Surface Normal Estimation with Hierarchical RGB-D Fusion](https://mlsanity.com/paper/1904.03405/) (2019)
- [GraphFit: Learning Multi-scale Graph-Convolutional Representation for Point Cloud Normal Estimation](https://mlsanity.com/paper/2207.11484/) (2022)
- [Three-Filters-to-Normal: An Accurate and Ultrafast Surface Normal Estimator](https://mlsanity.com/paper/2005.08165/) (2020)

## Built on by (most cited first, 2 of 2)

- [LiSu: A Dataset and Method for LiDAR Surface Normal Estimation](https://mlsanity.com/paper/2503.08601/) (2025)
- [Entropy and Attention Dynamics in Small Language Models: A Trace-Level Structural Analysis on the TruthfulQA Benchmark](https://mlsanity.com/paper/2604.03589/) (2026)

## Abstract (arXiv)

High-quality surface normal can help improve geometry estimation in problems faced by autonomous vehicles, such as collision avoidance and occlusion inference. While a considerable volume of literature focuses on densely scanned indoor scenarios, normal estimation during autonomous driving remains an intricate problem due to the sparse, non-uniform, and noisy nature of real-world LiDAR scans. In this paper, we introduce a multi-modal technique that leverages 3D point clouds and 2D colour images obtained from LiDAR and camera sensors for surface normal estimation. We present the Hybrid Geometric Transformer (HGT), a novel transformer-based neural network architecture that proficiently fuses visual semantic and 3D geometric information. Furthermore, we developed an effective learning strategy for the multi-modal data. Experimental results demonstrate the superior effectiveness of our information fusion approach compared to existing methods. It has also been verified that the proposed model can learn from a simulated 3D environment that mimics a traffic scene. The learned geometric knowledge is transferable and can be applied to real-world 3D scenes in the KITTI dataset. Further tasks built upon the estimated normal vectors in the KITTI dataset show that the proposed estimator has an advantage over existing methods.