# Retro-FPN: Retrospective Feature Pyramid Network for Point Cloud Semantic Segmentation

arXiv: https://arxiv.org/abs/2308.09314 · HTML: https://mlsanity.com/paper/2308.09314/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 92 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Builds on (most cited first, 30 of 78)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)

## Built on by (most cited first, 6 of 6)

- [Learning a More Continuous Zero Level Set in Unsigned Distance Fields through Level Set Projection](https://mlsanity.com/paper/2308.11441/) (2023)
- [MonoInstance: Enhancing Monocular Priors via Multi-view Instance Alignment for Neural Rendering and Reconstruction](https://mlsanity.com/paper/2503.18363/) (2025)
- [Omni-Supervised Motion Editing: Balancing Change and Invariance through Positive-Negative Learning](https://mlsanity.com/paper/2605.30969/) (2026)
- [Spatial Temporal Synergy: Balancing Change and Invariance in Text Driven 3D Human Motion Editing](https://mlsanity.com/paper/2608.16008/) (2026)
- [Point Neighborhood Embeddings](https://mlsanity.com/paper/2310.02083/) (2023)
- [BAFPN: Bi directional alignment of features to improve localization accuracy](https://mlsanity.com/paper/2412.01859/) (2024)

## Abstract (arXiv)

Learning per-point semantic features from the hierarchical feature pyramid is essential for point cloud semantic segmentation. However, most previous methods suffered from ambiguous region features or failed to refine per-point features effectively, which leads to information loss and ambiguous semantic identification. To resolve this, we propose Retro-FPN to model the per-point feature prediction as an explicit and retrospective refining process, which goes through all the pyramid layers to extract semantic features explicitly for each point. Its key novelty is a retro-transformer for summarizing semantic contexts from the previous layer and accordingly refining the features in the current stage. In this way, the categorization of each point is conditioned on its local semantic pattern. Specifically, the retro-transformer consists of a local cross-attention block and a semantic gate unit. The cross-attention serves to summarize the semantic pattern retrospectively from the previous layer. And the gate unit carefully incorporates the summarized contexts and refines the current semantic features. Retro-FPN is a pluggable neural network that applies to hierarchical decoders. By integrating Retro-FPN with three representative backbones, including both point-based and voxel-based methods, we show that Retro-FPN can significantly improve performance over state-of-the-art backbones. Comprehensive experiments on widely used benchmarks can justify the effectiveness of our design. The source is available at https://github.com/AllenXiangX/Retro-FPN