# LATFormer: Locality-Aware Point-View Fusion Transformer for 3D Shape Recognition

arXiv: https://arxiv.org/abs/2109.01291 · HTML: https://mlsanity.com/paper/2109.01291/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 87 references that are in the map.
- First posted on arXiv 3 Sep 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis](https://mlsanity.com/paper/2105.01288/) (2021)
- [Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline](https://mlsanity.com/paper/2106.05304/) (2021)
- [Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models](https://mlsanity.com/paper/1704.01222/) (2017)
- [DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing](https://mlsanity.com/paper/1909.03669/) (2019)
- [Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network](https://mlsanity.com/paper/1811.02565/) (2018)
- [MeshNet: Mesh Neural Network for 3D Shape Representation](https://mlsanity.com/paper/1811.11424/) (2018)
- [Triplet-Center Loss for Multi-View 3D Object Retrieval](https://mlsanity.com/paper/1803.06189/) (2018)
- [PointAugment: an Auto-Augmentation Framework for Point Cloud Classification](https://mlsanity.com/paper/2002.10876/) (2020)
- [PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition](https://mlsanity.com/paper/1808.07659/) (2018)
- [Point Attention Network for Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/1909.12663/) (2019)
- [View N-gram Network for 3D Object Retrieval](https://mlsanity.com/paper/1908.01958/) (2019)
- [FusionNet: 3D Object Classification Using Multiple Data Representations](https://mlsanity.com/paper/1607.05695/) (2016)
- [3DViewGraph: Learning Global Features for 3D Shapes from A Graph of Unordered Views with Attention](https://mlsanity.com/paper/1905.07503/) (2019)

## Built on by (most cited first, 4 of 4)

- [LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modeling](https://mlsanity.com/paper/2405.17149/) (2024)
- [Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval](https://mlsanity.com/paper/2507.21489/) (2025)
- [TACO-Net: Topological Signatures Triumph in 3D Object Classification](https://mlsanity.com/paper/2509.24802/) (2025)
- [DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrieval](https://mlsanity.com/paper/2604.19432/) (2026)

## Abstract (arXiv)

Recently, 3D shape understanding has achieved significant progress due to the advances of deep learning models on various data formats like images, voxels, and point clouds. Among them, point clouds and multi-view images are two complementary modalities of 3D objects and learning representations by fusing both of them has been proven to be fairly effective. While prior works typically focus on exploiting global features of the two modalities, herein we argue that more discriminative features can be derived by modeling ``where to fuse''. To investigate this, we propose a novel Locality-Aware Point-View Fusion Transformer (LATFormer) for 3D shape retrieval and classification. The core component of LATFormer is a module named Locality-Aware Fusion (LAF) which integrates the local features of correlated regions across the two modalities based on the co-occurrence scores. We further propose to filter out scores with low values to obtain salient local co-occurring regions, which reduces redundancy for the fusion process. In our LATFormer, we utilize the LAF module to fuse the multi-scale features of the two modalities both bidirectionally and hierarchically to obtain more informative features. Comprehensive experiments on four popular 3D shape benchmarks covering 3D object retrieval and classification validate its effectiveness.