# A Recipe for Geometry-Aware 3D Mesh Transformers

arXiv: https://arxiv.org/abs/2411.00164 · HTML: https://mlsanity.com/paper/2411.00164/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Structure-Aware Transformer for Graph Representation Learning](https://mlsanity.com/paper/2202.03036/) (2022)
- [DiffusionNet: Discretization Agnostic Learning on Surfaces](https://mlsanity.com/paper/2012.00888/) (2020)
- [SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels](https://mlsanity.com/paper/1711.08920/) (2017)
- [Point Convolutional Neural Networks by Extension Operators](https://mlsanity.com/paper/1803.10091/) (2018)
- [MeshNet: Mesh Neural Network for 3D Shape Representation](https://mlsanity.com/paper/1811.11424/) (2018)
- [Learning shape correspondence with anisotropic convolutional neural networks](https://mlsanity.com/paper/1605.06437/) (2016)
- [A Generalization of ViT/MLP-Mixer to Graphs](https://mlsanity.com/paper/2212.13350/) (2022)
- [MeshCNN: A Network with an Edge](https://mlsanity.com/paper/1809.05910/) (2018)
- [Subdivision-Based Mesh Convolution Networks](https://mlsanity.com/paper/2106.02285/) (2021)
- [Effective Rotation-invariant Point CNN with Spherical Harmonics kernels](https://mlsanity.com/paper/1906.11555/) (2019)
- [Deep Geometric Texture Synthesis](https://mlsanity.com/paper/2007.00074/) (2020)
- [Primal-Dual Mesh Convolutional Neural Networks](https://mlsanity.com/paper/2010.12455/) (2020)
- [CNNs on Surfaces using Rotation-Equivariant Features](https://mlsanity.com/paper/2006.01570/) (2020)
- [MeshWalker: Deep Mesh Understanding by Random Walks](https://mlsanity.com/paper/2006.05353/) (2020)
- [MeshMAE: Masked Autoencoders for 3D Mesh Data Analysis](https://mlsanity.com/paper/2207.10228/) (2022)

## Built on by (most cited first, 1 of 1)

- [Hierarchical Mesh Transformers with Topology-Guided Pretraining for Morphometric Analysis of Brain Structures](https://mlsanity.com/paper/2604.05215/) (2026)

## Abstract (arXiv)

Utilizing patch-based transformers for unstructured geometric data such as polygon meshes presents significant challenges, primarily due to the absence of a canonical ordering and variations in input sizes. Prior approaches to handling 3D meshes and point clouds have either relied on computationally intensive node-level tokens for large objects or resorted to resampling to standardize patch size. Moreover, these methods generally lack a geometry-aware, stable Structural Embedding (SE), often depending on simplistic absolute SEs such as 3D coordinates, which compromise isometry invariance essential for tasks like semantic segmentation. In our study, we meticulously examine the various components of a geometry-aware 3D mesh transformer, from tokenization to structural encoding, assessing the contribution of each. Initially, we introduce a spectral-preserving tokenization rooted in algebraic multigrid methods. Subsequently, we detail an approach for embedding features at the patch level, accommodating patches with variable node counts. Through comparative analyses against a baseline model employing simple point-wise Multi-Layer Perceptrons (MLP), our research highlights critical insights: 1) the importance of structural and positional embeddings facilitated by heat diffusion in general 3D mesh transformers; 2) the effectiveness of novel components such as geodesic masking and feature interaction via cross-attention in enhancing learning; and 3) the superior performance and efficiency of our proposed methods in challenging segmentation and classification tasks.