# Geometric Algebra Attention Networks for Small Point Clouds

arXiv: https://arxiv.org/abs/2110.02393 · HTML: https://mlsanity.com/paper/2110.02393/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 68 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://mlsanity.com/paper/2104.09459/) (2021)
- [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](https://mlsanity.com/paper/1810.00825/) (2018)
- [N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials](https://mlsanity.com/paper/1803.01588/) (2018)
- [Embed Me If You Can: A Geometric Perceptron](https://mlsanity.com/paper/2006.06507/) (2020)
- [freud: A Software Suite for High Throughput Analysis of Particle Simulation Data](https://mlsanity.com/paper/1906.06317/) (2019)

## Built on by (most cited first, 10 of 10)

- [Clifford Neural Layers for PDE Modeling](https://mlsanity.com/paper/2209.04934/) (2022)
- [Clifford Group Equivariant Neural Networks](https://mlsanity.com/paper/2305.11141/) (2023)
- [Geometric Algebra Transformer](https://mlsanity.com/paper/2305.18415/) (2023)
- [Geometric Clifford Algebra Networks](https://mlsanity.com/paper/2302.06594/) (2023)
- [Clifford Group Equivariant Simplicial Message Passing Networks](https://mlsanity.com/paper/2402.10011/) (2024)
- [The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Science](https://mlsanity.com/paper/2210.17484/) (2022)
- [MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling](https://mlsanity.com/paper/2309.05934/) (2023)
- [Using Multiple Vector Channels Improves E(n)-Equivariant Graph Neural Networks](https://mlsanity.com/paper/2309.03139/) (2023)
- [A Clifford Algebraic Approach to E(n)-Equivariant High-order Graph Neural Networks](https://mlsanity.com/paper/2410.04692/) (2024)
- [Self-Supervised Learning for Ordered Three-Dimensional Structures](https://mlsanity.com/paper/2411.14680/) (2024)

## Abstract (arXiv)

Much of the success of deep learning is drawn from building architectures that properly respect underlying symmetry and structure in the data on which they operate - a set of considerations that have been united under the banner of geometric deep learning. Often problems in the physical sciences deal with relatively small sets of points in two- or three-dimensional space wherein translation, rotation, and permutation equivariance are important or even vital for models to be useful in practice. In this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields a powerful way to impose permutation equivariance. We demonstrate the usefulness of these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.