# GeoAggregator: An Efficient Transformer Model for Geo-Spatial Tabular Data

arXiv: https://arxiv.org/abs/2502.15032 · HTML: https://mlsanity.com/paper/2502.15032/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [STAN: Spatio-Temporal Attention Network for Next Location Recommendation](https://mlsanity.com/paper/2102.04095/) (2021)
- [Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells](https://mlsanity.com/paper/2003.00824/) (2020)
- [TorchGeo: Deep Learning With Geospatial Data](https://mlsanity.com/paper/2111.08872/) (2021)
- [Inducing Point Operator Transformer: A Flexible and Scalable Architecture for Solving PDEs](https://mlsanity.com/paper/2312.10975/) (2023)
- [A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space](https://mlsanity.com/paper/2311.04434/) (2023)
- [Understanding Gaussian Attention Bias of Vision Transformers Using Effective Receptive Fields](https://mlsanity.com/paper/2305.04722/) (2023)
- [SSIN: Self-Supervised Learning for Rainfall Spatial Interpolation](https://mlsanity.com/paper/2311.15530/) (2023)

## Built on by (most cited first, 2 of 2)

- [Improving the Computational Efficiency and Explainability of GeoAggregator](https://mlsanity.com/paper/2507.17977/) (2025)
- [Quantifying geographic domain shift to decouple the geospatial transferability of human mobility flow generation models](https://mlsanity.com/paper/2608.21567/) (2026)

## Abstract (arXiv)

Modeling geospatial tabular data with deep learning has become a promising alternative to traditional statistical and machine learning approaches. However, existing deep learning models often face challenges related to scalability and flexibility as datasets grow. To this end, this paper introduces GeoAggregator, an efficient and lightweight algorithm based on transformer architecture designed specifically for geospatial tabular data modeling. GeoAggregators explicitly account for spatial autocorrelation and spatial heterogeneity through Gaussian-biased local attention and global positional awareness. Additionally, we introduce a new attention mechanism that uses the Cartesian product to manage the size of the model while maintaining strong expressive power. We benchmark GeoAggregator against spatial statistical models, XGBoost, and several state-of-the-art geospatial deep learning methods using both synthetic and empirical geospatial datasets. The results demonstrate that GeoAggregators achieve the best or second-best performance compared to their competitors on nearly all datasets. GeoAggregator's efficiency is underscored by its reduced model size, making it both scalable and lightweight. Moreover, ablation experiments offer insights into the effectiveness of the Gaussian bias and Cartesian attention mechanism, providing recommendations for further optimizing the GeoAggregator's performance.