# Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

arXiv: https://arxiv.org/abs/2601.08882 · HTML: https://mlsanity.com/paper/2601.08882/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [Geometry-aware training of factorized layers in tensor Tucker format](https://mlsanity.com/paper/2305.19059/) (2023)
- [OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery](https://mlsanity.com/paper/2410.19965/) (2024)
- [Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees](https://mlsanity.com/paper/2406.17887/) (2024)
- [Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks](https://mlsanity.com/paper/2505.08022/) (2025)
- [GeoLoRA: Geometric integration for parameter efficient fine-tuning](https://mlsanity.com/paper/2410.18720/) (2024)
- [A geometric framework for momentum-based optimizers for low-rank training](https://mlsanity.com/paper/2506.17475/) (2025)
- [Harnessing Orthogonality to Train Low-Rank Neural Networks](https://mlsanity.com/paper/2401.08505/) (2024)
- [An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training](https://mlsanity.com/paper/2502.03006/) (2025)

## Abstract (arXiv)

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption. In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.