# AdaptFly: Prompt-Guided Adaptation of Foundation Models for Low-Altitude UAV Networks

arXiv: https://arxiv.org/abs/2511.11720 · HTML: https://mlsanity.com/paper/2511.11720/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [A Tutorial on Environment-Aware Communications via Channel Knowledge Map for 6G](https://mlsanity.com/paper/2309.07460/) (2023)
- [VDD: Varied Drone Dataset for Semantic Segmentation](https://mlsanity.com/paper/2305.13608/) (2023)
- [Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction](https://mlsanity.com/paper/2303.09792/) (2023)
- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.08372/) (2024)
- [UAV3D: A Large-scale 3D Perception Benchmark for Unmanned Aerial Vehicles](https://mlsanity.com/paper/2410.11125/) (2024)
- [Semantic-Aware Sensing Information Transmission for Metaverse: A Contest Theoretic Approach](https://mlsanity.com/paper/2211.12783/) (2022)
- [A Unified Framework for Guiding Generative AI with Wireless Perception in Resource Constrained Mobile Edge Networks](https://mlsanity.com/paper/2309.01426/) (2023)
- [Generative AI-enabled Wireless Communications for Robust Low-Altitude Economy Networking](https://mlsanity.com/paper/2502.18118/) (2025)
- [Toward Realization of Low-Altitude Economy Networks: Core Architecture, Integrated Technologies, and Future Directions](https://mlsanity.com/paper/2504.21583/) (2025)
- [Multi-Objective Optimization for UAV Swarm-Assisted IoT with Virtual Antenna Arrays](https://mlsanity.com/paper/2308.01511/) (2023)
- [Physics-informed Generalizable Wireless Channel Modeling with Segmentation and Deep Learning: Fundamentals, Methodologies, and Challenges](https://mlsanity.com/paper/2401.01288/) (2024)
- [From Question to Exploration: Test-Time Adaptation in Semantic Segmentation?](https://mlsanity.com/paper/2310.05341/) (2023)
- [Embodied AI-empowered Low Altitude Economy: Integrated Sensing, Communications, Computation, and Control (ISC3)](https://mlsanity.com/paper/2412.19996/) (2024)
- [Reliable and Efficient Data Collection in UAV-based IoT Networks](https://mlsanity.com/paper/2311.05303/) (2023)
- [High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous Flight](https://mlsanity.com/paper/2503.15676/) (2025)
- [Satellite-Assisted Low-Altitude Economy Networking: Concepts, Applications, and Opportunities](https://mlsanity.com/paper/2505.04098/) (2025)
- [Co-Design of Sensing, Communications, and Control for Low-Altitude Wireless Networks](https://mlsanity.com/paper/2506.20970/) (2025)

## Abstract (arXiv)

Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on robust semantic segmentation as a foundational enabler for distributed sensing-communication-control co-design across heterogeneous agents within the network. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAVs cannot run gradient-based test-time adaptation, while resource-massive UAVs adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAVs, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAVs, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.