mlsanityData to 14 Oct 2026

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

arXiv 2405.20348 ↗First posted 24 May 2024physics.ao-ph cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 44 of its 63 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

44 of 63 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  8. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  9. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  12. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
32 more
  1. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  2. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  3. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  4. 2019Federated Learning with Personalization Layerscited by 541
  5. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  6. 2022Transformers in Time Series: A Surveycited by 497
  7. 2021Towards Personalized Federated Learningcited by 464
  8. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  9. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  10. 2023ClimaX: A foundation model for weather and climatecited by 337
  11. 2020WeatherBench: A benchmark dataset for data-driven weather forecastingcited by 275
  12. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 267
  13. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  14. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  15. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 174
  16. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 172
  17. 2022Forecasting Global Weather with Graph Neural Networkscited by 168
  18. 2022Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structurescited by 160
  19. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 148
  20. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  21. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 123
  22. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
  23. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  24. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  25. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  26. 2023LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Versioncited by 50
  27. 2023Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Datacited by 40
  28. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 34
  29. 2023W-MAE: Pre-trained weather model with masked autoencoder for multi-variable weather forecastingcited by 29
  30. 2023TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regressioncited by 12
  31. 2024Anomaly Detection Based on Isolation Mechanisms: A Surveycited by 10
  32. 2023Self-Supervised Pre-Training for Precipitation Post-Processorcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 8.

15 papers.

Sort:

By area:

  1. 2024Federated Foundation Models on Heterogeneous Time Seriescited by 13
  2. 2025FeDaL: Federated Dataset Learning for General Time Series Foundation Modelscited by 10
  3. 2024Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approachcited by 9
  4. 2024Federated Distillation for Medical Image Classification: Towards Trustworthy Computer-Aided Diagnosiscited by 4
  5. 2025Foundation Models for Environmental Science: A Survey of Emerging Frontierscited by 4
  6. 2025FairTune: A Bias-Aware Fine-Tuning Framework Towards Fair Heart Rate Prediction from PPGcited by 2
  7. 2026Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approachcited by 2
  8. 2024A Comparative Study of Pruning Methods in Transformer-based Time Series Forecastingcited by 1
  9. 2026Spectral-Aware Text-to-Time Series Generation with Billion-Scale Multimodal Meteorological Datacited by 1
  10. 2026Discrete Prototypical Memories for Federated Time Series Foundation Modelscited by 1
  11. 2026PiXTime: A Model for Federated Time Series Forecasting with Heterogeneous Data across Nodescited by 0
  12. 2026Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusioncited by 0
3 more
  1. 2026Federated Weather Modeling on Sensor Datacited by 0
  2. 2026FM2: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imagingcited by 0
  3. 2026Lethe: How Hard Is It to Forget? A Benchmark for Federated Unlearning in Medical Imagingcited by 0