# When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting

arXiv: https://arxiv.org/abs/2606.19363 · HTML: https://mlsanity.com/paper/2606.19363/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [AdaRNN: Adaptive Learning and Forecasting of Time Series](https://mlsanity.com/paper/2108.04443/) (2021)
- [A Comprehensive Survey of Deep Learning for Time Series Forecasting: Architectural Diversity and Open Challenges](https://mlsanity.com/paper/2411.05793/) (2024)
- [One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers](https://mlsanity.com/paper/2106.01023/) (2021)
- [Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning](https://mlsanity.com/paper/2306.06634/) (2023)
- [Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition](https://mlsanity.com/paper/2502.18510/) (2025)
- [TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillation](https://mlsanity.com/paper/2502.15016/) (2025)
- [HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting](https://mlsanity.com/paper/2401.05012/) (2024)
- [Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Papers](https://mlsanity.com/paper/2511.21739/) (2025)
- [Knowledge-Guided Masked Autoencoder with Linear Spectral Mixing and Spectral-Angle-Aware Reconstruction](https://mlsanity.com/paper/2512.12445/) (2025)

## Built on by (most cited first, 1 of 1)

- [SGA: Uncertainty Quantification for Multi-Step Forecasting in Time Series Foundation Models](https://mlsanity.com/paper/2609.28582/) (2026)

## Abstract (arXiv)

The deployment of Time-Series Foundation Models (TSFMs) in physical sciences is hindered by a critical trade-off: while these models encode rich, universal temporal dynamics, they suffer from severe distributional misalignment when applied zero-shot to specific scientific domains, and their computational cost prohibits deployment in edge-computing sensor networks. We address a fundamental challenge: How can we extract latent structural knowledge from misaligned foundation models (FM) to train lightweight, specialized forecasters? We propose Gated Uncertainty-Aware Routing for Distillation (Guard), a novel framework that reframes multiteacher distillation as an instance-wise decision process with two adaptive mechanisms: (1) a Contextual Router that dynamically selects the most relevant teacher based on local input statistics, exploiting complementarity across diverse foundation models; and (2) an Uncertainty-Gated Temperature mechanism that acts as a "circuit-breaker," automatically attenuating distillation strength when teacher confidence diverges from domain reality. We evaluate our proposed lightweight framework on four climate-critical domains: meteorology, ecosystem carbon flux, soil moisture, and energy grids. Our method significantly reduces RMSE relative to a fixed-weight multi-teacher distillation baseline, successfully distilling knowledge from pretrained FMs (teachers) even when they exhibit suboptimal zero-shot accuracy due to distribution shift between the original and target data domains. We demonstrate that these domain-misaligned teachers can still serve as critical correctives, outperforming the globally superior FMs on 28.5% of the hardest instances. Ultimately, this enables high-precision scientific forecasting suitable for resource-constrained edge deployment. Code is available at https://github.com/RupasreeDey/GUARD-KDD2026.