mlsanityData to 14 Oct 2026

When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting

arXiv 2606.19363 ↗First posted 10 Jun 2026cs.LGcited by 1

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

The deployment of Time-Series Foundation Models (TSFMs) in physical sciences is hindered by a critical trade-off: while these models encode rich, universal temporal dynamics, they suffer from severe distributional misalignment when applied zero-shot to specific scientific domains, and their computational cost prohibits deployment in edge-computing sensor networks. We address a fundamental challenge: How can we extract latent structural knowledge from misaligned foundation models (FM) to train lightweight, specialized forecasters? We propose Gated Uncertainty-Aware Routing for Distillation (Guard), a novel framework that reframes multiteacher distillation as an instance-wise decision process with two adaptive mechanisms: (1) a Contextual Router that dynamically selects the most relevant teacher based on local input statistics, exploiting complementarity across diverse foundation models; and (2) an Uncertainty-Gated Temperature mechanism that acts as a "circuit-breaker," automatically attenuating distillation strength when teacher confidence diverges from domain reality. We evaluate our proposed lightweight framework on four climate-critical domains: meteorology, ecosystem carbon flux, soil moisture, and energy grids. Our method significantly reduces RMSE relative to a fixed-weight multi-teacher distillation baseline, successfully distilling knowledge from pretrained FMs (teachers) even when they exhibit suboptimal zero-shot accuracy due to distribution shift between the original and target data domains. We demonstrate that these domain-misaligned teachers can still serve as critical correctives, outperforming the globally superior FMs on 28.5% of the hardest instances. Ultimately, this enables high-precision scientific forecasting suitable for resource-constrained edge deployment. Code is available at https://github.com/RupasreeDey/GUARD-KDD2026.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 13 of its 39 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

13 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition22 Feb
  2. 2024HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting10 Jan
  3. 2023Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning11 Jun

Other references · 10

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  5. 2021AdaRNN: Adaptive Learning and Forecasting of Time Seriescited by 121
  6. 2024A Comprehensive Survey of Deep Learning for Time Series Forecasting: Architectural Diversity and Open Challengescited by 47
  7. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 40
  8. 2025TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillationcited by 12
  9. 2025Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Paperscited by 5
  10. 2025Knowledge-Guided Masked Autoencoder with Linear Spectral Mixing and Spectral-Angle-Aware Reconstructioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Distilling Time Series Foundation Models for Efficient Forecasting
  2. 2026DynG-Diff: A State-Aware Dynamic Guidance Diffusion Framework for Probabilistic Time Series Forecasting
  3. 2026Aurora-X: Built for Extreme Time Series Forecasting
  4. 2026TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Models
  5. 2026AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting
  6. 2026GRAM: Correcting Frozen Time-Series Foundation Models via Graph-Retrieved Amplitude Memory
  7. 2026Distillation of Synthetic Data for Time Series Foundation Models
  8. 2025Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models