# LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Version

arXiv: https://arxiv.org/abs/2302.12721 · HTML: https://mlsanity.com/paper/2302.12721/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 71 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 30 of 42)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [Benchmarking Multivariate Time Series Classification Algorithms](https://mlsanity.com/paper/2007.13156/) (2020)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining](https://mlsanity.com/paper/2006.09191/) (2020)
- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [A Time Series Forest for Classification and Feature Extraction](https://mlsanity.com/paper/1302.2277/) (2013)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [An Overview of Neural Network Compression](https://mlsanity.com/paper/2006.03669/) (2020)
- [Towards Spatio-Temporal Aware Traffic Time Series Forecasting--Full Version](https://mlsanity.com/paper/2203.15737/) (2022)
- [Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Version](https://mlsanity.com/paper/2204.13767/) (2022)
- [The Canonical Interval Forest (CIF) Classifier for Time Series Classification](https://mlsanity.com/paper/2008.09172/) (2020)
- [AutoCTS: Automated Correlated Time Series Forecasting -- Extended Version](https://mlsanity.com/paper/2112.11174/) (2021)
- [Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces](https://mlsanity.com/paper/2111.01186/) (2021)

## Built on by (most cited first, 30 of 50)

- [CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patching](https://mlsanity.com/paper/2410.12261/) (2024)
- [Spatial-Temporal Large Language Model for Traffic Prediction](https://mlsanity.com/paper/2401.10134/) (2024)
- [TAB: Unified Benchmarking of Time Series Anomaly Detection Methods](https://mlsanity.com/paper/2506.18046/) (2025)
- [A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Data](https://mlsanity.com/paper/2404.14999/) (2024)
- [LightPath: Lightweight and Scalable Path Representation Learning](https://mlsanity.com/paper/2307.10171/) (2023)
- [Air Quality Prediction with Physics-Guided Dual Neural ODEs in Open Systems](https://mlsanity.com/paper/2410.19892/) (2024)
- [TimeRAG: BOOSTING LLM Time Series Forecasting via Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.16643/) (2024)
- [Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillation](https://mlsanity.com/paper/2505.02138/) (2025)
- [xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories](https://mlsanity.com/paper/2410.16928/) (2024)
- [Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models](https://mlsanity.com/paper/2405.20348/) (2024)
- [TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillation](https://mlsanity.com/paper/2502.15016/) (2025)
- [Learning Soft Sparse Shapes for Efficient Time-Series Classification](https://mlsanity.com/paper/2505.06892/) (2025)
- [MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters](https://mlsanity.com/paper/2410.02081/) (2024)
- [Vision-Enhanced Time Series Forecasting via Latent Diffusion Models](https://mlsanity.com/paper/2502.14887/) (2025)
- [HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting](https://mlsanity.com/paper/2401.05012/) (2024)
- [LLM-Mixer: Multiscale Mixing in LLMs for Time Series Forecasting](https://mlsanity.com/paper/2410.11674/) (2024)
- [K2VAE: A Koopman-Kalman Enhanced Variational AutoEncoder for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2505.23017/) (2025)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [QCore: Data-Efficient, On-Device Continual Calibration for Quantized Models -- Extended Version](https://mlsanity.com/paper/2404.13990/) (2024)
- [FTS: A Framework to Find a Faithful TimeSieve](https://mlsanity.com/paper/2405.19647/) (2024)
- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [Orca: Ocean Significant Wave Height Estimation with Spatio-temporally Aware Large Language Models](https://mlsanity.com/paper/2407.20053/) (2024)
- [PPG-Distill: Efficient Photoplethysmography Signals Analysis via Foundation Model Distillation](https://mlsanity.com/paper/2509.19215/) (2025)
- [WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge](https://mlsanity.com/paper/2312.09507/) (2023)
- [DELPHYNE: A Pre-Trained Model for General and Financial Time Series](https://mlsanity.com/paper/2506.06288/) (2025)
- [Towards Lightweight Time Series Forecasting: a Patch-wise Transformer with Weak Data Enriching](https://mlsanity.com/paper/2501.10448/) (2025)
- [Large Language models for Time Series Analysis: Techniques, Applications, and Challenges](https://mlsanity.com/paper/2506.11040/) (2025)
- [Augur: Modeling Covariate Causal Associations in Time Series via Large Language Models](https://mlsanity.com/paper/2510.07858/) (2025)
- [LTSM-Bundle: A Toolbox and Benchmark on Large Language Models for Time Series Forecasting](https://mlsanity.com/paper/2406.14045/) (2024)
- [Data Driven Decision Making with Time Series and Spatio-temporal Data](https://mlsanity.com/paper/2503.08473/) (2025)

## Abstract (arXiv)

Due to the sweeping digitalization of processes, increasingly vast amounts of time series data are being produced. Accurate classification of such time series facilitates decision making in multiple domains. State-of-the-art classification accuracy is often achieved by ensemble learning where results are synthesized from multiple base models. This characteristic implies that ensemble learning needs substantial computing resources, preventing their use in resource-limited environments, such as in edge devices. To extend the applicability of ensemble learning, we propose the LightTS framework that compresses large ensembles into lightweight models while ensuring competitive accuracy. First, we propose adaptive ensemble distillation that assigns adaptive weights to different base models such that their varying classification capabilities contribute purposefully to the training of the lightweight model. Second, we propose means of identifying Pareto optimal settings w.r.t. model accuracy and model size, thus enabling users with a space budget to select the most accurate lightweight model. We report on experiments using 128 real-world time series sets and different types of base models that justify key decisions in the design of LightTS and provide evidence that LightTS is able to outperform competitors.