# Hybrid Attention Model Using Feature Decomposition and Knowledge Distillation for Glucose Forecasting

arXiv: https://arxiv.org/abs/2411.10703 · HTML: https://mlsanity.com/paper/2411.10703/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 56 references that are in the map.
- First posted on arXiv 16 Nov 2024.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables](https://mlsanity.com/paper/2402.19072/) (2024)
- [Convolutional Recurrent Neural Networks for Glucose Prediction](https://mlsanity.com/paper/1807.03043/) (2018)
- [Gluformer: Transformer-Based Personalized Glucose Forecasting with Uncertainty Quantification](https://mlsanity.com/paper/2209.04526/) (2022)
- [Multivariate Variational Mode Decomposition](https://mlsanity.com/paper/1907.04509/) (2019)
- [Stacked LSTM Based Deep Recurrent Neural Network with Kalman Smoothing for Blood Glucose Prediction](https://mlsanity.com/paper/2101.06850/) (2021)
- [Deep Personalized Glucose Level Forecasting Using Attention-based Recurrent Neural Networks](https://mlsanity.com/paper/2106.00884/) (2021)
- [Self-Supervised Learning and Opportunistic Inference for Continuous Monitoring of Freezing of Gait in Parkinson's Disease](https://mlsanity.com/paper/2410.21326/) (2024)
- [An Edge-based WiFi Fingerprinting Indoor Localization Using Convolutional Neural Network and Convolutional Auto-Encoder](https://mlsanity.com/paper/2303.03699/) (2023)

## Built on by (most cited first, 7 of 7)

- [AttenGluco: Multimodal Transformer-Based Blood Glucose Forecasting on AI-READI Dataset](https://mlsanity.com/paper/2502.09919/) (2025)
- [LLM-Powered Prediction of Hyperglycemia and Discovery of Behavioral Treatment Pathways from Wearables and Diet](https://mlsanity.com/paper/2503.03935/) (2025)
- [GlyRAG: Context-Aware Retrieval-Augmented Framework for Blood Glucose Forecasting](https://mlsanity.com/paper/2601.05353/) (2026)
- [CLAD-Net: Continual Activity Recognition in Multi-Sensor Wearable Systems](https://mlsanity.com/paper/2509.23077/) (2025)
- [From Prediction to Intervention: Personalized Meal-Level Glucose Regulation via an LLM Agent](https://mlsanity.com/paper/2608.13581/) (2026)
- [GluMind: Multimodal Parallel Attention and Knowledge Retention for Robust Cross-Population Blood Glucose Forecasting](https://mlsanity.com/paper/2509.18457/) (2025)
- [Label-Efficient Bilateral Attention for Parkinson's Disease Screening from Wrist-Worn IMU Signals](https://mlsanity.com/paper/2604.18372/) (2026)

## Abstract (arXiv)

The availability of continuous glucose monitors as over-the-counter commodities have created a unique opportunity to monitor a person's blood glucose levels, forecast blood glucose trajectories and provide automated interventions to prevent devastating chronic complications that arise from poor glucose control. However, forecasting blood glucose levels is challenging because blood glucose changes consistently in response to food intake, medication intake, physical activity, sleep, and stress. It is particularly difficult to accurately predict BGL from multimodal and irregularly sampled data and over long prediction horizons. Furthermore, these forecasting models must operate in real-time on edge devices to provide in-the-moment interventions. To address these challenges, we propose GlucoNet, an AI-powered sensor system for continuously monitoring behavioral and physiological health and robust forecasting of blood glucose patterns. GlucoNet devises a feature decomposition-based transformer model that incorporates patients' behavioral and physiological data and transforms sparse and irregular patient data (e.g., diet and medication intake data) into continuous features using a mathematical model, facilitating better integration with the BGL data. Given the non-linear and non-stationary nature of BG signals, we propose a decomposition method to extract both low and high-frequency components from the BGL signals, thus providing accurate forecasting. To reduce the computational complexity, we also propose to employ knowledge distillation to compress the transformer model. GlucoNet achieves a 60% improvement in RMSE and a 21% reduction in the number of parameters, improving RMSE and MAE by 51% and 57%, using data obtained involving 12 participants with T1-Diabetes. These results underscore GlucoNet's potential as a compact and reliable tool for real-world diabetes prevention and management.