# Transformer-based few-shot learning for modeling Electricity Consumption Profiles with minimal data across thousands of domains

arXiv: https://arxiv.org/abs/2408.08399 · HTML: https://mlsanity.com/paper/2408.08399/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 72 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [Few-shot Image Generation via Cross-domain Correspondence](https://mlsanity.com/paper/2104.06820/) (2021)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Free Lunch for Few-shot Learning: Distribution Calibration](https://mlsanity.com/paper/2101.06395/) (2021)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Transferring GANs: generating images from limited data](https://mlsanity.com/paper/1805.01677/) (2018)
- [Few-shot Image Generation with Elastic Weight Consolidation](https://mlsanity.com/paper/2012.02780/) (2020)
- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction](https://mlsanity.com/paper/1906.03631/) (2019)
- [Learning feed-forward one-shot learners](https://mlsanity.com/paper/1606.05233/) (2016)
- [DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation](https://mlsanity.com/paper/2002.11603/) (2020)
- [A Closer Look at Few-shot Image Generation](https://mlsanity.com/paper/2205.03805/) (2022)
- [Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning](https://mlsanity.com/paper/2211.03044/) (2022)
- [Few-shot Image Generation via Adaptation-Aware Kernel Modulation](https://mlsanity.com/paper/2210.16559/) (2022)
- [Recurrent Mixture Density Network for Spatiotemporal Visual Attention](https://mlsanity.com/paper/1603.08199/) (2016)
- [Privacy-Preserving Synthetic Data Generation for Recommendation Systems](https://mlsanity.com/paper/2209.13133/) (2022)
- [AffinityNet: semi-supervised few-shot learning for disease type prediction](https://mlsanity.com/paper/1805.08905/) (2018)
- [Graph Mixture Density Networks](https://mlsanity.com/paper/2012.03085/) (2020)
- [Smart Home Energy Management: VAE-GAN synthetic dataset generator and Q-learning](https://mlsanity.com/paper/2305.08885/) (2023)
- [Customized Load Profiles Synthesis for Electricity Customers Based on Conditional Diffusion Models](https://mlsanity.com/paper/2304.12076/) (2023)
- [EnergyDiff: Universal Time-Series Energy Data Generation using Diffusion Models](https://mlsanity.com/paper/2407.13538/) (2024)

## Abstract (arXiv)

Electricity Consumption Profiles (ECPs) are crucial for operating and planning power distribution systems, especially with the increasing number of low-carbon technologies such as solar panels and electric vehicles. Traditional ECP modeling methods typically assume the availability of sufficient ECP data. However, in practice, the accessibility of ECP data is limited due to privacy issues or the absence of metering devices. Few-shot learning (FSL) has emerged as a promising solution for ECP modeling in data-scarce scenarios. Nevertheless, standard FSL methods, such as those used for images, are unsuitable for ECP modeling because (1) these methods usually assume several source domains with sufficient data and several target domains. However, in the context of ECP modeling, there may be thousands of source domains, e.g., households with a moderate amount of data, and thousands of target domains, e.g., households that ECP are required to be modeled. (2) Standard FSL methods usually involve cumbersome knowledge transfer mechanisms, such as pre-training and fine-tuning. To address these limitations, this paper proposes a novel FSL framework that integrates Transformers with Gaussian Mixture Models (GMMs) for ECP modeling. The proposed approach is fine-tuning-free, computationally efficient, and robust even with extremely limited data. Results show that our method can accurately restore the complex ECP distribution with a minimal amount of ECP data (e.g., only 1.6% of the complete domain dataset) and outperforms state-of-the-art time series modeling methods in the context of ECP modeling.