# BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles

arXiv: https://arxiv.org/abs/2310.17742 · HTML: https://mlsanity.com/paper/2310.17742/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [ProfileSR-GAN: A GAN based Super-Resolution Method for Generating High-Resolution Load Profiles](https://mlsanity.com/paper/2107.09523/) (2021)
- [Load Profile Inpainting for Missing Load Data Restoration and Baseline Estimation](https://mlsanity.com/paper/2211.16332/) (2022)
- [A Modified Sequence-to-point HVAC Load Disaggregation Algorithm](https://mlsanity.com/paper/2212.04886/) (2022)

## Built on by (most cited first, 2 of 2)

- [Applying Fine-Tuned LLMs for Reducing Data Needs in Load Profile Analysis](https://mlsanity.com/paper/2406.02479/) (2024)
- [LAMPER: LanguAge Model and Prompt EngineeRing for zero-shot time series classification](https://mlsanity.com/paper/2403.15875/) (2024)

## Abstract (arXiv)

Inspired by the success of the Transformer model in natural language processing and computer vision, this paper introduces BERT-PIN, a Bidirectional Encoder Representations from Transformers (BERT) powered Profile Inpainting Network. BERT-PIN recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. To adopt a standard Transformer model structure for profile inpainting, we segment the load and temperature profiles into line segments, treating each segment as a word and the entire profile as a sentence. We incorporate a top candidates selection process in BERT-PIN, enabling it to produce a sequence of probability distributions, based on which users can generate multiple plausible imputed data sets, each reflecting different confidence levels. We develop and evaluate BERT-PIN using real-world dataset for two applications: multiple MDSs recovery and demand response baseline estimation. Simulation results show that BERT-PIN outperforms the existing methods in accuracy while is capable of restoring multiple MDSs within a longer window. BERT-PIN, served as a pre-trained model, can be fine-tuned for conducting many downstream tasks, such as classification and super resolution.