# Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM

arXiv: https://arxiv.org/abs/2410.12861 · HTML: https://mlsanity.com/paper/2410.12861/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 22 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Builds on (most cited first, 4 of 4)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural NILM: Deep Neural Networks Applied to Energy Disaggregation](https://mlsanity.com/paper/1507.06594/) (2015)
- [Sequence-to-point learning with neural networks for nonintrusive load monitoring](https://mlsanity.com/paper/1612.09106/) (2016)
- [Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation](https://mlsanity.com/paper/2203.16537/) (2022)

## Abstract (arXiv)

Transformers have demonstrated exceptional performance across various domains due to their self-attention mechanism, which captures complex relationships in data. However, training on smaller datasets poses challenges, as standard attention mechanisms can over-smooth attention scores and overly prioritize intra-token relationships, reducing the capture of meaningful inter-token dependencies critical for tasks like Non-Intrusive Load Monitoring (NILM). To address this, we propose a novel transformer architecture with two key innovations: inter-token relation enhancement and dynamic temperature tuning. The inter-token relation enhancement mechanism removes diagonal entries in the similarity matrix to improve attention focus on inter-token relations. The dynamic temperature tuning mechanism, a learnable parameter, adapts attention sharpness during training, preventing over-smoothing and enhancing sensitivity to token relationships. We validate our method on the REDD dataset and show that it outperforms the original transformer and state-of-the-art models by 10-15% in F1 score across various appliance types, demonstrating its efficacy for training on smaller datasets.