# Soft Sensing Transformer: Hundreds of Sensors are Worth a Single Word

arXiv: https://arxiv.org/abs/2111.05973 · HTML: https://mlsanity.com/paper/2111.05973/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Auto-encoder based Model for High-dimensional Imbalanced Industrial Data](https://mlsanity.com/paper/2108.02083/) (2021)

## Built on by (most cited first, 5 of 5)

- [GraSSNet: Graph Soft Sensing Neural Networks](https://mlsanity.com/paper/2111.06980/) (2021)
- [Soft-Sensing ConFormer: A Curriculum Learning-based Convolutional Transformer](https://mlsanity.com/paper/2111.06981/) (2021)
- [Soft Sensing Model Visualization: Fine-tuning Neural Network from What Model Learned](https://mlsanity.com/paper/2111.06982/) (2021)
- [Soft Sensing Regression Model: from Sensor to Wafer Metrology Forecasting](https://mlsanity.com/paper/2301.08974/) (2023)
- [Self-supervised Learning Method Using Transformer for Multi-dimensional Sensor Data Processing](https://mlsanity.com/paper/2505.21918/) (2025)

## Abstract (arXiv)

With the rapid development of AI technology in recent years, there have been many studies with deep learning models in soft sensing area. However, the models have become more complex, yet, the data sets remain limited: researchers are fitting million-parameter models with hundreds of data samples, which is insufficient to exercise the effectiveness of their models and thus often fail to perform when implemented in industrial applications. To solve this long-lasting problem, we are providing large scale, high dimensional time series manufacturing sensor data from Seagate Technology to the public. We demonstrate the challenges and effectiveness of modeling industrial big data by a Soft Sensing Transformer model on these data sets. Transformer is used because, it has outperformed state-of-the-art techniques in Natural Language Processing, and since then has also performed well in the direct application to computer vision without introduction of image-specific inductive biases. We observe the similarity of a sentence structure to the sensor readings and process the multi-variable sensor readings in a time series in a similar manner of sentences in natural language. The high-dimensional time-series data is formatted into the same shape of embedded sentences and fed into the transformer model. The results show that transformer model outperforms the benchmark models in soft sensing field based on auto-encoder and long short-term memory (LSTM) models. To the best of our knowledge, we are the first team in academia or industry to benchmark the performance of original transformer model with large-scale numerical soft sensing data.