# Generative Pre-Training of Time-Series Data for Unsupervised Fault Detection in Semiconductor Manufacturing

arXiv: https://arxiv.org/abs/2309.11427 · HTML: https://mlsanity.com/paper/2309.11427/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 64 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection](https://mlsanity.com/paper/1607.00148/) (2016)
- [Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress](https://mlsanity.com/paper/2009.13807/) (2020)
- [Neural Machine Translation in Linear Time](https://mlsanity.com/paper/1610.10099/) (2016)
- [Anomaly Detection with Generative Adversarial Networks for Multivariate Time Series](https://mlsanity.com/paper/1809.04758/) (2018)
- [TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks](https://mlsanity.com/paper/2009.07769/) (2020)
- [An Explainable Artificial Intelligence Approach for Unsupervised Fault Detection and Diagnosis in Rotating Machinery](https://mlsanity.com/paper/2102.11848/) (2021)
- [Random Dilated Shapelet Transform: A New Approach for Time Series Shapelets](https://mlsanity.com/paper/2109.13514/) (2021)
- [Sintel: A Machine Learning Framework to Extract Insights from Signals](https://mlsanity.com/paper/2204.09108/) (2022)
- [RTFN: A Robust Temporal Feature Network for Time Series Classification](https://mlsanity.com/paper/2011.11829/) (2020)
- [MTV: Visual Analytics for Detecting, Investigating, and Annotating Anomalies in Multivariate Time Series](https://mlsanity.com/paper/2112.05734/) (2021)
- [AER: Auto-Encoder with Regression for Time Series Anomaly Detection](https://mlsanity.com/paper/2212.13558/) (2022)

## Built on by (most cited first, 1 of 1)

- [Diffuse to Detect: Generative Diffusion Models for Unsupervised IC Anomaly Detection](https://mlsanity.com/paper/2605.26468/) (2026)

## Abstract (arXiv)

This paper introduces TRACE-GPT, which stands for Time-seRies Anomaly-detection with Convolutional Embedding and Generative Pre-trained Transformers. TRACE-GPT is designed to pre-train univariate time-series sensor data and detect faults on unlabeled datasets in semiconductor manufacturing. In semiconductor industry, classifying abnormal time-series sensor data from normal data is important because it is directly related to wafer defect. However, small, unlabeled, and even mixed training data without enough anomalies make classification tasks difficult. In this research, we capture features of time-series data with temporal convolutional embedding and Generative Pre-trained Transformer (GPT) to classify abnormal sequences from normal sequences using cross entropy loss. We prove that our model shows better performance than previous unsupervised models with both an open dataset, the University of California Riverside (UCR) time-series classification archive, and the process log of our Chemical Vapor Deposition (CVD) equipment. Our model has the highest F1 score at Equal Error Rate (EER) across all datasets and is only 0.026 below the supervised state-of-the-art baseline on the open dataset.