# Implementing NLPs in industrial process modeling: Addressing Categorical Variables

arXiv: https://arxiv.org/abs/2409.19097 · HTML: https://mlsanity.com/paper/2409.19097/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 65 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Talking About Large Language Models](https://mlsanity.com/paper/2212.03551/) (2022)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [Is Cosine-Similarity of Embeddings Really About Similarity?](https://mlsanity.com/paper/2403.05440/) (2024)
- [An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation](https://mlsanity.com/paper/1607.05368/) (2016)
- [The role of surrogate models in the development of digital twins of dynamic systems](https://mlsanity.com/paper/2001.09292/) (2020)
- [A Machine Learning Framework for Real-time Inverse Modeling and Multi-objective Process Optimization of Composites for Active Manufacturing Control](https://mlsanity.com/paper/2104.11342/) (2021)
- [Model-Free Adaptive Optimal Control of Episodic Fixed-Horizon Manufacturing Processes using Reinforcement Learning](https://mlsanity.com/paper/1809.06646/) (2018)
- [The Daunting Dilemma with Sentence Encoders: Success on Standard Benchmarks, Failure in Capturing Basic Semantic Properties](https://mlsanity.com/paper/2309.03747/) (2023)
- [Evaluating Pretrained Transformer Models for Entity Linking in Task-Oriented Dialog](https://mlsanity.com/paper/2112.08327/) (2021)
- [Integrating supervised and unsupervised learning approaches to unveil critical process inputs](https://mlsanity.com/paper/2405.07751/) (2024)

## Built on by (most cited first, 1 of 1)

- [Learning to Choose: An Empowerment-Guided Multi-Agent System with semantic communication for Adaptive Method Selection](https://mlsanity.com/paper/2605.30042/) (2026)

## Abstract (arXiv)

Important variables of processes are often categorical, i.e. names or labels representing, e.g. categories of inputs, or types of reactors or a sequence of steps. In this work, we use Natural Language Processing Models to derive embeddings of such inputs that represent their actual meaning, or reflect the "distances" between categories, i.e. how similar or dissimilar they are. This is a marked difference from the current standard practice of using binary, or one-hot encoding to replace categorical variables with sequences of ones and zeros. Combined with dimensionality reduction techniques, either linear such as Principal Component Analysis, or nonlinear such as Uniform Manifold Approximation and Projection, the proposed approach leads to a meaningful, low-dimensional feature space. The significance of obtaining meaningful embeddings is illustrated in the context of an industrial coating process for cutting tools that includes both numerical and categorical inputs. In this industrial process, subject matter expertise suggests that the categorical inputs are critical for determining the final outcome but this cannot be taken into account with the current state-of-the-art. The proposed approach enables feature importance which is a marked improvement compared to the current state-of-the-art in the encoding of categorical variables. The proposed approach is not limited to the case-study presented here and is suitable for applications with similar mix of categorical and numerical critical inputs.