# Introducing Hybrid Modeling with Time-series-Transformers: A Comparative Study of Series and Parallel Approach in Batch Crystallization

arXiv: https://arxiv.org/abs/2308.05749 · HTML: https://mlsanity.com/paper/2308.05749/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 78 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Sparse identification of nonlinear dynamics for model predictive control in the low-data limit](https://mlsanity.com/paper/1711.05501/) (2017)
- [A physics-informed variational DeepONet for predicting the crack path in brittle materials](https://mlsanity.com/paper/2108.06905/) (2021)
- [Physics-informed neural networks for inverse problems in supersonic flows](https://mlsanity.com/paper/2202.11821/) (2022)
- [The Depth-to-Width Interplay in Self-Attention](https://mlsanity.com/paper/2006.12467/) (2020)
- [Learning from flowsheets: A generative transformer model for autocompletion of flowsheets](https://mlsanity.com/paper/2208.00859/) (2022)
- [Lipschitz Constrained Parameter Initialization for Deep Transformers](https://mlsanity.com/paper/1911.03179/) (2019)
- [Data-Centric Engineering: integrating simulation, machine learning and statistics. Challenges and Opportunities](https://mlsanity.com/paper/2111.06223/) (2021)
- [Machine Learning Benchmarks for the Classification of Equivalent Circuit Models from Electrochemical Impedance Spectra](https://mlsanity.com/paper/2302.03362/) (2023)
- [CrystalGPT: Enhancing system-to-system transferability in crystallization prediction and control using time-series-transformers](https://mlsanity.com/paper/2306.03099/) (2023)
- [Require Process Control? LSTMc is all you need!](https://mlsanity.com/paper/2306.07510/) (2023)

## Built on by (most cited first, 5 of 5)

- [Physics-Informed Neural Networks for Dynamic Process Operations with Limited Physical Knowledge and Data](https://mlsanity.com/paper/2406.01528/) (2024)
- [A Mechanistic Analysis of Transformers for Dynamical Systems](https://mlsanity.com/paper/2512.21113/) (2025)
- [Modern, Efficient, and Differentiable Transport Equation Models using JAX: Applications to Population Balance Equations](https://mlsanity.com/paper/2411.00742/) (2024)
- [BITS for GAPS: Bayesian Information-Theoretic Sampling for hierarchical GAussian Process Surrogates](https://mlsanity.com/paper/2511.16815/) (2025)
- [Can Transformers overcome the lack of data in the simulation of history-dependent flows?](https://mlsanity.com/paper/2512.16305/) (2025)

## Abstract (arXiv)

Most existing digital twins rely on data-driven black-box models, predominantly using deep neural recurrent, and convolutional neural networks (DNNs, RNNs, and CNNs) to capture the dynamics of chemical systems. However, these models have not seen the light of day, given the hesitance of directly deploying a black-box tool in practice due to safety and operational issues. To tackle this conundrum, hybrid models combining first-principles physics-based dynamics with machine learning (ML) models have increased in popularity as they are considered a 'best of both worlds' approach. That said, existing simple DNN models are not adept at long-term time-series predictions and utilizing contextual information on the trajectory of the process dynamics. Recently, attention-based time-series transformers (TSTs) that leverage multi-headed attention mechanism and positional encoding to capture long-term and short-term changes in process states have shown high predictive performance. Thus, a first-of-a-kind, TST-based hybrid framework has been developed for batch crystallization, demonstrating improved accuracy and interpretability compared to traditional black-box models. Specifically, two different configurations (i.e., series and parallel) of TST-based hybrid models are constructed and compared, which show a normalized-mean-square-error (NMSE) in the range of [10, 50]×10-4 and an R2 value over 0.99. Given the growing adoption of digital twins, next-generation attention-based hybrid models are expected to play a crucial role in shaping the future of chemical manufacturing.