# A comparative study of transformer models and recurrent neural networks for path-dependent composite materials

arXiv: https://arxiv.org/abs/2603.00092 · HTML: https://mlsanity.com/paper/2603.00092/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 56 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Neural Architecture Search with Bayesian Optimisation and Optimal Transport](https://mlsanity.com/paper/1802.07191/) (2018)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [An FE-DMN method for the multiscale analysis of fiber reinforced plastic components](https://mlsanity.com/paper/2103.08253/) (2021)
- [A micromechanics-based recurrent neural networks model for path-dependent cyclic deformation of short fiber composites](https://mlsanity.com/paper/2210.00842/) (2022)
- [A Neural Network Transformer Model for Composite Microstructure Homogenization](https://mlsanity.com/paper/2304.07877/) (2023)
- [Recurrent neural networks and transfer learning for elasto-plasticity in woven composites](https://mlsanity.com/paper/2311.13434/) (2023)
- [A novel Taguchi-based approach for optimizing neural network architectures: application to elastic short fiber composites](https://mlsanity.com/paper/2407.19802/) (2024)
- [Test-time data augmentation: improving predictions of recurrent neural network models of composites](https://mlsanity.com/paper/2409.02478/) (2024)

## Abstract (arXiv)

Accurate modeling of Short Fiber Reinforced Composites (SFRCs) remains computationally expensive for full-field simulations. Data-driven surrogate models using Artificial Neural Networks (ANNs) have been proposed as an efficient alternative to numerical modeling, where Recurrent Neural Networks (RNNs) are increasingly being used for path-dependent multiscale modeling by predicting the homogenized response of a Representative Volume Element (RVE). However, recently, transformer models have been developed and they offer scalability and efficient parallelization, yet have not been systematically compared with RNNs in this field. In this study, we perform a systematic comparison between RNNs and transformer models trained on sequences of homogenized response of SFRC RVEs. We study the effect on two types of hyperparameters, namely architectural hyperparameters (such as the number of GRU layers, hidden size, number of attention heads, and encoder blocks) and training hyperparameters (such as learning rate and batch size). Both sets of hyperparameters are tuned using Bayesian optimization. We then analyze scaling laws with respect to dataset size and inference accuracy in interpolation and extrapolation regimes. The results show that while transformer models remain competitive in terms of accuracy on large datasets, the RNNs demonstrate better accuracy on small datasets and show better extrapolation performance. Furthermore, under extrapolation, there is a clear difference, where the RNN remains accurate, while the transformer model performs poorly. On the other hand, the transformer model is 7 times faster at inference, requiring 0.5 ms per prediction compared to the 3.5 ms per prediction for the RNN model.