# Foundation Models for Structural Health Monitoring

arXiv: https://arxiv.org/abs/2404.02944 · HTML: https://mlsanity.com/paper/2404.02944/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [A Survey of Deep Learning and Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2401.13912/) (2024)
- [DIME-FM: DIstilling Multimodal and Efficient Foundation Models](https://mlsanity.com/paper/2303.18232/) (2023)
- [Foundation Models for Natural Language Processing -- Pre-trained Language Models Integrating Media](https://mlsanity.com/paper/2302.08575/) (2023)
- [Exploring Scalable, Distributed Real-Time Anomaly Detection for Bridge Health Monitoring](https://mlsanity.com/paper/2203.02380/) (2022)

## Built on by (most cited first, 3 of 3)

- [Transformer self-attention encoder-decoder with multimodal deep learning for response time series forecasting and digital twin support in wind structural health monitoring](https://mlsanity.com/paper/2604.01712/) (2026)
- [Transformer-Based Approach to Optimal Sensor Placement for Structural Health Monitoring of Probe Cards](https://mlsanity.com/paper/2509.07603/) (2025)
- [Hybrid Spectro-Temporal Fusion Framework for Structural Health Monitoring](https://mlsanity.com/paper/2604.16589/) (2026)

## Abstract (arXiv)

Structural Health Monitoring (SHM) is a critical task for ensuring the safety and reliability of civil infrastructures, typically realized on bridges and viaducts by means of vibration monitoring. In this paper, we propose for the first time the use of Transformer neural networks, with a Masked Auto-Encoder architecture, as Foundation Models for SHM. We demonstrate the ability of these models to learn generalizable representations from multiple large datasets through self-supervised pre-training, which, coupled with task-specific fine-tuning, allows them to outperform state-of-the-art traditional methods on diverse tasks, including Anomaly Detection (AD) and Traffic Load Estimation (TLE). We then extensively explore model size versus accuracy trade-offs and experiment with Knowledge Distillation (KD) to improve the performance of smaller Transformers, enabling their embedding directly into the SHM edge nodes. We showcase the effectiveness of our foundation models using data from three operational viaducts. For AD, we achieve a near-perfect 99.9% accuracy with a monitoring time span of just 15 windows. In contrast, a state-of-the-art method based on Principal Component Analysis (PCA) obtains its first good result (95.03% accuracy), only considering 120 windows. On two different TLE tasks, our models obtain state-of-the-art performance on multiple evaluation metrics (R2 score, MAE% and MSE%). On the first benchmark, we achieve an R2 score of 0.97 and 0.90 for light and heavy vehicle traffic, respectively, while the best previous approach (a Random Forest) stops at 0.91 and 0.84. On the second one, we achieve an R2 score of 0.54 versus the 0.51 of the best competitor method, a Long-Short Term Memory network.