# Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing

arXiv: https://arxiv.org/abs/2408.05307 · HTML: https://mlsanity.com/paper/2408.05307/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 87 references that are in the map.
- First posted on arXiv 9 Aug 2024.

## Builds on (most cited first, 20 of 20)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Unified Deep Supervised Domain Adaptation and Generalization](https://mlsanity.com/paper/1709.10190/) (2017)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Audio-Adaptive Activity Recognition Across Video Domains](https://mlsanity.com/paper/2203.14240/) (2022)
- [Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimising Global Loss Functions](https://mlsanity.com/paper/1512.09272/) (2015)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Audio Spectrogram Representations for Processing with Convolutional Neural Networks](https://mlsanity.com/paper/1706.09559/) (2017)
- [XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification](https://mlsanity.com/paper/1709.00572/) (2017)
- [Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment](https://mlsanity.com/paper/2106.01689/) (2021)
- [Predicting Defects in Laser Powder Bed Fusion using in-situ Thermal Imaging Data and Machine Learning](https://mlsanity.com/paper/2112.11212/) (2021)
- [In-situ crack and keyhole pore detection in laser directed energy deposition through acoustic signal and deep learning](https://mlsanity.com/paper/2304.04598/) (2023)
- [In-situ process monitoring and adaptive quality enhancement in laser additive manufacturing: a critical review](https://mlsanity.com/paper/2404.13673/) (2024)
- [Multisensor fusion-based digital twin in additive manufacturing for in-situ quality monitoring and defect correction](https://mlsanity.com/paper/2304.05685/) (2023)
- [Limits of Transfer Learning](https://mlsanity.com/paper/2006.12694/) (2020)
- [Transferability analysis of data-driven additive manufacturing knowledge: a case study between powder bed fusion and directed energy deposition](https://mlsanity.com/paper/2309.06286/) (2023)
- [RNG: Reducing Multi-level Noise and Multi-grained Semantic Gap for Joint Multimodal Aspect-Sentiment Analysis](https://mlsanity.com/paper/2405.13059/) (2024)
- [Multimodal sensor fusion for real-time location-dependent defect detection in laser-directed energy deposition](https://mlsanity.com/paper/2305.13596/) (2023)

## Built on by (most cited first, 2 of 2)

- [Redundancy Analysis and Mitigation for Machine Learning-Based Process Monitoring of Additive Manufacturing](https://mlsanity.com/paper/2504.21317/) (2025)
- [Linking heterogeneous microstructure informatics with expert characterization knowledge through customized and hybrid vision-language representations for industrial qualification](https://mlsanity.com/paper/2508.20243/) (2025)

## Abstract (arXiv)

Various machine learning (ML)-based in-situ monitoring systems have been developed to detect anomalies and defects in laser additive manufacturing (LAM) processes. While multimodal fusion, which integrates data from visual, audio, and other modalities, can improve monitoring performance, it also increases hardware, computational, and operational costs. This paper introduces a cross-modality knowledge transfer (CMKT) methodology for LAM in-situ monitoring, which transfers knowledge from a source modality to a target modality. CMKT enhances the representativeness of the features extracted from the target modality, allowing the removal of source modality sensors during prediction. This paper proposes three CMKT methods: semantic alignment, fully supervised mapping, and semi-supervised mapping. The semantic alignment method establishes a shared encoded space between modalities to facilitate knowledge transfer. It employs a semantic alignment loss to align the distributions of identical groups (e.g., visual and audio defective groups) and a separation loss to distinguish different groups (e.g., visual defective and audio defect-free groups). The two mapping methods transfer knowledge by deriving features from one modality to another using fully supervised and semi-supervised learning approaches. In a case study for LAM in-situ defect detection, the proposed CMKT methods were compared with multimodal audio-visual fusion. The semantic alignment method achieved an accuracy of 98.6% while removing the audio modality during the prediction phase, which is comparable to the 98.2% accuracy obtained through multimodal fusion. Using explainable artificial intelligence, we discovered that semantic alignment CMKT can extract more representative features while reducing noise by leveraging the inherent correlations between modalities.