# Multimodal Spatio-Temporal Deep Learning Approach for Neonatal Postoperative Pain Assessment

arXiv: https://arxiv.org/abs/2012.02175 · HTML: https://mlsanity.com/paper/2012.02175/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [Review of multimodal machine learning approaches in healthcare](https://mlsanity.com/paper/2402.02460/) (2024)
- [CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection](https://mlsanity.com/paper/2506.01890/) (2025)
- [Intelligent Sight and Sound: A Chronic Cancer Pain Dataset](https://mlsanity.com/paper/2204.04214/) (2022)
- [Multimodal Machine Learning in Precision Health](https://mlsanity.com/paper/2204.04777/) (2022)
- [DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing Data](https://mlsanity.com/paper/2205.00701/) (2022)
- [Can AI detect pain and express pain empathy? A review from emotion recognition and a human-centered AI perspective](https://mlsanity.com/paper/2110.04249/) (2021)
- [Robust Neonatal Face Detection in Real-world Clinical Settings](https://mlsanity.com/paper/2204.00655/) (2022)
- [Pattern Recognition in Vital Signs Using Spectrograms](https://mlsanity.com/paper/2108.03168/) (2021)
- [BiDepth: A Bidirectional-Depth Neural Network for Spatio-Temporal Prediction](https://mlsanity.com/paper/2501.08411/) (2025)
- [Exploring Remote Photoplethysmography for Neonatal Pain Detection from Facial Videos](https://mlsanity.com/paper/2604.25680/) (2026)

## Abstract (arXiv)

The current practice for assessing neonatal postoperative pain relies on bedside caregivers. This practice is subjective, inconsistent, slow, and discontinuous. To develop a reliable medical interpretation, several automated approaches have been proposed to enhance the current practice. These approaches are unimodal and focus mainly on assessing neonatal procedural (acute) pain. As pain is a multimodal emotion that is often expressed through multiple modalities, the multimodal assessment of pain is necessary especially in case of postoperative (acute prolonged) pain. Additionally, spatio-temporal analysis is more stable over time and has been proven to be highly effective at minimizing misclassification errors. In this paper, we present a novel multimodal spatio-temporal approach that integrates visual and vocal signals and uses them for assessing neonatal postoperative pain. We conduct comprehensive experiments to investigate the effectiveness of the proposed approach. We compare the performance of the multimodal and unimodal postoperative pain assessment, and measure the impact of temporal information integration. The experimental results, on a real-world dataset, show that the proposed multimodal spatio-temporal approach achieves the highest AUC (0.87) and accuracy (79%), which are on average 6.67% and 6.33% higher than unimodal approaches. The results also show that the integration of temporal information markedly improves the performance as compared to the non-temporal approach as it captures changes in the pain dynamic. These results demonstrate that the proposed approach can be used as a viable alternative to manual assessment, which would tread a path toward fully automated pain monitoring in clinical settings, point-of-care testing, and homes.