# Intelligent Sight and Sound: A Chronic Cancer Pain Dataset

arXiv: https://arxiv.org/abs/2204.04214 · HTML: https://mlsanity.com/paper/2204.04214/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 95 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://mlsanity.com/paper/1907.06724/) (2019)
- [Recurrent Convolutional Neural Network Regression for Continuous Pain Intensity Estimation in Video](https://mlsanity.com/paper/1605.00894/) (2016)
- [Multimodal Spatio-Temporal Deep Learning Approach for Neonatal Postoperative Pain Assessment](https://mlsanity.com/paper/2012.02175/) (2020)
- [Limitations and Biases in Facial Landmark Detection -- An Empirical Study on Older Adults with Dementia](https://mlsanity.com/paper/1905.07446/) (2019)
- [Automated Pain Detection from Facial Expressions using FACS: A Review](https://mlsanity.com/paper/1811.07988/) (2018)
- [Multi-Channel Neural Network for Assessing Neonatal Pain from Videos](https://mlsanity.com/paper/1908.09254/) (2019)
- [The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data](https://mlsanity.com/paper/2009.10589/) (2020)
- [Unobtrusive Pain Monitoring in Older Adults with Dementia using Pairwise and Contrastive Training](https://mlsanity.com/paper/2101.03251/) (2021)
- [Neonatal Pain Expression Recognition Using Transfer Learning](https://mlsanity.com/paper/1807.01631/) (2018)
- [Using Hankel Matrices for Dynamics-based Facial Emotion Recognition and Pain Detection](https://mlsanity.com/paper/1506.05001/) (2015)

## Built on by (most cited first, 3 of 3)

- [DDxT: Deep Generative Transformer Models for Differential Diagnosis](https://mlsanity.com/paper/2312.01242/) (2023)
- [EEG4Students: An Experimental Design for EEG Data Collection and Machine Learning Analysis](https://mlsanity.com/paper/2208.11743/) (2022)
- [A Generative Approach for Image Registration of Visible-Thermal (VT) Cancer Faces](https://mlsanity.com/paper/2308.12271/) (2023)

## Abstract (arXiv)

Cancer patients experience high rates of chronic pain throughout the treatment process. Assessing pain for this patient population is a vital component of psychological and functional well-being, as it can cause a rapid deterioration of quality of life. Existing work in facial pain detection often have deficiencies in labeling or methodology that prevent them from being clinically relevant. This paper introduces the first chronic cancer pain dataset, collected as part of the Intelligent Sight and Sound (ISS) clinical trial, guided by clinicians to help ensure that model findings yield clinically relevant results. The data collected to date consists of 29 patients, 509 smartphone videos, 189,999 frames, and self-reported affective and activity pain scores adopted from the Brief Pain Inventory (BPI). Using static images and multi-modal data to predict self-reported pain levels, early models show significant gaps between current methods available to predict pain today, with room for improvement. Due to the especially sensitive nature of the inherent Personally Identifiable Information (PII) of facial images, the dataset will be released under the guidance and control of the National Institutes of Health (NIH).