# A Video-based End-to-end Pipeline for Non-nutritive Sucking Action Recognition and Segmentation in Young Infants

arXiv: https://arxiv.org/abs/2303.16867 · HTML: https://mlsanity.com/paper/2303.16867/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [InfAnFace: Bridging the infant-adult domain gap in facial landmark estimation in the wild](https://mlsanity.com/paper/2110.08935/) (2021)
- [Infant Contact-less Non-Nutritive Sucking Pattern Quantification via Facial Gesture Analysis](https://mlsanity.com/paper/1906.01821/) (2019)

## Built on by (most cited first, 3 of 3)

- [Multiple Toddler Tracking in Indoor Videos](https://mlsanity.com/paper/2311.17656/) (2023)
- [Subtle Signals: Video-based Detection of Infant Non-nutritive Sucking as a Neurodevelopmental Cue](https://mlsanity.com/paper/2310.16138/) (2023)
- [Challenges in Video-Based Infant Action Recognition: A Critical Examination of the State of the Art](https://mlsanity.com/paper/2311.12300/) (2023)

## Abstract (arXiv)

We present an end-to-end computer vision pipeline to detect non-nutritive sucking (NNS) -- an infant sucking pattern with no nutrition delivered -- as a potential biomarker for developmental delays, using off-the-shelf baby monitor video footage. One barrier to clinical (or algorithmic) assessment of NNS stems from its sparsity, requiring experts to wade through hours of footage to find minutes of relevant activity. Our NNS activity segmentation algorithm solves this problem by identifying periods of NNS with high certainty -- up to 94.0% average precision and 84.9% average recall across 30 heterogeneous 60 s clips, drawn from our manually annotated NNS clinical in-crib dataset of 183 hours of overnight baby monitor footage from 19 infants. Our method is based on an underlying NNS action recognition algorithm, which uses spatiotemporal deep learning networks and infant-specific pose estimation, achieving 94.9% accuracy in binary classification of 960 2.5 s balanced NNS vs. non-NNS clips. Tested on our second, independent, and public NNS in-the-wild dataset, NNS recognition classification reaches 92.3% accuracy, and NNS segmentation achieves 90.8% precision and 84.2% recall.