# Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder

arXiv: https://arxiv.org/abs/2409.13606 · HTML: https://mlsanity.com/paper/2409.13606/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning](https://mlsanity.com/paper/2403.07376/) (2024)
- [Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology](https://mlsanity.com/paper/2402.14252/) (2024)
- [One-shot action recognition in challenging therapy scenarios](https://mlsanity.com/paper/2102.08997/) (2021)
- [DOMINO: A Dual-System for Multi-step Visual Language Reasoning](https://mlsanity.com/paper/2310.02804/) (2023)
- [Guided Weak Supervision for Action Recognition with Scarce Data to Assess Skills of Children with Autism](https://mlsanity.com/paper/1911.04140/) (2019)
- [A Context-Aware Computational Approach for Measuring Vocal Entrainment in Dyadic Conversations](https://mlsanity.com/paper/2211.03279/) (2022)

## Built on by (most cited first, 2 of 2)

- [Neurodiversity-Aware Multimodal Affective Computing for Neurodevelopmental Assessment: From Norm-Referenced Classification to Context-Sensitive Decision Support](https://mlsanity.com/paper/2610.04705/) (2026)
- [A Deterministic Evidence Layer for Vision-Language Autism Screening from Naturalistic Home Video](https://mlsanity.com/paper/2610.09217/) (2026)

## Abstract (arXiv)

Clinical videos in the context of Autism Spectrum Disorder are often long-form interactions between children and caregivers/clinical professionals, encompassing complex verbal and non-verbal behaviors. Objective analyses of these videos could provide clinicians and researchers with nuanced insights into the behavior of children with Autism Spectrum Disorder. Manually coding these videos is a time-consuming task and requires a high level of domain expertise. Hence, the ability to capture these interactions computationally can augment the manual effort and enable supporting the diagnostic procedure. In this work, we investigate the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions. We propose a unified methodology to combine multiple modalities by using large language models as reasoning agents. We evaluate their performance on two tasks with different information granularity: activity recognition and abnormal behavior detection. We find that the proposed multimodal pipeline provides robustness to modality-specific limitations and improves performance on the clinical video analysis compared to unimodal settings.