# Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement

arXiv: https://arxiv.org/abs/2411.09413 · HTML: https://mlsanity.com/paper/2411.09413/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 93 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2406.15704/) (2024)
- [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [Large Language Models to Identify Social Determinants of Health in Electronic Health Records](https://mlsanity.com/paper/2308.06354/) (2023)
- [Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models](https://mlsanity.com/paper/2406.10507/) (2024)
- [Large language models in medicine: the potentials and pitfalls](https://mlsanity.com/paper/2309.00087/) (2023)
- [Vision-Based Activity Recognition in Children with Autism-Related Behaviors](https://mlsanity.com/paper/2208.04206/) (2022)
- [Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition](https://mlsanity.com/paper/2406.02554/) (2024)

## Built on by (most cited first, 1 of 1)

- [Neurodiversity-Aware Multimodal Affective Computing for Neurodevelopmental Assessment: From Norm-Referenced Classification to Context-Sensitive Decision Support](https://mlsanity.com/paper/2610.04705/) (2026)

## Abstract (arXiv)

The early diagnosis of autism spectrum disorder (ASD) is critically dependent on systematic observation and analysis of children's social behaviors. While current methodologies predominantly utilize supervised learning approaches, their clinical adoption faces two principal limitations: insufficient ASD diagnostic samples and inadequate interpretability of the detection outcomes. This paper presents a novel zero-shot ASD detection framework based on script-centric behavioral understanding with emotional enhancement, which is designed to overcome the aforementioned clinical constraints. The proposed pipeline automatically converts audio-visual data into structured behavioral text scripts through computer vision techniques, subsequently capitalizing on the generalization capabilities of large language models (LLMs) for zero-shot/few-shot ASD detection. Three core technical contributions are introduced: (1) A multimodal script transcription module transforming behavioral cues into structured textual representations. (2) An emotion textualization module encoding emotional dynamics as the contextual features to augment behavioral understanding. (3) A domain-specific prompt engineering strategy enables the injection of clinical knowledge into LLMs. Our method achieves an F1-score of 95.24% in diagnosing ASD in children with an average age of two years while generating interpretable detection rationales. This work opens up new avenues for leveraging the power of LLMs in analyzing and understanding ASD-related human behavior, thereby enhancing the accuracy of assisted autism diagnosis.