# Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos

arXiv: https://arxiv.org/abs/2606.27484 · HTML: https://mlsanity.com/paper/2606.27484/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 59 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning](https://mlsanity.com/paper/2408.07981/) (2024)
- [Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models](https://mlsanity.com/paper/2401.12215/) (2024)
- [Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models](https://mlsanity.com/paper/2410.09750/) (2024)
- [A Review of and Roadmap for Data Science and Machine Learning for the Neuropsychiatric Phenotype of Autism](https://mlsanity.com/paper/2303.03577/) (2023)
- [Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos](https://mlsanity.com/paper/2402.08875/) (2024)
- [TempT: Temporal consistency for Test-time adaptation](https://mlsanity.com/paper/2303.10536/) (2023)
- [Computer Vision Estimation of Emotion Reaction Intensity in the Wild](https://mlsanity.com/paper/2303.10741/) (2023)
- [Video-based Analysis Reveals Atypical Social Gaze in People with Autism Spectrum Disorder](https://mlsanity.com/paper/2409.00664/) (2024)
- [Exploiting ChatGPT for Diagnosing Autism-Associated Language Disorders and Identifying Distinct Features](https://mlsanity.com/paper/2405.01799/) (2024)
- [Ensemble Modeling of Multiple Physical Indicators to Dynamically Phenotype Autism Spectrum Disorder](https://mlsanity.com/paper/2408.13255/) (2024)
- [FIESTA: Fisher Information-based Efficient Selective Test-time Adaptation](https://mlsanity.com/paper/2503.23257/) (2025)
- [LDP: Parameter-Efficient Fine-Tuning of Multimodal LLM for Medical Report Generation](https://mlsanity.com/paper/2512.10750/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Deterministic Evidence Layer for Vision-Language Autism Screening from Naturalistic Home Video](https://mlsanity.com/paper/2610.09217/) (2026)

## Abstract (arXiv)

Autism spectrum disorder (ASD) affects 1 in 31 US children, yet median age at diagnosis exceeds four years. Artificial intelligence pipelines that provide quantified diagnosis using easy to access observational data (e.g., home videos) could help with earlier diagnosis, and timely delivery of early treatments. We fine-tuned Gemini 2.5 Pro on 400 clinician-rated home videos with low-rank adaptation, training only on 30 behavioral features previously validated to produce reliable predictions when passed to various ML models. On 99 held-out children (49 ASD, 50 neurotypical), inter-rater reliability with clinicians (per-feature weighted Cohen's kappa) improved by 40% (p<0.001), with 27 of 28 evaluable features improving. As an emergent zero-shot capability, direct ASD diagnosis F1 improved by 53% (p<0.001), matching or exceeding clinician outcomes. Classifier-assisted pipelines using fine-tuned LLM-derived behavioral features matched clinician-scored inputs across all tested pathways and achieved 77% accuracy (95% CI: 68-85%) and an AUC of 86% (95% CI: 78-92%). Fine-tuned multimodal LLMs can serve as scalable behavioral feature extractors for use in autism assessment and diagnosis.