Towards Aligning Multimodal LLMs with Human Experts: A Focus on Parent-Child Interaction
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
While multimodal large language models (MLLMs) are increasingly applied in human-centred AI systems, their ability to understand complex social interactions remains uncertain. We present an exploratory study on aligning MLLMs with speech-language pathologists (SLPs) in analysing joint attention in parent-child interactions, a key construct in early social-communicative development. Drawing on interviews and video annotations with three SLPs, we characterise how observational cues of gaze, action, and vocalisation inform their reasoning processes. We then test whether an MLLM can approximate this workflow through a two-stage prompting approach, separating observation from judgement. Our findings reveal that alignment is more robust at the observation layer, where experts share common descriptors, than at the judgement layer, where interpretive criteria diverge. We position this work as a case-based probe into expert-AI alignment in complex social behaviour, highlighting both the feasibility and the challenges of applying MLLMs to socially situated interaction analysis.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 57 references that are in the map. First posted on arXiv 6 Nov 2025.
Builds on
13 of 57 references.
Closest ancestors · 1
Same direction, newest first
Other references · 12
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
- 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
- 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
- 2024Can LLM be a Personalized Judge?cited by 100
- 2024Exploring Parent's Needs for Children-Centered AI to Support Preschoolers' Interactive Storytelling and Reading Activitiescited by 8
- 2024AACessTalk: Fostering Communication between Minimally Verbal Autistic Children and Parents with Contextual Guidance and Card Recommendationcited by 7
- 2025Towards Multimodal Large-Language Models for Parent-Child Interaction: A Focus on Joint Attentioncited by 5
- 2022Captivate! Contextual Language Guidance for Parent-Child Interactioncited by 3
- 2025Exploring the Perspectives of Social VR-Aware Non-Parent Adults and Parents on Children's Use of Social Virtual Realitycited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Human-AI Alignment of Multimodal Large Language Models with Speech-Language Pathologists in Parent-Child Interactions
- 2026More Than 1v1: Human-AI Alignment in Early Developmental Communities with Multimodal LLMs
- 2025Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments
- 2025HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs
- 2024Towards Social AI: A Survey on Understanding Social Interactions
- 2025Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach
- 2026SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
- 2026MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings