# Leveraging LLMs with Iterative Loop Structure for Enhanced Social Intelligence in Video Question Answering

arXiv: https://arxiv.org/abs/2503.21190 · HTML: https://mlsanity.com/paper/2503.21190/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning](https://mlsanity.com/paper/2309.15091/) (2023)
- [VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding](https://mlsanity.com/paper/2406.09418/) (2024)
- [Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator](https://mlsanity.com/paper/2309.14494/) (2023)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models](https://mlsanity.com/paper/2306.11732/) (2023)
- [Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models](https://mlsanity.com/paper/2308.16529/) (2023)
- [DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding](https://mlsanity.com/paper/2310.18359/) (2023)

## Abstract (arXiv)

Social intelligence, the ability to interpret emotions, intentions, and behaviors, is essential for effective communication and adaptive responses. As robots and AI systems become more prevalent in caregiving, healthcare, and education, the demand for AI that can interact naturally with humans grows. However, creating AI that seamlessly integrates multiple modalities, such as vision and speech, remains a challenge. Current video-based methods for social intelligence rely on general video recognition or emotion recognition techniques, often overlook the unique elements inherent in human interactions. To address this, we propose the Looped Video Debating (LVD) framework, which integrates Large Language Models (LLMs) with visual information, such as facial expressions and body movements, to enhance the transparency and reliability of question-answering tasks involving human interaction videos. Our results on the Social-IQ 2.0 benchmark show that LVD achieves state-of-the-art performance without fine-tuning. Furthermore, supplementary human annotations on existing datasets provide insights into the model's accuracy, guiding future improvements in AI-driven social intelligence.