# Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition

arXiv: https://arxiv.org/abs/2607.25961 · HTML: https://mlsanity.com/paper/2607.25961/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 15 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Subject-Based Domain Adaptation for Facial Expression Recognition](https://mlsanity.com/paper/2312.05632/) (2023)
- [Personalized Feature Translation for Expression Recognition: An Efficient Source-Free Domain Adaptation Method](https://mlsanity.com/paper/2508.09202/) (2025)
- [Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2603.21309/) (2026)

## Abstract (arXiv)

Ambivalence and hesitancy (A/H) are conflicting affective states that precede the delay or abandonment of health behaviour change. Recognition of A/H at the video level is difficult, since the signal arises from disagreement across and within facial, vocal, linguistic, and bodily modalities, and manifests differently across individuals. The proposed PRISM-AH (Predictive Reasoning over Interacting Streams for Multimodal Ambivalence/Hesitancy Recognition), is a framework that treats A/H as a multimodal conflict that unfolds over time. Frozen vision, audio, and text encoders are aligned into short time windows and passed to a lightweight streaming model that scores cross-modal dissonance, predicts each next window to expose a hesitation surprise signal, discovers behaviour prototypes, and is conditioned on participant metadata. Dense window-level annotations supervise the model as an auxiliary objective, and the decision threshold is calibrated for macro F1. A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset, and its verdict is fused late only when validation performance improves. On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827. The reasoning gain is validated to transfer from validation to the larger test partition.