# Supporting Experts with a Multimodal Machine-Learning-Based Tool for Human Behavior Analysis of Conversational Videos

arXiv: https://arxiv.org/abs/2402.11145 · HTML: https://mlsanity.com/paper/2402.11145/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 89 references that are in the map.
- First posted on arXiv 17 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Recent Advance in Content-based Image Retrieval: A Literature Survey](https://mlsanity.com/paper/1706.06064/) (2017)
- [Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels](https://mlsanity.com/paper/1910.02993/) (2019)
- [Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists](https://mlsanity.com/paper/2006.12695/) (2020)
- [Hierarchical Summarization for Longform Spoken Dialog](https://mlsanity.com/paper/2108.09597/) (2021)
- [Construction of a Large-scale Japanese ASR Corpus on TV Recordings](https://mlsanity.com/paper/2103.14736/) (2021)
- [REsCUE: A framework for REal-time feedback on behavioral CUEs using multimodal anomaly detection](https://mlsanity.com/paper/1903.11485/) (2019)
- [Discourse Behavior of Older Adults Interacting With a Dialogue Agent Competent in Multiple Topics](https://mlsanity.com/paper/1907.06279/) (2019)
- [Computational Design with Crowds](https://mlsanity.com/paper/2002.08657/) (2020)
- [AI for human assessment: What do professional assessors need?](https://mlsanity.com/paper/2204.08471/) (2022)
- [Tool- and Domain-Agnostic Parameterization of Style Transfer Effects Leveraging Pretrained Perceptual Metrics](https://mlsanity.com/paper/2105.09207/) (2021)

## Built on by (most cited first, 2 of 2)

- [Coaching Copilot: Blended Form of an LLM-Powered Chatbot and a Human Coach to Effectively Support Self-Reflection for Leadership Growth](https://mlsanity.com/paper/2405.15250/) (2024)
- [From Cheap to Pro: A Learning-based Adaptive Camera Parameter Network for Professional-Style Imaging](https://mlsanity.com/paper/2510.20550/) (2025)

## Abstract (arXiv)

Multimodal scene search of conversations is essential for unlocking valuable insights into social dynamics and enhancing our communication. While experts in conversational analysis have their own knowledge and skills to find key scenes, a lack of comprehensive, user-friendly tools that streamline the processing of diverse multimodal queries impedes efficiency and objectivity. To solve it, we developed Providence, a visual-programming-based tool based on design considerations derived from a formative study with experts. It enables experts to combine various machine learning algorithms to capture human behavioral cues without writing code. Our study showed its preferable usability and satisfactory output with less cognitive load imposed in accomplishing scene search tasks of conversations, verifying the importance of its customizability and transparency. Furthermore, through the in-the-wild trial, we confirmed the objectivity and reusability of the tool transform experts' workflow, suggesting the advantage of expert-AI teaming in a highly human-contextual domain.