# Multimodal Fusion with Semi-Supervised Learning Minimizes Annotation Quantity for Modeling Videoconference Conversation Experience

arXiv: https://arxiv.org/abs/2506.13971 · HTML: https://mlsanity.com/paper/2506.13971/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 9 of 9)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Self-Training: A Survey](https://mlsanity.com/paper/2202.12040/) (2022)
- [Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions](https://mlsanity.com/paper/2109.09487/) (2021)
- [Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout](https://mlsanity.com/paper/2409.07078/) (2024)
- [3M-TRANSFORMER: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction](https://mlsanity.com/paper/2310.14859/) (2023)
- [Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment](https://mlsanity.com/paper/2501.03190/) (2025)

## Abstract (arXiv)

Group conversations over videoconferencing are a complex social behavior. However, the subjective moments of negative experience, where the conversation loses fluidity or enjoyment remain understudied. These moments are infrequent in naturalistic data, and thus training a supervised learning (SL) model requires costly manual data annotation. We applied semi-supervised learning (SSL) to leverage targeted labeled and unlabeled clips for training multimodal (audio, facial, text) deep features to predict non-fluid or unenjoyable moments in holdout videoconference sessions. The modality-fused co-training SSL achieved an ROC-AUC of 0.9 and an F1 score of 0.6, outperforming SL models by up to 4% with the same amount of labeled data. Remarkably, the best SSL model with just 8% labeled data matched 96% of the SL model's full-data performance. This shows an annotation-efficient framework for modeling videoconference experience.