How Video Meetings Change Your Expression
Not on the map. Why?
Abstract (arXiv)
Do our facial expressions change when we speak over video calls? Given two unpaired sets of videos of people, we seek to automatically find spatio-temporal patterns that are distinctive of each set. Existing methods use discriminative approaches and perform post-hoc explainability analysis. Such methods are insufficient as they are unable to provide insights beyond obvious dataset biases, and the explanations are useful only if humans themselves are good at the task. Instead, we tackle the problem through the lens of generative domain translation: our method generates a detailed report of learned, input-dependent spatio-temporal features and the extent to which they vary between the domains. We demonstrate that our method can discover behavioral differences between conversing face-to-face (F2F) and on video-calls (VCs). We also show the applicability of our method on discovering differences in presidential communication styles. Additionally, we are able to predict temporal change-points in videos that decouple expressions in an unsupervised way, and increase the interpretability and usefulness of our model. Finally, our method, being generative, can be used to transform a video call to appear as if it were recorded in a F2F setting. Experiments and visualizations show our approach is able to discover a range of behaviors, taking a step towards deeper understanding of human behaviors.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 50 of its 96 references that are in the map. First posted on arXiv 3 Jun 2024.
Builds on
50 of 96 references.
References · 50
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2015Learning Deep Features for Discriminative Localizationcited by 2,436
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2020Concept Bottleneck Modelscited by 1,032
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
- 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
- 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
- 2018Disentangling by Factorisingcited by 616
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
- 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
38 more
- 2018Video-to-Video Synthesiscited by 390
- 2018Understanding disentangling in β-VAEcited by 335
- 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 319
- 2019Counterfactual Visual Explanationscited by 313
- 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
- 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
- 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 241
- 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
- 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 126
- 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
- 2021Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypescited by 120
- 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
- 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
- 2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
- 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
- 2022Neural Basis Models for Interpretabilitycited by 56
- 2021Convolutional Dynamic Alignment Networks for Interpretable Classificationscited by 54
- 2020SCOUT: Self-aware Discriminant Counterfactual Explanationscited by 52
- 2023Can Language Models Learn to Listen?cited by 40
- 2020Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settingscited by 36
- 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 35
- 2020Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neuronscited by 33
- 2020One Explanation is Not Enough: Structured Attention Graphs for Image Classificationcited by 32
- 2018A Spectral Regularizer for Unsupervised Disentanglementcited by 30
- 2021Interpretable Data-Based Explanations for Fairness Debuggingcited by 30
- 2022Making Heads or Tails: Towards Semantically Consistent Visual Counterfactualscited by 30
- 2022Scalable Interpretability via Polynomialscited by 25
- 2019Learning Discrete and Continuous Factors of Data via Alternating Disentanglementcited by 23
- 2014Discovering Hidden Factors of Variation in Deep Networkscited by 21
- 2021Explaining Algorithmic Fairness Through Fairness-Aware Causal Path Decompositioncited by 17
- 2022VecGAN: Image-to-Image Translation with Interpretable Latent Directionscited by 17
- 2019InfoGAN-CR and ModelCentrality: Self-supervised Model Training and Selection for Disentangling GANscited by 16
- 2020Cross-Loss Influence Functions to Explain Deep Network Representationscited by 12
- 2023Sample based Explanations via Generalized Representerscited by 10
- 2019One-to-one Mapping for Unpaired Image-to-image Translationcited by 7
- 2022Disentangling Variational Autoencoderscited by 5
- 2016Understanding Visual Concepts with Continuation Learningcited by 4
- 2023Beta-VAE has 2 Behaviors: PCA or ICA?cited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.