# LaScA: Language-Conditioned Scalable Modelling of Affective Dynamics

arXiv: https://arxiv.org/abs/2604.07193 · HTML: https://mlsanity.com/paper/2604.07193/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition](https://mlsanity.com/paper/1811.07770/) (2018)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation](https://mlsanity.com/paper/2308.11509/) (2023)
- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [Deep Learning for Human Affect Recognition: Insights and New Developments](https://mlsanity.com/paper/1901.02884/) (2019)
- [EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction](https://mlsanity.com/paper/1808.07773/) (2018)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild](https://mlsanity.com/paper/2404.09010/) (2024)
- [From the Lab to the Wild: Affect Modeling via Privileged Information](https://mlsanity.com/paper/2305.10919/) (2023)
- [Supervised Contrastive Learning for Affect Modelling](https://mlsanity.com/paper/2208.12238/) (2022)
- [MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network](https://mlsanity.com/paper/2503.12623/) (2025)
- [The Invariant Ground Truth of Affect](https://mlsanity.com/paper/2210.07630/) (2022)
- [RankNEAT: Outperforming Stochastic Gradient Search in Preference Learning Tasks](https://mlsanity.com/paper/2204.06901/) (2022)
- [Knowing Your Annotator: Rapidly Testing the Reliability of Affect Annotation](https://mlsanity.com/paper/2308.16029/) (2023)
- [Privileged Contrastive Pretraining for Multimodal Affect Modelling](https://mlsanity.com/paper/2508.03729/) (2025)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [RankList -- A Listwise Preference Learning Framework for Predicting Subjective Preferences](https://mlsanity.com/paper/2508.09826/) (2025)

## Abstract (arXiv)

Predicting affect in unconstrained environments remains a fundamental challenge in human-centered AI. While deep neural embeddings dominate contemporary approaches, they often lack interpretability and limit expert-driven refinement. We propose a novel framework that uses Language Models (LMs) as semantic context conditioners over handcrafted affect descriptors to model changes in Valence and Arousal. Our approach begins with interpretable facial geometry and acoustic features derived from structured domain knowledge. These features are transformed into symbolic natural-language descriptions encoding their affective implications. A pretrained LM processes these descriptions to generate semantic context embeddings that act as high-level priors over affective dynamics. Unlike end-to-end black-box pipelines, our framework preserves feature transparency while leveraging the contextual abstraction capabilities of LMs. We evaluate the proposed method on the Aff-Wild2 and SEWA datasets for affect change prediction. Experimental results show consistent improvements in accuracy for both Valence and Arousal compared to handcrafted-only and deep-embedding baselines. Our findings demonstrate that semantic conditioning enables interpretable affect modelling without sacrificing predictive performance, offering a transparent and computationally efficient alternative to fully end-to-end architectures