# Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks

arXiv: https://arxiv.org/abs/2104.03899 · HTML: https://mlsanity.com/paper/2104.03899/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 68 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends](https://mlsanity.com/paper/2001.00378/) (2020)
- [Self-supervised audio representation learning for mobile devices](https://mlsanity.com/paper/1905.11796/) (2019)
- [Speaker-invariant Affective Representation Learning via Adversarial Training](https://mlsanity.com/paper/1911.01533/) (2019)
- [Neural Predictive Coding using Convolutional Neural Networks towards Unsupervised Learning of Speaker Characteristics](https://mlsanity.com/paper/1802.07860/) (2018)
- [Unspeech: Unsupervised Speech Context Embeddings](https://mlsanity.com/paper/1804.06775/) (2018)
- [Automatic prediction of suicidal risk in military couples using multimodal interaction cues from couples conversations](https://mlsanity.com/paper/1911.11927/) (2019)
- [Triplet Network with Attention for Speaker Diarization](https://mlsanity.com/paper/1808.01535/) (2018)
- [Sparsely Connected and Disjointly Trained Deep Neural Networks for Low Resource Behavioral Annotation: Acoustic Classification in Couples' Therapy](https://mlsanity.com/paper/1606.04518/) (2016)
- [Predicting Behavior in Cancer-Afflicted Patient and Spouse Interactions using Speech and Language](https://mlsanity.com/paper/1908.00908/) (2019)
- [Unsupervised Latent Behavior Manifold Learning from Acoustic Features: audio2behavior](https://mlsanity.com/paper/1701.03198/) (2017)
- [Towards an Unsupervised Entrainment Distance in Conversational Speech using Deep Neural Networks](https://mlsanity.com/paper/1804.08782/) (2018)
- [Linking emotions to behaviors through deep transfer learning](https://mlsanity.com/paper/1910.03641/) (2019)

## Built on by (most cited first, 1 of 1)

- [Personalized Student Knowledge Modeling for Future Learning Resource Prediction](https://mlsanity.com/paper/2505.14072/) (2025)

## Abstract (arXiv)

Speech encodes a wealth of information related to human behavior and has been used in a variety of automated behavior recognition tasks. However, extracting behavioral information from speech remains challenging including due to inadequate training data resources stemming from the often low occurrence frequencies of specific behavioral patterns. Moreover, supervised behavioral modeling typically relies on domain-specific construct definitions and corresponding manually-annotated data, rendering generalizing across domains challenging. In this paper, we exploit the stationary properties of human behavior within an interaction and present a representation learning method to capture behavioral information from speech in an unsupervised way. We hypothesize that nearby segments of speech share the same behavioral context and hence map onto similar underlying behavioral representations. We present an encoder-decoder based Deep Contextualized Network (DCN) as well as a Triplet-Enhanced DCN (TE-DCN) framework to capture the behavioral context and derive a manifold representation, where speech frames with similar behaviors are closer while frames of different behaviors maintain larger distances. The models are trained on movie audio data and validated on diverse domains including on a couples therapy corpus and other publicly collected data (e.g., stand-up comedy). With encouraging results, our proposed framework shows the feasibility of unsupervised learning within cross-domain behavioral modeling.