# Joint Learning of Context and Feedback Embeddings in Spoken Dialogue

arXiv: https://arxiv.org/abs/2406.07291 · HTML: https://mlsanity.com/paper/2406.07291/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [Conversational Contextual Cues: The Case of Personalization and History for Response Ranking](https://mlsanity.com/paper/1606.00372/) (2016)
- [Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor](https://mlsanity.com/paper/1706.01340/) (2017)
- [The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems](https://mlsanity.com/paper/2307.15493/) (2023)
- [It's not what you said, it's how you said it: discriminative perception of speech as a multichannel communication system](https://mlsanity.com/paper/2105.00260/) (2021)
- [Oh, Jeez! or Uh-huh? A Listener-aware Backchannel Predictor on ASR Transcriptions](https://mlsanity.com/paper/2304.04478/) (2023)

## Built on by (most cited first, 2 of 2)

- [Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models](https://mlsanity.com/paper/2509.20237/) (2025)
- [Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning](https://mlsanity.com/paper/2604.16622/) (2026)

## Abstract (arXiv)

Short feedback responses, such as backchannels, play an important role in spoken dialogue. So far, most of the modeling of feedback responses has focused on their timing, often neglecting how their lexical and prosodic form influence their contextual appropriateness and conversational function. In this paper, we investigate the possibility of embedding short dialogue contexts and feedback responses in the same representation space using a contrastive learning objective. In our evaluation, we primarily focus on how such embeddings can be used as a context-feedback appropriateness metric and thus for feedback response ranking in U.S. English dialogues. Our results show that the model outperforms humans given the same ranking task and that the learned embeddings carry information about the conversational function of feedback responses.