# Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models

arXiv: https://arxiv.org/abs/2509.20237 · HTML: https://mlsanity.com/paper/2509.20237/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 75 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)
- [On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers](https://mlsanity.com/paper/2010.02616/) (2020)
- [Stopwords in Technical Language Processing](https://mlsanity.com/paper/2006.02633/) (2020)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [Release of Pre-Trained Models for the Japanese Language](https://mlsanity.com/paper/2404.01657/) (2024)
- [Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis](https://mlsanity.com/paper/2309.15656/) (2023)
- [Backchannel Detection and Agreement Estimation from Video with Transformer Networks](https://mlsanity.com/paper/2306.01656/) (2023)
- [From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation](https://mlsanity.com/paper/2403.08890/) (2024)
- [Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives](https://mlsanity.com/paper/2301.10761/) (2023)
- [Building and curating conversational corpora for diversity-aware language science and technology](https://mlsanity.com/paper/2203.03399/) (2022)
- [Joint Learning of Context and Feedback Embeddings in Spoken Dialogue](https://mlsanity.com/paper/2406.07291/) (2024)

## Built on by (most cited first, 1 of 1)

- [Is Semi-Automatic Transcription Useful in Corpus Creation? Preliminary Considerations on the KIParla Corpus](https://mlsanity.com/paper/2603.16258/) (2026)

## Abstract (arXiv)

Backchannels and fillers are important linguistic expressions in dialogue, but often treated as 'noise' to be bypassed in modern transformer-based language models (LMs). Here, we study how they are represented in LMs using three fine-tuning strategies on three dialogue corpora in English and Japanese, in which backchannels and fillers are both preserved and annotated. This allows us to investigate how fine-tuning can help LMs learn these representations. We first apply clustering analysis to the learnt representation of backchannels and fillers, and find increased silhouette scores in representations from fine-tuned models, which suggests that fine-tuning enables LMs to distinguish the nuanced semantic variation in different backchannel and filler use. We also employ natural language generation metrics and qualitative analyses to verify that utterances produced by fine-tuned LMs resemble those produced by humans more closely. Our findings suggest the potential for transforming general LMs into conversational LMs that can produce human-like language more adequately.