mlsanityData to 7 Oct 2026

Improving Semantic Understanding in Speech Language Models via Brain-tuning

arXiv 2410.09230 ↗First posted 11 Oct 2024cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Speech language models align with human brain responses to natural language to an impressive degree. However, current models rely heavily on low-level speech features, indicating they lack brain-relevant semantics which limits their utility as model organisms of semantic processing in the brain. In this work, we address this limitation by inducing brain-relevant bias directly into the models via fine-tuning with fMRI recordings of people listening to natural stories, a process we name brain-tuning. After testing it on 3 different pretrained model families, we show that brain-tuning not only improves overall alignment with new brain recordings in semantic language regions, but also reduces the reliance on low-level speech features for this alignment. Excitingly, we further show that brain-tuning leads to 1) consistent improvements in performance on a range of downstream tasks and 2) a representational space with increased semantic preference. Our results provide converging evidence, for the first time, that incorporating brain signals into the training of language models improves the models' semantic understanding.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 24 of its 49 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

24 of 49 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  8. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  9. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  10. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  11. 2023Scaling laws for language encoding models in fMRIcited by 68
  12. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
12 more
  1. 2019Inducing brain-relevant bias in natural language processing modelscited by 38
  2. 2022Self-supervised models of audio effectively explain human cortical responses to speechcited by 37
  3. 2022Joint processing of linguistic properties in brains and language modelscited by 37
  4. 2021Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responsescited by 34
  5. 2022Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?cited by 27
  6. 2022Training language models to summarize narratives improves brain alignmentcited by 26
  7. 2023Speech language models lack important brain-relevant semanticscited by 19
  8. 2021Does injecting linguistic structure into language models lead to better alignment with brain recordings?cited by 11
  9. 2019Relating Simple Sentence Representations in Deep Neural Networks and the Braincited by 10
  10. 2022Same Cause; Different Effects in the Braincited by 10
  11. 2023Mapping Brains with Language Models: A Surveycited by 8
  12. 2023Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Languagecited by 6
  2. 2025Explanations of Large Language Models Explain Language Representations in the Braincited by 4
  3. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantagecited by 4
  4. 2025Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Modelscited by 4
  5. 2025AAD-LLM: Neural Attention-Driven Auditory Scene Understandingcited by 3
  6. 2025Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Braincited by 3
  7. 2025Fine-grained Analysis of Brain-LLM Alignment through Input Attributioncited by 3
  8. 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 3
  9. 2026Abstraction Induces the Brain Alignment of Language and Speech Modelscited by 2
  10. 2025Improve Language Model and Brain Alignment via Associative Memorycited by 1
  11. 2025The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friendscited by 1
  12. 2026MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encodingcited by 1
8 more
  1. 2025Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory taskscited by 0
  2. 2026A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Datacited by 0
  3. 2026Scaling Next-Brain-Token Prediction for MEGcited by 0
  4. 2026How Much is Brain Data Worth for Machine Learning?cited by 0
  5. 2026Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoGcited by 0
  6. 2026Platonic Projection Structures: Operator-Induced Observability in Representation Learningcited by 0
  7. 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognitioncited by 0
  8. 2026Two Routes to the Middle: Placement Search and Brain Readouts Converge on Where Continual Learners Should Specializecited by 0