mlsanityData to 7 Oct 2026

Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

arXiv 2605.00607 ↗First posted 1 May 2026cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Probing is widely used to study which features can be decoded from language model representations. However, the common decoding probe approach has two limitations that we aim to solve with our new encoding probe approach: contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results. We present an Encoding Probe that reverses this direction and reconstructs internal representations of models using interpretable features. We evaluate this method on text and speech transformer models, using feature sets spanning acoustics, phonetics, syntax, lexicon, and speaker identity. Our results suggest that speaker-related effects vary strongly across different training objectives and datasets, while syntactic and lexical features contribute independently to reconstruction. These results show that the Encoding Probe provides a complementary perspective on interpreting model representations beyond decodability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 45 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

23 of 45 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  8. 2019Designing and Interpreting Probes with Control Taskscited by 608
  9. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  11. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  12. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
11 more
  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  2. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  3. 2019Correlating neural and symbolic representations of languagecited by 32
  4. 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
  5. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  6. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  7. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  8. 2021Probing artificial neural networks: insights from neurosciencecited by 17
  9. 2025A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representationscited by 5
  10. 2025What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-trainingcited by 4
  11. 2025Word stress in self-supervised speech models: A cross-linguistic comparisoncited by 3

Built on by

Not cited yet.