mlsanityData to 14 Oct 2026

Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder

arXiv 2506.20083 ↗First posted 25 Jun 2025cs.CLcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Integrating compositional and symbolic properties into current distributional semantic spaces can enhance the interpretability, controllability, compositionality, and generalisation capabilities of Transformer-based auto-regressive language models (LMs). In this survey, we offer a novel perspective on latent space geometry through the lens of compositional semantics, a direction we refer to as textitsemantic representation learning. This direction enables a bridge between symbolic and distributional semantics, helping to mitigate the gap between them. We review and compare three mainstream autoencoder architectures-Variational AutoEncoder (VAE), Vector Quantised VAE (VQVAE), and Sparse AutoEncoder (SAE)-and examine the distinctive latent geometries they induce in relation to semantic structure and interpretability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 107 of its 181 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

107 of 181 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Cross-Layer Discrete Concept Discovery for Interpreting Language Models24 Jun
  2. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models22 May
  3. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Features4 Nov
  4. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering21 Oct
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 102

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,691
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2023Steering Language Models With Activation Engineeringcited by 922
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  11. 2018Disentangling by Factorisingcited by 616
  12. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  13. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  14. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  15. 2017Toward Controlled Generation of Textcited by 272
  16. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  17. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  18. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  19. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
  20. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  21. 2018Hyperspherical Variational Auto-Encoderscited by 189
  22. 2020Deep Learning for Text Style Transfer: A Surveycited by 162
  23. 2021Explaining Answers with Entailment Treescited by 152
  24. 2017Style Transfer in Text: Exploration and Evaluationcited by 146
  25. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  26. 2018Style Transfer Through Back-Translationcited by 124
  27. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  28. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  29. 2017Sentence Simplification with Deep Reinforcement Learningcited by 109
  30. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  31. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  32. 2018Learning Deep Disentangled Embeddings with the F-Statistic Losscited by 101
  33. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  34. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
  35. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  36. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  37. 2018Spherical Latent Spaces for Stable Variational Autoencoderscited by 67
  38. 2025How much do language models memorize?cited by 61

50 most cited of 102 All 102 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
  2. 2025Towards Atoms of Large Language Models
  3. 2023Compositional Semantics for Open Vocabulary Spatio-semantic Representations
  4. 2026Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders
  5. 2026A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability
  6. 2025Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
  7. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representations
  8. 2024Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning