mlsanityData to 7 Oct 2026

Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder

arXiv 2210.06230 ↗First posted 12 Oct 2022cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Formal/symbolic semantics can provide canonical, rigid controllability and interpretability to sentence representations due to their textitlocalisation or textitcomposition property. How can we deliver such property to the current distributional sentence representations to control and interpret the generation of language models (LMs)? In this work, we theoretically frame the sentence semantics as the composition of textitsemantic role - word content features and propose the formal semantic geometry. To inject such geometry into Transformer-based LMs (i.e. GPT2), we deploy Transformer-based Variational AutoEncoder with a supervision approach, where the sentence generation can be manipulated and explained over low-dimensional latent Gaussian space. In addition, we propose a new probing algorithm to guide the movement of sentence vectors over such geometry. Experimental results reveal that the formal semantic geometry can potentially deliver better control and interpretation to sentence generation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 79 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

35 of 79 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  5. 2018Disentangling by Factorisingcited by 614
  6. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  7. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  8. 2020Transformers as Soft Reasoners over Languagecited by 265
  9. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  10. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  11. 2021Explaining Answers with Entailment Treescited by 148
  12. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
23 more
  1. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  2. 2022The Geometry of Multilingual Language Model Representationscited by 94
  3. 2013Deep Learning of Representations: Looking Forwardcited by 84
  4. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  5. 2019Generating Sentences from Disentangled Syntactic and Semantic Spacescited by 59
  6. 2020A Disentangling Invertible Interpretation Network for Explaining Latent Representationscited by 57
  7. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  8. 2022A Causal Lens for Controllable Text Generationcited by 45
  9. 2022A Distributional Lens for Multi-Aspect Controllable Text Generationcited by 33
  10. 2019A Surprisingly Effective Fix for Deep Latent Variable Modeling of Textcited by 32
  11. 2020Differentiable Open-Ended Commonsense Reasoningcited by 29
  12. 2021Flexible Generation of Natural Language Deductionscited by 28
  13. 2022Composable Text Controls in Latent Space with ODEscited by 28
  14. 2017Answering Complex Questions Using Open Information Extractioncited by 25
  15. 2022Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generationcited by 20
  16. 2021Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoderscited by 19
  17. 2021Hybrid Autoregressive Inference for Scalable Multi-hop Explanation Regenerationcited by 17
  18. 2019Question Answering as Global Reasoning over Semantic Abstractionscited by 14
  19. 2022Learning Disentangled Representations of Negation and Uncertaintycited by 13
  20. 2022Learning Disentangled Representations for Natural Language Definitionscited by 12
  21. 2021SyGNS: A Systematic Generalization Testbed Based on Natural Language Semanticscited by 7
  22. 2019An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networkscited by 6
  23. 2022Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Rolescited by 5

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 1.

5 papers.

  1. 2023Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networkscited by 6
  2. 2024Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoderscited by 6
  3. 2023LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spacescited by 3
  4. 2023Towards Controllable Natural Language Inference through Lexical Inference Typescited by 1
  5. 2025Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncodercited by 0