# A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations

arXiv: https://arxiv.org/abs/1904.01173 · HTML: https://mlsanity.com/paper/1904.01173/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 30 of 37)

- [Evaluating Large Language Models on Controlled Generation Tasks](https://mlsanity.com/paper/2310.14542/) (2023)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems](https://mlsanity.com/paper/2310.05280/) (2023)
- [Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models](https://mlsanity.com/paper/2104.05115/) (2021)
- [Factorising Meaning and Form for Intent-Preserving Paraphrasing](https://mlsanity.com/paper/2105.15053/) (2021)
- [Hierarchical Sketch Induction for Paraphrase Generation](https://mlsanity.com/paper/2203.03463/) (2022)
- [Text Revision by On-the-Fly Representation Optimization](https://mlsanity.com/paper/2204.07359/) (2022)
- [Self-conditioning pre-trained language models](https://mlsanity.com/paper/2110.02802/) (2021)
- [Disentangling Representations of Text by Masking Transformers](https://mlsanity.com/paper/2104.07155/) (2021)
- [Debiasing Learning for Membership Inference Attacks Against Recommender Systems](https://mlsanity.com/paper/2206.12401/) (2022)
- [Learning Disentangled Representations for Natural Language Definitions](https://mlsanity.com/paper/2210.02898/) (2022)
- [Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2205.02309/) (2022)
- [Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension](https://mlsanity.com/paper/2204.00996/) (2022)
- [Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders](https://mlsanity.com/paper/2311.08579/) (2023)
- [Are there identifiable structural parts in the sentence embedding whole?](https://mlsanity.com/paper/2406.16563/) (2024)
- [Decoupling Content and Expression: Two-Dimensional Detection of AI-Generated Text](https://mlsanity.com/paper/2503.00258/) (2025)
- [Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles](https://mlsanity.com/paper/2206.11184/) (2022)
- [PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation](https://mlsanity.com/paper/2305.16701/) (2023)
- [Enhanced Coherence-Aware Network with Hierarchical Disentanglement for Aspect-Category Sentiment Analysis](https://mlsanity.com/paper/2403.10214/) (2024)
- [Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings](https://mlsanity.com/paper/2110.02030/) (2021)
- [Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs](https://mlsanity.com/paper/2205.05943/) (2022)
- [RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generators](https://mlsanity.com/paper/2205.12590/) (2022)
- [Mitigating Negative Style Transfer in Hybrid Dialogue System](https://mlsanity.com/paper/2212.07183/) (2022)
- [Syntactically Robust Training on Partially-Observed Data for Open Information Extraction](https://mlsanity.com/paper/2301.06841/) (2023)
- [Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers](https://mlsanity.com/paper/2502.20681/) (2025)
- [Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings](https://mlsanity.com/paper/2210.12857/) (2022)
- [Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply](https://mlsanity.com/paper/2509.01606/) (2025)
- [Time Travel Engine: A Shared Latent Chronological Manifold Enables Historical Navigation in Large Language Models](https://mlsanity.com/paper/2601.06437/) (2026)
- [Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets](https://mlsanity.com/paper/2106.03631/) (2021)
- [A Quality-based Syntactic Template Retriever for Syntactically-controlled Paraphrase Generation](https://mlsanity.com/paper/2310.13262/) (2023)

## Abstract (arXiv)

We propose a generative model for a sentence that uses two latent variables, with one intended to represent the syntax of the sentence and the other to represent its semantics. We show we can achieve better disentanglement between semantic and syntactic representations by training with multiple losses, including losses that exploit aligned paraphrastic sentences and word-order information. We also investigate the effect of moving from bag-of-words to recurrent neural network modules. We evaluate our models as well as several popular pretrained embeddings on standard semantic similarity tasks and novel syntactic similarity tasks. Empirically, we find that the model with the best performing syntactic and semantic representations also gives rise to the most disentangled representations.