# Hierarchical Sketch Induction for Paraphrase Generation

arXiv: https://arxiv.org/abs/2203.03463 · HTML: https://mlsanity.com/paper/2203.03463/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Continuous Hierarchical Representations with Poincar'e Variational Auto-Encoders](https://mlsanity.com/paper/1901.06033/) (2019)
- [Data Noising as Smoothing in Neural Network Language Models](https://mlsanity.com/paper/1703.02573/) (2017)
- [Generating Sentences from Disentangled Syntactic and Semantic Spaces](https://mlsanity.com/paper/1907.05789/) (2019)
- [Hierarchical Quantized Autoencoders](https://mlsanity.com/paper/2002.08111/) (2020)
- [Neural Syntactic Preordering for Controlled Paraphrase Generation](https://mlsanity.com/paper/2005.02013/) (2020)
- [Neural Paraphrase Generation with Stacked Residual LSTM Networks](https://mlsanity.com/paper/1610.03098/) (2016)
- [Learning the Predictability of the Future](https://mlsanity.com/paper/2101.01600/) (2021)
- [Syntax-guided Controlled Generation of Paraphrases](https://mlsanity.com/paper/2005.08417/) (2020)
- [Learning to Paraphrase for Question Answering](https://mlsanity.com/paper/1708.06022/) (2017)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [Controllable Paraphrase Generation with a Syntactic Exemplar](https://mlsanity.com/paper/1906.00565/) (2019)
- [A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations](https://mlsanity.com/paper/1904.01173/) (2019)
- [Adversarial Domain Adaptation for Duplicate Question Detection](https://mlsanity.com/paper/1809.02255/) (2018)
- [Paraphrase Generation with Latent Bag of Words](https://mlsanity.com/paper/2001.01941/) (2020)
- [Factorising Meaning and Form for Intent-Preserving Paraphrasing](https://mlsanity.com/paper/2105.15053/) (2021)
- [Unsupervised Paraphrasing without Translation](https://mlsanity.com/paper/1905.12752/) (2019)
- [ConRPG: Paraphrase Generation using Contexts as Regularizer](https://mlsanity.com/paper/2109.00363/) (2021)
- [Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach](https://mlsanity.com/paper/2109.01862/) (2021)

## Built on by (most cited first, 22 of 22)

- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [RADAR: Robust AI-Text Detection via Adversarial Learning](https://mlsanity.com/paper/2307.03838/) (2023)
- [Human Feedback is not Gold Standard](https://mlsanity.com/paper/2309.16349/) (2023)
- [CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks](https://mlsanity.com/paper/2209.08773/) (2022)
- [Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing](https://mlsanity.com/paper/2502.15208/) (2025)
- [Bias in Opinion Summarisation from Pre-training to Adaptation: A Case Study in Political Bias](https://mlsanity.com/paper/2402.00322/) (2024)
- [GPTs Are Multilingual Annotators for Sequence Generation Tasks](https://mlsanity.com/paper/2402.05512/) (2024)
- [Diversify Question Generation with Retrieval-Augmented Style Transfer](https://mlsanity.com/paper/2310.14503/) (2023)
- [Evaluation of Question Generation Needs More References](https://mlsanity.com/paper/2305.16626/) (2023)
- [Attributable and Scalable Opinion Summarization](https://mlsanity.com/paper/2305.11603/) (2023)
- [Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs](https://mlsanity.com/paper/2205.05943/) (2022)
- [Vector-Quantized Prompt Learning for Paraphrase Generation](https://mlsanity.com/paper/2311.14949/) (2023)
- [Continuous Decomposition of Granularity for Neural Paraphrase Generation](https://mlsanity.com/paper/2209.01765/) (2022)
- [BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems](https://mlsanity.com/paper/2211.11982/) (2022)
- [Language as a Latent Sequence: deep latent variable models for semi-supervised paraphrase generation](https://mlsanity.com/paper/2301.02275/) (2023)
- [A Quality-based Syntactic Template Retriever for Syntactically-controlled Paraphrase Generation](https://mlsanity.com/paper/2310.13262/) (2023)
- [Object Recognition from Scientific Document based on Compartment Refinement Framework](https://mlsanity.com/paper/2312.09038/) (2023)
- [Linguistically-Controlled Paraphrase Generation](https://mlsanity.com/paper/2410.24199/) (2024)
- [Interpretable Sentence Representation with Variational Autoencoders and Attention](https://mlsanity.com/paper/2305.02810/) (2023)
- [Action Controlled Paraphrasing](https://mlsanity.com/paper/2405.11277/) (2024)
- [Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder](https://mlsanity.com/paper/2506.20083/) (2025)
- [SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation](https://mlsanity.com/paper/2507.03415/) (2025)

## Abstract (arXiv)

We propose a generative model of paraphrase generation, that encourages syntactic diversity by conditioning on an explicit syntactic sketch. We introduce Hierarchical Refinement Quantized Variational Autoencoders (HRQ-VAE), a method for learning decompositions of dense encodings as a sequence of discrete latent variables that make iterative refinements of increasing granularity. This hierarchy of codes is learned through end-to-end training, and represents fine-to-coarse grained information about the input. We use HRQ-VAE to encode the syntactic form of an input sentence as a path through the hierarchy, allowing us to more easily predict syntactic sketches at test time. Extensive experiments, including a human evaluation, confirm that HRQ-VAE learns a hierarchical representation of the input space, and generates paraphrases of higher quality than previous systems.