# Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency

arXiv: https://arxiv.org/abs/2305.20018 · HTML: https://mlsanity.com/paper/2305.20018/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Neural AMR: Sequence-to-Sequence Models for Parsing and Generation](https://mlsanity.com/paper/1704.08381/) (2017)
- [Graph Pre-training for AMR Parsing and Generation](https://mlsanity.com/paper/2203.07836/) (2022)
- [CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training](https://mlsanity.com/paper/2006.04702/) (2020)
- [Semi-Supervised Learning for Neural Machine Translation](https://mlsanity.com/paper/1606.04596/) (2016)
- [Cycle-consistency training for end-to-end speech recognition](https://mlsanity.com/paper/1811.01690/) (2018)
- [Improving AMR Parsing with Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2010.01771/) (2020)
- [Maximum Bayes Smatch Ensemble Distillation for AMR Parsing](https://mlsanity.com/paper/2112.07790/) (2021)
- [Language as a Latent Variable: Discrete Generative Models for Sentence Compression](https://mlsanity.com/paper/1609.07317/) (2016)
- [Semantic Parsing with Dual Learning](https://mlsanity.com/paper/1907.05343/) (2019)
- [Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing](https://mlsanity.com/paper/2110.15534/) (2021)
- [StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing](https://mlsanity.com/paper/1806.07832/) (2018)
- [Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder](https://mlsanity.com/paper/1807.09875/) (2018)
- [Pushing the Limits of AMR Parsing with Self-Learning](https://mlsanity.com/paper/2010.10673/) (2020)
- [ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models](https://mlsanity.com/paper/2108.12472/) (2021)
- [End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimator](https://mlsanity.com/paper/1810.13107/) (2018)
- [A Survey of Unsupervised Dependency Parsing](https://mlsanity.com/paper/2010.01535/) (2020)
- [Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsing](https://mlsanity.com/paper/2305.04346/) (2023)

## Abstract (arXiv)

We introduce Logical Offline Cycle Consistency Optimization (LOCCO), a scalable, semi-supervised method for training a neural semantic parser. Conceptually, LOCCO can be viewed as a form of self-learning where the semantic parser being trained is used to generate annotations for unlabeled text that are then used as new supervision. To increase the quality of annotations, our method utilizes a count-based prior over valid formal meaning representations and a cycle-consistency score produced by a neural text generation model as additional signals. Both the prior and semantic parser are updated in an alternate fashion from full passes over the training data, which can be seen as approximating the marginalization of latent structures through stochastic variational inference. The use of a count-based prior, frozen text generation model, and offline annotation process yields an approach with negligible complexity and latency increases as compared to conventional self-learning. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model. We demonstrate the utility of LOCCO on the well-known WebNLG benchmark where we obtain an improvement of 2 points against a self-learning parser under equivalent conditions, an improvement of 1.3 points against the previous state-of-the-art parser, and competitive text generation performance in terms of BLEU score.