# Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Models

arXiv: https://arxiv.org/abs/2309.12294 · HTML: https://mlsanity.com/paper/2309.12294/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 63 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing](https://mlsanity.com/paper/2009.13845/) (2020)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [Residual Energy-Based Models for Text Generation](https://mlsanity.com/paper/2004.11714/) (2020)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [A Graph-to-Sequence Model for AMR-to-Text Generation](https://mlsanity.com/paper/1805.02473/) (2018)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Few-Shot Semantic Parsing with Language Models Trained On Code](https://mlsanity.com/paper/2112.08696/) (2021)
- [Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models](https://mlsanity.com/paper/2205.11503/) (2022)
- [Structural Adapters in Pretrained Language Models for AMR-to-text Generation](https://mlsanity.com/paper/2103.09120/) (2021)
- [Logic2Text: High-Fidelity Natural Language Generation from Logical Forms](https://mlsanity.com/paper/2004.14579/) (2020)
- [Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models](https://mlsanity.com/paper/2009.13267/) (2020)

## Built on by (most cited first, 1 of 1)

- [Preference Optimization for Molecule Synthesis with Conditional Residual Energy-based Models](https://mlsanity.com/paper/2406.02066/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in natural language generation. However, their output quality can be inconsistent, posing challenges for generating natural language from logical forms (LFs). This task requires the generated outputs to embody the exact semantics of LFs, without missing any LF semantics or creating any hallucinations. In this work, we tackle this issue by proposing a novel generate-and-rerank approach. Our approach involves initially generating a set of candidate outputs by prompting an LLM and subsequently reranking them using a task-specific reranker model. In addition, we curate a manually collected dataset to evaluate the alignment between different ranking metrics and human judgements. The chosen ranking metrics are utilized to enhance the training and evaluation of the reranker model. By conducting extensive experiments on three diverse datasets, we demonstrate that the candidates selected by our reranker outperform those selected by baseline methods in terms of semantic consistency and fluency, as measured by three comprehensive metrics. Our findings provide strong evidence for the effectiveness of our approach in improving the quality of generated outputs.