mlsanityData to 7 Oct 2026

Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Models

arXiv 2309.12294 ↗First posted 21 Sep 2023cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in natural language generation. However, their output quality can be inconsistent, posing challenges for generating natural language from logical forms (LFs). This task requires the generated outputs to embody the exact semantics of LFs, without missing any LF semantics or creating any hallucinations. In this work, we tackle this issue by proposing a novel generate-and-rerank approach. Our approach involves initially generating a set of candidate outputs by prompting an LLM and subsequently reranking them using a task-specific reranker model. In addition, we curate a manually collected dataset to evaluate the alignment between different ranking metrics and human judgements. The chosen ranking metrics are utilized to enhance the training and evaluation of the reranker model. By conducting extensive experiments on three diverse datasets, we demonstrate that the candidates selected by our reranker outperform those selected by baseline methods in terms of semantic consistency and fluency, as measured by three comprehensive metrics. Our findings provide strong evidence for the effectiveness of our approach in improving the quality of generated outputs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 63 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

44 of 63 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  12. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
32 more
  1. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  2. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  3. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  4. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  5. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  6. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  7. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  8. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  9. 2020Residual Energy-Based Models for Text Generationcited by 95
  10. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  11. 2021Learning Compact Metrics for MTcited by 93
  12. 2018A Graph-to-Sequence Model for AMR-to-Text Generationcited by 56
  13. 2019Few-Shot NLG with Pre-Trained Language Modelcited by 52
  14. 2021Few-Shot Semantic Parsing with Language Models Trained On Codecited by 50
  15. 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
  16. 2021Structural Adapters in Pretrained Language Models for AMR-to-text Generationcited by 48
  17. 2020Logic2Text: High-Fidelity Natural Language Generation from Logical Formscited by 45
  18. 2020Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Modelscited by 44
  19. 2021Learning to Synthesize Data for Semantic Parsingcited by 41
  20. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
  21. 2019Modeling Graph Structure in Transformer for Better AMR-to-Text Generationcited by 35
  22. 2020Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelitycited by 35
  23. 2018SQL-to-Text Generation with Graph-to-Sequence Modelcited by 31
  24. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  25. 2019Don't paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsingcited by 24
  26. 2019Enhancing AMR-to-Text Generation with Dual Graph Representationscited by 24
  27. 2022Extrinsic Evaluation of Machine Translation Metricscited by 14
  28. 2022Compositionality as Lexical Symmetrycited by 8
  29. 2022T5QL: Taming language models for SQL generationcited by 8
  30. 2023Learning to Substitute Spans towards Improving Compositional Generalizationcited by 8
  31. 2022Paraphrasing Techniques for Maritime QA systemcited by 7
  32. 2022Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generationcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Preference Optimization for Molecule Synthesis with Conditional Residual Energy-based Modelscited by 6