mlsanityData to 7 Oct 2026

Generative Pretraining for Paraphrase Evaluation

arXiv 2107.08251 ↗First posted 17 Jul 2021cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce ParaBLEU, a paraphrase representation learning model and evaluation metric for text generation. Unlike previous approaches, ParaBLEU learns to understand paraphrasis using generative conditioning as a pretraining objective. ParaBLEU correlates more strongly with human judgements than existing metrics, obtaining new state-of-the-art results on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only 50% of the available training data and surpassing BLEU, ROUGE and METEOR with only 40 labelled examples. Finally, we demonstrate that ParaBLEU can be used to conditionally generate novel paraphrases from a single demonstration, which we use to confirm our hypothesis that it learns abstract, generalized paraphrase representations.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 17 Jul 2021.

Builds on

35 of 71 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
23 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  10. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  11. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  12. 2017Challenges in Data-to-Document Generationcited by 226
  13. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  14. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  15. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
  16. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  17. 2018Unsupervised Text Style Transfer using Language Models as Discriminatorscited by 70
  18. 2020Neural Syntactic Preordering for Controlled Paraphrase Generationcited by 56
  19. 2017ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translationscited by 44
  20. 2019Encoding Musical Style with Transformer Autoencoderscited by 35
  21. 2019Decomposable Neural Paraphrase Generationcited by 30
  22. 2021ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generationcited by 28
  23. 2018Language Style Transfer from Sentences with Arbitrary Unknown Stylescited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1.

4 papers.

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2022DATScore: Evaluating Translation with Data Augmented Translationscited by 2
  3. 2022Dense Paraphrasing for Textual Enrichmentcited by 1
  4. 2023Improving Metrics for Speech Translationcited by 1