mlsanityData to 7 Oct 2026

SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation

arXiv 2507.03415 ↗First posted 4 Jul 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantically meaningful text representation as an initial embedding in the autoregressive training and generation processes. The extensive empirical study demonstrates that the SMCLM approach makes autoregressive models capable of learning robust and high-quality paraphrase generation. The proposed method is competitive with the supervised method and achieves state-of-the-art results in unsupervised approaches. This article also presents a comprehensive set of automatic metrics that cover a wide range of autogenerated paraphrase evaluation aspects. Simultaneously, this article highlights the low reliability of the metrics that are widely used in paraphrase generation evaluation, including BLEU, ROUGE, and BERTScore.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 93 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

50 of 93 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  12. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
38 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  6. 2015Generating Sentences from a Continuous Spacecited by 607
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  8. 2018Universal Sentence Encodercited by 485
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  11. 2015Skip-Thought Vectorscited by 321
  12. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  13. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  14. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  15. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  16. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  17. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  18. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  19. 2017Paraphrase Generation with Deep Reinforcement Learningcited by 58
  20. 2020Neural Syntactic Preordering for Controlled Paraphrase Generationcited by 56
  21. 2017A Deep Generative Framework for Paraphrase Generationcited by 54
  22. 2020Limitations of Autoregressive Models and Their Alternativescited by 54
  23. 2016Neural Paraphrase Generation with Stacked Residual LSTM Networkscited by 52
  24. 2021Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairscited by 45
  25. 2019Unsupervised Paraphrasing by Simulated Annealingcited by 37
  26. 2019Paraphrasing with Large Language Modelscited by 35
  27. 2022Quality Controlled Paraphrase Generationcited by 31
  28. 2017Learning Paraphrastic Sentence Embeddings from Back-Translated Bitextcited by 27
  29. 2024Numerical Pruning for Efficient Autoregressive Modelscited by 24
  30. 2022Hierarchical Sketch Induction for Paraphrase Generationcited by 21
  31. 2022Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue Systemcited by 16
  32. 2018A task in a suit and a tie: paraphrase generation with semantic augmentationcited by 15
  33. 2020Unsupervised Paraphrase Generation using Pre-trained Language Modelscited by 15
  34. 2020Unsupervised Paraphrasing with Pretrained Language Modelscited by 15
  35. 2021From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decodingcited by 14
  36. 2022Training Effective Neural Sentence Encoders from Automatically Mined Paraphrasescited by 11
  37. 2021Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learningcited by 10
  38. 2023Multilingual Sentence Transformer as A Multilingual Word Alignercited by 8

Built on by

Not cited yet.