# FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models

arXiv: https://arxiv.org/abs/2406.00839 · HTML: https://mlsanity.com/paper/2406.00839/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation](https://mlsanity.com/paper/2304.01746/) (2023)
- [Analyzing Information Leakage of Updates to Natural Language Models](https://mlsanity.com/paper/1912.07942/) (2019)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [How Large Language Models are Transforming Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2210.03568/) (2022)
- [PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation](https://mlsanity.com/paper/2010.02301/) (2020)
- [Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance](https://mlsanity.com/paper/2305.13225/) (2023)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore](https://mlsanity.com/paper/2405.04286/) (2024)
- [On the use of BERT for Neural Machine Translation](https://mlsanity.com/paper/1909.12744/) (2019)
- [Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model](https://mlsanity.com/paper/2404.16766/) (2024)
- [A Deep Network Model for Paraphrase Detection in Short Text Messages](https://mlsanity.com/paper/1712.02820/) (2017)
- [Deep Investigation of Cross-Language Plagiarism Detection Methods](https://mlsanity.com/paper/1705.08828/) (2017)

## Built on by (most cited first, 3 of 3)

- [DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios](https://mlsanity.com/paper/2410.23746/) (2024)
- [What is the Best Way for ChatGPT to Translate Poetry?](https://mlsanity.com/paper/2406.03450/) (2024)
- [Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry](https://mlsanity.com/paper/2509.01620/) (2025)

## Abstract (arXiv)

Pre-trained Language Models (PLMs) have shown impressive results in various Natural Language Generation (NLG) tasks, such as powering chatbots and generating stories. However, an ethical concern arises due to their potential to produce verbatim copies of paragraphs from their training data. This is problematic as PLMs are trained on corpora constructed by human authors. As such, there is a pressing need for research to promote the generation of original content by these models. In this study, we introduce a unique "self-plagiarism" contrastive decoding strategy, aimed at boosting the originality of text produced by PLMs. Our method entails modifying prompts in LLMs to develop an amateur model and a professional model. Specifically, the amateur model is urged to plagiarize using three plagiarism templates we have designed, while the professional model maintains its standard language model status. This strategy employs prompts to stimulate the model's capacity to identify non-original candidate token combinations and subsequently impose penalties. The application of this strategy is integrated prior to the model's final layer, ensuring smooth integration with most existing PLMs (T5, GPT, LLaMA) without necessitating further adjustments. Implementing our strategy, we observe a significant decline in non-original sequences comprised of more than three words in the academic AASC dataset and the story-based ROCStories dataset.