# SynSciPass: detecting appropriate uses of scientific text generation arXiv: https://arxiv.org/abs/2209.03742 ยท HTML: https://mlsanity.com/paper/2209.03742/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 16 of its 33 references that are in the map. - First posted on arXiv 7 Sep 2022. ## Builds on (most cited first, 16 of 16) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020) - [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021) - [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019) - [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018) - [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021) - [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021) - [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020) - [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021) - [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021) - [The Limitations of Stylometry for Detecting Machine-Generated Fake News](https://mlsanity.com/paper/1908.09805/) (2019) - [Attacking Neural Text Detectors](https://mlsanity.com/paper/2002.11768/) (2020) - [A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications](https://mlsanity.com/paper/2202.02013/) (2022) - [Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals](https://mlsanity.com/paper/2107.06751/) (2021) - [Are You Robert or RoBERTa? Deceiving Online Authorship Attribution Models Using Neural Text Generators](https://mlsanity.com/paper/2203.09813/) (2022) - [A Large Parallel Corpus of Full-Text Scientific Articles](https://mlsanity.com/paper/1905.01852/) (2019) ## Built on by (most cited first, 10 of 10) - [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022) - [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024) - [Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective](https://mlsanity.com/paper/2210.10488/) (2022) - [AI vs. Human -- Differentiation Analysis of Scientific Content Generation](https://mlsanity.com/paper/2301.10416/) (2023) - [Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights](https://mlsanity.com/paper/2403.03506/) (2024) - [TOPFORMER: Topology-Aware Authorship Attribution of Deepfake Texts with Diverse Writing Styles](https://mlsanity.com/paper/2309.12934/) (2023) - [Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Texts](https://mlsanity.com/paper/2410.14677/) (2024) - ['Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews](https://mlsanity.com/paper/2410.09770/) (2024) - [Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors](https://mlsanity.com/paper/2503.15128/) (2025) - [Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?](https://mlsanity.com/paper/2407.17870/) (2024) ## Abstract (arXiv) Approaches to machine generated text detection tend to focus on binary classification of human versus machine written text. In the scientific domain where publishers might use these models to examine manuscripts under submission, misclassification has the potential to cause harm to authors. Additionally, authors may appropriately use text generation models such as with the use of assistive technologies like translation tools. In this setting, a binary classification scheme might be used to flag appropriate uses of assistive text generation technology as simply machine generated which is a cause of concern. In our work, we simulate this scenario by presenting a state-of-the-art detector trained on the DAGPap22 with machine translated passages from Scielo and find that the model performs at random. Given this finding, we develop a framework for dataset development that provides a nuanced approach to detecting machine generated text by having labels for the type of technology used such as for translation or paraphrase resulting in the construction of SynSciPass. By training the same model that performed well on DAGPap22 on SynSciPass, we show that not only is the model more robust to domain shifts but also is able to uncover the type of technology used for machine generated text. Despite this, we conclude that current datasets are neither comprehensive nor realistic enough to understand how these models would perform in the wild where manuscript submissions can come from many unknown or novel distributions, how they would perform on scientific full-texts rather than small passages, and what might happen when there is a mix of appropriate and inappropriate uses of natural language generation.