mlsanityData to 7 Oct 2026

Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition

arXiv 2402.12255 ↗First posted 19 Feb 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Numerous AI-assisted scholarly applications have been developed to aid different stages of the research process. We present an analysis of AI-assisted scholarly writing generated with ScholaCite, a tool we built that is designed for organizing literature and composing Related Work sections for academic papers. Our evaluation method focuses on the analysis of citation graphs to assess the structural complexity and inter-connectedness of citations in texts and involves a three-way comparison between (1) original human-written texts, (2) purely GPT-generated texts, and (3) human-AI collaborative texts. We find that GPT-4 can generate reasonable coarse-grained citation groupings to support human users in brainstorming, but fails to perform detailed synthesis of related works without human intervention. We suggest that future writing assistant tools should not be used to draft text independently of the human author.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 13 of its 34 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

13 of 34 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  3. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  4. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  5. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  6. 2021When Does Translation Require Context? A Data-driven, Multilingual Explorationcited by 36
  7. 2022Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge networkcited by 34
  8. 2022Target-aware Abstractive Related Work Generation with Contrastive Learningcited by 20
  9. 2023Do PLMs Know and Understand Ontological Knowledge?cited by 13
  10. 2022Automatic Related Work Generation: A Meta Studycited by 8
  11. 2023Faithful Low-Resource Data-to-Text Generation through Cycle Trainingcited by 7
  12. 2023Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparencycited by 6
1 more
  1. 2021Generating Related Workcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 4.

12 papers.

  1. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  2. 2024Related Work and Citation Text Generation: A Surveycited by 17
  3. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11
  4. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  5. 2024OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sourcescited by 6
  6. 2025Expert Preference-based Evaluation of Automated Related Work Generationcited by 6
  7. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  8. 2025Select, Read, and Write: A Multi-Agent Framework of Full-Text-based Related Work Generationcited by 4
  9. 2025Modelling and Classifying the Components of a Literature Reviewcited by 2
  10. 2026GRASP: Graph-Reasoning Aided Survey Planning for High-Fidelity Related Work Generationcited by 0
  11. 2026Does AI-Generated Scientific Text Follow Human Argumentation Patterns? A CARS-Based Comparison of Research Article Introductionscited by 0
  12. 2026Ream: Unfolding Mutual Awareness in Human-Agent Workspacescited by 0