# Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition

arXiv: https://arxiv.org/abs/2402.12255 · HTML: https://mlsanity.com/paper/2402.12255/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network](https://mlsanity.com/paper/2210.00881/) (2022)
- [Target-aware Abstractive Related Work Generation with Contrastive Learning](https://mlsanity.com/paper/2205.13339/) (2022)
- [Do PLMs Know and Understand Ontological Knowledge?](https://mlsanity.com/paper/2309.05936/) (2023)
- [Automatic Related Work Generation: A Meta Study](https://mlsanity.com/paper/2201.01880/) (2022)
- [Faithful Low-Resource Data-to-Text Generation through Cycle Training](https://mlsanity.com/paper/2305.14793/) (2023)
- [Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparency](https://mlsanity.com/paper/2304.11079/) (2023)
- [Generating Related Work](https://mlsanity.com/paper/2104.08668/) (2021)

## Built on by (most cited first, 12 of 12)

- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Related Work and Citation Text Generation: A Survey](https://mlsanity.com/paper/2404.11588/) (2024)
- [KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions](https://mlsanity.com/paper/2403.03866/) (2024)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources](https://mlsanity.com/paper/2405.01930/) (2024)
- [Expert Preference-based Evaluation of Automated Related Work Generation](https://mlsanity.com/paper/2508.07955/) (2025)
- [Select, Read, and Write: A Multi-Agent Framework of Full-Text-based Related Work Generation](https://mlsanity.com/paper/2505.19647/) (2025)
- [Modelling and Classifying the Components of a Literature Review](https://mlsanity.com/paper/2508.04337/) (2025)
- [GRASP: Graph-Reasoning Aided Survey Planning for High-Fidelity Related Work Generation](https://mlsanity.com/paper/2607.03709/) (2026)
- [Does AI-Generated Scientific Text Follow Human Argumentation Patterns? A CARS-Based Comparison of Research Article Introductions](https://mlsanity.com/paper/2610.01353/) (2026)
- [Ream: Unfolding Mutual Awareness in Human-Agent Workspaces](https://mlsanity.com/paper/2610.09497/) (2026)

## Abstract (arXiv)

Numerous AI-assisted scholarly applications have been developed to aid different stages of the research process. We present an analysis of AI-assisted scholarly writing generated with ScholaCite, a tool we built that is designed for organizing literature and composing Related Work sections for academic papers. Our evaluation method focuses on the analysis of citation graphs to assess the structural complexity and inter-connectedness of citations in texts and involves a three-way comparison between (1) original human-written texts, (2) purely GPT-generated texts, and (3) human-AI collaborative texts. We find that GPT-4 can generate reasonable coarse-grained citation groupings to support human users in brainstorming, but fails to perform detailed synthesis of related works without human intervention. We suggest that future writing assistant tools should not be used to draft text independently of the human author.