mlsanityData to 14 Oct 2026

Writing as a testbed for open ended agents

arXiv 2503.19711 ↗First posted 25 Mar 2025cs.CL cs.AI cs.HCcited by 8

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Open-ended tasks are particularly challenging for LLMs due to the vast solution space, demanding both expansive exploration and adaptable strategies, especially when success lacks a clear, objective definition. Writing, with its vast solution space and subjective evaluation criteria, provides a compelling testbed for studying such problems. In this paper, we investigate the potential of LLMs to act as collaborative co-writers, capable of suggesting and implementing text improvements autonomously. We analyse three prominent LLMs - Gemini 1.5 Pro, Claude 3.5 Sonnet, and GPT-4o - focusing on how their action diversity, human alignment, and iterative improvement capabilities impact overall performance. This work establishes a framework for benchmarking autonomous writing agents and, more broadly, highlights fundamental challenges and potential solutions for building systems capable of excelling in diverse open-ended domains.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 27 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

21 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 20

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  5. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  6. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  8. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  9. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  10. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  11. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  12. 2022PEER: A Collaborative Language Modelcited by 86
8 more
  1. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 83
  2. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  3. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 68
  4. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  5. 2020Neurosymbolic Reinforcement Learning with Formally Verified Explorationcited by 49
  6. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  7. 2022Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revisioncited by 46
  8. 2024From Pen to Prompt: How Creative Writers Integrate AI into their Writing Practicecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  2. 2025Measuring AI "Slop" in Textcited by 6
  3. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  4. 2025Interaction Dynamics as a Reward Signal for LLMscited by 5
  5. 2025Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefactscited by 1
  6. 2025Do Language Models Agree with Human Perceptions of Suspense in Stories?cited by 1
  7. 2025Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoningcited by 1
  8. 2026Triviality Corrected Endogenous Rewardcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild
  2. 2023Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub
  3. 2026PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf
  4. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROI
  5. 2025Creativity in LLM-based Multi-Agent Systems: A Survey
  6. 2024A Survey of Useful LLM Evaluation
  7. 2025AI Agents with Human-Like Collaborative Tools: Adaptive Strategies for Enhanced Problem-Solving
  8. 2026Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimization