# Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generation

arXiv: https://arxiv.org/abs/2310.10706 · HTML: https://mlsanity.com/paper/2310.10706/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models](https://mlsanity.com/paper/2202.04173/) (2022)
- [Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning](https://mlsanity.com/paper/1805.10627/) (2018)
- [Perfection Not Required? Human-AI Partnerships in Code Translation](https://mlsanity.com/paper/2104.03820/) (2021)
- [Generating Representative Headlines for News Stories](https://mlsanity.com/paper/2001.09386/) (2020)
- [Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting](https://mlsanity.com/paper/2303.03199/) (2023)
- [AI Song Contest: Human-AI Co-Creation in Songwriting](https://mlsanity.com/paper/2010.05388/) (2020)
- [Visual Interaction with Deep Learning Models through Collaborative Semantic Inference](https://mlsanity.com/paper/1907.10739/) (2019)
- [Fluid Transformers and Creative Analogies: Exploring Large Language Models' Capacity for Augmenting Cross-Domain Analogical Creativity](https://mlsanity.com/paper/2302.12832/) (2023)
- [Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning](https://mlsanity.com/paper/1909.03582/) (2019)
- [Self-Attentive Model for Headline Generation](https://mlsanity.com/paper/1901.07786/) (2019)
- [Mapping the Design Space of Human-AI Interaction in Text Summarization](https://mlsanity.com/paper/2206.14863/) (2022)
- [Attractive or Faithful? Popularity-Reinforced Learning for Inspired Headline Generation](https://mlsanity.com/paper/2002.02095/) (2020)
- [GenNI: Human-AI Collaboration for Data-Backed Text Generation](https://mlsanity.com/paper/2110.10185/) (2021)
- [Generating News Headlines with Recurrent Neural Networks](https://mlsanity.com/paper/1512.01712/) (2015)
- [A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation](https://mlsanity.com/paper/1805.01553/) (2018)
- [THEaiTRE: Artificial Intelligence to Write a Theatre Play](https://mlsanity.com/paper/2006.14668/) (2020)
- [Changing the Mind of Transformers for Topically-Controllable Language Generation](https://mlsanity.com/paper/2103.15335/) (2021)
- [Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks](https://mlsanity.com/paper/2303.06430/) (2023)
- [Hone as You Read: A Practical Type of Interactive Summarization](https://mlsanity.com/paper/2105.02923/) (2021)

## Built on by (most cited first, 14 of 14)

- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [KG-Retriever: Efficient Knowledge Indexing for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2412.05547/) (2024)
- [SPHERE: An Evaluation Card for Human-AI Systems](https://mlsanity.com/paper/2504.07971/) (2025)
- [PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction](https://mlsanity.com/paper/2602.06714/) (2026)
- [LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content](https://mlsanity.com/paper/2410.15308/) (2024)
- [LLeMpower: Understanding Disparities in the Control and Access of Large Language Models](https://mlsanity.com/paper/2404.09356/) (2024)
- ["The Diagram is like Guardrails": Structuring GenAI-assisted Hypotheses Exploration with an Interactive Shared Representation](https://mlsanity.com/paper/2503.16791/) (2025)
- [Frontend Diffusion: Exploring Intent-Based User Interfaces through Abstract-to-Detailed Task Transitions](https://mlsanity.com/paper/2408.00778/) (2024)
- [Facilitating Opinion Diversity through Hybrid NLP Approaches](https://mlsanity.com/paper/2405.09439/) (2024)
- [FeedQUAC: Quick Unobtrusive AI-Generated Commentary](https://mlsanity.com/paper/2504.16416/) (2025)
- [Multilingual Fine-Grained News Headline Hallucination Detection](https://mlsanity.com/paper/2407.15975/) (2024)
- [Co-creation for Sign Language Processing and Machine Translation](https://mlsanity.com/paper/2503.01553/) (2025)
- [Trust in Disinformation Narratives: a Trust in the News Experiment](https://mlsanity.com/paper/2503.11116/) (2025)
- [When Less Is Enough: Context Selection and Prompting Strategies for Bengali News Headline Generation](https://mlsanity.com/paper/2608.15879/) (2026)

## Abstract (arXiv)

To explore how humans can best leverage LLMs for writing and how interacting with these models affects feelings of ownership and trust in the writing process, we compared common human-AI interaction types (e.g., guiding system, selecting from system outputs, post-editing outputs) in the context of LLM-assisted news headline generation. While LLMs alone can generate satisfactory news headlines, on average, human control is needed to fix undesirable model outputs. Of the interaction methods, guiding and selecting model output added the most benefit with the lowest cost (in time and effort). Further, AI assistance did not harm participants' perception of control compared to freeform editing.