Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing?
Not on the map. Why?
Abstract (arXiv)
AI-text detection generally asks whether a machine wrote a text. As AI becomes an increasingly integral and accepted part of writing, the more relevant question is no longer whether AI was used, but how much of the text reflects human contribution. A prominent line of work estimates this by measuring how predictive a person's prompt is of the final text. We show that this measure can be easily manipulated. We introduce SPECIFICITY PADDING that simply asks the model to copy a few random IDs from the prompt, and that alone makes a title-only request look more human than one that supplies the article's key ideas. A context-swap audit then holds each document fixed, and only shows or hides the IDs to the evaluator. The score of the unchanged text swings by a third to half of its scale, and it rises for every document we test, on news and scientific abstracts, across varied LLMs from 3.8B to 753B parameters. The same articles can average 85% human or 15% human depending only on which prompt is on record, and even with no prompt at all, the output-only variant rewards the random strings themselves. An exact identity, the credit ledger, explains why. The score is a surprisal-weighted average over the text, so rare strings that the context makes copyable collect the credit, whatever they mean. A two-point bound marks a deeper limit. If the prompt could have been machine-written, every transcript-only method must, in the worst case, miss by half the contribution at stake. Likelihood-based scores alone cannot reliably establish human contribution without evidence of what the person added and how the text was produced.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
31 of 47 references.
References · 31
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
- 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
- 2023Enabling Large Language Models to Generate Text with Citationscited by 595
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
- 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 496
- 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 429
- 2019Measurement and Fairnesscited by 308
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
- 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 287
19 more
- 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 263
- 2021Measuring Attribution in Natural Language Generation Modelscited by 256
- 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
- 2020A Theory of Usable Information Under Computational Constraintscited by 162
- 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
- 2019On the Use of ArXiv as a Datasetcited by 103
- 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
- 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
- 2024LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected?cited by 49
- 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
- 2022Reward Gaming in Conditional Text Generationcited by 30
- 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
- 2023The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualizationcited by 24
- 2024Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurementcited by 17
- 2025HACo-Det: A Study Towards Fine-Grained Machine-Generated Text Detection under Human-AI Coauthoringcited by 15
- 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 13
- 2024Measuring Human Contribution in AI-Assisted Content Generationcited by 6
- 2025FAID: Fine-Grained AI-Generated Text Detection Using Multi-Task Auxiliary and Multi-Level Contrastive Learningcited by 6
- 2025Measuring Human Involvement in AI-Generated Text: A Case Study on Academic Writingcited by 1
Built on by
Not cited yet.