mlsanityData to 14 Oct 2026

Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes

arXiv 2608.23640 ↗First posted 23 Aug 2026cs.AI cs.CL cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

When a large language model (LLM) is asked to write a person's life, how much of what it writes actually happened? We present a scene-level case-study audit - the first quantified audit of LLM-generated autobiography against a subject-specific ground-truth corpus that we are aware of, based on an unsystematic literature search. The subject and the author of this paper are the same person: a 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM whose documented inputs were a template, two exemplar days, and each day's quote - not her corpus - and every day was subsequently audited at the anecdote-scene level against an independent verification corpus using a four-level rubric fixed before analysis. We define the verification-failure rate as the share of days not rated VERIFIED (scene positively corroborated): 354 of 366 days fail, 96.7% (Wilson 95% CI 94.4-98.1%). Only 12 days contain a corroborated scene; 19 days (5.2%) assert claims actively contradicted by the record; the dominant failure mode is grounded drift - real people, employers, and settings inside invented scenes - though its measured share varies across raters. Independent re-rating replicates the headline (no evidence the original rate was inflated) while showing that the four-way taxonomy has only fair-to-moderate reliability. Regenerating the same days with current named models reproduces 100% verification failure under the same inputs; grounding generation in the subject's corpus significantly improves the verification rate while leaving substantial residual failure (83.3%). We contribute the measurement, a reusable audit instrument whose WEAK/UNVERIFIED boundary we show to be unreliable, and a grounding remedy with quantified effect.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 17 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

17 of 17 references.

References · 17

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  6. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  7. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 249
  8. 2024Long-form factuality in large language modelscited by 155
  9. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  10. 2024Confabulation: The Surprising Value of Large Language Model Hallucinationscited by 43
  11. 2024PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counselingcited by 38
  12. 2024RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systemscited by 21
5 more
  1. 2025Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMscited by 9
  2. 2024Conversational AI Powered by Large Language Models Amplifies False Memories in Witness Interviewscited by 5
  3. 2024SHARP: Unlocking Interactive Hallucination via Stance Transfer in Role-Playing LLMscited by 5
  4. 2025Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answeringcited by 5
  5. 2025ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotationscited by 4

Built on by

Not cited yet.