mlsanityData to 7 Oct 2026

Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow

arXiv 2402.09894 ↗First posted 15 Feb 2024cs.HC cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Generative AI brings novel and impressive abilities to help people in everyday tasks. There are many AI workflows that solve real and complex problems by chaining AI outputs together with human interaction. Although there is an undeniable lure of AI, it is uncertain how useful generative AI workflows are after the novelty wears off. Additionally, workflows built with generative AI have the potential to be easily customized to fit users' individual needs, but do users take advantage of this? We conducted a three-week longitudinal study with 12 users to understand the familiarization and customization of generative AI tools for science communication. Our study revealed that there exists a familiarization phase, during which users were exploring the novel capabilities of the workflow and discovering which aspects they found useful. After this phase, users understood the workflow and were able to anticipate the outputs. Surprisingly, after familiarization the perceived utility of the system was rated higher than before, indicating that the perceived utility of AI is not just a novelty effect. The increase in benefits mainly comes from end-users' ability to customize prompts, and thus potentially appropriate the system to their own needs. This points to a future where generative AI systems can allow us to design for appropriation.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 49 of its 128 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

49 of 128 references.

References · 49

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  9. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  10. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  11. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  12. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
37 more
  1. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  2. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 95
  3. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  4. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 81
  5. 2021Sparks: Inspiration for Science Writing using Language Modelscited by 80
  6. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  7. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 65
  8. 2023Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creationcited by 63
  9. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  10. 2023Teach LLMs to Personalize -- An Approach inspired by Writing Educationcited by 48
  11. 2023VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototypingcited by 46
  12. 20223DALL-E: Integrating Text-to-Image AI in 3D Design Workflowscited by 41
  13. 2022Opal: Multimodal Image Generation for News Illustrationcited by 36
  14. 2023CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Modelscited by 34
  15. 2023CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIcited by 34
  16. 2023ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principlescited by 33
  17. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31
  18. 2023ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Eventscited by 31
  19. 2023LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMscited by 27
  20. 2021PopBlends: Strategies for Conceptual Blending with Large Language Modelscited by 22
  21. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  22. 2023Generative Disco: Text-to-Video Generation for Music Visualizationcited by 19
  23. 2023ReelFramer: Human-AI Co-Creation for News-to-Video Translationcited by 19
  24. 2023Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflowscited by 17
  25. 2023GenQuery: Supporting Expressive Visual Search with Generative Modelscited by 14
  26. 2023Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastinationcited by 12
  27. 2023Tweetorial Hooks: Generative AI Tools to Motivate Science on Social Mediacited by 11
  28. 2022Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Modelscited by 10
  29. 2024TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generationcited by 10
  30. 2022The Placebo Effect of Artificial Intelligence in Human-Computer Interactioncited by 9
  31. 2024Canvil: Designerly Adaptation for LLM-Powered User Experiencescited by 9
  32. 2024Keyframer: Empowering Animation Design using Large Language Modelscited by 9
  33. 2023The Creative Frontier of Generative AI: Managing the Novelty-Usefulness Tradeoffcited by 6
  34. 2021Novelty and Primacy: A Long-Term Estimator for Online Experimentscited by 5
  35. 2023Perceptual Pat: A Virtual Human System for Iterative Visualization Designcited by 4
  36. 2024GraphiMind: LLM-centric Interface for Information Graphics Designcited by 4
  37. 2023Challenges and Opportunities for the Design of Smart Speakerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  2. 2025Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Makingcited by 9
  3. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  4. 2025Phraselette: A Poet's Procedural Palettecited by 4
  5. 2025FeedQUAC: Quick Unobtrusive AI-Generated Commentarycited by 2
  6. 2025Scaffolding Recursive Divergence and Convergence in Story Ideationcited by 2
  7. 2025More-than-Human Storytelling: Designing Longitudinal Narrative Engagements with Generative AIcited by 1
  8. 2025Facilitating Longitudinal Interaction Studies of AI Systemscited by 1
  9. 2025DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflowcited by 1
  10. 2026Priming, Path-dependence, and Plasticity: Understanding the molding of user-LLM interaction and its implications from (many) chat logs in the wildcited by 1
  11. 2026Adopt neq Adapt: Longitudinal Analyses of LLM Conversations in the Wildcited by 1
  12. 2026"Looking for Something Weird to Happen": How Humans Sustain AI Agent Novelty Amid Semantic Collapsecited by 0
1 more
  1. 2026ParallelPilot: Supporting Coordination and Monitoring in Parallel AI Codingcited by 0