mlsanityData to 14 Oct 2026

Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation

arXiv 2503.16789 ↗First posted 21 Mar 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Human-LLM conversations are increasingly becoming more pervasive in peoples' professional and personal lives, yet many users still struggle to elicit helpful responses from LLM Chatbots. One of the reasons for this issue is users' lack of understanding in crafting effective prompts that accurately convey their information needs. Meanwhile, the existence of real-world conversational datasets on the one hand, and the text understanding faculties of LLMs on the other, present a unique opportunity to study this problem, and its potential solutions at scale. Thus, in this paper we present the first LLM-centric study of real human-AI chatbot conversations, focused on investigating aspects in which user queries fall short of expressing information needs, and the potential of using LLMs to rewrite suboptimal user prompts. Our findings demonstrate that rephrasing ineffective prompts can elicit better responses from a conversational system, while preserving the user's original intent. Notably, the performance of rewrites improves in longer conversations, where contextual inferences about user needs can be made more accurately. Additionally, we observe that LLMs often need to -- and inherently do -- make plausible assumptions about a user's intentions and goals when interpreting prompts. Our findings largely hold true across conversational domains, user intents, and LLMs of varying sizes and families, indicating the promise of using prompt rewriting as a solution for better human-AI interactions.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

33 of 51 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  8. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  9. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  10. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 74
  11. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  12. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
21 more
  1. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  2. 2024Understanding User Experience in Large Language Model Interactionscited by 44
  3. 2023Learning to Rewrite Prompts for Personalized Text Generationcited by 40
  4. 2023Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomiescited by 34
  5. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  6. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  7. 2020Elaborative Simplification: Content Addition and Explanation Generation in Text Simplificationcited by 27
  8. 2024InterrogateLLM: Zero-Resource Hallucination Detection in LLM-Generated Answerscited by 27
  9. 2024Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seekingcited by 26
  10. 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 20
  11. 2023Natural Language Decompositions of Implicit Content Enable Better Text Representationscited by 19
  12. 2024Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Taskscited by 18
  13. 2024LLM Targeted Underperformance Disproportionately Impacts Vulnerable Userscited by 15
  14. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12
  15. 2023FOLLOWUPQG: Towards Information-Seeking Follow-up Question Generationcited by 12
  16. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  17. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  18. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  19. 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
  20. 2020Characterizing Stage-Aware Writing Assistance in Collaborative Document Authoringcited by 6
  21. 2024RepEval: Effective Text Evaluation with LLM Representationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  2. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  3. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 25
  4. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 16
  5. 2025Role-Augmented Intent-Driven Generative Search Engine Optimizationcited by 5
  6. 2025Show or Tell? Modeling the evolution of request-making in Human-LLM conversationscited by 3
  7. 2026AInterviewer: A Platform for Designing and Conducting AI-led Qualitative Interviewscited by 0
  8. 2026Automatically Evolving Prompt Guidelines for Task-Specific Optimizationcited by 0
  9. 2026How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversationscited by 0
  10. 2026SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RLcited by 0