mlsanityData to 7 Oct 2026

The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions

arXiv 2310.12418 ↗First posted 19 Oct 2023cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Recent progress in Large Language Models (LLMs) has produced models that exhibit remarkable performance across a variety of NLP tasks. However, it remains unclear whether the existing focus of NLP research accurately captures the genuine requirements of human users. This paper provides a comprehensive analysis of the divergence between current NLP research and the needs of real-world NLP applications via a large-scale collection of user-GPT conversations. We analyze a large-scale collection of real user queries to GPT. We compare these queries against existing NLP benchmark tasks and identify a significant gap between the tasks that users frequently request from LLMs and the tasks that are commonly studied in academic research. For example, we find that tasks such as ``design'' and ``planning'' are prevalent in user interactions but are largely neglected or different from traditional NLP benchmarks. We investigate these overlooked tasks, dissect the practical challenges they pose, and provide insights toward a roadmap to make LLMs better aligned with user needs.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 33 of its 81 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

33 of 81 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2018Deep contextualized word representationscited by 2,285
21 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  6. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2023The False Promise of Imitating Proprietary LLMscited by 254
  9. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 246
  10. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  11. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  12. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  13. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  14. 2018Interpretation of Natural Language Rules in Conversational Machine Readingcited by 72
  15. 2019The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Servicecited by 47
  16. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  17. 2023When do you need Chain-of-Thought Prompting for ChatGPT?cited by 34
  18. 2019ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generationcited by 32
  19. 2020Dialogue Graph Modeling for Conversational Machine Readingcited by 26
  20. 2022Multimedia Generative Script Learning for Task Planningcited by 12
  21. 2017A Recorded Debating Datasetcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 21; 2026: 7.

45 papers.

Sort:

By area:

  1. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  2. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  3. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 88
  4. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  5. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66
  6. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  7. 2024Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrievalcited by 49
  8. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  9. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  10. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  11. 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 27
  12. 2024Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Modelscited by 26
33 more
  1. 2024WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queriescited by 26
  2. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  3. 2025Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Modelscited by 25
  4. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  5. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 22
  6. 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
  7. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  8. 2025Benchmarking Retrieval-Augmented Generation for Chemistrycited by 19
  9. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  10. 2023NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMscited by 14
  11. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11
  12. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 11
  13. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 10
  14. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8
  15. 2025Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactionscited by 8
  16. 2025Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classificationcited by 7
  17. 2025What Is The Political Content in LLMs' Pre- and Post-Training Data?cited by 7
  18. 2025Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personascited by 7
  19. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  20. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  21. 2024Exploring Human-LLM Conversations: Mental Models and the Originator of Toxicitycited by 4
  22. 2025Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMscited by 3
  23. 2026Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Taskscited by 3
  24. 2024Facilitating Opinion Diversity through Hybrid NLP Approachescited by 2
  25. 2025Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Casescited by 2
  26. 2025Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Contextcited by 2
  27. 2026AI Fiction in the Wildcited by 2
  28. 2026Adopt neq Adapt: Longitudinal Analyses of LLM Conversations in the Wildcited by 1
  29. 2025Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Modelscited by 0
  30. 2026LLM Benchmark-User Need Misalignment for Climate Changecited by 0
  31. 2026Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Researchcited by 0
  32. 2026How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversationscited by 0
  33. 2026WildSEEK: Evaluating Language Models for Information-Seekingcited by 0