mlsanityData to 7 Oct 2026

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

arXiv 2605.00964 ↗First posted 1 May 2026cs.IR cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human-AI workflows has stayed behind. This work evaluates a chatbot-style assistant based on Retrieval-Augmented Generation (RAG) in a realistic multi-turn information-seeking scenario inspired by workplace settings where compliance with local legislation and secure handling of sensitive data are often key. Specifically, we examine the performance of humans (N=112) assisted by RAG-assistants compared to LLM-only or LLM+RAG baselines. In this setting, we investigate how underlying model size (3B, 8B, and 70B) shapes the human-AI collaborative dynamic and how it influences perceived usability and satisfaction. Results show that the performance gain of human-AI collaboration over the model-only baselines is significant, irrespective of model size, suggesting that hybrid systems are beneficial in information-seeking scenarios. Interestingly, however, perceived usability and satisfaction among participants showed little difference across model sizes. This demonstrates a nuanced trade-off between model size, performance, and user perception. Our work highlights the added value of evaluating AI applications in actual multi-turn interactions with human users, looking at usability and satisfaction besides accuracy, rather than focusing on benchmark performance only.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 68 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

46 of 68 references.

References · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  12. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
34 more
  1. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  2. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  3. 2025Small Language Models are the Future of Agentic AIcited by 254
  4. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  5. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  6. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  7. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  8. 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 100
  9. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  10. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  11. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  12. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  13. 2023Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Makingcited by 67
  14. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  15. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  16. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  17. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  18. 2024A Taxonomy for Human-LLM Interaction Modes: An Initial Explorationcited by 38
  19. 2024Trust in AI: Progress, Challenges, and Future Directionscited by 35
  20. 2023WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpuscited by 29
  21. 2022Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Makingcited by 28
  22. 2025MiniRAG: Towards Extremely Simple Retrieval-Augmented Generationcited by 28
  23. 2024GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulationcited by 22
  24. 2024A Survey of Conversational Searchcited by 22
  25. 2024Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Modelscited by 19
  26. 2024Small Language Models are Good Too: An Empirical Study of Zero-Shot Classificationcited by 17
  27. 2022Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translationcited by 14
  28. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  29. 2024Evaluating Explanations Through LLMs: Beyond Traditional User Studiescited by 8
  30. 2024A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Casecited by 6
  31. 2025When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaborationcited by 4
  32. 2024A Speaker Turn-Aware Multi-Task Adversarial Network for Joint User Satisfaction Estimation and Sentiment Analysiscited by 2
  33. 2022Go-tuning: Improving Zero-shot Learning Abilities of Smaller Language Modelscited by 1
  34. 2025Human-Centered Evaluation of RAG outputs: a framework and questionnaire for human-AI collaborationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks
  2. 2025A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
  3. 2024Deploying Large Language Models With Retrieval Augmented Generation
  4. 2024FACTS About Building Retrieval Augmented Generation-based Chatbots
  5. 2024A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
  6. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
  7. 2025Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation
  8. 2023Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbook