mlsanityData to 14 Oct 2026

A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case

arXiv 2510.21933 ↗First posted 24 Oct 2025cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 47 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

15 of 47 references.

References · 15

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Using an LLM to Help With Code Understandingcited by 228
  3. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  4. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  5. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  6. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  7. 2023Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Studycited by 44
  8. 2024Test-Driven Development for Code Generationcited by 34
  9. 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
  10. 2024Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practicecited by 25
  11. 2024Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Taskscited by 13
  12. 2023Large Language Models for Education: Grading Open-Ended Questions Using ChatGPTcited by 11
3 more
  1. 2024What You Need is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistantcited by 8
  2. 2021Can I Solve It? Identifying APIs Required to Complete OSS Taskcited by 7
  3. 2025The Multifaceted Nature of Mentoring in OSS: Strategies, Qualities, and Ideal Outcomescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
  2. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
  3. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
  4. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
  5. 2024Deploying Large Language Models With Retrieval Augmented Generation
  6. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
  7. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  8. 2024Adaptive Retrieval-Augmented Generation for Conversational Systems