mlsanityData to 7 Oct 2026

Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Study

arXiv 2404.11792 ↗First posted 17 Apr 2024cs.AIcited by 11

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

This paper investigates the impact of domain-specific model fine-tuning and of reasoning mechanisms on the performance of question-answering (Q&A) systems powered by large language models (LLMs) and Retrieval-Augmented Generation (RAG). Using the FinanceBench SEC financial filings dataset, we observe that, for RAG, combining a fine-tuned embedding model with a fine-tuned LLM achieves better accuracy than generic models, with relatively greater gains attributable to fine-tuned embedding models. Additionally, employing reasoning iterations on top of RAG delivers an even bigger jump in performance, enabling the Q&A systems to get closer to human-expert quality. We discuss the implications of such findings, propose a structured technical design space capturing major technical components of Q&A AI, and provide recommendations for making high-impact technical choices for such components. We plan to follow up on this work with actionable guides for AI teams and further investigations into the impact of domain-specific augmentation in RAG and into agentic AI capabilities such as advanced planning and reasoning.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 27 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

24 of 27 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
12 more
  1. 2023Active Retrieval Augmented Generationcited by 506
  2. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  3. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  4. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  5. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  6. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  7. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 108
  8. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  9. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  10. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  11. 2023Retrieval-Generation Synergy Augmented Large Language Modelscited by 26
  12. 2023Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Taskscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 5.

11 papers.

  1. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoningcited by 42
  2. 2025KAQG: A Knowledge-Graph-Enhanced RAG for Difficulty-Controlled Question Generationcited by 2
  3. 2026Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documentscited by 2
  4. 2025CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generationcited by 1
  5. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAGcited by 1
  6. 2024A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Sciencecited by 0
  7. 2025Smarter, not Bigger: Fine-Tuned RAG-Enhanced LLMs for Automotive HIL Testingcited by 0
  8. 2026A Human-Centred Architecture for Large Language Models-Cognitive Assistants in Manufacturing within Quality Management Systemscited by 0
  9. 2026Assessment of RAG and Fine-Tuning for Industrial Question-Answering-Applicationscited by 0
  10. 2026HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Supportcited by 0
  11. 2026MimirRAG: A Multi-Agent RAG Framework for Financial Data Retrieval with Metadata Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient In-Domain Question Answering for Resource-Constrained Environments
  2. 2025One More Question is Enough, Expert Question Decomposition (EQD) Model for Domain Quantitative Reasoning
  3. 2025Integrating Domain Knowledge for Financial QA: A Multi-Retriever RAG Approach with LLMs
  4. 2024Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems
  5. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains
  6. 2024Aggregated Knowledge Model: Enhancing Domain-Specific QA with Fine-Tuned and Retrieval-Augmented Generation Models
  7. 2025Beyond QA Pairs: Assessing Parameter-Efficient Fine-Tuning for Fact Embedding in LLMs
  8. 2024Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization