# Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Study

arXiv: https://arxiv.org/abs/2404.11792 · HTML: https://mlsanity.com/paper/2404.11792/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [Self-Evaluation Guided Beam Search for Reasoning](https://mlsanity.com/paper/2305.00633/) (2023)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks](https://mlsanity.com/paper/2305.18395/) (2023)
- [A Multi-Task Embedder For Retrieval Augmented LLMs](https://mlsanity.com/paper/2310.07554/) (2023)
- [RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models](https://mlsanity.com/paper/2308.07922/) (2023)
- [Retrieval-Generation Synergy Augmented Large Language Models](https://mlsanity.com/paper/2310.05149/) (2023)
- [Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks](https://mlsanity.com/paper/2305.17653/) (2023)

## Built on by (most cited first, 11 of 11)

- [BioRAG: A RAG-LLM Framework for Biological Question Reasoning](https://mlsanity.com/paper/2408.01107/) (2024)
- [KAQG: A Knowledge-Graph-Enhanced RAG for Difficulty-Controlled Question Generation](https://mlsanity.com/paper/2505.07618/) (2025)
- [Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents](https://mlsanity.com/paper/2603.11772/) (2026)
- [CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.13993/) (2025)
- [RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAG](https://mlsanity.com/paper/2511.04502/) (2025)
- [A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Science](https://mlsanity.com/paper/2412.15404/) (2024)
- [Smarter, not Bigger: Fine-Tuned RAG-Enhanced LLMs for Automotive HIL Testing](https://mlsanity.com/paper/2511.22584/) (2025)
- [A Human-Centred Architecture for Large Language Models-Cognitive Assistants in Manufacturing within Quality Management Systems](https://mlsanity.com/paper/2603.16325/) (2026)
- [Assessment of RAG and Fine-Tuning for Industrial Question-Answering-Applications](https://mlsanity.com/paper/2605.09533/) (2026)
- [HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Support](https://mlsanity.com/paper/2605.16347/) (2026)
- [MimirRAG: A Multi-Agent RAG Framework for Financial Data Retrieval with Metadata Integration](https://mlsanity.com/paper/2605.25030/) (2026)

## Abstract (arXiv)

This paper investigates the impact of domain-specific model fine-tuning and of reasoning mechanisms on the performance of question-answering (Q&A) systems powered by large language models (LLMs) and Retrieval-Augmented Generation (RAG). Using the FinanceBench SEC financial filings dataset, we observe that, for RAG, combining a fine-tuned embedding model with a fine-tuned LLM achieves better accuracy than generic models, with relatively greater gains attributable to fine-tuned embedding models. Additionally, employing reasoning iterations on top of RAG delivers an even bigger jump in performance, enabling the Q&A systems to get closer to human-expert quality. We discuss the implications of such findings, propose a structured technical design space capturing major technical components of Q&A AI, and provide recommendations for making high-impact technical choices for such components. We plan to follow up on this work with actionable guides for AI teams and further investigations into the impact of domain-specific augmentation in RAG and into agentic AI capabilities such as advanced planning and reasoning.