# Establishing Performance Baselines in Fine-Tuning, Retrieval-Augmented Generation and Soft-Prompting for Non-Specialist LLM Users

arXiv: https://arxiv.org/abs/2311.05903 · HTML: https://mlsanity.com/paper/2311.05903/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities](https://mlsanity.com/paper/2310.10436/) (2023)
- [Theory of Mind for Multi-Agent Collaboration via Large Language Models](https://mlsanity.com/paper/2310.10701/) (2023)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets](https://mlsanity.com/paper/2310.04793/) (2023)
- [FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions](https://mlsanity.com/paper/2305.17718/) (2023)
- [Exploring the Intersection of Large Language Models and Agent-Based Modeling via Prompt Engineering](https://mlsanity.com/paper/2308.07411/) (2023)
- [AmadeusGPT: a natural language interface for interactive animal behavioral analysis](https://mlsanity.com/paper/2307.04858/) (2023)
- [Towards reducing hallucination in extracting information from financial reports using Large Language Models](https://mlsanity.com/paper/2310.10760/) (2023)
- [Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems](https://mlsanity.com/paper/2310.01991/) (2023)
- [The Trip to The Enterprise Gourmet Data Product Marketplace through a Self-service Data Platform](https://mlsanity.com/paper/2107.13212/) (2021)

## Built on by (most cited first, 6 of 6)

- [Instruction Fine-Tuning: Does Prompt Loss Matter?](https://mlsanity.com/paper/2401.13586/) (2024)
- [MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector](https://mlsanity.com/paper/2408.08661/) (2024)
- [Overcoming LLM Challenges using RAG-Driven Precision in Coffee Leaf Disease Remediation](https://mlsanity.com/paper/2405.01310/) (2024)
- [AutoPureData: Automated Filtering of Undesirable Web Data to Update LLM Knowledge](https://mlsanity.com/paper/2406.19271/) (2024)
- [Capturing Bias Diversity in LLMs](https://mlsanity.com/paper/2410.12839/) (2024)
- [Vision Meets Language: A RAG-Augmented YOLOv8 Framework for Coffee Disease Diagnosis and Farmer Assistance](https://mlsanity.com/paper/2505.21544/) (2025)

## Abstract (arXiv)

Research into methods for improving the performance of large language models (LLMs) through fine-tuning, retrieval-augmented generation (RAG) and soft-prompting has tended to focus on the use of highly technical or high-cost techniques, making many of the newly discovered approaches comparatively inaccessible to non-technical users. In this paper we tested an unmodified version of GPT 3.5, a fine-tuned version, and the same unmodified model when given access to a vectorised RAG database, both in isolation and in combination with a basic, non-algorithmic soft prompt. In each case we tested the model's ability to answer a set of 100 questions relating primarily to events that occurred after September 2021 (the point at which GPT 3.5's training data set ends). We found that if commercial platforms are used and default settings are applied with no iteration in order to establish a baseline set of outputs, a fine-tuned model outperforms GPT 3.5 Turbo, while the RAG approach out-performed both. The application of a soft prompt significantly improved the performance of each approach.