# Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together

arXiv: https://arxiv.org/abs/2407.10930 · HTML: https://mlsanity.com/paper/2407.10930/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 15 of 15)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval](https://mlsanity.com/paper/2101.00436/) (2021)
- [Answering Open-Domain Questions of Varying Reasoning Steps from Text](https://mlsanity.com/paper/2010.12527/) (2020)
- [Prompt2Model: Generating Deployable Models from Natural Language Instructions](https://mlsanity.com/paper/2308.12261/) (2023)

## Built on by (most cited first, 30 of 40)

- [StructuredRAG: JSON Response Formatting with Large Language Models](https://mlsanity.com/paper/2408.11061/) (2024)
- [Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval](https://mlsanity.com/paper/2410.23214/) (2024)
- [Abacus: A Cost-Based Optimizer for Semantic Operator Systems](https://mlsanity.com/paper/2505.14661/) (2025)
- [Tuning LLM Judge Design Decisions for 1/1000 of the Cost](https://mlsanity.com/paper/2501.17178/) (2025)
- [WHALE: A Simple Recipe for Joint Harness-Weight Optimization](https://mlsanity.com/paper/2609.00196/) (2026)
- [PRL: Prompts from Reinforcement Learning](https://mlsanity.com/paper/2505.14412/) (2025)
- [DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discovery](https://mlsanity.com/paper/2508.06960/) (2025)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [A Survey of Automatic Prompt Optimization with Instruction-focused Heuristic-based Search Algorithm](https://mlsanity.com/paper/2502.18746/) (2025)
- [promptolution: A Unified, Modular Framework for Prompt Optimization](https://mlsanity.com/paper/2512.02840/) (2025)
- [Learning, Fast and Slow: Towards LLMs That Adapt Continually](https://mlsanity.com/paper/2605.12484/) (2026)
- [Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2506.08234/) (2025)
- [Composing Policy Gradients and Prompt Optimization for Language Model Programs](https://mlsanity.com/paper/2508.04660/) (2025)
- [Code-enabled language models can outperform reasoning models on diverse tasks](https://mlsanity.com/paper/2510.20909/) (2025)
- [Auto prompting without training labels: An LLM cascade for product quality assessment in e-commerce catalogs](https://mlsanity.com/paper/2510.23941/) (2025)
- [Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge](https://mlsanity.com/paper/2506.22644/) (2025)
- [MedVAL: Toward Expert-Level Medical Text Validation with Language Models](https://mlsanity.com/paper/2507.03152/) (2025)
- [TextBO: Bayesian Optimization in Language Space for Eval-Efficient Self-Improving AI](https://mlsanity.com/paper/2511.12063/) (2025)
- [Agentic Policy Optimization via Instruction-Policy Co-Evolution](https://mlsanity.com/paper/2512.01945/) (2025)
- [OpenJarvis: Personal AI, On Personal Devices](https://mlsanity.com/paper/2605.17172/) (2026)
- [Clinical trial cohort selection using Large Language Models on n2c2 Challenges](https://mlsanity.com/paper/2501.11114/) (2025)
- [LangProBe: a Language Programs Benchmark](https://mlsanity.com/paper/2502.20315/) (2025)
- [Prompt and Parameter Co-Optimization for Large Language Models](https://mlsanity.com/paper/2509.24245/) (2025)
- [PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data](https://mlsanity.com/paper/2512.11013/) (2025)
- [COSMOS: Predictable and Cost-Effective Adaptation of LLMs](https://mlsanity.com/paper/2505.01449/) (2025)
- [Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models](https://mlsanity.com/paper/2507.10103/) (2025)
- [Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data](https://mlsanity.com/paper/2508.18244/) (2025)
- [Neurosymbolic LoRA: Why and When to Tune Weights vs. Rewrite Prompts](https://mlsanity.com/paper/2601.12711/) (2026)
- [Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning](https://mlsanity.com/paper/2601.23027/) (2026)
- [TATRA: Training-Free Instance-Adaptive Prompting Through Rephrasing and Aggregation](https://mlsanity.com/paper/2603.03298/) (2026)

## Abstract (arXiv)

Natural Language Processing (NLP) systems are increasingly taking the form of sophisticated modular pipelines, e.g., Retrieval Augmented Generation (RAG), where each module may involve a distinct Language Model (LM) and an associated prompt template. These compound systems often lack intermediate labels or gradient flow to optimize each module, making their end-to-end optimization challenging. Here we seek strategies to optimize both the module-level LM weights and the associated prompt templates of such systems to maximize a downstream task metric. We propose for the first time combining the weight and prompt optimization strategies to optimize a modular LM pipeline by alternating between the two to get the same LM to teach itself. In experiments with multi-hop QA, mathematical reasoning, and feature-based classification using mistral-7b, llama-2-7b, and llama-3-8b, these BetterTogether strategies optimizing the weights and prompts of a pipeline together outperform directly optimizing weights alone and prompts alone by up to 60% and 6%, respectively, on average across LMs and tasks. BetterTogether optimizer is released in DSPy at http://dspy.ai