mlsanityData to 14 Oct 2026

A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation

arXiv 2412.15298 ↗First posted 19 Dec 2024cs.CL cs.AI cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 9 of its 15 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

9 of 15 references.

References · 9

  1. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  5. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  6. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  7. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  8. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 57
  9. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026Spec-Harness: Assessing and Guiding LLM-Synthesized Formal Specifications Beyond Verifiabilitycited by 2
  2. 2025Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translationcited by 1
  3. 2026FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Textscited by 1
  4. 2026Prompt Segmentation and Annotation Optimisation: Controlling LLM Behaviour via Optimised Segment-Level Annotationscited by 1
  5. 2025Data Diversification Methods In Alignment Enhance Math Performance In LLMscited by 0
  6. 2026From Errors to Rules: Iterative Prompt Optimization for Text Classificationcited by 0
  7. 2026Benchmarking Prompt Optimization of Large Language Models With Chesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models
  2. 2024Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
  3. 2026Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
  4. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  5. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  6. 2025Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
  8. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models