mlsanityData to 14 Oct 2026

To Write or to Automate Linguistic Prompts, That Is the Question

arXiv 2603.25169 ↗First posted 26 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM performance is highly sensitive to prompt design, yet whether automatic prompt optimization can replace expert prompt engineering in linguistic tasks remains unexplored. We present the first systematic comparison of hand-crafted zero-shot expert prompts, base DSPy signatures, and GEPA-optimized DSPy signatures across translation, terminology insertion, and language quality assessment, evaluating five model configurations. Results are task-dependent. In terminology insertion, optimized and manual prompts produce mostly statistically indistinguishable quality. In translation, each approach wins on different models. In LQA, expert prompts achieve stronger error detection while optimization improves characterization. Across all tasks, GEPA elevates minimal DSPy signatures, and the majority of expert-optimized comparisons show no statistically significant difference. We note that the comparison is asymmetric: GEPA optimization searches programmatically over gold-standard splits, whereas expert prompts require in principle no labeled data, relying instead on domain expertise and iterative refinement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

31 of 41 references.

References · 31

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  8. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  9. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  10. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  12. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
19 more
  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  2. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  3. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  4. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 193
  5. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  6. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  7. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  8. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  9. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  10. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  11. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  12. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  13. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  14. 2023Domain Terminology Integration into Machine Translation: Leveraging Large Language Modelscited by 17
  15. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  16. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  17. 2024Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levelscited by 9
  18. 2023Revisiting Automated Prompting: Are We Actually Doing Better?cited by 7
  19. 2024Efficient Terminology Integration for LLM-based Translation in Specialized Domainscited by 7

Built on by

Not cited yet.