mlsanityData to 7 Oct 2026

Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary cases

arXiv 2402.03099 ↗First posted 5 Feb 2024cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Prompt engineering is a challenging and important task due to the high sensitivity of Large Language Models (LLMs) to the given prompt and the inherent ambiguity of a textual task instruction. Automatic prompt engineering is essential to achieve optimized performance from LLMs. Recent studies have demonstrated the capabilities of LLMs to automatically conduct prompt engineering by employing a meta-prompt that incorporates the outcomes of the last trials and proposes an improved prompt. However, this requires a high-quality benchmark to compare different prompts, which is difficult and expensive to acquire in many real-world use cases. In this work, we introduce a new method for automatic prompt engineering, using a calibration process that iteratively refines the prompt to the user intent. During the optimization process, the system jointly generates synthetic data of boundary use cases and optimizes the prompt according to the generated dataset. We demonstrate the effectiveness of our method with respect to strong proprietary models on real-world tasks such as moderation and generation. Our method outperforms state-of-the-art methods with a limited number of annotated samples. Furthermore, we validate the advantages of each one of the system's key components. Our system is built in a modular way, facilitating easy adaptation to other tasks. The code is available hrefhttps://github.com/Eladlev/AutoPrompthere.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

32 of 47 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2023Large Language Models as Optimizerscited by 830
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  10. 2023Large Language Modelscited by 774
  11. 2021GPT Understands, Toocited by 715
  12. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
20 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2023Textbooks Are All You Needcited by 561
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  5. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  7. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  8. 2023Improving Text Embeddings with Large Language Modelscited by 295
  9. 2023How is ChatGPT's behavior changing over time?cited by 265
  10. 2021Curriculum Learning: A Surveycited by 257
  11. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  12. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  13. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  14. 2015Webly Supervised Learning of Convolutional Networkscited by 95
  15. 2017Curriculum Learning and Minibatch Bucketing in Neural Machine Translationcited by 63
  16. 2017How hard can it be? Estimating the difficulty of visual search in an imagecited by 52
  17. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  18. 2016Self Paced Deep Learning for Weakly Supervised Object Detectioncited by 25
  19. 2016Weakly Supervised Object Localization Using Size Estimatescited by 16
  20. 2023Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Datacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 1.

13 papers.

Sort:

By area:

  1. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  2. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  3. 2025AIP: Subverting Retrieval-Augmented Generation via Adversarial Instructional Promptcited by 5
  4. 2024What is the Role of Large Language Models in the Evolution of Astronomy Research?cited by 4
  5. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  6. 2025GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimizationcited by 3
  7. 2025Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanationscited by 3
  8. 2024Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Promptingcited by 0
  9. 2024Can we only use guideline instead of shot in prompt?cited by 0
  10. 2024Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Designcited by 0
  11. 2025PARASITE: Conditional System Prompt Poisoning to Hijack LLMscited by 0
  12. 2025PRAISE: Enhancing Product Descriptions with LLM-Driven Structured Insightscited by 0
1 more
  1. 2026BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debatecited by 0