# Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary cases

arXiv: https://arxiv.org/abs/2402.03099 · HTML: https://mlsanity.com/paper/2402.03099/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://mlsanity.com/paper/2305.08844/) (2023)
- [Webly Supervised Learning of Convolutional Networks](https://mlsanity.com/paper/1505.01554/) (2015)
- [Curriculum Learning and Minibatch Bucketing in Neural Machine Translation](https://mlsanity.com/paper/1707.09533/) (2017)
- [How hard can it be? Estimating the difficulty of visual search in an image](https://mlsanity.com/paper/1705.08280/) (2017)
- [TEMPERA: Test-Time Prompting via Reinforcement Learning](https://mlsanity.com/paper/2211.11890/) (2022)
- [Self Paced Deep Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1605.07651/) (2016)

## Built on by (most cited first, 13 of 13)

- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [Large Language Models Are Active Critics in NLG Evaluation](https://mlsanity.com/paper/2410.10724/) (2024)
- [Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models](https://mlsanity.com/paper/2504.20157/) (2025)
- [AIP: Subverting Retrieval-Augmented Generation via Adversarial Instructional Prompt](https://mlsanity.com/paper/2509.15159/) (2025)
- [What is the Role of Large Language Models in the Evolution of Astronomy Research?](https://mlsanity.com/paper/2409.20252/) (2024)
- [Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations](https://mlsanity.com/paper/2510.21631/) (2025)
- [GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization](https://mlsanity.com/paper/2504.03975/) (2025)
- [Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Prompting](https://mlsanity.com/paper/2408.01423/) (2024)
- [Can we only use guideline instead of shot in prompt?](https://mlsanity.com/paper/2409.12979/) (2024)
- [Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design](https://mlsanity.com/paper/2412.06432/) (2024)
- [PARASITE: Conditional System Prompt Poisoning to Hijack LLMs](https://mlsanity.com/paper/2505.16888/) (2025)
- [PRAISE: Enhancing Product Descriptions with LLM-Driven Structured Insights](https://mlsanity.com/paper/2506.17314/) (2025)
- [BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate](https://mlsanity.com/paper/2604.25203/) (2026)

## Abstract (arXiv)

Prompt engineering is a challenging and important task due to the high sensitivity of Large Language Models (LLMs) to the given prompt and the inherent ambiguity of a textual task instruction. Automatic prompt engineering is essential to achieve optimized performance from LLMs. Recent studies have demonstrated the capabilities of LLMs to automatically conduct prompt engineering by employing a meta-prompt that incorporates the outcomes of the last trials and proposes an improved prompt. However, this requires a high-quality benchmark to compare different prompts, which is difficult and expensive to acquire in many real-world use cases. In this work, we introduce a new method for automatic prompt engineering, using a calibration process that iteratively refines the prompt to the user intent. During the optimization process, the system jointly generates synthetic data of boundary use cases and optimizes the prompt according to the generated dataset. We demonstrate the effectiveness of our method with respect to strong proprietary models on real-world tasks such as moderation and generation. Our method outperforms state-of-the-art methods with a limited number of annotated samples. Furthermore, we validate the advantages of each one of the system's key components. Our system is built in a modular way, facilitating easy adaptation to other tasks. The code is available hrefhttps://github.com/Eladlev/AutoPrompthere.