# Paraphrase Types Elicit Prompt Engineering Capabilities

arXiv: https://arxiv.org/abs/2406.19898 · HTML: https://mlsanity.com/paper/2406.19898/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews](https://mlsanity.com/paper/1303.4402/) (2013)
- [An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels](https://mlsanity.com/paper/2203.11364/) (2022)
- [The language of prompting: What linguistic properties make a prompt successful?](https://mlsanity.com/paper/2311.01967/) (2023)
- [How Large Language Models are Transforming Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2210.03568/) (2022)
- [How are Prompts Different in Terms of Sensitivity?](https://mlsanity.com/paper/2311.07230/) (2023)

## Built on by (most cited first, 15 of 15)

- [Benchmarking Linguistic Diversity of Large Language Models](https://mlsanity.com/paper/2412.10271/) (2024)
- [When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas](https://mlsanity.com/paper/2505.19212/) (2025)
- [Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs](https://mlsanity.com/paper/2504.02789/) (2025)
- [SPaRC: A Spatial Pathfinding Reasoning Challenge](https://mlsanity.com/paper/2505.16686/) (2025)
- [LLM Library Learning Fails: A LEGO-Prover Case Study](https://mlsanity.com/paper/2504.03048/) (2025)
- [Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework](https://mlsanity.com/paper/2505.03563/) (2025)
- [How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs](https://mlsanity.com/paper/2507.14307/) (2025)
- [One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation](https://mlsanity.com/paper/2605.22544/) (2026)
- [Towards Human Understanding of Paraphrase Types in Large Language Models](https://mlsanity.com/paper/2407.02302/) (2024)
- [How Small Transformation Expose the Weakness of Semantic Similarity Measures](https://mlsanity.com/paper/2509.09714/) (2025)
- [RCScore: Quantifying Response Consistency in Large Language Models](https://mlsanity.com/paper/2510.26193/) (2025)
- [Zero-Shot Anomaly Detection in Battery Thermal Images Using Visual Question Answering with Prior Knowledge](https://mlsanity.com/paper/2505.16674/) (2025)
- [Emergent retokenization symmetry in large language models: phenomenology and applications](https://mlsanity.com/paper/2606.15521/) (2026)
- [Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute](https://mlsanity.com/paper/2608.09351/) (2026)
- [Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility](https://mlsanity.com/paper/2608.10315/) (2026)

## Abstract (arXiv)

Much of the success of modern language models depends on finding a suitable prompt to instruct the model. Until now, it has been largely unknown how variations in the linguistic expression of prompts affect these models. This study systematically and empirically evaluates which linguistic features influence models through paraphrase types, i.e., different linguistic changes at particular positions. We measure behavioral changes for five models across 120 tasks and six families of paraphrases (i.e., morphology, syntax, lexicon, lexico-syntax, discourse, and others). We also control for other prompt engineering factors (e.g., prompt length, lexical diversity, and proximity to training data). Our results show a potential for language models to improve tasks when their prompts are adapted in specific paraphrase types (e.g., 6.7% median gain in Mixtral 8x7B; 5.5% in LLaMA 3 8B). In particular, changes in morphology and lexicon, i.e., the vocabulary used, showed promise in improving prompts. These findings contribute to developing more robust language models capable of handling variability in linguistic expression.