mlsanityData to 7 Oct 2026

Unveiling the Lexical Sensitivity of LLMs: Combinatorial Optimization for Prompt Enhancement

arXiv 2405.20701 ↗First posted 31 May 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) demonstrate exceptional instruct-following ability to complete various downstream tasks. Although this impressive ability makes LLMs flexible task solvers, their performance in solving tasks also heavily relies on instructions. In this paper, we reveal that LLMs are over-sensitive to lexical variations in task instructions, even when the variations are imperceptible to humans. By providing models with neighborhood instructions, which are closely situated in the latent representation space and differ by only one semantically similar word, the performance on downstream tasks can be vastly different. Following this property, we propose a black-box Combinatorial Optimization framework for Prompt Lexical Enhancement (COPLE). COPLE performs iterative lexical optimization according to the feedback from a batch of proxy tasks, using a search strategy related to word influence. Experiments show that even widely-used human-crafted prompts for current benchmarks suffer from the lexical sensitivity of models, and COPLE recovers the declined model ability in both instruct-following and solving downstream tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 30 of its 63 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

30 of 63 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
18 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2018Neural Network Acceptability Judgmentscited by 952
  7. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  8. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  9. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  10. 2018Universal Sentence Encodercited by 485
  11. 2023Larger language models do in-context learning differentlycited by 386
  12. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  13. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  14. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  15. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  16. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  17. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  18. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 61
  2. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Timecited by 12
  3. 2025No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimizationcited by 7
  4. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  5. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  6. 2025Neuron-Level Analysis of Cultural Understanding in Large Language Modelscited by 1
  7. 2026Prompt Segmentation and Annotation Optimisation: Controlling LLM Behaviour via Optimised Segment-Level Annotationscited by 0
  8. 2026Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuningcited by 0
  9. 2026When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variationscited by 0
  10. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  11. 2026Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluationcited by 0
  12. 2026Prompt-Robust Language Models: Which Training Strategies Work?cited by 0
1 more
  1. 2026STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verificationcited by 0