mlsanityData to 7 Oct 2026

TEMPERA: Test-Time Prompting via Reinforcement Learning

arXiv 2211.11890 ↗First posted 21 Nov 2022cs.CL cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Careful prompt design is critical to the use of large language models in zero-shot or few-shot learning. As a consequence, there is a growing interest in automated methods to design optimal prompts. In this work, we propose Test-time Prompt Editing using Reinforcement learning (TEMPERA). In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge, is adaptive to different queries and provides an interpretable prompt for every query. To achieve this, we design a novel action space that allows flexible editing of the initial prompts covering a wide set of commonly-used components like instructions, few-shot exemplars, and verbalizers. The proposed method achieves significant gains compared with recent SoTA approaches like prompt tuning, AutoPrompt, and RLPrompt, across a variety of tasks including sentiment analysis, topic classification, natural language inference, and reading comprehension. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

30 of 34 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
18 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2021GPT Understands, Toocited by 715
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  10. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  11. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  12. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  13. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  14. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  15. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  16. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  17. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  18. 2022Instance-wise Prompt Tuning for Pretrained Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 13; 2025: 12; 2026: 5.

50 papers.

Sort:

By area:

  1. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  2. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  3. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  4. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  5. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  6. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  7. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  8. 2023In-context Example Selection with Influencescited by 60
  9. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  10. 2023What's the Magic Word? A Control Theory of LLM Promptingcited by 41
  11. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Modelscited by 37
  12. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
38 more
  1. 2024PromptWizard: Task-Aware Prompt Optimization Frameworkcited by 32
  2. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  3. 2023AutoHint: Automatic Prompt Optimization with Hint Generationcited by 29
  4. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 28
  5. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  6. 2024Task Facet Learning: A Structured Approach to Prompt Optimizationcited by 17
  7. 2023Pre-Trained Large Language Models for Industrial Controlcited by 15
  8. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  9. 2025MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimizationcited by 15
  10. 2023Universal Self-Adaptive Promptingcited by 14
  11. 2023SPELL: Semantic Prompt Evolution based on a LLMcited by 14
  12. 2024Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary casescited by 13
  13. 2023Explanation Selection Using Unlabeled Data for Chain-of-Thought Promptingcited by 12
  14. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  15. 2023Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actorscited by 9
  16. 2024Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Modelscited by 7
  17. 2024Large Language Models are Interpretable Learnerscited by 7
  18. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  19. 2023Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiencycited by 5
  20. 2025Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMscited by 5
  21. 2023A Bayesian approach for prompt optimization in pre-trained language modelscited by 4
  22. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  23. 2024CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM Code Assistantscited by 3
  24. 2025TAPO: Task-Referenced Adaptation for Prompt Optimizationcited by 3
  25. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  26. 2023Matching Table Metadata with Business Glossaries Using Large Language Modelscited by 2
  27. 2024Diffusion-Enhanced Test-time Adaptation with Text and Image Augmentationcited by 2
  28. 2025Efficient Algorithms for Logistic Contextual Slate Bandits with Bandit Feedbackcited by 2
  29. 2025TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewardscited by 2
  30. 2025Better by Comparison: Retrieval-Augmented Contrastive Reasoning for Automatic Prompt Optimizationcited by 2
  31. 2025Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregationcited by 1
  32. 2025Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoningcited by 1
  33. 2026Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshapingcited by 1
  34. 2024TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Promptscited by 0
  35. 2025Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRIcited by 0
  36. 2026PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responsescited by 0
  37. 2026In-Context Learning as Implicit Policy Gradientcited by 0
  38. 2026On the Capability and Limitation of Hard Promptcited by 0