mlsanityData to 7 Oct 2026

Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following

arXiv 2302.14691 ↗First posted 28 Feb 2023cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

In this paper, we present our finding that prepending a Task-Agnostic Prefix Prompt (TAPP) to the input improves the instruction-following ability of various Large Language Models (LLMs) during inference. TAPP is different from canonical prompts for LLMs in that it is a fixed prompt prepended to the beginning of every input regardless of the target task for zero-shot generalization. We observe that both base LLMs (i.e. not fine-tuned to follow instructions) and instruction-tuned models benefit from TAPP, resulting in 34.58% and 12.26% improvement on average, respectively. This implies that the instruction-following ability of LLMs can be improved during inference time with a fixed prompt constructed with simple heuristics. We hypothesize that TAPP assists language models to better estimate the output distribution by focusing more on the instruction of the target task during inference. In other words, such ability does not seem to be sufficiently activated in not only base LLMs but also many instruction-fine-tuned LLMs. All experiments are reproducible from https://github.com/seonghyeonye/TAPP.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

24 of 36 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
12 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2021GPT Understands, Toocited by 715
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  7. 2021MetaICL: Learning to Learn In Contextcited by 450
  8. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  10. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  11. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  12. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 13; 2025: 5; 2026: 1.

33 papers.

Sort:

By area:

  1. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  2. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  3. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  4. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  5. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  6. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoningcited by 49
  7. 2023Interactive Natural Language Processingcited by 31
  8. 2024CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informaticscited by 25
  9. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  10. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  11. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  12. 2024Nudging: Inference-time Alignment of LLMs via Guided Decodingcited by 22
21 more
  1. 2023Generative Calibration for In-context Learningcited by 20
  2. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  3. 2025Causally Reliable Concept Bottleneck Modelscited by 17
  4. 2023From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuningcited by 14
  5. 2023Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineeringcited by 13
  6. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  7. 2024Say More with Less: Understanding Prompt Learning Behaviors through Gist Compressioncited by 8
  8. 2024Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hackcited by 8
  9. 2024Challenges Faced by Large Language Models in Solving Multi-Agent Flockingcited by 7
  10. 2025The Few-shot Dilemma: Over-prompting Large Language Modelscited by 7
  11. 2023JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuningcited by 6
  12. 2024How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignmentcited by 6
  13. 2024FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarioscited by 6
  14. 2024Improving Language Model Reasoning with Self-motivated Learningcited by 4
  15. 2024Linearly-Interpretable Concept Embedding Models for Text Analysiscited by 3
  16. 2023Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Samplingcited by 2
  17. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Modelscited by 1
  18. 2025Enhancing Clinical Note Generation with ICD-10, Clinical Ontology Knowledge Graphs, and Chain-of-Thought Prompting Using GPT-4cited by 1
  19. 2024textitGeoHard: Towards Measuring Class-wise Hardness through Modelling Class Semanticscited by 0
  20. 2025The Impact of Role Design in In-Context Learning for Large Language Modelscited by 0
  21. 2026Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICLcited by 0