mlsanityData to 7 Oct 2026

Does Prompt Formatting Have Any Impact on LLM Performance?

arXiv 2411.10541 ↗First posted 15 Nov 2024cs.CL cs.LGcited by 132

Not on the map. Why?

Cited by 21 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

20 of 34 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
8 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  5. 2024Stealing Part of a Production Language Modelcited by 152
  6. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  7. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  8. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 83; 2026: 47.

132 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  3. 2025Self-Supervised Prompt Optimizationcited by 37
  4. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 33
  5. 2025Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Modelscited by 29
  6. 2025Learning to Generate Structured Output with Schema Reinforcement Learningcited by 27
  7. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 26
  8. 2025Towards LLMs Robustness to Changes in Prompt Format Stylescited by 21
  9. 2025The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMscited by 19
  10. 2025Unable to Forget: Proactive Interference Reveals Working Memory Limits in LLMs Beyond Context Lengthcited by 18
  11. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
  12. 2025Adaptive Self-improvement LLM Agentic System for ML Library Developmentcited by 16
38 more
  1. 2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 15
  2. 2025Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systemscited by 14
  3. 2025Token embeddings violate the manifold hypothesiscited by 14
  4. 2025Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analysescited by 13
  5. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  6. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
  7. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 12
  8. 2025Tuning LLM Judge Design Decisions for 1/1000 of the Costcited by 11
  9. 2025Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Datacited by 10
  10. 2025Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Modelscited by 10
  11. 2025Distributive Fairness in Large Language Models: Evaluating Alignment with Human Valuescited by 9
  12. 2025Automated Profile Inference with Language Model Agentscited by 9
  13. 2025InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognitioncited by 8
  14. 2025Decoupling Task-Solving and Output Formatting in LLM Generationcited by 8
  15. 2025Joint Localization and Activation Editing for Low-Resource Fine-Tuningcited by 7
  16. 2025Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulationcited by 7
  17. 2025SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedbackcited by 7
  18. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 7
  19. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?cited by 7
  20. 2025A Taxonomy of Prompt Defects in LLM Systemscited by 7
  21. 2025ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detectioncited by 7
  22. 2026SkCC: Portable and Secure Skill Compilation for Cross-Framework LLM Agentscited by 7
  23. 2025Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level IIIcited by 6
  24. 2025The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formattingcited by 5
  25. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  26. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  27. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  28. 2025Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue Historycited by 4
  29. 2025Decoupling Template Bias in CLIP: Harnessing Empty Prompts for Enhanced Few-Shot Learningcited by 4
  30. 2026Structured Context Engineering for File-Native Agentic Systems: Evaluating Schema Accuracy, Format Effectiveness, and Multi-File Navigation at Scalecited by 4
  31. 2024StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputscited by 3
  32. 2025User-Assistant Bias in LLMscited by 3
  33. 2025Textual interpretation of transient image classifications from large language modelscited by 3
  34. 2025Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priorscited by 3
  35. 2025Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoningcited by 3
  36. 2025Can LLMs Make (Personalized) Access Control Decisions?cited by 3
  37. 2026Within-Model vs Between-Prompt Variability in Large Language Models for Creative Taskscited by 3
  38. 2026Measuring Pragmatic Influence in Large Language Model Instructionscited by 3

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study on Prompt Compression for Large Language Models
  2. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
  3. 2024The Impact of Prompt Programming on Function-Level Code Generation
  4. 2024What do Large Language Models Need for Machine Translation Evaluation?
  5. 2025Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs
  6. 2023Establishing Performance Baselines in Fine-Tuning, Retrieval-Augmented Generation and Soft-Prompting for Non-Specialist LLM Users
  7. 2025NL in the Middle: Code Translation with LLMs and Intermediate Representations
  8. 2023Prompt Sketching for Large Language Models