mlsanityData to 7 Oct 2026

Exploring the Curious Case of Code Prompts

arXiv 2304.13250 ↗First posted 26 Apr 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that prompting language models with code-like representations of natural language leads to performance improvements on structured reasoning tasks. However, such tasks comprise only a small subset of all natural language tasks. In our work, we seek to answer whether or not code-prompting is the preferred way of interacting with language models in general. We compare code and text prompts across three popular GPT models (davinci, code-davinci-002, and text-davinci-002) on a broader selection of tasks (e.g., QA, sentiment, summarization) and find that with few exceptions, code prompts do not consistently outperform text prompts. Furthermore, we show that the style of code prompt has a large effect on performance for some but not all tasks and that fine-tuning on text instructions leads to better relative performance of code prompts.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

23 of 40 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
11 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2022PAL: Program-aided Language Modelscited by 635
  6. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  7. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  8. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  9. 2020A Dataset for Tracking Entities in Open Domain Procedural Textcited by 37
  10. 2023CodeKGC: Code Language Model for Generative Knowledge Graph Constructioncited by 28
  11. 2023Causal Reasoning of Entities and Events in Procedural Textscited by 23

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3.

6 papers.

  1. 2023Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Educationcited by 38
  2. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  3. 2023Prompting with Pseudo-Code Instructionscited by 13
  4. 2023Leveraging Code to Improve In-context Learning for Semantic Parsingcited by 10
  5. 2024Learning to Reason via Program Generation, Emulation, and Searchcited by 9
  6. 2024RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructionscited by 2