# Prompting with Pseudo-Code Instructions

arXiv: https://arxiv.org/abs/2305.11790 · HTML: https://mlsanity.com/paper/2305.11790/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 107 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 30 of 71)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)

## Built on by (most cited first, 14 of 14)

- [CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models](https://mlsanity.com/paper/2402.16717/) (2024)
- [SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)](https://mlsanity.com/paper/2306.00739/) (2023)
- [Exploring Autonomous Agents through the Lens of Large Language Models: A Review](https://mlsanity.com/paper/2404.04442/) (2024)
- [UniCoder: Scaling Code Large Language Model via Universal Code](https://mlsanity.com/paper/2406.16441/) (2024)
- [Evaluating LLM-based Personal Information Extraction and Countermeasures](https://mlsanity.com/paper/2408.07291/) (2024)
- [Learning to Reason via Program Generation, Emulation, and Search](https://mlsanity.com/paper/2405.16337/) (2024)
- [Chandomitra: Towards Generating Structured Sanskrit Poetry from Natural Language Inputs](https://mlsanity.com/paper/2506.00815/) (2025)
- [RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions](https://mlsanity.com/paper/2402.16431/) (2024)
- [Multi-agent Self-triage System with Medical Flowcharts](https://mlsanity.com/paper/2511.12439/) (2025)
- [Extracting chemical food safety hazards from the scientific literature automatically using large language models](https://mlsanity.com/paper/2405.15787/) (2024)
- [Training with Pseudo-Code for Instruction Following](https://mlsanity.com/paper/2505.18011/) (2025)
- [Learning Evidence of Depression Symptoms via Prompt Induction](https://mlsanity.com/paper/2604.24376/) (2026)
- [Scaling Automatic Extraction of Pseudocode](https://mlsanity.com/paper/2406.04635/) (2024)
- [PromptMN: Pseudo Prompting Language](https://mlsanity.com/paper/2606.17164/) (2026)

## Abstract (arXiv)

Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models. Given the inherent ambiguity present in natural language, it is intuitive to consider the possible advantages of prompting with less ambiguous prompt styles, such as the use of pseudo-code. In this paper we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models. We manually create a dataset of pseudo-code prompts for 132 different tasks spanning classification, QA and generative language tasks, sourced from the Super-NaturalInstructions dataset. Using these prompts along with their counterparts in natural language, we study their performance on two LLM families - BLOOM and CodeGen. Our experiments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in F1 scores for classification tasks and an improvement (relative) of 12-38% in aggregate ROUGE-L scores across all tasks. We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded in pseudo-code all contribute towards the improvement in performance. To the best of our knowledge our work is the first to demonstrate how pseudo-code prompts can be helpful in improving the performance of pre-trained LMs.