Data to 14 Oct 2026

The Turking Test: Can Language Models Understand Instructions?

arXiv 2010.11982 ↗First posted 22 Oct 2020cs.CL cs.AI cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Supervised machine learning provides the learner with a set of input-output examples of the target task. Humans, however, can also learn to perform new tasks from instructions in natural language. Can machines learn to understand instructions as well? We present the Turking Test, which examines a model's ability to follow natural language instructions of varying complexity. These range from simple tasks, like retrieving the nth word of a sentence, to ones that require creativity, such as generating examples for SNLI and SQuAD in place of human intelligence workers ("turkers"). Despite our lenient evaluation methodology, we observe that a large pretrained language model performs poorly across all tasks. Analyzing the model's error patterns reveals that the model tends to ignore explicit instructions and often generates outputs that cannot be construed as an attempt to solve the task. While it is not yet clear whether instruction understanding can be captured by traditional language models, the sheer expressivity of instruction understanding makes it an appealing alternative to the rising few-shot inference paradigm.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 22; 2023: 32; 2024: 12; 2025: 5.

82 papers.

Sort:

By area:

2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
2023LIMA: Less Is More for Alignmentcited by 1,100
2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326

38 more

2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
2021Generating Datasets with Pretrained Language Modelscited by 193
2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
2021Reframing Instructional Prompts to GPTk's Languagecited by 176
2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
2023Finding Support Examples for In-Context Learningcited by 104
2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 45
2024Language Models for Text Classification: Is In-Context Learning Enough?cited by 40
2023Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?cited by 39
2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
2023Prompting Large Language Models for Counterfactual Generation: An Empirical Studycited by 37
2022Robustness of Learning from Task Instructionscited by 35
2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
2022ConTinTin: Continual Learning from Task Instructionscited by 26
2023A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Taskscited by 25
2023BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processingcited by 25
2023InstructABSA: Instruction Learning for Aspect Based Sentiment Analysiscited by 23
2023Working Memory Capacity of ChatGPT: An Empirical Studycited by 23
2023Training-free Zero-shot Composed Image Retrieval with Local Concept Rerankingcited by 23
2023Model Leeching: An Extraction Attack Targeting LLMscited by 22
2023ScatterShot: Interactive In-context Example Curation for Text Transformationcited by 20

50 most cited of 82 All 82 (JSON)