mlsanityData to 14 Oct 2026

Learning with Latent Language

arXiv 1711.00482 ↗First posted 1 Nov 2017cs.CL cs.NEcited by 75

Not on the map. Why?

Abstract (arXiv)

The named concepts and compositional operators present in natural language provide a rich source of information about the kinds of abstractions humans use to navigate the world. Can this linguistic background knowledge improve the generality and efficiency of learned classifiers and control policies? This paper aims to show that using the space of natural language strings as a parameter space is an effective way to capture natural task structure. In a pretraining phase, we learn a language interpretation model that transforms inputs (e.g. images) into outputs (e.g. labels) given natural language descriptions. To learn a new concept (e.g. a classifier), we search directly in the space of descriptions to minimize the interpreter's loss on training examples. Crucially, our models do not require language data to learn these concepts: language is used only in pretraining to impose structure on subsequent learning. Results on image classification, text editing, and reinforcement learning show that, in all settings, models with a linguistic parameterization outperform those without.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 17; 2022: 25; 2023: 20; 2024: 5; 2025: 4; 2026: 3.

75 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  5. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  7. 2022Can language models learn from explanations in context?cited by 249
  8. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  9. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  10. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  11. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 124
  12. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
38 more
  1. 2021Skill Induction and Planning with Latent Languagecited by 99
  2. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  3. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  4. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  5. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  6. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  7. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 45
  8. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  9. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  10. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  11. 2021Communicating Natural Programs to Humans and Machinescited by 35
  12. 2023Language Model Alignment with Elastic Resetcited by 34
  13. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33
  14. 2022Contrastive Audio-Language Learning for Musiccited by 33
  15. 2022Fixing Model Bugs with Natural Language Patchescited by 33
  16. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  17. 2021Interactive Learning from Activity Descriptioncited by 26
  18. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
  19. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  20. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  21. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  22. 2023Human-like Few-Shot Learning via Bayesian Reasoning over Natural Languagecited by 22
  23. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
  24. 2022Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machinescited by 20
  25. 2022Abstraction for Deep Reinforcement Learningcited by 19
  26. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  27. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  28. 2021Rich Semantics Improve Few-shot Learningcited by 15
  29. 2021Zero-shot Task Adaptation using Natural Languagecited by 15
  30. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  31. 2022G^3: Geolocation via Guidebook Groundingcited by 12
  32. 2023FLamE: Few-shot Learning from Natural Language Explanationscited by 10
  33. 2023Predictive Chemistry Augmented with Text Retrievalcited by 10
  34. 2022CLUES: A Benchmark for Learning Classifiers using Natural Language Explanationscited by 9
  35. 2023Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actorscited by 9
  36. 2021Multitasking Inhibits Semantic Driftcited by 8
  37. 2023Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standardscited by 8
  38. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8

50 most cited of 75 All 75 (JSON)