# Draw Me a Flower: Processing and Grounding Abstraction in Natural Language

arXiv: https://arxiv.org/abs/2106.14321 · HTML: https://mlsanity.com/paper/2106.14321/

## Key facts

- Cited by 14 papers in the map (data through 13 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 27 Jun 2021.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction](https://mlsanity.com/paper/1809.00786/) (2018)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://mlsanity.com/paper/1906.01530/) (2019)
- [Large-Scale QA-SRL Parsing](https://mlsanity.com/paper/1805.05377/) (2018)
- [Learning Language Games through Interaction](https://mlsanity.com/paper/1606.02447/) (2016)
- [A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context](https://mlsanity.com/paper/1907.03399/) (2019)
- [Controlled Crowdsourcing for High-Quality QA-SRL Annotation](https://mlsanity.com/paper/1911.03243/) (2019)
- [QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://mlsanity.com/paper/2010.02815/) (2020)
- [Simpler Context-Dependent Logical Forms via Model Projections](https://mlsanity.com/paper/1606.05378/) (2016)
- [Naturalizing a Programming Language via Interactive Learning](https://mlsanity.com/paper/1704.06956/) (2017)
- [Learning Interpretable Spatial Operations in a Rich 3D Blocks World](https://mlsanity.com/paper/1712.03463/) (2017)
- [Planning, Inference and Pragmatics in Sequential Language Games](https://mlsanity.com/paper/1805.11774/) (2018)
- [RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation](https://mlsanity.com/paper/1909.08970/) (2019)

## Built on by (most cited first, 14 of 14)

- [Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models](https://mlsanity.com/paper/2310.06117/) (2023)
- [In-Context Analogical Reasoning with Pre-Trained Language Models](https://mlsanity.com/paper/2305.17626/) (2023)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via Dialogue](https://mlsanity.com/paper/2303.11108/) (2023)
- [ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models](https://mlsanity.com/paper/2305.05050/) (2023)
- [Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models](https://mlsanity.com/paper/2301.05149/) (2022)
- ["We Demand Justice!": Towards Social Context Grounding of Political Texts](https://mlsanity.com/paper/2311.09106/) (2023)
- [A Dialogue Game for Eliciting Balanced Collaboration](https://mlsanity.com/paper/2406.08202/) (2024)
- [Where Do We Go from Here? Multi-scale Allocentric Relational Inference from Natural Spatial Descriptions](https://mlsanity.com/paper/2402.16364/) (2024)
- [On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directions](https://mlsanity.com/paper/2406.10885/) (2024)
- [Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming](https://mlsanity.com/paper/2409.11041/) (2024)
- [Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents](https://mlsanity.com/paper/2604.17019/) (2026)
- [Sketch-Plan-Generalize: Learning and Planning with Neuro-Symbolic Programmatic Representations for Inductive Spatial Concepts](https://mlsanity.com/paper/2404.07774/) (2024)

## Abstract (arXiv)

Abstraction is a core tenet of human cognition and communication. When composing natural language instructions, humans naturally evoke abstraction to convey complex procedures in an efficient and concise way. Yet, interpreting and grounding abstraction expressed in NL has not yet been systematically studied in NLP, with no accepted benchmarks specifically eliciting abstraction in NL. In this work, we set the foundation for a systematic study of processing and grounding abstraction in NLP. First, we deliver a novel abstraction elicitation method and present Hexagons, a 2D instruction-following game. Using Hexagons we collected over 4k naturally-occurring visually-grounded instructions rich with diverse types of abstractions. From these data, we derive an instruction-to-execution task and assess different types of neural models. Our results show that contemporary models and modeling practices are substantially inferior to human performance, and that models' performance is inversely correlated with the level of abstraction, showing less satisfying performance on higher levels of abstraction. These findings are consistent across models and setups, confirming that abstraction is a challenging phenomenon deserving further attention and study in NLP/AI research.