mlsanityData to 14 Oct 2026

Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?

arXiv 2109.08634 ↗First posted 17 Sep 2021cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Models designed for intelligent process automation are required to be capable of grounding user interface elements. This task of interface element grounding is centred on linking instructions in natural language to their target referents. Even though BERT and similar pre-trained language models have excelled in several NLP tasks, their use has not been widely explored for the UI grounding domain. This work concentrates on testing and probing the grounding abilities of three different transformer-based models: BERT, RoBERTa and LayoutLM. Our primary focus is on these models' spatial reasoning skills, given their importance in this domain. We observe that LayoutLM has a promising advantage for applications in this domain, even though it was created for a different original purpose (representing scanned documents): the learned spatial features appear to be transferable to the UI grounding setting, especially as they demonstrate the ability to discriminate between target directions in natural language instructions.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 17 Sep 2021.

Builds on

16 of 23 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  5. 2019Designing and Interpreting Probes with Control Taskscited by 640
  6. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  7. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  8. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  9. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  10. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  11. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  12. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
4 more
  1. 2018Mapping Natural Language Commands to Web Elementscited by 23
  2. 2018Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysiscited by 17
  3. 2021Does My Representation Capture X? Probe-Ablycited by 8
  4. 2020On the Evaluation of Intelligent Process Automationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 6; 2025: 2.

10 papers.

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  3. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  5. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 66
  6. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  7. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 34
  8. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  9. 2024Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfacescited by 3
  10. 2023Referring to Screen Texts with Voice Assistantscited by 2