# Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments

arXiv: https://arxiv.org/abs/2302.11649 · HTML: https://mlsanity.com/paper/2302.11649/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications](https://mlsanity.com/paper/1807.06333/) (2018)
- [Environment-Independent Task Specifications via GLTL](https://mlsanity.com/paper/1704.04341/) (2017)
- [CAPE: Corrective Actions from Precondition Errors using Large Language Models](https://mlsanity.com/paper/2211.09935/) (2022)
- [Learning a natural-language to LTL executable semantic parser for grounded robotics](https://mlsanity.com/paper/2008.03277/) (2020)
- [From Spot 2.0 to Spot 2.10: What's New?](https://mlsanity.com/paper/2206.11366/) (2022)
- [Specification Patterns for Robotic Missions](https://mlsanity.com/paper/1901.02077/) (2019)
- [Planning With Uncertain Specifications (PUnS)](https://mlsanity.com/paper/1906.03218/) (2019)
- [Interactive Robot Training for Non-Markov Tasks](https://mlsanity.com/paper/2003.02232/) (2020)
- [Planning with State Abstractions for Non-Markovian Task Specifications](https://mlsanity.com/paper/1905.12096/) (2019)
- [LTL-Transfer: Skill Transfer for Temporal Task Specification](https://mlsanity.com/paper/2206.05096/) (2022)

## Built on by (most cited first, 30 of 53)

- [Large Language Models Can Learn Temporal Reasoning](https://mlsanity.com/paper/2401.06853/) (2024)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [Safety Guardrails for LLM-Enabled Robots](https://mlsanity.com/paper/2503.07885/) (2025)
- [Are Large Language Models Temporally Grounded?](https://mlsanity.com/paper/2311.08398/) (2023)
- [Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models](https://mlsanity.com/paper/2403.17359/) (2024)
- [A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings](https://mlsanity.com/paper/2405.13245/) (2024)
- [VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification](https://mlsanity.com/paper/2503.18492/) (2025)
- [SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models](https://mlsanity.com/paper/2409.19471/) (2024)
- [DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL](https://mlsanity.com/paper/2410.04631/) (2024)
- [VeriPlan: Integrating Formal Verification and LLMs into End-User Planning](https://mlsanity.com/paper/2502.17898/) (2025)
- [SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments](https://mlsanity.com/paper/2410.03035/) (2024)
- [Constrained Discrete Diffusion](https://mlsanity.com/paper/2503.09790/) (2025)
- [Verifiably Following Complex Robot Instructions with Foundation Models](https://mlsanity.com/paper/2402.11498/) (2024)
- [ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees](https://mlsanity.com/paper/2504.21022/) (2025)
- [Air-Ground Collaboration for Language-Specified Missions in Unknown Environments](https://mlsanity.com/paper/2505.09108/) (2025)
- [Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification](https://mlsanity.com/paper/2411.16718/) (2024)
- [Grammar-Forced Translation of Natural Language to Temporal Logic using LLMs](https://mlsanity.com/paper/2512.16814/) (2025)
- [Generative AI Agents in Autonomous Machines: A Safety Perspective](https://mlsanity.com/paper/2410.15489/) (2024)
- [Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind](https://mlsanity.com/paper/2409.10849/) (2024)
- [Jailbreaking LLM-Controlled Robots](https://mlsanity.com/paper/2410.13691/) (2024)
- [Joint Verification and Refinement of Language Models for Safety-Constrained Planning](https://mlsanity.com/paper/2410.14865/) (2024)
- [LogicGuard: Improving Embodied LLM agents through Temporal Logic based Critics](https://mlsanity.com/paper/2507.03293/) (2025)
- [Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data](https://mlsanity.com/paper/2507.10741/) (2025)
- [Closed Loop Interactive Embodied Reasoning for Robot Manipulation](https://mlsanity.com/paper/2404.15194/) (2024)
- [LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Models](https://mlsanity.com/paper/2407.05434/) (2024)
- [Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite](https://mlsanity.com/paper/2507.00877/) (2025)
- [CoT-TL: Low-Resource Temporal Knowledge Representation of Planning Instructions Using Chain-of-Thought Reasoning](https://mlsanity.com/paper/2410.16207/) (2024)
- [Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation](https://mlsanity.com/paper/2501.01367/) (2025)
- [TRAVEL: Training-Free Retrieval and Alignment for Vision-and-Language Navigation](https://mlsanity.com/paper/2502.07306/) (2025)
- [VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents](https://mlsanity.com/paper/2606.05395/) (2026)

## Abstract (arXiv)

Grounding navigational commands to linear temporal logic (LTL) leverages its unambiguous semantics for reasoning about long-horizon tasks and verifying the satisfaction of temporal constraints. Existing approaches require training data from the specific environment and landmarks that will be used in natural language to understand commands in those environments. We propose Lang2LTL, a modular system and a software package that leverages large language models (LLMs) to ground temporal navigational commands to LTL specifications in environments without prior language data. We comprehensively evaluate Lang2LTL for five well-defined generalization behaviors. Lang2LTL demonstrates the state-of-the-art ability of a single model to ground navigational commands to diverse temporal specifications in 21 city-scaled environments. Finally, we demonstrate a physical robot using Lang2LTL can follow 52 semantically diverse navigational commands in two indoor environments.