# Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation

arXiv: https://arxiv.org/abs/1805.10209 · HTML: https://mlsanity.com/paper/1805.10209/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 12 of 12)

- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [LISA: Learning Interpretable Skill Abstractions from Language](https://mlsanity.com/paper/2203.00054/) (2022)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing](https://mlsanity.com/paper/2101.01686/) (2021)
- [A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment](https://mlsanity.com/paper/2101.07891/) (2021)
- [LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training](https://mlsanity.com/paper/2201.08081/) (2022)
- [Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models](https://mlsanity.com/paper/2603.06828/) (2026)
- [GoalNet: Inferring Conjunctive Goal Predicates from Human Plan Demonstrations for Robot Instruction Following](https://mlsanity.com/paper/2205.07081/) (2022)
- [Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling](https://mlsanity.com/paper/2306.12387/) (2023)
- [Semantic Parsing in Limited Resource Conditions](https://mlsanity.com/paper/2309.07429/) (2023)
- [Improving Agent Interactions in Virtual Environments with Language Models](https://mlsanity.com/paper/2402.05440/) (2024)

## Abstract (arXiv)

We propose a learning approach for mapping context-dependent sequential instructions to actions. We address the problem of discourse and state dependencies with an attention-based model that considers both the history of the interaction and the state of the world. To train from start and goal states without access to demonstrations, we propose SESTRA, a learning algorithm that takes advantage of single-step reward observations and immediate expected reward maximization. We evaluate on the SCONE domains, and show absolute accuracy improvements of 9.8%-25.3% across the domains over approaches that use high-level logical representations.