# Factorizing Perception and Policy for Interactive Instruction Following

arXiv: https://arxiv.org/abs/2012.03208 · HTML: https://mlsanity.com/paper/2012.03208/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2020.

## Built on by (most cited first, 29 of 29)

- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [LangNav: Language as a Perceptual Representation for Navigation](https://mlsanity.com/paper/2310.07889/) (2023)
- [Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks](https://mlsanity.com/paper/2503.21696/) (2025)
- [Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agents](https://mlsanity.com/paper/2308.07241/) (2023)
- [One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones](https://mlsanity.com/paper/2202.07028/) (2022)
- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [ThinkBot: Embodied Instruction Following with Thought Chain Reasoning](https://mlsanity.com/paper/2312.07062/) (2023)
- [Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API](https://mlsanity.com/paper/2310.04716/) (2023)
- [ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments](https://mlsanity.com/paper/2407.18550/) (2024)
- [OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following](https://mlsanity.com/paper/2403.03017/) (2024)
- [Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples](https://mlsanity.com/paper/2412.17288/) (2024)
- [MAEA: Multimodal Attribution for Embodied AI](https://mlsanity.com/paper/2307.13850/) (2023)
- [Multitask Multimodal Prompted Training for Interactive Embodied Task Completion](https://mlsanity.com/paper/2311.04067/) (2023)
- [Learning to Act with Affordance-Aware Multimodal Neural SLAM](https://mlsanity.com/paper/2201.09862/) (2022)
- [LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following](https://mlsanity.com/paper/2310.12344/) (2023)
- [Embodied Instruction Following in Unknown Environments](https://mlsanity.com/paper/2406.11818/) (2024)
- [Multi-Level Compositional Reasoning for Interactive Instruction Following](https://mlsanity.com/paper/2308.09387/) (2023)
- [Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following](https://mlsanity.com/paper/2404.15190/) (2024)
- [A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search](https://mlsanity.com/paper/2206.13396/) (2022)
- [DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control](https://mlsanity.com/paper/2407.14758/) (2024)
- [Learning the Effects of Physical Actions in a Multi-modal Environment](https://mlsanity.com/paper/2301.11845/) (2023)
- [RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recovery](https://mlsanity.com/paper/2503.17703/) (2025)
- [tagE: Enabling an Embodied Agent to Understand Human Instructions](https://mlsanity.com/paper/2310.15605/) (2023)
- [ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulation](https://mlsanity.com/paper/2604.13633/) (2026)
- [Improving the Robustness to Variations of Objects and Instructions with a Neuro-Symbolic Approach for Interactive Instruction Following](https://mlsanity.com/paper/2110.07031/) (2021)
- [HAPFI: History-Aware Planning based on Fused Information](https://mlsanity.com/paper/2407.16533/) (2024)
- [ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints](https://mlsanity.com/paper/2604.14902/) (2026)
- [Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments](https://mlsanity.com/paper/2605.24020/) (2026)
- [GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planning](https://mlsanity.com/paper/2608.07905/) (2026)

## Abstract (arXiv)

Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.