mlsanityData to 7 Oct 2026

Factorizing Perception and Policy for Interactive Instruction Following

arXiv 2012.03208 ↗First posted 6 Dec 2020cs.AI cs.CV cs.ROcited by 28

Not on the map. Why?

Abstract (arXiv)

Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 10; 2024: 8; 2025: 2; 2026: 3.

28 papers.

Sort:

By area:

  1. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  2. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  3. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  4. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  5. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  6. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  7. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  8. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18
  9. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  10. 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Followingcited by 14
  11. 2023MAEA: Multimodal Attribution for Embodied AIcited by 9
  12. 2023Multitask Multimodal Prompted Training for Interactive Embodied Task Completioncited by 9
16 more
  1. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  2. 2022Learning to Act with Affordance-Aware Multimodal Neural SLAMcited by 8
  3. 2023LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Followingcited by 8
  4. 2024Embodied Instruction Following in Unknown Environmentscited by 8
  5. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  6. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  7. 2022A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Searchcited by 6
  8. 2024DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Controlcited by 4
  9. 2023Learning the Effects of Physical Actions in a Multi-modal Environmentcited by 3
  10. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  11. 2023tagE: Enabling an Embodied Agent to Understand Human Instructionscited by 1
  12. 2026ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulationcited by 1
  13. 2021Improving the Robustness to Variations of Objects and Instructions with a Neuro-Symbolic Approach for Interactive Instruction Followingcited by 0
  14. 2024HAPFI: History-Aware Planning based on Fused Informationcited by 0
  15. 2026ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraintscited by 0
  16. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planningcited by 0