mlsanityData to 7 Oct 2026

ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes

arXiv 2304.04321 ↗First posted 9 Apr 2023cs.AI cs.CL cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

Understanding the continuous states of objects is essential for task learning and planning in the real world. However, most existing task learning benchmarks assume discrete (e.g., binary) object goal states, which poses challenges for the learning of complex tasks and transferring learned policy from simulated environments to the real world. Furthermore, state discretization limits a robot's ability to follow human instructions based on the grounding of actions and states. To tackle these challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes. ARNOLD is comprised of 8 language-conditioned tasks that involve understanding object states and learning policies for continuous goals. To promote language-instructed learning, we provide expert demonstrations with template-generated language descriptions. We assess task performance by utilizing the latest language-conditioned policy learning models. Our results indicate that current models for language-conditioned manipulations continue to experience significant challenges in novel goal-state generalizations, scene generalizations, and object generalizations. These findings highlight the need to develop new algorithms that address this gap and underscore the potential for further research in this area. Project website: https://arnold-benchmark.github.io.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 79 of its 102 references that are in the map. First posted on arXiv 9 Apr 2023.

Builds on

79 of 102 references.

References · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
38 more
  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  2. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  3. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  6. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  7. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  11. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  13. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  14. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  15. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  16. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  17. 2017Embodied Question Answeringcited by 335
  18. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  19. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  20. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  21. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  22. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  23. 2019Vision-and-Dialog Navigationcited by 246
  24. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  25. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  26. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  27. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  28. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  29. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  30. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  31. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  32. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  33. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  34. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  35. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  36. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  37. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  38. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 135

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 22; 2025: 24; 2026: 16.

68 papers.

Sort:

By area:

  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  2. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  3. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  4. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 157
  5. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  6. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  7. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  9. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  10. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  11. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  12. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
38 more
  1. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  2. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  3. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrationscited by 51
  4. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  5. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  6. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  7. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  8. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  9. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  10. 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluationcited by 31
  11. 2025Decompositional Neural Scene Reconstruction with Generative Diffusion Priorcited by 30
  12. 2024PhysPart: Physically Plausible Part Completion for Interactable Objectscited by 24
  13. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  14. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  15. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  16. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  17. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  18. 2023LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulationcited by 16
  19. 2025AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learningcited by 12
  20. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12
  21. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  22. 2025ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Taskscited by 9
  23. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  24. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  25. 2025AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learningcited by 8
  26. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  27. 2023SAGE: Bridging Semantic and Actionable Parts for GEneralizable Manipulation of Articulated Objectscited by 5
  28. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 5
  29. 2024GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulationcited by 4
  30. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  31. 2025Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulationcited by 4
  32. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  33. 2025Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulationcited by 3
  34. 2025Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Groundingcited by 3
  35. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  36. 2026SPEAR: A Simulator for Photorealistic Embodied AI Researchcited by 3
  37. 2026MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalizationcited by 3
  38. 2025IRIS: An Immersive Robot Interaction Systemcited by 2

50 most cited of 68 All 68 (JSON)