mlsanityData to 7 Oct 2026

Interaction is all You Need? A Study of Robots Ability to Understand and Execute

arXiv 2311.07150 ↗First posted 13 Nov 2023cs.RO cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper aims to address a critical challenge in robotics, which is enabling them to operate seamlessly in human environments through natural language interactions. Our primary focus is to equip robots with the ability to understand and execute complex instructions in coherent dialogs to facilitate intricate task-solving scenarios. To explore this, we build upon the Execution from Dialog History (EDH) task from the Teach benchmark. We employ a multi-transformer model with BART LM. We observe that our best configuration outperforms the baseline with a success rate score of 8.85 and a goal-conditioned success rate score of 14.02. In addition, we suggest an alternative methodology for completing this task. Moreover, we introduce a new task by expanding the EDH task and making predictions about game plans instead of individual actions. We have evaluated multiple BART models and an LLaMA2 LLM, which has achieved a ROGUE-L score of 46.77 for this task.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

24 of 33 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
12 more
  1. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  2. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  3. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  4. 2022When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialoguescited by 32
  5. 2019Context-Aware Self-Attention Networkscited by 25
  6. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  7. 2022Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?cited by 16
  8. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  9. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  10. 2021How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpuscited by 11
  11. 2022Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructionscited by 9
  12. 2017Generalized Grounding Graphs: A Probabilistic Framework for Understanding Grounded Commandscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Makingcited by 9