mlsanityData to 7 Oct 2026

Dialog Acts for Task-Driven Embodied Agents

arXiv 2209.12953 ↗First posted 26 Sep 2022cs.CL cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Embodied agents need to be able to interact in natural language understanding task descriptions and asking appropriate follow up questions to obtain necessary information to be effective at successfully accomplishing tasks for a wide range of users. In this work, we propose a set of dialog acts for modelling such dialogs and annotate the TEACh dataset that includes over 3,000 situated, task oriented conversations (consisting of 39.5k utterances in total) with dialog acts. TEACh-DA is one of the first large scale dataset of dialog act annotations for embodied task completion. Furthermore, we demonstrate the use of this annotated dataset in training models for tagging the dialog acts of a given utterance, predicting the dialog act of the next response given a dialog history, and use the dialog acts to guide agent's non-dialog behaviour. In particular, our experiments on the TEACh Execution from Dialog History task where the model predicts the sequence of low level actions to be executed in the environment for embodied task completion, demonstrate that dialog acts can improve end task success rate by up to 2 points compared to the system without dialog acts.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

19 of 32 references.

References · 19

  1. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  2. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  3. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  4. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  5. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 124
  6. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  7. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  8. 2020doc2dial: A Goal-Oriented Document-Grounded Dialogue Datasetcited by 88
  9. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  10. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 54
  11. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  12. 2016Sequential Short-Text Classification with Recurrent and Convolutional Neural Networkscited by 28
7 more
  1. 2021VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigatorcited by 13
  2. 2018ISO-Standard Domain-Independent Dialogue Act Tagging for Conversational Agentscited by 12
  3. 2021How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpuscited by 11
  4. 2022Learning to Act with Affordance-Aware Multimodal Neural SLAMcited by 8
  5. 2013Recurrent Convolutional Neural Networks for Discourse Compositionalitycited by 7
  6. 2019Towards Universal Dialogue Act Tagging for Task-Oriented Dialoguescited by 7
  7. 2018Deep Dialog Act Recognition using Multiple Token, Segment, and Context Information Representationscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 4; 2025: 4.

13 papers.

Sort:

By area:

  1. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  2. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 14
  3. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  4. 2023The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systemscited by 9
  5. 2024Simulating User Agents for Embodied Conversational-AIcited by 6
  6. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  7. 2025Referential ambiguity and clarification requests: comparing human and LLM behaviourcited by 5
  8. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  9. 2025Plan Verification for LLM-Based Embodied Task Completion Agentscited by 4
  10. 2023Who's in Charge? Roles and Responsibilities of Decision-Making Components in Conversational Robotscited by 1
  11. 2023Multimodal Contextualized Plan Prediction for Embodied Task Completioncited by 1
  12. 2023Interaction is all You Need? A Study of Robots Ability to Understand and Executecited by 1
1 more
  1. 2025A Fully Automated Pipeline for Conversational Discourse Annotation: Tree Scheme Generation and Labeling with Large Language Modelscited by 1