mlsanityData to 7 Oct 2026

Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests

arXiv 2401.17039 ↗First posted 30 Jan 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Clarification requests are a mechanism to help solve communication problems, e.g. due to ambiguity or underspecification, in instruction-following interactions. Despite their importance, even skilful models struggle with producing or interpreting such repair acts. In this work, we test three hypotheses concerning the effects of action taking as an auxiliary task in modelling iCR policies. Contrary to initial expectations, we conclude that its contribution to learning an iCR policy is limited, but some information can still be extracted from prediction uncertainty. We present further evidence that even well-motivated, Transformer-based models fail to learn good policies for when to ask Instruction CRs (iCRs), while the task of determining what to ask about can be more successfully modelled. Considering the implications of these findings, we further discuss the shortcomings of the data-driven paradigm for learning meta-communication acts.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 38 of its 76 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

38 of 76 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2019Vision-and-Dialog Navigationcited by 246
  7. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  8. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  9. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  10. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  11. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  12. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
26 more
  1. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  2. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 54
  3. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  4. 2023Grounding Gaps in Language Model Generationscited by 45
  5. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 44
  6. 2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
  7. 2021Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledgecited by 40
  8. 2020ClarQ: A large-scale and diverse dataset for Clarification Question Generationcited by 31
  9. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  10. 2023A Survey on Asking Clarification Questions Datasets in Conversational Systemscited by 28
  11. 2022Python Code Generation by Asking Clarification Questionscited by 16
  12. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  13. 2021A recipe for annotating grounded clarificationscited by 13
  14. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  15. 2023Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructionscited by 13
  16. 2021How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpuscited by 11
  17. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  18. 2022Don't Copy the Teacher: Data and Model Challenges in Embodied Dialoguecited by 10
  19. 2022Collecting Interactive Multi-modal Datasets for Grounded Language Understandingcited by 10
  20. 2023When and What to Ask Through World States and Text Instructions: IGLU NLP Challenge Solutioncited by 5
  21. 2020Interactive Question Clarification in Dialogue via Reinforcement Learningcited by 4
  22. 2018Ask No More: Deciding when to guess in referential visual dialoguecited by 3
  23. 2023"Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Datasetcited by 3
  24. 2023'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchangescited by 3
  25. 2023Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examplescited by 1
  26. 2023Investigating Reinforcement Learning for Communication Strategies in a Task-Initiative Settingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  2. 2024Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Modelscited by 3
  3. 2024It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learningcited by 1
  4. 2026XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?cited by 1