mlsanityData to 14 Oct 2026

Summarizing a virtual robot's past actions in natural language

arXiv 2203.06671 ↗First posted 13 Mar 2022cs.CL cs.ROcited by 3

Not on the map. Why?

Abstract (arXiv)

We propose and demonstrate the task of giving natural language summaries of the actions of a robotic agent in a virtual environment. We explain why such a task is important, what makes it difficult, and discuss how it might be addressed. To encourage others to work on this, we show how a popular existing dataset that matches robot actions with natural language descriptions designed for an instruction following task can be repurposed to serve as a training ground for robot action summarization work. We propose and test several methods of learning to generate such summaries, starting from either egocentric video frames of the robot taking actions or intermediate text representations of the actions used by an automatic planner. We provide quantitative and qualitative evaluations of our results, which can serve as a baseline for future work.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 13 Mar 2022.

Builds on

31 of 48 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
19 more
  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  2. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  3. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  4. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  6. 2020Experience Grounds Languagecited by 242
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  9. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  10. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  11. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  12. 2021Video Summarization Using Deep Neural Networks: A Surveycited by 85
  13. 2020An Exploration of Embodied Visual Explorationcited by 58
  14. 2020Word meaning in minds and machinescited by 51
  15. 2021Grounding 'Grounding' in NLPcited by 44
  16. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  17. 2021A Survey on Multi-modal Summarizationcited by 26
  18. 2015Robot Language Learning, Generation, and Comprehensioncited by 3
  19. 2020Caption Generation of Robot Behaviors based on Unsupervised Learning of Action Segmentscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
  2. 2024Temporal and Semantic Evaluation Metrics for Foundation Models in Post-Hoc Analysis of Robotic Sub-taskscited by 1
  3. 2024Instructional Text Across Disciplines: A Survey of Representations, Downstream Tasks, and Open Challenges Toward Capable AI Agentscited by 1