mlsanityData to 14 Oct 2026

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

arXiv 2406.13787 ↗First posted 19 Jun 2024cs.RO cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Language Models (LLM) and Vision Language Models (VLM) enable robots to ground natural language prompts into control actions to achieve tasks in an open world. However, when applied to a long-horizon collaborative task, this formulation results in excessive prompting for initiating or clarifying robot actions at every step of the task. We propose Language-driven Intention Tracking (LIT), leveraging LLMs and VLMs to model the human user's long-term behavior and to predict the next human intention to guide the robot for proactive collaboration. We demonstrate smooth coordination between a LIT-based collaborative robot and the human user in collaborative cooking tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 39 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

27 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedback12 Oct
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 24

  1. 2023Segment Anythingcited by 8,234
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  11. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  12. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
12 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  2. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  3. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  4. 2020Deep Open Intent Classification with Adaptive Decision Boundarycited by 56
  5. 2020Deep Affordance Foresight: Planning Through What Can Be Done in the Futurecited by 41
  6. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  7. 2020Goal-driven Long-Term Trajectory Predictioncited by 24
  8. 2022STAP: Sequencing Task-Agnostic Policiescited by 18
  9. 2022Learning Discriminative Representations and Decision Boundaries for Open Intent Detectioncited by 15
  10. 2020Long-term Pedestrian Trajectory Prediction using Mutable Intention Filter and Warp LSTMcited by 10
  11. 2024MOSAIC: Modular Foundation Models for Assistive and Interactive Cookingcited by 9
  12. 2022Hierarchical Intention Tracking for Robust Human-Robot Collaboration in Industrial Assembly Taskscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Modelscited by 1
  2. 2025Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistancecited by 0
  3. 2026IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration
  2. 2024Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task
  3. 2023Interactive Task Planning with Language Models
  4. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
  5. 2025BT-ACTION: A Test-Driven Approach for Modular Understanding of User Instruction Leveraging Behaviour Trees and LLMs ↗
  6. 2025Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models
  7. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Models
  8. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models