mlsanityData to 7 Oct 2026

Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning

arXiv 2603.18495 ↗First posted 19 Mar 2026cs.AIcited by 0

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Recent advances in Vision-Language Models (VLMs) have enabled video-instructed robotic programming, allowing agents to interpret video demonstrations and generate executable control code. We formulate video-instructed robotic programming as a cross-domain adaptation problem, where perceptual and physical differences between demonstration and deployment induce procedural mismatches. However, current VLMs lack the procedural understanding needed to reformulate causal dependencies and achieve task-compatible behavior under such domain shifts. We introduce NeSyCR, a neurosymbolic counterfactual reasoning framework that enables verifiable adaptation of task procedures, providing a reliable synthesis of code policies. NeSyCR abstracts video demonstrations into symbolic trajectories that capture the underlying task procedure. Given deployment observations, it derives counterfactual states that reveal cross-domain incompatibilities. By exploring the symbolic state space with verifiable checks, NeSyCR proposes procedural revisions that restore compatibility with the demonstrated procedure. NeSyCR achieves a 31.14% improvement in task success over the strongest baseline Statler, showing robust cross-domain adaptation across both simulated and real-world manipulation tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 70 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

59 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoning22 Oct
  2. 2025Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation8 Jan
  3. 2023Statler: State-Maintaining Language Models for Embodied Reasoning30 Jun
  4. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought26 May
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 54

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  12. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
38 more
  1. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  2. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  3. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  4. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  5. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  6. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  7. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  8. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  9. 2017Multi-shot ASP solving with clingocited by 115
  10. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  11. 2020Visual Imitation Made Easycited by 111
  12. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  13. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  14. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  15. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  16. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  17. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  18. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  19. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  20. 2021Cross-domain Imitation from Observationscited by 42
  21. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planningcited by 37
  22. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  23. 2023Leveraging Large Language Models to Generate Answer Set Programscited by 33
  24. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  25. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  26. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 21
  27. 2024Recover: A Neuro-Symbolic Framework for Failure Detection and Recoverycited by 15
  28. 2025NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domainscited by 15
  29. 2023Domain Adaptive Imitation Learning with Visual Observationcited by 11
  30. 2024From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Modelscited by 11
  31. 2025Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generationcited by 10
  32. 2025Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planningcited by 10
  33. 2024GenCHiP: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Taskscited by 9
  34. 2025LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Studycited by 9
  35. 2024One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skillcited by 7
  36. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  37. 2025Bilevel Learning for Bilevel Planningcited by 6
  38. 2024Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agentscited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Models
  2. 2025UniDomain: Pretraining a Unified PDDL Domain from Real-World Demonstrations for Generalizable Robot Task Planning
  3. 2025From Code to Action: Hierarchical Learning of Diffusion-VLM Policies
  4. 2026Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation
  5. 2026CAPER: Constrained and Procedural Reasoning for Robotic Scientific Experiments
  6. 2025HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
  7. 2026Explore, Execute, Evolve: A Skill Acquisition and Reuse Loop for Embodied Agents
  8. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs