# H-GAR: A Hierarchical Interaction Framework via Goal-Driven Observation-Action Refinement for Robotic Manipulation

arXiv: https://arxiv.org/abs/2511.17079 · HTML: https://mlsanity.com/paper/2511.17079/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2025.

## Built on by (most cited first, 17 of 17)

- [PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records](https://mlsanity.com/paper/2601.09636/) (2026)
- [HiconAgent: History Context-aware Policy Optimization for GUI Agents](https://mlsanity.com/paper/2512.01763/) (2025)
- [DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation](https://mlsanity.com/paper/2603.08361/) (2026)
- [HATS: Hardness-Aware Trajectory Synthesis for GUI Agents](https://mlsanity.com/paper/2603.12138/) (2026)
- [AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition](https://mlsanity.com/paper/2604.12735/) (2026)
- [ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation](https://mlsanity.com/paper/2605.05126/) (2026)
- [From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Model](https://mlsanity.com/paper/2605.22671/) (2026)
- [AULLM++: Structured-Token-Conditioned Large Language Models for Micro-Expression Action Unit Detection](https://mlsanity.com/paper/2603.08387/) (2026)
- [RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning](https://mlsanity.com/paper/2606.28266/) (2026)
- [Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification](https://mlsanity.com/paper/2604.10695/) (2026)
- [HydraPrompt: An Adaptive and Asymmetric Framework of Vision-Language Models for Synthetic Image Detection](https://mlsanity.com/paper/2605.26421/) (2026)
- [MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding](https://mlsanity.com/paper/2606.09641/) (2026)
- [UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling](https://mlsanity.com/paper/2609.11875/) (2026)
- [Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models](https://mlsanity.com/paper/2609.06578/) (2026)
- [FLOW: Feature-Level Optimal Warping for Generalized Remote Physiological Measurement](https://mlsanity.com/paper/2609.38913/) (2026)
- [ATI-VLA: Action-Centric Predictive Vision-Language-Action Models via Actionable Alignment Then Adaptive Injection](https://mlsanity.com/paper/2610.01741/) (2026)
- [World-Calibrated Proposal-to-Action Flow for Vision-Language-Action Models](https://mlsanity.com/paper/2610.02323/) (2026)

## Abstract (arXiv)

Unified video and action prediction models hold great potential for robotic manipulation, as future observations offer contextual cues for planning, while actions reveal how interactions shape the environment. However, most existing approaches treat observation and action generation in a monolithic and goal-agnostic manner, often leading to semantically misaligned predictions and incoherent behaviors. To this end, we propose H-GAR, a Hierarchical interaction framework via Goal-driven observation-Action Refinement.To anchor prediction to the task objective, H-GAR first produces a goal observation and a coarse action sketch that outline a high-level route toward the goal. To enable explicit interaction between observation and action under the guidance of the goal observation for more coherent decision-making, we devise two synergistic modules. (1) Goal-Conditioned Observation Synthesizer (GOS) synthesizes intermediate observations based on the coarse-grained actions and the predicted goal observation. (2) Interaction-Aware Action Refiner (IAAR) refines coarse actions into fine-grained, goal-consistent actions by leveraging feedback from the intermediate observations and a Historical Action Memory Bank that encodes prior actions to ensure temporal consistency. By integrating goal grounding with explicit action-observation interaction in a coarse-to-fine manner, H-GAR enables more accurate manipulation. Extensive experiments on both simulation and real-world robotic manipulation tasks demonstrate that H-GAR achieves state-of-the-art performance.