# Reflex First, Reflect Later: Latency-Aware Embodied LLM Agents for Dynamic Response

arXiv: https://arxiv.org/abs/2506.07223 · HTML: https://mlsanity.com/paper/2506.07223/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making](https://mlsanity.com/paper/2410.07166/) (2024)
- [HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments](https://mlsanity.com/paper/2401.12975/) (2024)
- [Robotouille: An Asynchronous Planning Benchmark for LLM Agents](https://mlsanity.com/paper/2502.05227/) (2025)
- [EPO: Hierarchical LLM Agents with Environment Preference Optimization](https://mlsanity.com/paper/2408.16090/) (2024)
- [TANGO: Training-free Embodied AI Agents for Open-world Tasks](https://mlsanity.com/paper/2412.10402/) (2024)

## Built on by (most cited first, 4 of 4)

- [Real-Time Reasoning Agents in Evolving Environments](https://mlsanity.com/paper/2511.04898/) (2025)
- [Engagement Process: Rethinking the Temporal Interface of Action and Observation](https://mlsanity.com/paper/2605.11484/) (2026)
- [Self-supervised Hierarchical Visual Reasoning with World Model](https://mlsanity.com/paper/2605.17537/) (2026)
- [Separating Decision Time from Decision Quality in the Real-Time Gap of Distilled Deciders: Evidence from a Game and a Conveyor Simulator](https://mlsanity.com/paper/2610.04810/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have substantially improved the planning capabilities of embodied agents, enabling their deployment in dynamic and safety-critical environments. However, these settings expose a critical limitation: inference latency. Delayed LLM responses can weaken real-time responsiveness and misalign agent reasoning with rapidly changing environmental states. This paper systematically studies the impact of inference latency on LLM-based embodied agents in dynamic environments. We introduce an FPS-based Time Conversion Mechanism (TCM) that maps inference time to elapsed simulation time, allowing computational delays to directly affect environmental evolution and agent outcomes. We instantiate this protocol in HAZARD and introduce Response Latency (RL) and Latency-to-Action Ratio (LAR) to evaluate agent responsiveness. Building on this framework, we propose the Rapid-Reflex Async-Reflect Agent (RRARA), which integrates rapid reflexive actions with asynchronous LLM reflection to mitigate latency-induced errors. We further introduce an LLM-based PrePlanner that generates cached object-centric subgoals, reducing repeated LLM calls while retaining the model's high-level reasoning capability. Experiments show that accounting for inference latency substantially changes embodied-agent performance and that RRARA achieves a stronger balance between decision quality and responsiveness.