# Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving

arXiv: https://arxiv.org/abs/2605.20072 · HTML: https://mlsanity.com/paper/2605.20072/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 8 of 8)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023)
- [RBO Hand 3 -- A Platform for Soft Dexterous Manipulation](https://mlsanity.com/paper/2201.10883/) (2022)
- [BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment](https://mlsanity.com/paper/2207.08192/) (2022)
- [Addressing LLM Diversity by Infusing Random Concepts](https://mlsanity.com/paper/2601.18053/) (2026)
- [Using Memory-Based Learning to Solve Tasks with State-Action Constraints](https://mlsanity.com/paper/2303.04327/) (2023)
- [A Biologically Inspired Design Principle for Building Robust Robotic Systems](https://mlsanity.com/paper/2408.10192/) (2024)

## Built on by (most cited first, 1 of 1)

- [Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail](https://mlsanity.com/paper/2608.23651/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly proposed as cognitive components for robotic systems, yet their opaque decision processes make it difficult to explain success or failure in closed-loop embodied tasks. Following an empirical AI methodology, we study an embodied LLM agent behaviorally by varying the available information and measuring the resulting changes in behavior. Using the Lockbox, a sequential mechanical puzzle with hidden interdependencies, we evaluate LLMs across RGB, RGB-D, and ground-truth symbolic observations in a physical robotic setup and use simulation to probe the resulting behavior. Counterintuitively, agents perform best under raw RGB input and worst under perfect ground-truth observations. In simulation, we probe this effect by randomly flipping perceived action outcomes and find that moderate noise improves performance, peaking at a 40% flip probability with a 2.85-fold success rate increase over the noise-free baseline. Further analysis links this gain to a reduction in repetitive action loops. These findings suggest that success rates alone are insufficient for evaluating LLMs, as measured performance may reflect the interaction between perceptual errors and reasoning failures rather than robust problem solving.