mlsanityData to 14 Oct 2026

Beyond Needle(s) in the Embodied Haystack: Environment, Architecture, and Training Considerations for Long Context Reasoning

arXiv 2505.16928 ↗First posted 22 May 2025cs.AI cs.LG cs.ROcited by 3

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We introduce ∞-THOR, a new framework for long-horizon embodied tasks that advances long-context understanding in embodied AI. ∞-THOR provides: (1) a generation framework for synthesizing scalable, reproducible, and unlimited long-horizon trajectories; (2) a novel embodied QA task, Needle(s) in the Embodied Haystack, where multiple scattered clues across extended trajectories test agents' long-context reasoning ability; and (3) a long-horizon dataset and benchmark suite featuring complex tasks that span hundreds of environment steps, each paired with ground-truth action sequences. To enable this capability, we explore architectural adaptations, including interleaved Goal-State-Action modeling, context extension techniques, and Context Parallelism, to equip LLM-based agents for extreme long-context reasoning and interaction. Experimental results and analyses highlight the challenges posed by our benchmark and provide insights into training strategies and model behaviors under long-horizon conditions. Our work provides a foundation for the next generation of embodied AI systems capable of robust, long-term reasoning and planning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 45 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

33 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
20 more
  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  2. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  4. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  8. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  10. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  11. 2017Embodied Question Answeringcited by 336
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  13. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  14. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  15. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  16. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  17. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  18. 2024Needle In A Multimodal Haystackcited by 58
  19. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 55
  20. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 7
  2. 2025DeliveryBench: Can Agents Earn Profit in Real World?cited by 6
  3. 2025Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scalecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agents
  2. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
  3. 2024Long-horizon Embodied Planning with Implicit Logical Inference and Hallucination Mitigation
  4. 2026ACC: Compiling Agent Trajectories for Long-Context Training
  5. 2026Long-Horizon Embodied Decision-Making via Multimodal Memory Compression
  6. 2026ParallelWorld: Test-Time Scaling for Embodied Reasoning
  7. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning
  8. 2026SAM: State-Adaptive Memory for Long-Horizon Reasoning Agent