mlsanityData to 7 Oct 2026

AGI Is Coming... Right After AI Learns to Play Wordle

arXiv 2504.15434 ↗First posted 21 Apr 2025cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates multimodal agents, in particular, OpenAI's Computer-User Agent (CUA), trained to control and complete tasks through a standard computer interface, similar to humans. We evaluated the agent's performance on the New York Times Wordle game to elicit model behaviors and identify shortcomings. Our findings revealed a significant discrepancy in the model's ability to recognize colors correctly depending on the context. The model had a 5.36% success rate over several hundred runs across a week of Wordle. Despite the immense enthusiasm surrounding AI agents and their potential to usher in Artificial General Intelligence (AGI), our findings reinforce the fact that even simple tasks present substantial challenges for today's frontier AI models. We conclude with a discussion of the potential underlying causes, implications for future development, and research directions to improve these AI systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

18 of 29 references.

References · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  9. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  10. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
6 more
  1. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  2. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  3. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 72
  4. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  5. 2024Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Taskscited by 24
  6. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13

Built on by

Not cited yet.