# Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI

arXiv: https://arxiv.org/abs/2503.21668 · HTML: https://mlsanity.com/paper/2503.21668/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 19 of its 86 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 19 of 19)

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making](https://mlsanity.com/paper/2410.07166/) (2024)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Learning Affordance Landscapes for Interaction Exploration in 3D Environments](https://mlsanity.com/paper/2008.09241/) (2020)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [NEWTON: Are Large Language Models Capable of Physical Reasoning?](https://mlsanity.com/paper/2310.07018/) (2023)
- [A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming](https://mlsanity.com/paper/2404.16038/) (2024)
- [A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment](https://mlsanity.com/paper/2410.23242/) (2024)
- [Rediscovering Affordance: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2112.12886/) (2021)
- [On Reality and the Limits of Language Data: Aligning LLMs with Human Norms](https://mlsanity.com/paper/2208.11981/) (2022)

## Built on by (most cited first, 1 of 1)

- [Exploring Diagnostic Prompting Approach for Multimodal LLM-based Visual Complexity Assessment: A Case Study of Amazon Search Result Pages](https://mlsanity.com/paper/2512.00082/) (2025)

## Abstract (arXiv)

One of the core components of our world models is 'intuitive physics' - an understanding of objects, space, and causality. This capability enables us to predict events, plan action and navigate environments, all of which rely on a composite sense of objecthood. Despite its importance, there is no single, unified account of objecthood, though multiple theoretical frameworks provide insights. In the first part of this paper, we present a comprehensive overview of the main theoretical frameworks in objecthood research - Gestalt psychology, enactive cognition, and developmental psychology - and identify the core capabilities each framework attributes to object understanding, as well as what functional roles they play in shaping world models in biological agents. Given the foundational role of objecthood in world modelling, understanding objecthood is also essential in AI. In the second part of the paper, we evaluate how current AI paradigms approach and test objecthood capabilities compared to those in cognitive science. We define an AI paradigm as a combination of how objecthood is conceptualised, the methods used for studying objecthood, the data utilised, and the evaluation techniques. We find that, whilst benchmarks can detect that AI systems model isolated aspects of objecthood, the benchmarks cannot detect when AI systems lack functional integration across these capabilities, not solving the objecthood challenge fully. Finally, we explore novel evaluation approaches that align with the integrated vision of objecthood outlined in this paper. These methods are promising candidates for advancing from isolated object capabilities toward general-purpose AI with genuine object understanding in real-world contexts.