# Testing System Intelligence

arXiv: https://arxiv.org/abs/2305.11472 · HTML: https://mlsanity.com/paper/2305.11472/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 42 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 4 of 4)

- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [Specification and Validation of Autonomous Driving Systems: A Multilevel Semantic Framework](https://mlsanity.com/paper/2109.06478/) (2021)

## Abstract (arXiv)

We discuss the adequacy of tests for intelligent systems and practical problems raised by their implementation. We propose the replacement test as the ability of a system to replace successfully another system performing a task in a given context. We show how it can characterize salient aspects of human intelligence that cannot be taken into account by the Turing test. We argue that building intelligent systems passing the replacement test involves a series of technical problems that are outside the scope of current AI. We present a framework for implementing the proposed test and validating the properties of the intelligent systems. We discuss the inherent limitations of intelligent system validation and advocate new theoretical foundations for extending existing rigorous test methods. We suggest that the replacement test, based on the complementarity of skills between human and machine, can lead to a multitude of intelligence concepts reflecting the ability to combine data-based and symbolic knowledge to varying degrees.