mlsanityData to 7 Oct 2026

Integration of cognitive tasks into artificial general intelligence test for large models

arXiv 2402.02547 ↗First posted 4 Feb 2024cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

During the evolution of large models, performance evaluation is necessarily performed to assess their capabilities and ensure safety before practical application. However, current model evaluations mainly rely on specific tasks and datasets, lacking a united framework for assessing the multidimensional intelligence of large models. In this perspective, we advocate for a comprehensive framework of cognitive science-inspired artificial general intelligence (AGI) tests, aimed at fulfilling the testing needs of large models with enhanced capabilities. The cognitive science-inspired AGI tests encompass the full spectrum of intelligence facets, including crystallized intelligence, fluid intelligence, social intelligence, and embodied intelligence. To assess the multidimensional intelligence of large models, the AGI tests consist of a battery of well-designed cognitive tests adopted from human intelligence tests, and then naturally encapsulates into an immersive virtual community. We propose increasing the complexity of AGI testing tasks commensurate with advancements in large models and emphasizing the necessity for the interpretation of test results to avoid false negatives and false positives. We believe that cognitive science-inspired AGI tests will effectively guide the targeted improvement of large models in specific dimensions of intelligence and accelerate the integration of large models into human society.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 81 of its 177 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

81 of 177 references.

References · 81

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Segment Anythingcited by 8,158
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  13. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  14. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  15. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  16. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  17. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  18. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  19. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  20. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  21. 2023PaLM 2 Technical Reportcited by 1,154
  22. 2015Teaching Machines to Read and Comprehendcited by 1,039
  23. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  24. 2021Measuring Coding Challenge Competence With APPScited by 879
  25. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  26. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  27. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  28. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  29. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  30. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  31. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  32. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  33. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  34. 2022PAL: Program-aided Language Modelscited by 635
  35. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  36. 2019Object Detection in 20 Years: A Surveycited by 528
  37. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  38. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 1.

5 papers.

  1. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  2. 2024YOLO advances to its genesis: a decadal and comprehensive review of the You Only Look Once (YOLO) seriescited by 22
  3. 2025Visual Large Language Models Exhibit Human-Level Cognitive Flexibility in the Wisconsin Card Sorting Testcited by 5
  4. 2024Unleashing Artificial Cognition: Integrating Multiple AI Systemscited by 3
  5. 2024Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospectscited by 1