# MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models

arXiv: https://arxiv.org/abs/2403.01116 · HTML: https://mlsanity.com/paper/2403.01116/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 58 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- [Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities](https://mlsanity.com/paper/2305.17214/) (2023)
- [CogniVal: A Framework for Cognitive Word Embedding Evaluation](https://mlsanity.com/paper/1909.09001/) (2019)
- [Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps](https://mlsanity.com/paper/2207.03380/) (2022)
- [Robust Evaluation of Language-Brain Encoding Experiments](https://mlsanity.com/paper/1904.02547/) (2019)
- [Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning](https://mlsanity.com/paper/2310.10318/) (2023)
- [Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity](https://mlsanity.com/paper/1711.09285/) (2017)
- [From Language to Language-ish: How Brain-Like is an LSTM's Representation of Nonsensical Language Stimuli?](https://mlsanity.com/paper/2010.07435/) (2020)

## Built on by (most cited first, 9 of 9)

- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers](https://mlsanity.com/paper/2505.01482/) (2025)
- [Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency](https://mlsanity.com/paper/2507.08309/) (2025)
- [Computational Models to Study Language Processing in the Human Brain: A Survey](https://mlsanity.com/paper/2403.13368/) (2024)
- [Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models](https://mlsanity.com/paper/2404.19364/) (2024)
- [THiNK: Can Large Language Models Think-aloud?](https://mlsanity.com/paper/2505.20184/) (2025)
- [Discovering Semantic Subdimensions through Disentangled Conceptual Representations](https://mlsanity.com/paper/2508.21436/) (2025)
- [The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese](https://mlsanity.com/paper/2607.10745/) (2026)

## Abstract (arXiv)

Pre-trained computational language models have recently made remarkable progress in harnessing the language abilities which were considered unique to humans. Their success has raised interest in whether these models represent and process language like humans. To answer this question, this paper proposes MulCogBench, a multi-modal cognitive benchmark dataset collected from native Chinese and English participants. It encompasses a variety of cognitive data, including subjective semantic ratings, eye-tracking, functional magnetic resonance imaging (fMRI), and magnetoencephalography (MEG). To assess the relationship between language models and cognitive data, we conducted a similarity-encoding analysis which decodes cognitive data based on its pattern similarity with textual embeddings. Results show that language models share significant similarities with human cognitive data and the similarity patterns are modulated by the data modality and stimuli complexity. Specifically, context-aware models outperform context-independent models as language stimulus complexity increases. The shallow layers of context-aware models are better aligned with the high-temporal-resolution MEG signals whereas the deeper layers show more similarity with the high-spatial-resolution fMRI. These results indicate that language models have a delicate relationship with brain language representations. Moreover, the results between Chinese and English are highly consistent, suggesting the generalizability of these findings across languages.