# CogniVal: A Framework for Cognitive Word Embedding Evaluation

arXiv: https://arxiv.org/abs/1909.09001 · HTML: https://mlsanity.com/paper/1909.09001/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 27 of 27)

- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?](https://mlsanity.com/paper/2205.01404/) (2022)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?](https://mlsanity.com/paper/2205.10226/) (2022)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [Do Large Language Models Mirror Cognitive Language Processing?](https://mlsanity.com/paper/2402.18023/) (2024)
- [MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models](https://mlsanity.com/paper/2403.01116/) (2024)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Dataset](https://mlsanity.com/paper/2303.17876/) (2023)
- [LAST at CMCL 2021 Shared Task: Predicting Gaze Data During Reading with a Gradient Boosting Decision Tree Approach](https://mlsanity.com/paper/2104.13043/) (2021)
- [Reading Task Classification Using EEG and Eye-Tracking Data](https://mlsanity.com/paper/2112.06310/) (2021)
- [Language with Vision: a Study on Grounded Word and Sentence Embeddings](https://mlsanity.com/paper/2206.08823/) (2022)
- [BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision](https://mlsanity.com/paper/2309.12056/) (2023)
- [Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase](https://mlsanity.com/paper/2412.04936/) (2024)
- [Every word counts: A multilingual analysis of individual human alignment with model attention](https://mlsanity.com/paper/2210.04963/) (2022)
- [Attention-aware semantic relevance predicting Chinese sentence reading](https://mlsanity.com/paper/2403.18542/) (2024)
- [Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models](https://mlsanity.com/paper/2404.19364/) (2024)
- [Non-invasive Neural Decoding in Source Reconstructed Brain Space](https://mlsanity.com/paper/2410.19838/) (2024)
- [Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings](https://mlsanity.com/paper/2508.13729/) (2025)
- [Cross-view Brain Decoding](https://mlsanity.com/paper/2204.09564/) (2022)
- [Syntactic Structure Processing in the Brain while Listening](https://mlsanity.com/paper/2302.08589/) (2023)
- [Evaluating Word Embeddings with Categorical Modularity](https://mlsanity.com/paper/2106.00877/) (2021)
- [How direct is the link between words and images?](https://mlsanity.com/paper/2206.15381/) (2022)
- [Changes from Classical Statistics to Modern Statistics and Data Science](https://mlsanity.com/paper/2211.03756/) (2022)
- [Bridging Natural Language Processing and Psycholinguistics: computationally grounded semantic similarity datasets for Basque and Spanish](https://mlsanity.com/paper/2304.09616/) (2023)
- [Riding Brainwaves in LLM Space: Understanding Activation Patterns Using Individual Neural Signatures](https://mlsanity.com/paper/2603.21847/) (2026)

## Abstract (arXiv)

An interesting method of evaluating word representations is by how much they reflect the semantic representations in the human brain. However, most, if not all, previous works only focus on small datasets and a single modality. In this paper, we present the first multi-modal framework for evaluating English word representations based on cognitive lexical semantics. Six types of word embeddings are evaluated by fitting them to 15 datasets of eye-tracking, EEG and fMRI signals recorded during language processing. To achieve a global score over all evaluation hypotheses, we apply statistical significance testing accounting for the multiple comparisons problem. This framework is easily extensible and available to include other intrinsic and extrinsic evaluation methods. We find strong correlations in the results between cognitive datasets, across recording modalities and to their performance on extrinsic NLP tasks.