# CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding

arXiv: https://arxiv.org/abs/2107.09847 · HTML: https://mlsanity.com/paper/2107.09847/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 59 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)
- [KnowIT VQA: Answering Knowledge-Based Questions about Videos](https://mlsanity.com/paper/1910.10706/) (2019)
- [Towards a Better Metric for Evaluating Question Generation Systems](https://mlsanity.com/paper/1808.10192/) (2018)
- [DramaQA: Character-Centered Video Story Understanding with Hierarchical QA](https://mlsanity.com/paper/2005.03356/) (2020)
- [A Survey on Machine Reading Comprehension Systems](https://mlsanity.com/paper/2001.01582/) (2020)
- [To Test Machine Comprehension, Start by Defining Comprehension](https://mlsanity.com/paper/2005.01525/) (2020)
- [Recent Advances in Video Question Answering: A Review of Datasets and Methods](https://mlsanity.com/paper/2101.05954/) (2021)
- [Constructing Hierarchical Q&A Datasets for Video Story Understanding](https://mlsanity.com/paper/1904.00623/) (2019)
- [Co-attentional Transformers for Story-Based Video Understanding](https://mlsanity.com/paper/2010.14104/) (2020)

## Built on by (most cited first, 1 of 1)

- [Commonsense for Zero-Shot Natural Language Video Localization](https://mlsanity.com/paper/2312.17429/) (2023)

## Abstract (arXiv)

We introduce CogME, a cognition-inspired, multi-dimensional evaluation metric designed for AI models focusing on story understanding. CogME is a framework grounded in human thinking strategies and story elements that involve story understanding. With a specific breakdown of the questions, this approach provides a nuanced assessment revealing not only AI models' particular strengths and weaknesses but also the characteristics of the benchmark dataset. Our case study with the DramaQA dataset demonstrates a refined analysis of the model and the benchmark dataset. We argue the need for metrics based on understanding the nature of tasks and designed to align closely with human cognitive processes. This approach provides insights beyond traditional overall scores and paves the way for more sophisticated AI development targeting higher cognitive functions.