mlsanityData to 7 Oct 2026

CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment

arXiv 2203.06228 ↗First posted 11 Mar 2022cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natural language understanding (NLU) capabilities of PLMs: Given a definition and a context each for k words, but not the words themselves, the task is to align the k definitions with the k contexts. CoDA21 requires a deep understanding of contexts and definitions, including complex inference and world knowledge. We find that there is a large gap between human and PLM performance, suggesting that CoDA21 measures an aspect of NLU that is not sufficiently covered in existing benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

18 of 26 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
6 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  3. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  4. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  5. 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 40
  6. 2021Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Efficient Zero-shot Event Extraction with Context-Definition Alignmentcited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracity
  2. 2023Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?
  3. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
  4. 2026CL-bench: A Benchmark for Context Learning
  5. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluation
  6. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs
  7. 2023Aligning Large Language Models with Human: A Survey
  8. 2025A Survey on Post-training of Large Language Models