mlsanityData to 7 Oct 2026

Towards explainable evaluation of language models on the semantic similarity of visual concepts

arXiv 2209.03723 ↗First posted 8 Sep 2022cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent breakthroughs in NLP research, such as the advent of Transformer models have indisputably contributed to major advancements in several tasks. However, few works research robustness and explainability issues of their evaluation strategies. In this work, we examine the behavior of high-performing pre-trained language models, focusing on the task of semantic similarity for visual vocabularies. First, we address the need for explainable evaluation metrics, necessary for understanding the conceptual quality of retrieved instances. Our proposed metrics provide valuable insights in local and global level, showcasing the inabilities of widely used approaches. Secondly, adversarial interventions on salient query semantics expose vulnerabilities of opaque metrics and highlight patterns in learned linguistic representations.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

21 of 36 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
9 more
  1. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  3. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  4. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  5. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  6. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  7. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  8. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  9. 2019Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?cited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 2.

5 papers.

  1. 2024Optimal and efficient text counterfactuals using Graph Neural Networkscited by 1
  2. 2023Counterfactual Edits for Generative Evaluationcited by 0
  3. 2023Knowledge-Based Counterfactual Queries for Visual Question Answeringcited by 0
  4. 2025HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioningcited by 0
  5. 2025Conceptual Contrastive Edits in Textual and Vision-Language Retrievalcited by 0