mlsanityData to 7 Oct 2026

Evaluating Compositionality in Sentence Embeddings

arXiv 1802.04302 ↗First posted 12 Feb 2018cs.CL stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

An important challenge for human-like AI is compositional semantics. Recent research has attempted to address this by using deep neural networks to learn vector space embeddings of sentences, which then serve as input to other tasks. We present a new dataset for one such task, `natural language inference' (NLI), that cannot be solved using only word-level knowledge and requires some compositionality. We find that the performance of state of the art sentence embeddings (InferSent; Conneau et al., 2017) on our new dataset is poor. We analyze the decision rules learned by InferSent and find that they are consistent with simple heuristics that are ecologically valid in its training dataset. Further, we find that augmenting training with our dataset improves test performance on our dataset without loss of performance on the original training dataset. This highlights the importance of structured datasets in better understanding and improving AI systems.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 9; 2023: 6; 2024: 3; 2025: 4.

35 papers.

Sort:

By area:

  1. 2022Language models show human-like content effects on reasoning taskscited by 171
  2. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  3. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  4. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  5. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  6. 2021Figurative Language in Recognizing Textual Entailmentcited by 26
  7. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  8. 2021End-to-End Self-Debiasing Framework for Robust NLU Trainingcited by 24
  9. 2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
  10. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  11. 2021Exploring Transitivity in Neural NLI Models through Veridicalitycited by 15
  12. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
23 more
  1. 2022L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathicited by 10
  2. 2022Feature-Level Debiased Natural Language Understandingcited by 10
  3. 2022Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Languagecited by 9
  4. 2021SyGNS: A Systematic Generalization Testbed Based on Natural Language Semanticscited by 7
  5. 2022Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learningcited by 6
  6. 2025Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modescited by 5
  7. 2023A Simple yet Effective Self-Debiasing Framework for Transformer Modelscited by 4
  8. 2023REFER: An End-to-end Rationale Extraction Framework for Explanation Regularizationcited by 4
  9. 2025Understanding Subword Compositionality of Large Language Modelscited by 3
  10. 2021Variation and generality in encoding of syntactic anomaly information in sentence embeddingscited by 2
  11. 2022To Find Waldo You Need Contextual Cues: Debiasing Who's Waldocited by 2
  12. 2023IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU modelscited by 2
  13. 2021Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmarkcited by 1
  14. 2021Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classificationcited by 1
  15. 2022A Survey of Parameters Associated with the Quality of Benchmarks in NLPcited by 1
  16. 2024Sequential Compositional Generalization in Multimodal Modelscited by 1
  17. 2021Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modellingcited by 0
  18. 2022Polish Natural Language Inference and Factivity -- an Expert-based Dataset and Benchmarkscited by 0
  19. 2023Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairscited by 0
  20. 2023Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inferencecited by 0
  21. 2023Compositional Fusion of Signals in Data Embeddingcited by 0
  22. 2025Large Language Models Are Still Misled by Simple Bias Ensemblescited by 0
  23. 2025L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Modelscited by 0