# Evaluation Benchmarks for Spanish Sentence Representations

arXiv: https://arxiv.org/abs/2204.07571 · HTML: https://mlsanity.com/paper/2204.07571/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [MLSUM: The Multilingual Summarization Corpus](https://mlsanity.com/paper/2004.14900/) (2020)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Translation Artifacts in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2004.04721/) (2020)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://mlsanity.com/paper/1909.02597/) (2019)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [SLM: Learning a Discourse Language Representation with Sentence Unshuffling](https://mlsanity.com/paper/2010.16249/) (2020)
- [LINSPECTOR: Multilingual Probing Tasks for Word Representations](https://mlsanity.com/paper/1903.09442/) (2019)
- [Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations](https://mlsanity.com/paper/1909.00142/) (2019)
- [Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation](https://mlsanity.com/paper/1807.03121/) (2018)
- [Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks](https://mlsanity.com/paper/2002.06261/) (2020)
- [Probing Multilingual Sentence Representations With X-Probe](https://mlsanity.com/paper/1906.05061/) (2019)
- [Evaluating Discourse in Structured Text Representations](https://mlsanity.com/paper/1906.01472/) (2019)
- [The Gutenberg Dialogue Dataset](https://mlsanity.com/paper/2004.12752/) (2020)
- [SILT: Efficient transformer training for inter-lingual inference](https://mlsanity.com/paper/2103.09635/) (2021)
- [Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations](https://mlsanity.com/paper/2109.04602/) (2021)
- [RuSentEval: Linguistic Source, Encoder Force!](https://mlsanity.com/paper/2103.00573/) (2021)

## Built on by (most cited first, 6 of 6)

- [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment](https://mlsanity.com/paper/2407.14878/) (2024)
- [Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models](https://mlsanity.com/paper/2410.12011/) (2024)
- [Unveiling Social Media Comments with a Novel Named Entity Recognition System for Identity Groups](https://mlsanity.com/paper/2405.13011/) (2024)

## Abstract (arXiv)

Due to the success of pre-trained language models, versions of languages other than English have been released in recent years. This fact implies the need for resources to evaluate these models. In the case of Spanish, there are few ways to systematically assess the models' quality. In this paper, we narrow the gap by building two evaluation benchmarks. Inspired by previous work (Conneau and Kiela, 2018; Chen et al., 2019), we introduce Spanish SentEval and Spanish DiscoEval, aiming to assess the capabilities of stand-alone and discourse-aware sentence representations, respectively. Our benchmarks include considerable pre-existing and newly constructed datasets that address different tasks from various domains. In addition, we evaluate and analyze the most recent pre-trained Spanish language models to exhibit their capabilities and limitations. As an example, we discover that for the case of discourse evaluation tasks, mBERT, a language model trained on multiple languages, usually provides a richer latent representation than models trained only with documents in Spanish. We hope our contribution will motivate a fairer, more comparable, and less cumbersome way to evaluate future Spanish language models.