mlsanityData to 7 Oct 2026

RuSentEval: Linguistic Source, Encoder Force!

arXiv 2103.00573 ↗First posted 28 Feb 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The success of pre-trained transformer language models has brought a great deal of interest on how these models work, and what they learn about language. However, prior research in the field is mainly devoted to English, and little is known regarding other languages. To this end, we introduce RuSentEval, an enhanced set of 14 probing tasks for Russian, including ones that have not been explored yet. We apply a combination of complementary probing methods to explore the distribution of various linguistic properties in five multilingual transformers for two typologically contrasting languages -- Russian and English. Our results provide intriguing findings that contradict the common understanding of how linguistic knowledge is represented, and demonstrate that some properties are learned in a similar manner despite the language differences.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 28 Feb 2021.

Builds on

37 of 61 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
25 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  4. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  9. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  10. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  11. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  12. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  13. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  14. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  15. 2018UniMorph 2.0: Universal Morphologycited by 40
  16. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  17. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  18. 2020Finding Experts in Transformer Modelscited by 28
  19. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  20. 2019LINSPECTOR: Multilingual Probing Tasks for Word Representationscited by 22
  21. 2020Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extractioncited by 18
  22. 2019Probing Multilingual Sentence Representations With X-Probecited by 13
  23. 2020How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluationcited by 7
  24. 2019AGRR-2019: A Corpus for Gapping Resolution in Russiancited by 1
  25. 2020Cross-neutralising: Probing for joint encoding of linguistic information in multilingual modelscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1; 2025: 1.

4 papers.

  1. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 23
  2. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  3. 2022Evaluation Benchmarks for Spanish Sentence Representationscited by 6
  4. 2022Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluationcited by 3