# The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

arXiv: https://arxiv.org/abs/1707.08172 · HTML: https://mlsanity.com/paper/1707.08172/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2017.

## Built on by (most cited first, 23 of 23)

- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://mlsanity.com/paper/2110.06696/) (2021)
- [Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrifice](https://mlsanity.com/paper/2105.14553/) (2021)
- [Evaluating and Modeling Attribution for Cross-Lingual Question Answering](https://mlsanity.com/paper/2305.14332/) (2023)
- [Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models](https://mlsanity.com/paper/2104.08066/) (2021)
- [Learning Better Masking for Better Language Model Pre-training](https://mlsanity.com/paper/2208.10806/) (2022)
- [BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples](https://mlsanity.com/paper/2210.14814/) (2022)
- [Task Compass: Scaling Multi-task Pre-training with Task Prefix](https://mlsanity.com/paper/2210.06277/) (2022)
- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embedding](https://mlsanity.com/paper/2210.07547/) (2022)
- [TRUE: Re-evaluating Factual Consistency Evaluation](https://mlsanity.com/paper/2204.04991/) (2022)
- [Rethinking Textual Adversarial Defense for Pre-trained Language Models](https://mlsanity.com/paper/2208.10251/) (2022)
- [When LLMs Learn to be Students: The SOEI Framework for Modeling and Evaluating Virtual Student Agents in Educational Interaction](https://mlsanity.com/paper/2410.15701/) (2024)
- [Distinguishing Non-natural from Natural Adversarial Samples for More Robust Pre-trained Language Model](https://mlsanity.com/paper/2203.11199/) (2022)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Instance Regularization for Discriminative Language Model Pre-training](https://mlsanity.com/paper/2210.05471/) (2022)
- [Ensembling Finetuned Language Models for Text Classification](https://mlsanity.com/paper/2410.19889/) (2024)
- [Span Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2108.12848/) (2021)
- [Compliance Checking with NLI: Privacy Policies vs. Regulations](https://mlsanity.com/paper/2204.01845/) (2022)
- [Rethinking Offensive Text Detection as a Multi-Hop Reasoning Problem](https://mlsanity.com/paper/2204.10521/) (2022)
- [Language Model Pre-training on True Negatives](https://mlsanity.com/paper/2212.00460/) (2022)
- [Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences](https://mlsanity.com/paper/2110.00697/) (2021)
- [CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation](https://mlsanity.com/paper/2608.12842/) (2026)

## Abstract (arXiv)

This paper presents the results of the RepEval 2017 Shared Task, which evaluated neural network sentence representation learning models on the Multi-Genre Natural Language Inference corpus (MultiNLI) recently introduced by Williams et al. (2017). All of the five participating teams beat the bidirectional LSTM (BiLSTM) and continuous bag of words baselines reported in Williams et al.. The best single model used stacked BiLSTMs with residual connections to extract sentence features and reached 74.5% accuracy on the genre-matched test set. Surprisingly, the results of the competition were fairly consistent across the genre-matched and genre-mismatched test sets, and across subsets of the test data representing a variety of linguistic phenomena, suggesting that all of the submitted systems learned reasonably domain-independent representations for sentence meaning.