# Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

arXiv: https://arxiv.org/abs/1711.01505 · HTML: https://mlsanity.com/paper/1711.01505/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2017.

## Built on by (most cited first, 24 of 24)

- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study](https://mlsanity.com/paper/2106.00872/) (2021)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [The King is Naked: on the Notion of Robustness for Natural Language Processing](https://mlsanity.com/paper/2112.07605/) (2021)
- [Improving Generalizability in Implicitly Abusive Language Detection with Concept Activation Vectors](https://mlsanity.com/paper/2204.02261/) (2022)
- [Metaethical Perspectives on 'Benchmarking' AI Ethics](https://mlsanity.com/paper/2204.05151/) (2022)
- [Verifying the Robustness of Automatic Credibility Assessment](https://mlsanity.com/paper/2303.08032/) (2023)
- [DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation](https://mlsanity.com/paper/2108.13140/) (2021)
- [Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs](https://mlsanity.com/paper/2201.08318/) (2022)
- [Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks](https://mlsanity.com/paper/2310.16955/) (2023)
- [Event Coreference Resolution for Contentious Politics Events](https://mlsanity.com/paper/2203.10123/) (2022)
- [A Multilingual Evaluation of NER Robustness to Adversarial Inputs](https://mlsanity.com/paper/2305.18933/) (2023)
- [Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks](https://mlsanity.com/paper/2105.01192/) (2021)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation](https://mlsanity.com/paper/2401.11185/) (2024)
- [Using Word Embeddings to Analyze Protests News](https://mlsanity.com/paper/2203.05875/) (2022)
- [Analyzing the Generalizability of Deep Contextualized Language Representations For Text Classification](https://mlsanity.com/paper/2303.12936/) (2023)
- [Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks](https://mlsanity.com/paper/2602.12759/) (2026)

## Abstract (arXiv)

This paper presents a summary of the first Workshop on Building Linguistically Generalizable Natural Language Processing Systems, and the associated Build It Break It, The Language Edition shared task. The goal of this workshop was to bring together researchers in NLP and linguistics with a shared task aimed at testing the generalizability of NLP systems beyond the distributions of their training data. We describe the motivation, setup, and participation of the shared task, provide discussion of some highlighted results, and discuss lessons learned.