# BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models

arXiv: https://arxiv.org/abs/2510.19419 · HTML: https://mlsanity.com/paper/2510.19419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series](https://mlsanity.com/paper/2405.19327/) (2024)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Release of Pre-Trained Models for the Japanese Language](https://mlsanity.com/paper/2404.01657/) (2024)
- [Investigating Critical Period Effects in Language Acquisition through Neural Language Models](https://mlsanity.com/paper/2407.19325/) (2024)
- [SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT](https://mlsanity.com/paper/2305.19589/) (2023)
- [Second Language Acquisition of Neural Language Models](https://mlsanity.com/paper/2306.02920/) (2023)
- [Minimal Pair-Based Evaluation of Code-Switching](https://mlsanity.com/paper/2506.01840/) (2025)
- [On the Acquisition of Shared Grammatical Representations in Bilingual Language Models](https://mlsanity.com/paper/2503.03962/) (2025)

## Abstract (arXiv)

To bridge the gap between performance-oriented benchmarks and the evaluation of cognitively inspired models, we introduce BLiSS 1.0, a Benchmark of Learner Interlingual Syntactic Structure. Our benchmark operationalizes a new paradigm of selective tolerance, testing whether a model finds a naturalistic learner error more plausible than a matched, artificial error within the same sentence. Constructed from over 2.8 million naturalistic learner sentences, BLiSS provides 136,867 controlled triplets (corrected, learner, artificial) for this purpose. Experiments on a diverse suite of models demonstrate that selective tolerance is a distinct capability from standard grammaticality, with performance clustering strongly by training paradigm. This validates BLiSS as a robust tool for measuring how different training objectives impact a model's alignment with the systematic patterns of human language acquisition.