mlsanityData to 14 Oct 2026

Designing and Contextualising Probes for African Languages

arXiv 2505.10081 ↗First posted 15 May 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretrained language models (PLMs) for African languages are continually improving, but the reasons behind these advances remain unclear. This paper presents the first systematic investigation into probing PLMs for linguistic knowledge about African languages. We train layer-wise probes for six typologically diverse African languages to analyse how linguistic features are distributed. We also design control tasks, a way to interpret probe performance, for the MasakhaPOS dataset. We find PLMs adapted for African languages to encode more linguistic information about target languages than massively multilingual PLMs. Our results reaffirm previous findings that token-level syntactic information concentrates in middle-to-last layers, while sentence-level semantic information is distributed across all layers. Through control tasks and probing baselines, we confirm that performance reflects the internal knowledge of PLMs rather than probe memorisation. Our study applies established interpretability techniques to African-language PLMs. In doing so, we highlight the internal mechanisms underlying the success of strategies like active learning and multilingual adaptation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

19 of 29 references.

References · 19

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  7. 2019Designing and Interpreting Probes with Control Taskscited by 640
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  11. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  12. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
7 more
  1. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  2. 2022AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languagescited by 46
  3. 2021Subword Pooling Makes a Differencecited by 18
  4. 2020Analyzing analytical methods: The case of phonology in neural models of spoken languagecited by 16
  5. 2023Multilingual Nonce Dependency Treebanks: Understanding how Language Models represent and process syntactic structurecited by 5
  6. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  7. 2024What do Transformers Know about Government?cited by 1

Built on by

Not cited yet.