mlsanityData to 13 Oct 2026

It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

arXiv 2106.12066 ↗First posted 22 Jun 2021cs.CL cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Commonsense reasoning is one of the key problems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English. Pretrained cross-lingual models are a source of powerful language-agnostic representations, yet their inherent reasoning capabilities are still actively studied. In this work, we design a simple approach to commonsense reasoning which trains a linear classifier with weights of multi-head attention as features. To evaluate this approach, we create a multilingual Winograd Schema corpus by processing several datasets from prior work within a standardized pipeline and measure cross-lingual generalization ability in terms of out-of-sample performance. The method performs competitively with recent supervised and unsupervised approaches for commonsense reasoning, even when applied to other languages in a zero-shot manner. Also, we demonstrate that most of the performance is given by the same small subset of attention heads for all studied languages, which provides evidence of universal reasoning capabilities in multilingual encoders.

Key facts. Cited by 69 papers in the map (data through 13 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

32 of 49 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  6. 2020Scaling Laws for Neural Language Modelscited by 6,494
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,171
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,089
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
20 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 999
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 914
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 849
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 741
  5. 2019How multilingual is Multilingual BERT?cited by 643
  6. 2020Recipes for building an open-domain chatbotcited by 593
  7. 2020Towards a Human-like Open-Domain Chatbotcited by 469
  8. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 332
  9. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 254
  10. 2018A Simple Method for Commonsense Reasoningcited by 223
  11. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  12. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 109
  13. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  14. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  15. 2019Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsingcited by 46
  16. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  17. 2020Contrastive Self-Supervised Learning for Commonsense Reasoningcited by 25
  18. 2019Attention Is (not) All You Need for Commonsense Reasoningcited by 13
  19. 2020Heads-up! Unsupervised Constituency Parsing via Self-Attention Headscited by 12
  20. 2020Precise Task Formalization Matters in Winograd Schema Evaluationscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 8; 2024: 26; 2025: 17; 2026: 13.

69 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 644
  2. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  3. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 156
  4. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  5. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  6. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  7. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  8. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 80
  9. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  10. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  11. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
  12. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
38 more
  1. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  2. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  3. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  4. 2024Effectively Compress KV Heads for LLMcited by 30
  5. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  6. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  7. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 24
  8. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  9. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  10. 2024Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Modelscited by 20
  11. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 19
  12. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  13. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  14. 2024TuBA: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuningcited by 14
  15. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  16. 2024Agent Skill Acquisition for Large Language Models via CycleQDcited by 11
  17. 2025TokAlign: Efficient Vocabulary Adaptation via Token Alignmentcited by 11
  18. 2024Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?cited by 10
  19. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  20. 2024VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Modelscited by 10
  21. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 10
  22. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  23. 2024LiLiuM: eBay's Large Language Models for e-commercecited by 9
  24. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  25. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
  26. 2024Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalizationcited by 7
  27. 2024MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgettingcited by 6
  28. 2024DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimizationcited by 5
  29. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 5
  30. 2023Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attentioncited by 4
  31. 2024OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Modelscited by 4
  32. 2024DiPT: Enhancing LLM reasoning through diversified perspective-takingcited by 4
  33. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  34. 2025Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuningcited by 4
  35. 2026Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Modelscited by 3
  36. 2025Merging Language and Domain Specific Models: The Impact on Technical Vocabulary Acquisitioncited by 2
  37. 2026ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Lawscited by 2
  38. 2021EENLP: Cross-lingual Eastern European NLP Indexcited by 1

50 most cited of 69 All 69 (JSON)