mlsanityData to 7 Oct 2026

Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention

arXiv 2112.03254 ↗First posted 6 Dec 2021cs.CL cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Most of today's AI systems focus on using self-attention mechanisms and transformer architectures on large amounts of diverse data to achieve impressive performance gains. In this paper, we propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear. By integrating external information into the prediction process, we hope to reduce the need for ever-larger models and increase the democratization of AI systems. We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications. In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities. The proposed system, Knowledgeable External Attention for commonsense Reasoning (KEAR), reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4% in comparison to the human accuracy of 88.9%.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 50 of its 66 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

50 of 66 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  11. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  12. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  13. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  14. 2019Abductive Commonsense Reasoningcited by 300
  15. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  16. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  17. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  18. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  19. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  20. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  21. 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
  22. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  23. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  24. 2019Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answeringcited by 68
  25. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  26. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  27. 2020GenericsKB: A Knowledge Base of Generic Statementscited by 51
  28. 2020Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answeringcited by 44
  29. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  30. 2020Fusing Context Into Knowledge Graph for Commonsense Question Answeringcited by 33
  31. 2019Conversing by Reading: Contentful Neural Conversation with On-demand Machine Readingcited by 32
  32. 2021Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Taskscited by 24
  33. 2019How Additional Knowledge can Improve Natural Language Commonsense Question Answering?cited by 22
  34. 2020Posterior Differential Regularization with f-divergence for Improving Model Robustnesscited by 20
  35. 2020Learning Contextualized Knowledge Structures for Commonsense Reasoningcited by 18
  36. 2020Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sourcescited by 14
  37. 2021Leveraging Knowledge in Multilingual Commonsense Reasoningcited by 9
  38. 2019Asking the Right Question: Inferring Advice-Seeking Intentions from Personal Narrativescited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 11; 2023: 17; 2024: 8; 2025: 2.

40 papers.

Sort:

By area:

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  5. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  6. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  7. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  8. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  9. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  10. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  11. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  12. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
28 more
  1. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  2. 2022Revisiting a kNN-based Image Classification System with High-capacity Storagecited by 22
  3. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Modelscited by 18
  4. 2023Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarioscited by 18
  5. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  6. 2023Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Modelscited by 16
  7. 2023Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answeringcited by 14
  8. 2024GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language modelcited by 8
  9. 2022Improving Commonsense in Vision-Language Models via Knowledge Graph Riddlescited by 7
  10. 2023Self-Convinced Prompting: Few-Shot Question Answering with Repeated Introspectioncited by 7
  11. 2023A Case Study in Engineering a Conversational Programming Assistant's Personacited by 6
  12. 2023Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answeringcited by 6
  13. 2024Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedbackcited by 6
  14. 2024CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Modelscited by 5
  15. 2024Collaborative Knowledge Infusion for Low-resource Stance Detectioncited by 4
  16. 2023Thrust: Adaptively Propels Large Language Models with External Knowledgecited by 3
  17. 2022Impossible Triangle: What's Next for Pre-trained Language Models?cited by 2
  18. 2023KEPR: Knowledge Enhancement and Plausibility Ranking for Generative Commonsense Question Answeringcited by 2
  19. 2023CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Modelscited by 2
  20. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questionscited by 2
  21. 2023Retrieval-Augmented Meta Learning for Low-Resource Text Classificationcited by 2
  22. 2023Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrievalcited by 1
  23. 2024Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledgecited by 1
  24. 2025ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safetycited by 1
  25. 2023UFO: Unified Fact Obtaining for Commonsense Question Answeringcited by 0
  26. 2024A Conversational Brain-Artificial Intelligence Interfacecited by 0
  27. 2024Differentiating Choices via Commonality for Multiple-Choice Question Answeringcited by 0
  28. 2024Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)cited by 0