# RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms

arXiv: https://arxiv.org/abs/2005.00782 · HTML: https://mlsanity.com/paper/2005.00782/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 28 of 28)

- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources](https://mlsanity.com/paper/2103.11320/) (2021)
- [Commonsense-Focused Dialogues for Response Generation: An Empirical Study](https://mlsanity.com/paper/2109.06427/) (2021)
- [GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models](https://mlsanity.com/paper/2205.12247/) (2022)
- [Dimensions of Commonsense Knowledge](https://mlsanity.com/paper/2101.04640/) (2021)
- [RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning](https://mlsanity.com/paper/2205.12598/) (2022)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00930/) (2022)
- [Probing Commonsense Explanation in Dialogue Response Generation](https://mlsanity.com/paper/2104.09574/) (2021)
- [Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models](https://mlsanity.com/paper/2109.02837/) (2021)
- [PaCo: Preconditions Attributed to Commonsense Knowledge](https://mlsanity.com/paper/2104.08712/) (2021)
- [In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search](https://mlsanity.com/paper/2311.07237/) (2023)
- [A Survey on Symbolic Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2408.10210/) (2024)
- [Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge](https://mlsanity.com/paper/2112.08583/) (2021)
- [Contextualized Scene Imagination for Generative Commonsense Reasoning](https://mlsanity.com/paper/2112.06318/) (2021)
- [Empower Nested Boolean Logic via Self-Supervised Curriculum Learning](https://mlsanity.com/paper/2310.05450/) (2023)
- [Life after BERT: What do Other Muppets Understand about Language?](https://mlsanity.com/paper/2205.10696/) (2022)
- [The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systems](https://mlsanity.com/paper/2212.08192/) (2022)
- [QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering](https://mlsanity.com/paper/2310.11303/) (2023)
- [CoSe-Co: Text Conditioned Generative CommonSense Contextualizer](https://mlsanity.com/paper/2206.05706/) (2022)
- [Robust Text Classification: Analyzing Prototype-Based Networks](https://mlsanity.com/paper/2311.06647/) (2023)
- [Transferring Procedural Knowledge across Commonsense Tasks](https://mlsanity.com/paper/2304.13867/) (2023)
- [Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?](https://mlsanity.com/paper/2311.09109/) (2023)
- [Commonsense on Demand: Generating and Selectively Integrating Commonsense Knowledge for Natural Language Inference](https://mlsanity.com/paper/2507.15100/) (2025)
- [CoVeR: Conformal Calibration for Versatile and Reliable Autoregressive Next-Token Prediction](https://mlsanity.com/paper/2509.04733/) (2025)

## Abstract (arXiv)

Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference capability based on Commonsense Axioms, that evaluates robust commonsense inference despite textual perturbations. To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense.