# Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

arXiv: https://arxiv.org/abs/2607.16212 · HTML: https://mlsanity.com/paper/2607.16212/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 24 of 24)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks](https://mlsanity.com/paper/2204.05660/) (2022)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [FineSurE: Fine-grained Summarization Evaluation using LLMs](https://mlsanity.com/paper/2407.00908/) (2024)
- [Generating Scientific Claims for Zero-Shot Scientific Fact Checking](https://mlsanity.com/paper/2203.12990/) (2022)
- [SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes](https://mlsanity.com/paper/2504.11975/) (2025)
- [Ai2 Scholar QA: Organized Literature Synthesis with Attribution](https://mlsanity.com/paper/2504.10861/) (2025)
- [QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims](https://mlsanity.com/paper/2403.17169/) (2024)
- [Overview and Insights from the SciVer Shared Task on Scientific Claim Verification](https://mlsanity.com/paper/2107.08188/) (2021)
- [Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers](https://mlsanity.com/paper/2506.13342/) (2025)
- [EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verification](https://mlsanity.com/paper/2310.14508/) (2023)
- [VeriFastScore: Speeding up long-form factuality evaluation](https://mlsanity.com/paper/2505.16973/) (2025)
- [A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition](https://mlsanity.com/paper/2510.22055/) (2025)
- [HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification](https://mlsanity.com/paper/2512.22396/) (2025)
- [DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification](https://mlsanity.com/paper/2507.06195/) (2025)
- [ClaimIQ at CheckThat! 2025: Comparing Prompted and Fine-Tuned Language Models for Verifying Numerical Claims](https://mlsanity.com/paper/2509.11492/) (2025)
- [Think Right, Not More: Test-Time Scaling for Numerical Claim Verification](https://mlsanity.com/paper/2509.22101/) (2025)
- [From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors](https://mlsanity.com/paper/2509.20065/) (2025)

## Abstract (arXiv)

Large language models hallucinate numbers and units when summarizing scientific text, a failure mode that can silently invert a scientific claim. We recast the detection of such errors as typed verification: we introduce a five-class typed-quantity error taxonomy and a 1500-item benchmark, rewritten from PMC and arXiv sources and labeled by two independent LLM annotators with adjudication (Krippendorff's alpha = 0.882). A ModernBERT encoder fine-tuned on this benchmark reaches macro-F1 = 0.899, far above any off-the-shelf neural fact-checker, yet four probes expose a sharp structural blind spot: on canonical-equivalent rewrites of physically equivalent quantities (e.g., 95degC and 368.15 K) its accuracy collapses to 36.5%. We propose Symbolic Augmentation, a training-time framework that runs the modules of a symbolic verifier in reverse to generate label-preserving augmented training data. The augmentation lifts canonical-equivalence robustness to 98.2% while slightly improving in-distribution accuracy (macro-F1: 0.899 to 0.902); the augmented encoder matches a closed-frontier LLM at no inference cost and transfers to an external benchmark (SciFact-Open binary macro-F1: 0.791 to 0.828). Two negative results sharpen the claim: symbolic features as auxiliary encoder inputs add nothing, and symbolic silver labels scale negatively under teacher noise. Together these results identify training-time augmentation as the right integration point between symbolic and learned components.