Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers
Not on the map. Why?
Abstract (arXiv)
Large language models hallucinate numbers and units when summarizing scientific text, a failure mode that can silently invert a scientific claim. We recast the detection of such errors as typed verification: we introduce a five-class typed-quantity error taxonomy and a 1500-item benchmark, rewritten from PMC and arXiv sources and labeled by two independent LLM annotators with adjudication (Krippendorff's alpha = 0.882). A ModernBERT encoder fine-tuned on this benchmark reaches macro-F1 = 0.899, far above any off-the-shelf neural fact-checker, yet four probes expose a sharp structural blind spot: on canonical-equivalent rewrites of physically equivalent quantities (e.g., 95degC and 368.15 K) its accuracy collapses to 36.5%. We propose Symbolic Augmentation, a training-time framework that runs the modules of a symbolic verifier in reverse to generate label-preserving augmented training data. The augmentation lifts canonical-equivalence robustness to 98.2% while slightly improving in-distribution accuracy (macro-F1: 0.899 to 0.902); the augmented encoder matches a closed-frontier LLM at no inference cost and transfers to an external benchmark (SciFact-Open binary macro-F1: 0.791 to 0.828). Two negative results sharpen the claim: symbolic features as auxiliary encoder inputs add nothing, and symbolic silver labels scale negatively under teacher noise. Together these results identify training-time augmentation as the right integration point between symbolic and learned components.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 19 May 2026.
Builds on
24 of 48 references.
References · 24
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
- 2020Fact or Fiction: Verifying Scientific Claimscited by 554
- 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
- 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
- 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
- 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
- 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
- 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
- 2025SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakescited by 27
12 more
- 2025Ai2 Scholar QA: Organized Literature Synthesis with Attributioncited by 25
- 2024QuanTemp: A real-world open-domain benchmark for fact-checking numerical claimscited by 23
- 2021Overview and Insights from the SciVer Shared Task on Scientific Claim Verificationcited by 17
- 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 13
- 2023EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verificationcited by 9
- 2025VeriFastScore: Speeding up long-form factuality evaluationcited by 8
- 2025A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decompositioncited by 3
- 2025HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verificationcited by 3
- 2025DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verificationcited by 2
- 2025ClaimIQ at CheckThat! 2025: Comparing Prompted and Fine-Tuned Language Models for Verifying Numerical Claimscited by 2
- 2025Think Right, Not More: Test-Time Scaling for Numerical Claim Verificationcited by 2
- 2025From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errorscited by 1
Built on by
Not cited yet.