mlsanityData to 7 Oct 2026

CsFEVER and CTKFacts: Acquiring Czech data for fact verification

arXiv 2201.11115 ↗First posted 26 Jan 2022cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we examine several methods of acquiring Czech data for automated fact-checking, which is a task commonly modeled as a classification of textual claim veracity w.r.t. a corpus of trusted ground truths. We attempt to collect sets of data in form of a factual claim, evidence within the ground truth corpus, and its veracity label (supported, refuted or not enough info). As a first attempt, we generate a Czech version of the large-scale FEVER dataset built on top of Wikipedia corpus. We take a hybrid approach of machine translation and document alignment; the approach and the tools we provide can be easily applied to other languages. We discuss its weaknesses and inaccuracies, propose a future approach for their cleaning and publish the 127k resulting translations, as well as a version of such dataset reliably applicable for the Natural Language Inference task - the CsFEVER-NLI. Furthermore, we collect a novel dataset of 3,097 claims, which is annotated using the corpus of 2.2M articles of Czech News Agency. We present its extended annotation methodology based on the FEVER approach, and, as the underlying corpus is kept a trade secret, we also publish a standalone version of the dataset for the task of Natural Language Inference we call CTKFactsNLI. We analyze both acquired datasets for spurious cues - annotation patterns leading to model overfitting. CTKFacts is further examined for inter-annotator agreement, thoroughly cleaned, and a typology of common annotator errors is extracted. Finally, we provide baseline models for all stages of the fact-checking pipeline and publish the NLI datasets, as well as our annotation platform and other experimental data.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

32 of 65 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2019How multilingual is Multilingual BERT?cited by 642
  12. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
20 more
  1. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  2. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  3. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  4. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  5. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  6. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 166
  7. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  8. 2021Automated Fact-Checking for Assisting Human Fact-Checkerscited by 144
  9. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  10. 2018The Fact Extraction and VERification (FEVER) Shared Taskcited by 108
  11. 2018Automated Fact Checking: Task formulations, methods and future directionscited by 96
  12. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 90
  13. 2020FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19cited by 60
  14. 2019Neural Architectures for Nested NER through Linearizationcited by 56
  15. 2021Automated Fact-Checking: A Surveycited by 55
  16. 2021Claim Matching Beyond English to Scale Global Fact-Checkingcited by 42
  17. 2021RobeCzech: Czech RoBERTa, a monolingual contextualized language representation modelcited by 33
  18. 2019Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Modelscited by 24
  19. 2021Dataset of Fake News Detection and Fact Verification: A Surveycited by 11
  20. 2021Comparison of Czech Transformers on Text Classification Taskscited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3.

6 papers.

  1. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  2. 2024AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG taskcited by 8
  3. 2022Autonomation, Not Automation: Activities and Needs of European Fact-checkers as a Basis for Designing Human-Centered AI Systemscited by 7
  4. 2023Resources and Few-shot Learners for In-context Learning in Slavic Languagescited by 6
  5. 2023Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Languagecited by 4
  6. 2024Cross-Lingual Learning vs. Low-Resource Fine-Tuning: A Case Study with Fact-Checking in Turkishcited by 4