mlsanityData to 14 Oct 2026

JEC-QA: A Legal-Domain Question Answering Dataset

arXiv 1911.12011 ↗First posted 27 Nov 2019cs.CLcited by 116

Not on the map. Why?

Abstract (arXiv)

We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of professional skills for legal practitioners. College students are required to pass the examination to be certified as a lawyer or a judge. The dataset is challenging for existing question answering methods, because both retrieving relevant materials and answering questions require the ability of logic reasoning. Due to the high demand of multiple reasoning abilities to answer legal questions, the state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while skilled humans and unskilled humans can reach 81% and 64% accuracy respectively, which indicates a huge gap between humans and machines on this task. We will release JEC-QA and our baselines to help improve the reasoning ability of machine comprehension models. You can access the dataset from http://jecqa.thunlp.org/.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 8; 2023: 16; 2024: 33; 2025: 29; 2026: 19.

116 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,341
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  4. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  7. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  8. 2023Large Language Models in Law: A Surveycited by 128
  9. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
  10. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  11. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  12. 2024Language models scale reliably with over-training and on downstream taskscited by 113
38 more
  1. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  2. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69
  3. 2022GPT Takes the Bar Examcited by 65
  4. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 60
  5. 2023Lawyer LLaMA Technical Reportcited by 54
  6. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  7. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  8. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  9. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  10. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  11. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  12. 2024AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentationcited by 40
  13. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  14. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  15. 2024GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theorycited by 37
  16. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 37
  17. 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
  18. 2022LEVEN: A Large-Scale Chinese Legal Event Detection Datasetcited by 36
  19. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  20. 2024LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Modelcited by 34
  21. 2024Leveraging Large Language Models for Learning Complex Legal Concepts through Storytellingcited by 23
  22. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 22
  23. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
  24. 2024Chain of Logic: Rule-Based Reasoning with Large Language Modelscited by 18
  25. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  26. 2024CitaLaw: Enhancing LLM with Citations in Legal Domaincited by 16
  27. 2023VNHSGE: VietNamese High School Graduation Examination Dataset for Large Language Modelscited by 15
  28. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Modelscited by 15
  29. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
  30. 2025It's High Time: A Survey of Temporal Question Answeringcited by 14
  31. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13
  32. 2023Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on Chinese Legal Domaincited by 12
  33. 2025Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledgecited by 12
  34. 2025LegalDelta: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gaincited by 12
  35. 2021Equality before the Law: Legal Judgment Consistency Analysis for Fairnesscited by 11
  36. 2022HLDC: Hindi Legal Documents Corpuscited by 11
  37. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  38. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 11

50 most cited of 116 All 116 (JSON)