mlsanityData to 7 Oct 2026

ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark

arXiv 2304.12202 ↗First posted 9 Mar 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Following the hype around OpenAI's ChatGPT conversational agent, the last straw in the recent development of Large Language Models (LLMs) that demonstrate emergent unprecedented zero-shot capabilities, we audit the latest OpenAI's GPT-3.5 model, `gpt-3.5-turbo', the first available ChatGPT model, in the LexGLUE benchmark in a zero-shot fashion providing examples in a templated instruction-following format. The results indicate that ChatGPT achieves an average micro-F1 score of 47.6% across LexGLUE tasks, surpassing the baseline guessing rates. Notably, the model performs exceptionally well in some datasets, achieving micro-F1 scores of 62.8% and 70.2% in the ECtHR B and LEDGAR datasets, respectively. The code base and model predictions are available for review on https://github.com/coastalcph/zeroshot_lexglue.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

15 of 20 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  9. 2019Neural Legal Judgment Prediction in Englishcited by 208
  10. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  11. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  12. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
3 more
  1. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  2. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  3. 2023GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilitiescited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 16; 2025: 7; 2026: 2.

33 papers.

Sort:

By area:

  1. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  2. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  3. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  4. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48
  5. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  6. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  7. 2023Augmented Large Language Models with Parametric Knowledge Guidingcited by 37
  8. 2024IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoningcited by 16
  9. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Modelscited by 15
  10. 2024Promises and pitfalls of artificial intelligence for legal applicationscited by 14
  11. 2023Large Language Models are legal but they are not: Making the case for a powerful LegalLLMcited by 12
  12. 2025Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Expertscited by 9
21 more
  1. 2023Large Language Model Prompt Chaining for Long Legal Document Classificationcited by 7
  2. 2023Chatmap : Large Language Model Interaction with Cartographic Datacited by 7
  3. 2024Exploring the Nexus of Large Language Models and Legal Systems: A Short Surveycited by 6
  4. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  5. 2024Logic Rules as Explanations for Legal Case Retrievalcited by 5
  6. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
  7. 2024Aalap: AI Assistant for Legal & Paralegal Functions in Indiacited by 4
  8. 2024Legal Evalutions and Challenges of Large Language Modelscited by 4
  9. 2025Compressing Language Models for Specialized Domainscited by 4
  10. 2025Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penaltiescited by 3
  11. 2024Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4cited by 2
  12. 2024LAPIS: Language Model-Augmented Police Investigation Systemcited by 2
  13. 2025Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classificationcited by 2
  14. 2025Evaluating the Role of Large Language Models in Legal Practice in Indiacited by 2
  15. 2026Knowledge Graph-Assisted LLM Post-Training for Enhanced Legal Reasoningcited by 2
  16. 2024From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudencecited by 1
  17. 2023NESTLE: a No-Code Tool for Statistical Analysis of Legal Corpuscited by 0
  18. 2024Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?cited by 0
  19. 2024Impact of ChatGPT on the writing style of condensed matter physicistscited by 0
  20. 2025Explainable Statute Prediction via Attention-based Model and LLM Promptingcited by 0
  21. 2026Can Small Models Reason About Legal Documents? A Comparative Studycited by 0