mlsanityData to 14 Oct 2026

Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarization

arXiv 2509.00529 ↗First posted 30 Aug 2025cs.CL cs.CYcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to generate user-tailored summaries, adapting outputs to specific stakeholders. In legal contexts, this raises important questions about motivated reasoning -- how models strategically frame information to align with a stakeholder's position within the legal system. Building on theories of legal realism and recent trends in legal practice, we investigate how LLMs respond to prompts conditioned on different legal roles (e.g., judges, prosecutors, attorneys) when summarizing judicial decisions. We introduce an evaluation framework grounded in legal fact and reasoning inclusion, also considering favorability towards stakeholders. Our results show that even when prompts include balancing instructions, models exhibit selective inclusion patterns that reflect role-consistent perspectives. These findings raise broader concerns about how similar alignment may emerge as LLMs begin to infer user roles from prior interactions or context, even without explicit role instructions. Our results underscore the need for role-aware evaluation of LLM summarization behavior in high-stakes legal settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 19 of its 55 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

19 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Exams19 May

Other references · 18

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  5. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  6. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  7. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  8. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  9. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  10. 2023Eight Things to Know about Large Language Modelscited by 73
  11. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  12. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
6 more
  1. 2023Summarization is (Almost) Deadcited by 44
  2. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  3. 2024Promises and pitfalls of artificial intelligence for legal applicationscited by 14
  4. 2024Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerlandcited by 7
  5. 2025A Comprehensive Survey on Legal Summarization: Challenges and Future Directionscited by 6
  6. 2025Rehearse With User: Personalized Opinion Summarization via Role-Playing based on Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
  2. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  3. 2025Thinking Longer, Not Always Smarter: Evaluating LLM Capabilities in Hierarchical Legal Reasoning
  4. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
  5. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions
  6. 2026Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
  7. 2025When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
  8. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions