mlsanityData to 14 Oct 2026

HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

arXiv 2607.14349 ↗First posted 15 Jul 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

While Large Language Models (LLMs) excel in many general NLP tasks, their formal reasoning capabilities are often compromised by content effects, demonstrating a measurable bias towards real-world plausibility. In this paper, we present our system for SemEval-2026 Task 11, which evaluates the ability of models to disentangle formal logic from content across 12 languages with and without distractor premises. We address this challenge using mDeBERTa-v3 networks fine-tuned on a synthetic, rule-based dataset of syllogistic schemes to avoid the semantic noise of LLM-augmented data. To explicitly decouple plausibility from logical structure, our training pipeline employs a multi-objective loss function combining Adaptive Group Distributionally Robust Optimization (DRO), a scheduled differentiable bias penalty, and KL-Divergence consistency regularization. Our system achieved #1 ranks and perfect Ranking Scores (100.0) with 0.00% bias and 100.0% accuracy on Subtask 1 (English), Subtask 2 (Noisy English), and Subtask 3 (Multilingual). On the highly complex Subtask 4 (Noisy Multilingual), the system achieved the 6th rank with 89.06% Accuracy and F1-score, alongside a limited 2.89% Bias and a 37.78 Ranking Score. Our dataset generation engine and codebase are publicly available to facilitate future work on robust logical reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 19 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

10 of 19 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Survey21 Feb
  2. 2025SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Tasks20 Jan
  3. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models24 Jun
  4. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences17 Jun

Other references · 6

  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  2. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  4. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 96
  5. 2024Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Provingcited by 35
  6. 2025Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steeringcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs
  2. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
  3. 2026LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening
  4. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic
  5. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models
  6. 2025Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs
  7. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models
  8. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models