mlsanityData to 7 Oct 2026

When Harness Beats Scale, and When Reading Beats Both

arXiv 2609.34366 ↗First posted 28 Sep 2026cs.CL cs.AI cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

We describe our system for DocSem, the document-grounded quantitative reasoning shared task at DocInsights 2026, and analyze why it succeeded on labeled data and failed on the test set. The pipeline pairs hybrid block retrieval with Program-of-Thoughts (PoT) generation executed in a sandboxed interpreter, self-consistency sampling, and entity enrichment from chunk-level knowledge graphs. On our held-out split, application architecture moved the metrics far more than model scale did: PoT added 0.282 joint accuracy to a compact 7B model but at most 0.005 to a 72B model, and a 27B model with the full harness matched the 72B (0.884 vs. 0.873) at roughly 2.7× fewer parameters and a quarter of the CO2. We read this through a distinction between world knowledge, which scales steeply with parameters, and language knowledge, which scales gently, and show that structured-output training makes a compact model harness-ready rather than merely small. On the raster, watermarked test PDFs the same system collapsed to 13.58% joint (rank 149 of 163); a controlled re-rendering of the validation set reproduces the OCR half of the collapse while bounding what the simulation misses. Auditing the physical nature of evaluation inputs precedes architecture, and the leaderboard's bimodality is consistent with reading quality, not reasoning, having separated the field.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

15 of 32 references.

References · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  8. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
  9. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  10. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  11. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
  12. 2021NEREL: A Russian Dataset with Nested Named Entities, Relations and Eventscited by 13
3 more
  1. 2026RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generationcited by 3
  2. 2026GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentationscited by 2
  3. 2026RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLMcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
  2. 2026Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams
  3. 2026DocAtlas: Long-Document Understanding as Mutable-State Interaction
  4. 2026The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications
  5. 2026Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities
  6. 2026Meta-Harness: End-to-End Optimization of Model Harnesses
  7. 2026Scaling Multi-Hop Training Data via Graph-Constrained Path Selection
  8. 2025DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections