mlsanityData to 7 Oct 2026

Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit

arXiv 2609.32042 ↗First posted 25 Sep 2026cs.CLcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

We ask how GGUF quantization affects agentic tool use in Polish and whether the effects generalize across models. We introduce PolAgentBench, a deterministic benchmark with Polish prompts and English tool schemas: a 67-task main suite (15 adversarial probes, 52 hard-tier tasks) and a 46-task arithmetic isolation ladder. Three models span two axes of variation: Bielik-11B-v3.0 and its pruned, distilled child Bielik-Minitron-7B-v3.0 isolate model compression, and Llama-PLLuM-8B adds a change of pretraining family. Each is measured at six precisions, Q8_0 to Q2_K. Only the collapse threshold replicates. (1) All three models fall off a cliff between 3-bit and 2-bit (11B 0.716 to 0.045, 7B 0.463 to 0.149, PLLuM 0.224 to 0.015; paired McNemar p < 0.001 in each), across a fourfold capability spread and both axes. (2) Failure modes do not replicate: at 2-bit the 7B fails long (median 9.1k tokens, 4 steps) while the 11B mostly answers at the first step with a confabulated final answer (37 of 64 failures); PLLuM fails on content across precisions (71.8-92.0% of steps parse). (3) On the arithmetic ladder the unscaffolded rung is a floor, left standing by a rerun that states the no-tool rule; four explicit calls lift the 8-bit 11B from 1/10 to 9/10 and the 7B from 0/10 to 7/10 after format-only failures with the gold value are forgiven, an exploratory effect with eight distinct baseline inputs that does not survive multiplicity correction, while the order-trap arm separates the models at 8-bit (11B 6/6, 7B 0/6). (4) The Polish-versus-English gap is associated with degradation or with task family. We document four artifacts that shaped our conclusions (rounding-hostile gold values, strict answer typing, a no-tool rule the prompt never stated, priority-ordered failure labels), report affected results in strict and corrected form, and release the benchmark, trajectories and commit-stamped artifacts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 15 of its 18 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

15 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling30 Sep
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul

Other references · 13

  1. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  2. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  3. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  4. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  5. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  6. 2024Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluationcited by 14
  7. 2025PLLuM: A Family of Polish Large Language Modelscited by 7
  8. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 7
  9. 2025Bielik 11B v3: Multilingual Large Language Model for European Languagescited by 6
  10. 2025TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devicescited by 4
  11. 2026Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Languagecited by 4
  12. 2025Resource-Efficient Language Models: Quantization for Fast and Accessible Inferencecited by 1
1 more
  1. 2026Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Modelcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designs
  2. 2026Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
  3. 2026Agentic Tool Use in Large Language Models
  4. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use
  5. 2026Memory-Induced Tool-Drift in LLM Agents
  6. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability
  7. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  8. 2026Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents