mlsanityData to 7 Oct 2026

PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation

arXiv 2608.27716 ↗First posted 27 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

AI systems are being deployed on high-stakes, domain-specific workflows that demand correctness not just in the final output, but at every intermediate step. One such workflow is estimating a product carbon footprint (PCF), the greenhouse-gas emissions attributable to a physical product. AI agents are increasingly being used to generate PCFs, but existing evaluations score either total emissions (hiding error sources and cancelling mistakes) or sub-tasks in isolation (missing compositional interactions). We introduce PCFBench, the first benchmark to carve PCF modeling into independently-evaluable tasks that require decomposition, retrieval, ontology matching, and numerical extraction. It comprises 614 expert-labelled items across six tasks. Together they probe reasoning under under-specification, conflicting context, and numerical constraints. Across eight frontier LLMs from four providers, no single model dominates. Although the strongest models estimate total product emissions within 2 times of declared totals on 77% of products, this rate drops to 37-58% when the PCF is generated step by step, with only 45-75% obeying mass conservation. These failures undermine the transparency practitioners need to compare products and drive decarbonization. We release the dataset and evaluation harness to support targeted progress.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 77 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

38 of 77 references.

References · 38

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  4. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  5. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  6. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  7. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  8. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  9. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  10. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  11. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  12. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
26 more
  1. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  2. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  3. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  4. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
  5. 2021SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learningcited by 53
  6. 2025Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMscited by 32
  7. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 27
  8. 2024Belief Revision: The Adaptability of Large Language Models Reasoningcited by 26
  9. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  10. 2025The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMscited by 19
  11. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  12. 2025ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledgecited by 8
  13. 2024OAEI-LLM: A Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matchingcited by 3
  14. 2025Sustainability assessment using multimodal AI agentscited by 3
  15. 2025Automated Extraction of Material Properties using LLM-based AI Agentscited by 3
  16. 2023AutoPCF: Efficient Product Carbon Footprint Accounting with Large Language Modelscited by 2
  17. 2025Entity Linking using LLMs for Automated Product Carbon Footprint Estimationcited by 2
  18. 2025GenOM: Ontology Matching with Description Generation and Large Language Modelcited by 2
  19. 2025Criteria for Credible AI-assisted Carbon Footprinting Systems: The Cases of Mapping and Lifecycle Modelingcited by 2
  20. 2025Not Wrong, But Untrue: LLM Overconfidence in Document-Based Queriescited by 2
  21. 2026STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMscited by 2
  22. 2025OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matchingcited by 1
  23. 2025The Carbon Footprint Wizard: A Knowledge-Augmented AI Interface for Streamlining Food Carbon Footprint Analysiscited by 1
  24. 2025An Expert-grounded benchmark of General Purpose LLMs in LCAcited by 1
  25. 2026Eco-Amazon: Enriching E-commerce Datasets with Product Carbon Footprint for Sustainable Recommendationscited by 1
  26. 2026Mapping the Landscape of Artificial Intelligence in Life Cycle Assessment Using Large Language Modelscited by 1

Built on by

Not cited yet.