# EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals

arXiv: https://arxiv.org/abs/2608.24086 · HTML: https://mlsanity.com/paper/2608.24086/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [DeepMIMO: A Generic Deep Learning Dataset for Millimeter Wave and Massive MIMO Applications](https://mlsanity.com/paper/1902.06435/) (2019)
- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [MiniMax Sparse Attention](https://mlsanity.com/paper/2606.13392/) (2026)
- [WiSig: A Large-Scale WiFi Signal Dataset for Receiver and Channel Agnostic RF Fingerprinting](https://mlsanity.com/paper/2112.15363/) (2021)
- [RadioLLM: Introducing Large Language Model into Cognitive Radio via Hybrid Prompt and Token Reprogrammings](https://mlsanity.com/paper/2501.17888/) (2025)
- [RF-GPT: Teaching AI to See the Wireless World](https://mlsanity.com/paper/2602.14833/) (2026)
- [PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision](https://mlsanity.com/paper/2603.28183/) (2026)
- [MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals](https://mlsanity.com/paper/2603.08174/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as code agents for scientific and engineering analysis, but their ability to analyze raw physical-layer measurements remains untested. We introduce EMRB (Electromagnetic Reasoning Benchmark), which evaluates whether LLMs can analyze raw I/Q data by writing and running code. EMRB contains 200 problems across five difficulty levels and 27 question types, from signal detection to OFDM design, generated from 11 signal types with verified ground truth. Unlike benchmarks built on preprocessed features or structured tables, EMRB provides only the raw capture; the quantities each question refers to must first be discovered through code. We evaluate 14 LLMs spanning proprietary, open-weight, and reasoning-oriented families. Scores range from 24.1% to 78.9%, with the mean dropping from 84.9% on basic measurement to 21.2% on system design. We also propose ReconPilot, a structured method that separates signal reconnaissance, targeted analysis, and self-verification. Across three backbones, ReconPilot raises the overall score by 3.8 to 17.6 points and improves 13 of 15 backbone-level combinations tested. All data and code are publicly released in hrefhttps://github.com/mingxuZhang2/EMRBtextcolorblueour GitHub repository.