116 stories tagged Benchmark
A new benchmark called MERIT measures the marginal utility of long-term memory for tool-using LLM agents. The study finds that memory often fails to improve task performance, especially when access costs are high, challenging assumptions about AI memory in real-world applications.
Researchers introduced Harbor Adapters, a unified evaluation infrastructure that ports over 80 benchmarks for testing AI agents, and Harbor-Index, a curated meta-dataset of results from 8 models across 54 benchmarks.
Researchers introduced EvalDetectBench, an open benchmark that measures evaluation awareness in frontier large language models. If models behave differently in tests than in real-world deployment, it could undermine AI safety assessments.
Allen Institute for AI released BenchMIRT, a free tool that uses item response theory to reveal whether LLM benchmarks measure true capability or test-taker biases, helping developers and users choose better models.
Researchers introduced a benchmark to measure how often large multimodal reasoning models (LMRMs) agree with users who provide incorrect answers, a behavior known as sycophancy. The benchmark pairs visual questions with wrong answers to test model reliability.
OpenAI released HealthBench-Psych, a specialized benchmark to evaluate how well AI models handle mental health conversations. This tool helps developers improve AI's ability to provide psychological support, which is increasingly important as more people turn to AI for mental health assistance.
Researchers introduced FinSkillBench, a benchmark that evaluates AI agents on domain-specific financial skills—portfolio construction, risk management, and fundamental analysis—rather than just text generation, to ensure reliable and auditable outputs for investment management.
A new position paper on arXiv argues that AI leaderboards are structurally ill-suited to serving the Global South due to a lack of independent governance, conflict-of-interest policies, and metric evolution mechanisms. The paper emphasizes that the barrier is not missing data—high-quality regional benchmarks like IndicSUPERB, MILU, and LAHAJA for India, IrokoBench for Africa, and AlGhafa for Arabic already exist—but rather institutional design and commercial pressure.
A new ArXiv study challenges the common use of FLOPs (Floating Point Operations) to measure AI efficiency, finding that operations with the same FLOPs can have execution times varying by up to 30% due to differences in parallelization. The researchers replicated a previous study to demonstrate why FLOPs alone are a misleading metric, calling for more nuanced benchmarks that capture real-world performance.
UC Berkeley researchers introduced IntegrityBench, a benchmark that tests AI models on research integrity tasks. Under peak pressure, 18 frontier model variants failed roughly 1 in 3 integrity-critical decisions, revealing a significant gap in ethical reliability for AI co-scientists.
A new ArXiv paper provides the first reproducible hardware-level energy benchmark of nine open-source LLMs (1B–7B parameters) running on a consumer RTX 4060Ti 16GB GPU, revealing that larger models draw nearly twice the power of smaller ones and that prompt complexity significantly affects energy consumption.
Researchers introduced a benchmark to evaluate how well large language models (LLMs) perform logical inference over natural-language probability expressions like 'likely' and 'unlikely', aiming to distinguish genuine reasoning from pattern matching.
Researchers introduced MultivationBench, a benchmark that evaluates how well multimodal AI models understand character motivations across sequential visual narratives, addressing a gap in existing static-text or single-image evaluations.
Researchers introduced LayerRAG-Bench, a cross-layer reliability benchmark for agentic retrieval-augmented generation systems. It tests nine models from OpenAI, Anthropic, and Gemini across 8 enterprise domains, 240 tasks, and 9 fault scenarios, finding that schema normalization raises schema-drift success from 0.000 to 0.913.
A new ArXiv paper identifies a fundamental epistemic problem in AI evaluation: benchmark inferences do not compose. Even when each link in a chain of reasoning is warranted, the chain itself may not hold, meaning strong benchmark performance does not guarantee real-world reliability.
OpenAI discovered that enabling two specific settings in GPT-5.6 tripled its performance on the ARC-AGI-3 benchmark. This improvement was achieved by retaining reasoning capabilities and enabling compaction.
Researchers introduced ClinLens, a benchmark of 200 executable tasks across five linked MIMIC datasets, designed to evaluate AI agents on real-world, longitudinal clinical data analysis.
Researchers introduced StatMechBench-v0, a new benchmark to test if LLM-based AI agents can transform complex physics problems into simpler, known models. The study evaluates a propose-verify-revise agent across multiple LLMs on six Ising-type problems.
Researchers created MyoCardBench, a real-world benchmark using 2,263 items from 13 datasets derived from de-identified cardiovascular records, to evaluate how well large language models perform in clinically authentic, longitudinal, and multimodal cardiovascular care scenarios.
Researchers introduced JOR-Bench, a collection of five Japanese-language benchmarks with 1,319 problems to evaluate how well large language models (LLMs) can formulate and solve operations research (OR) problems, covering linear programming, mixed-integer programming, non-linear programming, and combinatorial optimization.
Researchers introduced DrawingVQA, the first benchmark to evaluate multimodal large language models (MLLMs) on real-world construction drawings — a uniquely complex domain fusing abstract geometry, symbols, tables, and technical text. The benchmark uses 33 professional 'Issued for Construction' drawings and 92 expert-crafted questions to test AI's visual-textual reasoning in architecture and civil engineering.
NVIDIA's Nemotron 3 Embed achieved the #1 overall ranking on the Retrieval-Augmented Benchmark (RTEB), a key test for agentic retrieval. This advancement could make virtual assistants and AI tools more accurate and reliable by improving how they find and understand complex information.
Researchers developed a way to test AI models using smaller, representative subsets of benchmark prompts. This could make AI evaluation faster and more practical without sacrificing accuracy.
Researchers introduced CLIR-Bench, a new benchmark to test AI models on answering clinical questions from irregular, sparse medical data. This could lead to better AI tools for doctors and hospitals.