128 stories tagged Benchmark · page 5 of 6
A new study introduces AgentFloor, a benchmark to test how well smaller AI models can handle routine tasks. The goal is to see which parts of AI workflows need big, advanced models and which can be done by smaller ones.
Researchers created ARMOR 2025 to test AI models for military use, ensuring they follow legal and ethical rules. This benchmark goes beyond civilian safety standards to address defense-specific needs.
Gemma 4 31B demonstrated superior efficiency and speed in a game development contest, completing the task in a fraction of the time compared to Qwen 3.6 27B. The results highlight significant performance differences between the two models.
A 200-person Chinese team released DeepSeek V4, surpassing models from larger labs. The model achieves state-of-the-art performance on key benchmarks, challenging the dominance of well-funded Western AI labs.
Researchers introduce BTF-2, a benchmark with 1,417 pastcasting questions, to analyze AI forecasters' reasoning. The study identifies accuracy differences as small as 0.004 Brier score and builds a more accurate composite forecaster.
Researchers introduce AgentSearchBench, a benchmark to evaluate AI agent search capabilities in realistic, unconstrained environments. The benchmark addresses gaps in existing research by focusing on compositional and execution-dependent agent capabilities.
Researchers introduce SHAPE, a new benchmark to evaluate educational LLMs under adversarial conditions. The study highlights 'pedagogical jailbreaks' where students manipulate LLMs to provide answers instead of learning guidance.
Researchers introduce a new benchmark, Math Takes Two, to evaluate whether language models truly understand math or just memorize patterns. The test focuses on emergent mathematical reasoning through communication, challenging models to construct abstract concepts from first principles.
Researchers introduced Deep FinResearch Bench to assess AI's financial research capabilities. The benchmark evaluates qualitative rigor, forecasting accuracy, and claim verifiability in investment reports.
ThermoQA is a new benchmark for evaluating LLMs on engineering thermodynamics problems. It shows significant performance drops as problem complexity increases, with top models like Claude Opus 4.6 leading at 94.1%.
DeepSeek has previewed two new AI models, DeepSeek V4 and DeepSeek V4+, that outperform its previous V3.2 model and significantly close the gap with leading models. The improvements come from architectural enhancements that boost efficiency and performance.
Researchers propose a new method for evaluating LLMs that considers individual user preferences, moving beyond aggregate benchmarks. This approach uses ELO ratings to rank models based on personal context and needs.
Researchers introduced DW-Bench, a benchmark for evaluating LLMs on data warehouse graph topology reasoning. Experiments show tool-augmented methods outperform static approaches but struggle with complex tasks.
Researchers introduce a framework to translate natural language into executable formal logic, addressing LLMs' limitations in symbolic reasoning. The NARS-Reasoning-v0.1 benchmark enables evaluation of this neuro-symbolic approach.
Researchers introduce AutomationBench, a new benchmark for evaluating AI agents on complex, cross-application workflows. It tests coordination, API discovery, and policy adherence in real-world business scenarios.
Researchers have introduced the first benchmark for extracting claims from multimodal social media posts, addressing a gap in automated fact-checking. The dataset includes text combined with images like memes and screenshots, challenging traditional methods.
Researchers introduce KWBench, a new benchmark for assessing whether large language models can identify professional scenarios without explicit prompting. This focuses on a critical yet often overlooked step in knowledge work: recognizing the structure of a situation before attempting to solve it.
Kimi K2.6 sets new standards in open-source coding with top-tier performance across multiple benchmarks. The model excels in long-horizon coding tasks, handling up to 4,000 tokens.
Researchers introduce RiskWebWorld, a realistic benchmark for evaluating GUI agents in high-stakes e-commerce risk management. It features 1,513 tasks from production risk-control pipelines, addressing a gap in current benchmarks.
Researchers introduce GeoAgentBench, a dynamic evaluation framework for LLM-based geographic information systems (GIS). It addresses gaps in static testing by assessing real-time, multimodal spatial analysis capabilities.
Researchers introduce a new evaluation method called Filtered Reasoning Score to assess the quality of reasoning in LLMs. This metric highlights that high accuracy doesn't always indicate sound reasoning, as models may rely on memorization or over-optimization.
Researchers introduce HORIZON, a diagnostic benchmark to analyze failures in LLM-based agents on long-horizon tasks. The study highlights the need for better characterization of these failures to improve agentic systems.
Researchers introduce a dynamic data curation pipeline to better evaluate how LVLMs handle conflicts between visual and textual evidence. The study underscores the importance of models admitting when they lack knowledge.
Researchers introduce SCBench, a rigorous new benchmark to evaluate AI's spatial reasoning. Existing tests fall short in assessing real-world navigation and planning. The benchmark spans three hierarchical capability buckets, testing executable outputs with deterministic checkers.