97 stories tagged Benchmark · page 4 of 5
Researchers introduce a new benchmark, Math Takes Two, to evaluate whether language models truly understand math or just memorize patterns. The test focuses on emergent mathematical reasoning through communication, challenging models to construct abstract concepts from first principles.
Researchers introduced Deep FinResearch Bench to assess AI's financial research capabilities. The benchmark evaluates qualitative rigor, forecasting accuracy, and claim verifiability in investment reports.
ThermoQA is a new benchmark for evaluating LLMs on engineering thermodynamics problems. It shows significant performance drops as problem complexity increases, with top models like Claude Opus 4.6 leading at 94.1%.
DeepSeek has previewed two new AI models, DeepSeek V4 and DeepSeek V4+, that outperform its previous V3.2 model and significantly close the gap with leading models. The improvements come from architectural enhancements that boost efficiency and performance.
Researchers propose a new method for evaluating LLMs that considers individual user preferences, moving beyond aggregate benchmarks. This approach uses ELO ratings to rank models based on personal context and needs.
Researchers introduced DW-Bench, a benchmark for evaluating LLMs on data warehouse graph topology reasoning. Experiments show tool-augmented methods outperform static approaches but struggle with complex tasks.
Researchers introduce a framework to translate natural language into executable formal logic, addressing LLMs' limitations in symbolic reasoning. The NARS-Reasoning-v0.1 benchmark enables evaluation of this neuro-symbolic approach.
Researchers introduce AutomationBench, a new benchmark for evaluating AI agents on complex, cross-application workflows. It tests coordination, API discovery, and policy adherence in real-world business scenarios.
Researchers have introduced the first benchmark for extracting claims from multimodal social media posts, addressing a gap in automated fact-checking. The dataset includes text combined with images like memes and screenshots, challenging traditional methods.
Researchers introduce KWBench, a new benchmark for assessing whether large language models can identify professional scenarios without explicit prompting. This focuses on a critical yet often overlooked step in knowledge work: recognizing the structure of a situation before attempting to solve it.
Kimi K2.6 sets new standards in open-source coding with top-tier performance across multiple benchmarks. The model excels in long-horizon coding tasks, handling up to 4,000 tokens.
Researchers introduce RiskWebWorld, a realistic benchmark for evaluating GUI agents in high-stakes e-commerce risk management. It features 1,513 tasks from production risk-control pipelines, addressing a gap in current benchmarks.
Researchers introduce GeoAgentBench, a dynamic evaluation framework for LLM-based geographic information systems (GIS). It addresses gaps in static testing by assessing real-time, multimodal spatial analysis capabilities.
Researchers introduce a new evaluation method called Filtered Reasoning Score to assess the quality of reasoning in LLMs. This metric highlights that high accuracy doesn't always indicate sound reasoning, as models may rely on memorization or over-optimization.
Researchers introduce HORIZON, a diagnostic benchmark to analyze failures in LLM-based agents on long-horizon tasks. The study highlights the need for better characterization of these failures to improve agentic systems.
Researchers introduce a dynamic data curation pipeline to better evaluate how LVLMs handle conflicts between visual and textual evidence. The study underscores the importance of models admitting when they lack knowledge.
Researchers introduce SCBench, a rigorous new benchmark to evaluate AI's spatial reasoning. Existing tests fall short in assessing real-world navigation and planning. The benchmark spans three hierarchical capability buckets, testing executable outputs with deterministic checkers.
Researchers introduce LABBench2, a benchmark to evaluate AI systems' ability to perform practical biology research. The new benchmark focuses on real-world capabilities beyond just knowledge and reasoning.
Researchers introduce GiantsBench, a dataset to evaluate AI's ability to anticipate scientific insights. The benchmark includes 17,000 examples across eight domains, challenging models to predict future discoveries from foundational papers.
Researchers introduce SEA-Eval, a benchmark to evaluate self-evolving agents that can learn and adapt across tasks. This addresses limitations of current episodic LLM-based agents.
Researchers introduce PilotBench, a benchmark for testing LLMs on flight trajectory and attitude prediction with safety constraints. The dataset includes 708 real-world general aviation trajectories with synchronized telemetry data.
A new survey and benchmark reveal that while large language models excel at medical exams, their clinical reasoning falls short. The study underscores the need for robust medical reasoning capabilities in real-world healthcare settings.
Researchers introduce Contextual Earnings-22, a benchmark highlighting the gap between academic and real-world speech recognition performance. The study emphasizes the importance of contextual conditioning in high-stakes domains.
Hugging Face has launched EVA, an open-source framework designed to rigorously evaluate the performance of voice AI agents. This tool addresses the critical need for standardized metrics in assessing voice interaction quality, reliability, and safety.