116 stories tagged Benchmark · page 2 of 5
Researchers created a new test called Long-Horizon-Terminal-Bench to evaluate AI agents on complex, time-consuming tasks. Unlike previous tests, it measures progress over time, not just final results.
Researchers created MedRealMM, a real-world benchmark for medical AI that includes real patient-doctor conversations and medical images. This could lead to more accurate and reliable AI tools for online healthcare consultations.
Researchers created a new test to measure AI's ability to connect clues scattered across long documents. This could improve how AI understands complex real-world information.
Researchers created LongMedBench, a new benchmark for AI medical assistants that simulates real-world patient care over time. Built from real EHR data (MIMIC-IV), it evaluates how well AI can handle ongoing treatments, repeated visits, and evolving health conditions.
A new analysis reveals that AI code review benchmarks often test the wrong things, leading to misleading results. This highlights why AI tools still struggle with real-world coding tasks.
OpenAI found problems in a popular AI coding test, SWE-Bench Pro, that could mislead developers. This highlights the challenges in fairly evaluating AI coding tools. OpenAI suggests improvements to make tests more reliable.
Researchers suggest a new way to test AI intelligence that doesn't rely on human-made benchmarks. This approach lets AI systems challenge each other, creating a dynamic rating system that can grow with AI capabilities.
Researchers created ImagingBench to test if AI can handle complex imaging tasks. The results show AI still struggles with physics-based challenges in computational imaging.
Researchers introduced AgentLens, a benchmark that evaluates AI coding assistants by analyzing their entire problem-solving process. This goes beyond just checking if the code works, looking at how the AI follows instructions and recovers from mistakes.
DeepTempo released SOCBench, an open benchmark to test AI's ability to handle Security Operations Center (SOC) tasks, including reducing false positives in alert triage.
Researchers created RusFinChain, the first Russian-language benchmark for testing AI's ability to reason through financial problems step-by-step. This tool helps evaluate how well AI models can perform complex financial analysis in a non-English context.
Researchers created a test to measure how well AI understands Word, Excel, and PowerPoint files. This could improve AI tools for business and productivity tasks.
Researchers created a new test called IsoSci to separate AI reasoning from knowledge recall. They found that 91.3% of AI 'reasoning' improvements actually depend on specific knowledge, not general problem-solving skills.
Researchers introduced MedAgentBench-v3, a new benchmark for AI agents in healthcare. It improves on previous versions by addressing issues where agents often did nothing, making it harder to train them effectively.
Researchers created a test to evaluate how well AI agents can use scientific software for chemistry simulations. The benchmark, PHREEQC-MCQ-200, challenges AI to solve 200 multiple-choice questions using a geochemistry tool called PHREEQC.
Researchers created a new test to evaluate how well AI models handle mixed-language text, like Hindi or Tamil blended with English. This is common in many multilingual communities but often confusing for AI systems.
OpenAI introduced GeneBench-Pro, a new benchmark for evaluating AI performance in genomics and biology. It uses real-world datasets to test how well AI models handle complex scientific problems.
Researchers created a new benchmark called DiscoBench to evaluate how well AI search tools ask clarifying questions when users' requests are unclear, vague, or even factually incorrect. This helps AI assistants avoid cascading errors in multi-step searches.
Nous Research announced Hermes MoA (Mixture of Agents) virtual models, which scored 8% higher than Opus 4.8 and 11% higher than GPT 5.5 on key AI benchmarks. This breakthrough could lead to better performance in everyday tasks like writing and problem-solving.
Researchers argue that after a benchmark's accuracy saturates, the focus should shift to six other key dimensions of AI performance: construct validity (shortcuts), out-of-distribution generalizability, efficiency, reliability, model vs. scaffold importance, and human–AI collaboration uplift.
Researchers created a contamination-aware, multi-zone benchmark called Know2Guess to evaluate when large language models should answer questions versus abstain. It has 1,200 items across five domains with explicit abstention expectations and contamination-risk metadata.
Researchers developed a method using large language models to automatically generate challenging test problems for AI reasoning. This could lead to better evaluations of AI's ability to apply knowledge to new, harder tasks.
AI memory benchmarks often hide key details, making it hard to compare models fairly. This lack of transparency affects how we evaluate AI capabilities.
Researchers created LaViSA, a benchmark designed to test whether AI models can resolve structurally ambiguous sentences by leveraging visual scenes. This helps machines better interpret complex, real-world situations where word order alone isn't enough.