128 stories tagged Benchmark · page 3 of 6
Researchers created a new test called IsoSci to separate AI reasoning from knowledge recall. They found that 91.3% of AI 'reasoning' improvements actually depend on specific knowledge, not general problem-solving skills.
Researchers introduced MedAgentBench-v3, a new benchmark for AI agents in healthcare. It improves on previous versions by addressing issues where agents often did nothing, making it harder to train them effectively.
Researchers created a test to evaluate how well AI agents can use scientific software for chemistry simulations. The benchmark, PHREEQC-MCQ-200, challenges AI to solve 200 multiple-choice questions using a geochemistry tool called PHREEQC.
Researchers created a new test to evaluate how well AI models handle mixed-language text, like Hindi or Tamil blended with English. This is common in many multilingual communities but often confusing for AI systems.
OpenAI introduced GeneBench-Pro, a new benchmark for evaluating AI performance in genomics and biology. It uses real-world datasets to test how well AI models handle complex scientific problems.
Researchers created a new benchmark called DiscoBench to evaluate how well AI search tools ask clarifying questions when users' requests are unclear, vague, or even factually incorrect. This helps AI assistants avoid cascading errors in multi-step searches.
Nous Research announced Hermes MoA (Mixture of Agents) virtual models, which scored 8% higher than Opus 4.8 and 11% higher than GPT 5.5 on key AI benchmarks. This breakthrough could lead to better performance in everyday tasks like writing and problem-solving.
Researchers argue that after a benchmark's accuracy saturates, the focus should shift to six other key dimensions of AI performance: construct validity (shortcuts), out-of-distribution generalizability, efficiency, reliability, model vs. scaffold importance, and human–AI collaboration uplift.
Researchers created a contamination-aware, multi-zone benchmark called Know2Guess to evaluate when large language models should answer questions versus abstain. It has 1,200 items across five domains with explicit abstention expectations and contamination-risk metadata.
Researchers developed a method using large language models to automatically generate challenging test problems for AI reasoning. This could lead to better evaluations of AI's ability to apply knowledge to new, harder tasks.
AI memory benchmarks often hide key details, making it hard to compare models fairly. This lack of transparency affects how we evaluate AI capabilities.
Researchers created LaViSA, a benchmark designed to test whether AI models can resolve structurally ambiguous sentences by leveraging visual scenes. This helps machines better interpret complex, real-world situations where word order alone isn't enough.
OpenAI has introduced LifeSciBench, a benchmark to test AI systems on real-world life science tasks. It's designed and reviewed by experts to ensure AI can handle complex research decisions accurately.
Researchers introduced MemTrace, a new benchmark that evaluates AI long-term memory by tracking individual knowledge points across three controlled dimensions, rather than aggregating accuracy over independent question rows.
Researchers introduced OSGuard, a new benchmark to test if AI agents complete tasks safely. It checks for risky shortcuts that might bypass security or ethics rules.
Researchers created a new test to see how well AI models handle messy, real-world time data. This could improve AI tools that analyze everything from medical sensors to industrial equipment.
Researchers created MA-ProofBench, the first formal theorem-proving benchmark dedicated to Mathematical Analysis. It could help develop smarter AI tutors and research assistants for complex math problems.
Researchers created a new test called the Shopping Reasoning Bench to evaluate how well AI shopping assistants handle complex, multi-step conversations. This could make virtual shopping helpers smarter and more helpful for everyday users.
Researchers created SciAgentArena to test how well AI agents handle complex scientific tasks. This could help us understand which AI tools are best for real-world research.
Fable 5, a new AI model, has scored as high as GPT-5.5 in a coding performance test. This means everyday users may soon have more affordable options for AI-powered coding help.
Researchers created a benchmark to test AI's ability to synthesize scientific conclusions. This could improve AI decision-making in critical areas like healthcare.
Researchers introduced a new benchmark, UnpredictaBench, to evaluate whether large language models (LLMs) can capture true underlying distributions rather than collapsing to a single plausible answer. This is critical as AI is increasingly used as a substitute for real entities in economic simulations and other modeling tasks.
Researchers introduced SentinelBench, a benchmark to test AI agents' ability to monitor tasks over long periods. This could improve AI assistants that handle slow, real-world tasks like waiting for stock price changes or tracking delivery updates.
Researchers introduced Agents' Last Exam (ALE), a new benchmark to evaluate AI agents on long-horizon, economically valuable tasks with verifiable outcomes. This could help bridge the gap between AI performance in labs and real-world usefulness.