97 stories tagged Benchmark · page 2 of 5
Researchers argue that after a benchmark's accuracy saturates, the focus should shift to six other key dimensions of AI performance: construct validity (shortcuts), out-of-distribution generalizability, efficiency, reliability, model vs. scaffold importance, and human–AI collaboration uplift.
Researchers created a contamination-aware, multi-zone benchmark called Know2Guess to evaluate when large language models should answer questions versus abstain. It has 1,200 items across five domains with explicit abstention expectations and contamination-risk metadata.
Researchers developed a method using large language models to automatically generate challenging test problems for AI reasoning. This could lead to better evaluations of AI's ability to apply knowledge to new, harder tasks.
AI memory benchmarks often hide key details, making it hard to compare models fairly. This lack of transparency affects how we evaluate AI capabilities.
Researchers created LaViSA, a benchmark designed to test whether AI models can resolve structurally ambiguous sentences by leveraging visual scenes. This helps machines better interpret complex, real-world situations where word order alone isn't enough.
OpenAI has introduced LifeSciBench, a benchmark to test AI systems on real-world life science tasks. It's designed and reviewed by experts to ensure AI can handle complex research decisions accurately.
Researchers introduced MemTrace, a new benchmark that evaluates AI long-term memory by tracking individual knowledge points across three controlled dimensions, rather than aggregating accuracy over independent question rows.
Researchers introduced OSGuard, a new benchmark to test if AI agents complete tasks safely. It checks for risky shortcuts that might bypass security or ethics rules.
Researchers created a new test to see how well AI models handle messy, real-world time data. This could improve AI tools that analyze everything from medical sensors to industrial equipment.
Researchers created MA-ProofBench, the first formal theorem-proving benchmark dedicated to Mathematical Analysis. It could help develop smarter AI tutors and research assistants for complex math problems.
Researchers created a new test called the Shopping Reasoning Bench to evaluate how well AI shopping assistants handle complex, multi-step conversations. This could make virtual shopping helpers smarter and more helpful for everyday users.
Researchers created SciAgentArena to test how well AI agents handle complex scientific tasks. This could help us understand which AI tools are best for real-world research.
Fable 5, a new AI model, has scored as high as GPT-5.5 in a coding performance test. This means everyday users may soon have more affordable options for AI-powered coding help.
Researchers created a benchmark to test AI's ability to synthesize scientific conclusions. This could improve AI decision-making in critical areas like healthcare.
Researchers introduced a new benchmark, UnpredictaBench, to evaluate whether large language models (LLMs) can capture true underlying distributions rather than collapsing to a single plausible answer. This is critical as AI is increasingly used as a substitute for real entities in economic simulations and other modeling tasks.
Researchers introduced SentinelBench, a benchmark to test AI agents' ability to monitor tasks over long periods. This could improve AI assistants that handle slow, real-world tasks like waiting for stock price changes or tracking delivery updates.
Researchers introduced Agents' Last Exam (ALE), a new benchmark to evaluate AI agents on long-horizon, economically valuable tasks with verifiable outcomes. This could help bridge the gap between AI performance in labs and real-world usefulness.
EVA-Bench Data 2.0 is a comprehensive dataset designed to test AI models' ability to use tools effectively. It includes 213 scenarios across 3 domains and 121 tools, making it a valuable resource for developers and researchers.
Researchers created VAMPS (Visual-Assisted Mathematical Problem Solving), a benchmark to test AI models' ability to solve math problems using visual tools like graphs. This is important because real-world science and engineering often rely on visual aids for problem-solving, and many current AI models struggle when they must use external tools and interpret their visual outputs.
Researchers created a benchmark to test if AI agents can handle the labor-intensive task of curating training data for other AI systems. This could drastically speed up AI development by automating a key bottleneck.
Researchers created BehaviorBench, a new AI benchmark that uses real-world behavioral data to test how well AI systems can personalize decisions. This could lead to more tailored AI assistants that understand individual preferences better.
Researchers created a new way to test AI's reasoning skills using interactive games. This method evaluates how well AI can gather information, adapt, and make decisions—just like solving a mystery.
Scientists developed a new method to spot mistakes in AI test data. Their approach finds likely errors with 95% precision in the top 200 examples across seven benchmarks, improving how we evaluate AI performance.
Researchers have created a new benchmark to test how well AI models make clinical decisions. This could help improve AI tools that assist doctors in diagnosing and treating patients.