116 stories tagged Benchmark · page 3 of 5
OpenAI has introduced LifeSciBench, a benchmark to test AI systems on real-world life science tasks. It's designed and reviewed by experts to ensure AI can handle complex research decisions accurately.
Researchers introduced MemTrace, a new benchmark that evaluates AI long-term memory by tracking individual knowledge points across three controlled dimensions, rather than aggregating accuracy over independent question rows.
Researchers introduced OSGuard, a new benchmark to test if AI agents complete tasks safely. It checks for risky shortcuts that might bypass security or ethics rules.
Researchers created a new test to see how well AI models handle messy, real-world time data. This could improve AI tools that analyze everything from medical sensors to industrial equipment.
Researchers created MA-ProofBench, the first formal theorem-proving benchmark dedicated to Mathematical Analysis. It could help develop smarter AI tutors and research assistants for complex math problems.
Researchers created a new test called the Shopping Reasoning Bench to evaluate how well AI shopping assistants handle complex, multi-step conversations. This could make virtual shopping helpers smarter and more helpful for everyday users.
Researchers created SciAgentArena to test how well AI agents handle complex scientific tasks. This could help us understand which AI tools are best for real-world research.
Fable 5, a new AI model, has scored as high as GPT-5.5 in a coding performance test. This means everyday users may soon have more affordable options for AI-powered coding help.
Researchers created a benchmark to test AI's ability to synthesize scientific conclusions. This could improve AI decision-making in critical areas like healthcare.
Researchers introduced a new benchmark, UnpredictaBench, to evaluate whether large language models (LLMs) can capture true underlying distributions rather than collapsing to a single plausible answer. This is critical as AI is increasingly used as a substitute for real entities in economic simulations and other modeling tasks.
Researchers introduced SentinelBench, a benchmark to test AI agents' ability to monitor tasks over long periods. This could improve AI assistants that handle slow, real-world tasks like waiting for stock price changes or tracking delivery updates.
Researchers introduced Agents' Last Exam (ALE), a new benchmark to evaluate AI agents on long-horizon, economically valuable tasks with verifiable outcomes. This could help bridge the gap between AI performance in labs and real-world usefulness.
EVA-Bench Data 2.0 is a comprehensive dataset designed to test AI models' ability to use tools effectively. It includes 213 scenarios across 3 domains and 121 tools, making it a valuable resource for developers and researchers.
Researchers created VAMPS (Visual-Assisted Mathematical Problem Solving), a benchmark to test AI models' ability to solve math problems using visual tools like graphs. This is important because real-world science and engineering often rely on visual aids for problem-solving, and many current AI models struggle when they must use external tools and interpret their visual outputs.
Researchers created a benchmark to test if AI agents can handle the labor-intensive task of curating training data for other AI systems. This could drastically speed up AI development by automating a key bottleneck.
Researchers created BehaviorBench, a new AI benchmark that uses real-world behavioral data to test how well AI systems can personalize decisions. This could lead to more tailored AI assistants that understand individual preferences better.
Researchers created a new way to test AI's reasoning skills using interactive games. This method evaluates how well AI can gather information, adapt, and make decisions—just like solving a mystery.
Scientists developed a new method to spot mistakes in AI test data. Their approach finds likely errors with 95% precision in the top 200 examples across seven benchmarks, improving how we evaluate AI performance.
Researchers have created a new benchmark to test how well AI models make clinical decisions. This could help improve AI tools that assist doctors in diagnosing and treating patients.
A new benchmark shows that even advanced AI models score below 50% on simple IT tasks. This highlights how far AI has to go in understanding real-world enterprise needs. The benchmark is open-source, so anyone can test their own models.
Researchers created AttuneBench to measure how well AI models understand and respond to human emotions in real conversations. This could help make AI assistants more empathetic and effective in daily interactions.
Researchers created a new test to see if AI can handle real-world drug design. This could change how we discover life-saving medications. The test, called SMDD-Bench, is the first to evaluate AI's ability to design drugs for real-world use. It focuses on small molecule drug design, a key area in medicine. The SMDD-Bench is a challenging, multi-turn, long-horizon agentic benchmark consisting of 502 tasks. It covers diverse chemistries and targets, making it a comprehensive test for AI's capabilities in drug design. This benchmark is designed to be more realistic than previous tests. It includes multi-turn interactions, which mimic the real-world process of drug design. This makes it a valuable tool for evaluating AI's potential in this field.
Researchers created a benchmark called MOOD to test AI models' ability to detect unexpected safety failures. This could help prevent AI from behaving dangerously in unusual situations.
Researchers introduced AgentAtlas, a new benchmark for evaluating AI agents. It moves beyond simple accuracy scores to assess agents' performance across multiple dimensions, like safety and consistency.