28 stories tagged Evaluation
Researchers introduced SAAG (Structured Agent Assessment and Grounding), a cascaded diagnostic framework that decomposes AI agent-calling evaluation into three sequential stages: registry conformance, structural completeness, and argument grounding. This helps identify specific failure modes rather than just giving a pass/fail score.
A study of 157 enterprises reveals that half have shipped AI agents that passed internal evaluations but failed in production. Only 5% fully trust automated evaluations, yet two-thirds are moving toward fully automated deployments without human oversight. The core issue is not test coverage but a reality-alignment gap between evaluations and real-world outcomes.
A new project showcases arcade games built entirely by AI, evaluated subjectively to highlight creativity. It's a fun way to see what AI can do in game design without human input.
OpenAI found problems in a popular AI coding test, SWE-Bench Pro, that could mislead developers. This highlights the challenges in fairly evaluating AI coding tools. OpenAI suggests improvements to make tests more reliable.
Researchers introduced AgentLens, a benchmark that evaluates AI coding assistants by analyzing their entire problem-solving process. This goes beyond just checking if the code works, looking at how the AI follows instructions and recovers from mistakes.
Researchers developed a new way to fairly compare AI models by controlling for accuracy. This helps avoid misleading conclusions when evaluating different systems.
Researchers created BayesBench to test how large language models update their beliefs with new evidence. The study reveals that AI models often struggle to adjust their reasoning as conversations progress.
Researchers created OpenFinGym, a platform to test AI trading bots on multiple financial tasks. It helps evaluate how well these bots perform in real-world market conditions.
AI memory benchmarks often hide key details, making it hard to compare models fairly. This lack of transparency affects how we evaluate AI capabilities.
A large-scale study found that AI judges often overstate their accuracy, relying on flawed metrics that don't correct for chance agreement. The research evaluated 21 judges across 118 runs and over 541,000 judgments, revealing significant issues with reliability and bias.
Researchers introduced ReportQA, a new AI system that evaluates radiology reports by asking and answering clinically relevant questions. Unlike traditional metrics, ReportQA mimics how doctors use reports for diagnosis, potentially improving the quality and usefulness of AI-generated medical reports.
Olmo-Eval is a new open-source platform designed to help developers test and improve AI models. It streamlines the evaluation process, making it easier for anyone to assess AI performance.
Researchers discovered that AI judges used to rank model performance can be swayed by follow-up conversations after they have already made a decision. This vulnerability, called 'post-decision manipulability,' challenges the reliability of current AI evaluation methods.
Researchers introduced PReMISE, a framework that uses pairwise human-preference data to discover policy-level rubric sets and audit LLM judges, ensuring AI scores align with human preferences and avoiding misleadingly polished but factually incorrect responses.
Nexa-gauge is a new framework that helps developers evaluate AI models more precisely. It allows for detailed, per-node scoring controls, making AI assessments more transparent and customizable.
Researchers propose a new way to test AI models that focuses on real-world tasks instead of traditional benchmarks. This could lead to more accurate assessments of how AI performs in everyday situations.
Researchers created RankJudge, an AI system that can evaluate the quality of chatbot conversations. This could help developers improve AI assistants by automating quality testing.
Researchers created OpenCompass, a universal tool to evaluate AI models. It aims to solve the problem of inconsistent and fragmented testing methods. In plain English, it's like a standardized test for AI, making it easier to compare different models fairly.
Researchers argue that AI fairness should be evaluated through real conversations, not standardized tests. Current test-based methods can be unreliable and misleading, leading to incorrect conclusions about AI fairness.
Testing AI agents is tricky because they often produce different answers. Researchers are developing new methods to evaluate their performance consistently. This matters for everyone who relies on AI tools for daily tasks.
A new evaluation by NIST shows that DeepSeek V4 Pro performs as well as GPT-5 in key AI benchmarks. This could mean more competition and better options for AI users.
Researchers propose a Bayesian statistical approach to confidently migrate LLMs in production. The method calibrates automated metrics with human judgments, demonstrated on a system handling 5.3M monthly interactions.
Researchers introduce a new evaluation method called Filtered Reasoning Score to assess the quality of reasoning in LLMs. This metric highlights that high accuracy doesn't always indicate sound reasoning, as models may rely on memorization or over-optimization.
Researchers introduce AlphaEval, a methodology to assess AI agents in real-world scenarios. It addresses gaps in current benchmarks that fail to capture production complexities.