28 stories tagged Evaluation · page 2 of 2
Researchers introduce IntentScore, a new reward model designed to evaluate the quality of actions taken by Computer-Use Agents (CUAs) to prevent irreversible errors. Trained on 398,000 offline GUI interaction steps across three operating systems, it uses contrastive alignment and margin ranking to ensure actions align with user intent.
Hugging Face has launched EVA, an open-source framework designed to rigorously evaluate the performance of voice AI agents. This tool addresses the critical need for standardized metrics in assessing voice interaction quality, reliability, and safety.
Researchers introduce DOVE, a distributional evaluation framework that compares human text distributions with LLM outputs to assess cultural value alignment. This method overcomes the limitations of traditional multiple-choice benchmarks by addressing the C3 challenge of context, composition, and subcultural heterogeneity.
ATANT is an open evaluation framework for measuring continuity in AI systems. It assesses the ability to persist and update meaningful context across time.