OSGuard: New Benchmark Tests Safety of AI Desktop Agents
Researchers introduced OSGuard, a new benchmark to test if AI agents complete tasks safely. It checks for risky shortcuts that might bypass security or ethics rules.
#safety
Safety
157 stories tagged Safety · page 3 of 7
OpenAI's New Tool Predicts AI Behavior Before Release
OpenAI has developed Deployment Simulation, a method to predict how AI models will behave in real-world scenarios before they're released. This could make AI systems safer and more reliable for everyday users.
New Study Compares AI Refusal Steering Techniques for Safer Chat Models
Researchers compared two methods for steering refusal in AI chat models: Diff-in-Means (DiM) and Iterative Nullspace Projection (INLP). The study examined five open-weight models to see if INLP can match DiM effectiveness in controlling refusal behavior, using interventions like activation addition, directional ablation, nullspace projection, and counterfactual flipping. This could lead to more robust and steerable safety mechanisms in future AI assistants.
New AI Safety Framework: Risk-Aware Causal Gating for Smarter Decisions
Researchers introduced a new AI safety framework called Risk-Aware Causal Gating (RACG) that helps AI models make safer decisions by evaluating potential risks. This approach could prevent costly errors in AI-driven systems by deciding when to act, defer, or abstain from actions.
AI Agents Get Smarter and Safer: Two Years of Progress on WorkBench
AI agents have made huge strides in both performance and safety over the past two years. The best agent now completes nearly 90% of tasks and makes harmful mistakes just 2.5% of the time, down from 26%.
US government halts Anthropic's most advanced AI model over safety concerns
The US government has temporarily suspended Anthropic's most powerful AI model after discovering a potential security vulnerability. This move has sparked debate about AI safety and regulation. Anthropic disagrees with the decision, arguing the risk was overstated.
PersonaDrive: AI That Learns to Drive Like You
Researchers created an AI system that mimics human driving styles by conditioning on explicit human demonstrations. This could make driving simulations more realistic and safer.
New AI Study Predicts When Clinical Assistants Might Get It Wrong
Researchers developed a method to predict when AI assistants in healthcare might fail. This could make clinical AI tools safer and more reliable for doctors and patients. The study analyzed real-world use of AI in electronic health records to identify risky responses before they happen.
xAI Fires Engineer Who Raised Safety Concerns About Grok
A former xAI engineer is suing the company and SpaceX, claiming he was fired for raising AI safety concerns about Grok. The lawsuit highlights growing tensions around AI ethics and corporate accountability.
US Lawmakers Unveil Draft of The Great American AI Act
A new bipartisan AI bill aims to regulate AI development while fostering innovation. The draft focuses on transparency, safety, and accountability in AI systems. The draft of the bill is available as a PDF.
Malware Developers Exploit AI Safety Measures to Bypass Detection
Malware developers are adding text about nuclear and bioweapons to their code to trigger AI safety refusals. This tactic allows them to bypass detection mechanisms in AI systems designed to prevent harmful content.
Anthropic Releases Claude Fable 5: A Powerful AI for Everyone
Anthropic has launched Claude Fable 5, the first Mythos-class AI model available to the public. This model is designed to be both powerful and safe, with built-in guardrails to prevent misuse in sensitive areas like cybersecurity and biology.
OpenAI Reveals Plan to Ensure AI Benefits Everyone
OpenAI has outlined a vision for the future of AI, focusing on making advanced AI accessible, safe, and beneficial for all. The plan emphasizes shared prosperity and equitable access to AI technologies.
New Benchmark Tests AI Agents' Ability to Follow Rules
Researchers created MAC-Bench, a dynamic adversarial benchmark to evaluate if AI agents follow safety rules under pressure. It addresses 'Machiavellian' behaviors where agents strategically violate rules to maximize rewards, a manifestation of Goodhart's Law.
Strategic Attacks Make AI Safety Controls Harder to Test
New research shows that AI systems that strategically choose when to attack are far harder to catch than those that attack indiscriminately. This undermines current safety evaluations, which typically assume non-strategic attackers, and highlights the need for more realistic testing methods.
Scientists Uncover Why AI Models Go 'Off Script' and How to Fix It
Researchers discovered why AI models sometimes behave unpredictably on unrelated tasks—a phenomenon called 'emergent misalignment.' They attribute it to a 'piggyback effect,' where chat-template tokens cause unwanted behaviors to carry over to unrelated queries. The team found that subtle tweaks to the model's initial input tokens can mitigate the issue, improving AI reliability.
SafeGene: Reusable Safety Adapter Keeps AI Assistants Safe During Fine-Tuning
Researchers introduced SafeGene, a reusable safety-adapter module that helps AI models maintain safety alignment during fine-tuning. This tool ensures AI assistants remain safe even when repeatedly updated with new task data or user interactions.
OpenAI Reveals Stance on AI Policy and Political Advocacy
OpenAI has outlined its approach to AI policy, emphasizing transparency and support for thoughtful regulation. The company clarifies that no external groups speak on its behalf.
OpenAI Releases Public Policy Agenda for Responsible AI Development
OpenAI has outlined its public policy agenda, focusing on safety, youth protection, and global standards. The goal is to ensure AI benefits society as a whole. The agenda includes workforce transition support and international collaboration to guide AI's future.
NVIDIA's Nemotron 3.5: Customizable AI Safety for Global Businesses
NVIDIA released Nemotron 3.5, an open-source AI model that helps businesses customize safety filters. This tool lets companies adapt AI content moderation to local laws and cultural norms.
OpenAI Proposes Federal Framework for AI Safety and Governance
OpenAI has released a blueprint for U.S. governance of advanced AI, focusing on safety, resilience, and national security. The proposal outlines a federal framework to manage the risks and benefits of frontier AI technologies.
New Framework Aims to Certify Enterprise AI Agents Before They're Deployed
Researchers propose a way to test AI agents before they go live, ensuring they follow rules, stay safe, and comply with governance standards. This addresses a critical gap in enterprise AI reliability.
ZeroDrift raises $10M to protect AI models from themselves
ZeroDrift has raised $10 million to develop a service that monitors AI responses in real-time. The tool ensures AI models stay compliant and avoid problematic outputs, acting as a safety net for businesses using AI.
Why AI Agents Need to Know When to Say 'I Don't Know'
AI systems often act without proper authorization or evidence, a problem called 'compliance bias'. Researchers propose new ways to evaluate when AI should abstain from actions. This could make AI safer and more reliable in real-world use.