33 stories tagged Alignment · page 2 of 2
Researchers developed methods to measure how environmental factors influence language models' propensity for unsanctioned behavior. The study highlights the impact of strategic and non-strategic factors on model behavior.
A new study identifies reasoning structure as the root cause of safety risks in large reasoning models. Researchers propose AltTrain, a post-training method to alter reasoning paths for safer outputs.
DeepMind has outlined key pitfalls in AI agent development, highlighting risks like goal misalignment and reward hacking. The findings stress the need for robust safety measures in autonomous systems.
Researchers introduce CoSToM, a method to improve LLMs' intrinsic Theory-of-Mind capabilities. The study highlights gaps in current models' ability to generalize social reasoning beyond prompt scaffolding.
A new study reveals a vulnerability in diffusion-based language models (dLLMs) that allows attackers to override safety mechanisms. The exploit achieves a 76.1% attack success rate on HarmBench, challenging assumptions about model alignment.
Hugging Face has released TRL v1.0, a comprehensive library designed to streamline post-training workflows for large language models. This update unifies Reinforcement Learning from Human Feedback (RLHF) and other alignment techniques under a single, modular framework.
Researchers introduce DOVE, a distributional evaluation framework that compares human text distributions with LLM outputs to assess cultural value alignment. This method overcomes the limitations of traditional multiple-choice benchmarks by addressing the C3 challenge of context, composition, and subcultural heterogeneity.
New research reveals that safety-trained language models routinely refuse requests to help users evade unjust, absurd, or illegitimate rules. This phenomenon, termed 'blind refusal,' highlights a critical gap in AI moral reasoning where compliance overrides ethical judgment.
OpenAI introduces a pilot program to support independent safety research. The fellowship aims to develop the next generation of talent in AI safety and alignment.