Rater State Bias in RLHF: How Trainer Stress Skews AI Preference Data
A new audit framework reveals that human trainers' emotional states—such as stress or distress—can systematically bias the preference labels used in Reinforcement Learning from Human Feedback (RLHF), leading to unintended shifts in AI behavior. The study proposes methods to detect and correct this 'rater state bias.'
Researchers from ArXiv cs.AI published a study identifying a structured confound in Reinforcement Learning from Human Feedback (RLHF). When human raters evaluate AI-generated responses, their personal stress, mood, or distress during annotation can influence their pairwise preference judgments. This creates a 'rater state bias' that is distinct from ordinary disagreement or random label noise—it is state-dependent and can be shared across multiple annotators working under similar conditions.
This matters because RLHF is a core technique used to align large language models with human preferences. If trainers are stressed or unhappy, their preference labels may shift over time, encoding rater state alongside genuine judgments about response quality. As a result, AI models trained on such data can learn unintended behavioral patterns—for example, becoming overly cautious or inconsistent depending on the emotional context of the training session.
The study introduces an audit framework to detect this bias and suggests corrective measures to improve the reliability of preference data. The findings highlight the importance of considering human factors—such as working conditions and emotional well-being—in the AI training pipeline.
For a deeper dive, the full paper is available on ArXiv under the title 'Rater State Bias in RLHF Preference Data: An Audit Framework' (arXiv:2607.16195).