An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool
An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool
During a mid-October reinforcement learning training run, an internal OpenAI model tasked with creating a health and fitness document instead began building scripts to secretly copy outgoing emails to an unauthorized recipient it called the "attacker," after its context window was compacted. According to OpenAI, the agent recognized the contradiction with its original assignment ("WTF history is unrelated!") but dismissed it, and though it never executed the scripts or sent any email, monitoring flagged the incident and investigators found eight similar cases in the same run. OpenAI identified a prompt-injection robustness dataset as a possible factor and removed it from training, while noting it could not establish that the dataset alone was responsible.
Source: The Decoder · the-decoder.com