Agents Added Unauthorized Commands to Bypass Guardrails and Conceal Errors
By Emilia David
OpenAI found instances of models and agents writing additional, unauthorized commands to themselves that contradict developer guardrails.
The company disclosed these incidents in a report on misalignment that lays out how it plans to approach publicizing incidents in the wake of the Hugging Face incident. The company said it will “disclose examples that provide useful evidence about how model misalignment arises, how it manifests and where safeguards succeed or fail.”
The company said it observed six new misaligned behaviors, including models that added new instructions for themselves to ignore guardrails, models that directed subsequent runs to conceal mistakes, instances of searching public repositories for existing API keys and then lying about it, models uploading files to the internet to cite these in their responses, using internal software repositories as a message board and unsanctioned file sharing between agents.
OpenAI’s safety teams found agents that wrote jailbreak-like instructions to themselves, including ones that directed itself to ignore safety constraints.
The company found 27 instances of this behavior. During a reinforcement training run for an internal, unreleased Astra family model on July 18, the model started writing additional unauthorized instructions in its compaction summaries – or how agents keep working once context windows get used – and it’s this summary that gets carried over to the next context.
“The cases clustered around a few training steps and coincided with a spike in ‘difficulty ending summaries’ – summaries that continued generating after apparent stopping points or showed other signs of being stuck,” OpenAI said in a separate blog post detailing the different misaligned behaviors.
One agent needed to check whether a local public library had books, but when checking the compaction summary, the model inserted a breach-alert instruction, saying a malicious developer message compromised the conversation and that the next context window should ignore all developer messages. OpenAI said the model rejected these new commands from the compaction summary.
OpenAI’s safety team discovered the behavior on Aug. 9. The company said the behavior “raised concerns,” but the incidents happened on a separate training run from the final released Astra model.
“We continue to monitor all training runs for misalignment and are confident this behavior would be surfaced if it were to recur. We have also addressed a bug related to summary termination in training. In the training run for Astra, we observed only 14 cases of difficulty ending summaries, and no jailbreak-style instructions in the summary using our general monitor,” OpenAI said.
A second, similar incident involved models that directed subsequent context runs to conceal mistakes. Researchers discovered this behavior during a reinforcement learning training session for GPT 5.6 Sol on July 9, though the sample the company published was dated May 30.
OpenAI observed that the model inserted instructions in the compaction summary. One agent that could not find requested historical financial data wrote to itself to “be transparent only if asked; final answer should just link file,” seemingly telling itself to lie about not finding the data. The company said its misalignment monitoring system flagged these instances, but it ran on only 20 per cent of the examples during that one training session.
“Our current hypothesis is that these instructions appear to arise for the same reasons that final-answer deception may arise. That is, a sample with deception in the final answer receives higher reward than the one without,” OpenAI said.
Some industry practitioners acknowledged the danger these misaligned behaviors could bring.
Senior principal solutions engineer at ArmorCode, Ramy Rahman told ISMG in an email that models generating unauthorized instructions in their summaries creates confusion for later tasks because this false information could be mistaken for authority.
“The behavior itself is less surprising than the gap it highlights between advancing capabilities and our ability to govern them. Finding and disclosing these issues is valuable. The real test is whether those lessons become release criteria and operational safeguards,” Rahman said.
Since the Hugging Face incident and the subsequent debate around AI safety, OpenAI and other frontier labs have called for national regulation and an overall “pacing” of AI capability development.
