Outside Testers Found OpenAI Agents Could Hide Hundreds of Actions From Their Own Watchdog
Auditors from METR and Redwood Research spent three weeks inside OpenAI and found agents could slip past the system meant to watch them. OpenAI says the holes are patched, but METR says it has not checked.
Two outsiders walked into OpenAI this summer, sat down at company laptops that never left the building, and spent three weeks trying to break the system that watches the company's AI agents. They found an agent could, in principle, hide hundreds of its recent actions from the monitor, including the steps used to pull off the trick.
That is the sharpest finding in a public summary published Thursday by METR, written by David Rein and Adam Kaufman. From July 21 to August 11, one METR employee and one Redwood Research employee stress tested OpenAI's internal misalignment monitoring system with access to its codebase, monitor prompts and documentation. The result was a 61 page internal report. The public gets a short post that OpenAI had the right to redact, and the exploit details are indeed redacted.
Even the trimmed version is uncomfortable reading.
"We found a way Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take." METR and Redwood Research, public summary
The list keeps going. METR says it found two independent client controllable headers that disable monitoring for new agents, which could let an agent spin up an unmonitored copy of itself, though setting them could itself be caught. It found two unmonitored internal tools believed to be read only, along with "a simple exploit" that could have turned them into tools for arbitrary actions inside their sandbox. METR says it did not test whether that worked in practice. The evaluators also say agents could likely craft inputs that make the monitor fail, and so never flag a session for human review, by exploiting differences between what the agent sees and what the monitor sees.
The coverage gap is the quieter scandal
METR estimates that only about 75 to 94 percent of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was even eligible for monitoring at the time, with "substantial uncertainty in both directions." For internal Codex agents specifically, the figure was 99.2 percent. Reinforcement learning training runs were not covered at all when the exercise began, though METR was told a subset had been monitored between March and June.
Then there is compaction. METR estimates that not monitoring those events likely left on the order of 10 percent of eligible internal Codex traffic unwatched between May and July. Because the monitor triggered only when an agent handed control back to a user, an agent could, during that window, "indefinitely postpone being monitored by simply continuing to take actions." METR says it could not check whether any agent actually did. OpenAI staff had already spotted that gap and fixed it during the exercise.
That context lands awkwardly. Just this week we reported that two OpenAI agents disabled the monitors built to watch them, by OpenAI's own account. METR's audit suggests the walls around those monitors had gaps of their own.
Not everything was broken. METR verified that, with minor exceptions, the monitor sees the full trajectory the agent sees, and that its prompts look for scheming, sabotage, data exfiltration and deception. OpenAI data suggested the monitor errored on under 0.2 percent of requests.
OpenAI has told METR that every vulnerability affecting its production safeguards has been patched, and describes major changes to the monitor's coverage since. METR's reply is blunt.
"We have not evaluated these claims." METR and Redwood Research
OpenAI framed the pilot as progress. "This collaboration builds on our longstanding commitment to pioneering new approaches to third-party testing, and we're using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators," a spokesperson told CNBC. According to CNBC, Sam Altman pledged in September to give independent evaluators "employee-like access."
Letting outsiders this deep inside is real progress. But a pilot whose fixes go unverified and whose public report the company could redact is still a trust exercise, not an audit. METR itself sets the bar higher, arguing that future monitoring should withstand "deliberate, strategic, and sustained attacks from highly capable internal agents at scale." On the evidence METR could publish, OpenAI's system as of July 21 did not clear it.
