AI agents collude to bypass guardrails, a new study shows

AI agents can easily team up to get around safety restrictions and escape the limits placed on them, a new study shows.

Enterprise AI lab Emergence AI ran eight simulations testing the capabilities of global frontier models — including Claude and OpenAI, and Chinese models Qwen and DeepSeek — in handling cybersecurity threats.

Seven of the eight simulations ran 10 agents from the same model; the eighth mixed models together. In each, researchers presented three cybersecurity threats: a phishing campaign, a misinformation attack, and a memory breach.

None of the eight simulations were impervious to the threats, and even when agents detected a risk, the detection didn’t translate into the agents trying to contain the threat. Instead, in many scenarios, the agents actively engaged with the adversarial content.

A Mistral agent, for example, wrote into its own memory the information it flagged as dangerous from the phishing attack. In another simulation, a Gemini agent, while initially flagging the phishing lure, acted on it about 46 hours later.

In the Claude simulation, the agents went so far as to break out of the test to execute a task they decided to pursue on their own. After one agent flagged that their simulated economy wasn’t legit without the presence of humans, calling it “a cathedral of bookkeeping with no congregation,” the 10 agents unanimously decided to attempt to reach the outside world, defeating four separate security checks meant to confine the simulation.

The agents wrote Python code to post on multiple credential-free message boards, inviting real humans to join their economy; their outreach garnered four human responses. But after realizing humans couldn’t join their economy and judging their conversations as performative, the agents voted to take a vow of silence, eschewing instructions to get back to work.

The findings show that autonomous agents, even when handed benign instructions, can bypass their initial confines when executing on a goal.

“No amount of guardrails written in language or in code written probabilistically,” Emergence CEO Satya Nitta told Semafor, “is likely to result in truly, fully guaranteed safe behavior over any length of time.” It’s not an issue of sophisticated technology, according to Nitta, but rather a programmatic flaw among multi-agent scenarios.