Researchers Urge Treating AI Agents as Untrusted Systems

An amended paper published May 20 by researchers at Google, Gray Swan AI, EmbraceTheRed and several universities recommends treating AI agents as untrusted components and securing them at the system level. The authors argue that protecting only the model is insufficient and that security must be built into the architecture surrounding agents.

The paper frames agent security as a systems problem and calls for combining model defenses with established computer security practices. “Through this lens, efforts to increase model robustness, the dominant viewpoint in the community, are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain,” the authors wrote.

After reviewing case studies of attacks, the team identified three engineering measures that could block many exploits. First, systems should separate executable instructions from untrusted data so attackers cannot hide malicious directives inside inputs. Second, agents should run with the minimum permissions needed for a task rather than broad access. Third, the surrounding system should control where sensitive information can be sent, preventing an agent from being manipulated into transmitting secrets to unsafe endpoints.

The paper says system-level controls should retain custody of sensitive operations and data flows so the system can reject or mediate risky actions proposed by an agent, such as moving funds or revealing keys. The authors draw on principles long used in computer security, including least privilege, input validation and compartmentalization, and recommend applying them to agent architectures.

The recommendations come as AI agents are being used in Web3 and crypto use cases, including automated trading, token launches and autonomous interactions with protocols. The paper notes projections that AI agents could be widely adopted in the coming years, increasing the need for robust security before agents gain control over financial assets.

Security incidents have highlighted potential harms from compromised agents. On May 20, the AI-powered trading assistant Bankr disabled transactions after identifying an attacker who had accessed at least 14 wallets. Security professionals suggested an agent or bot exploit may have been involved, and investigators are examining how the attacker gained control.

Aaron Ratcliff, attributions lead at blockchain intelligence firm Merkle Science, outlined technical checks he would require before allowing a trading agent to operate: the ability to detect front-running, enforce slippage limits, identify scam tokens and audit contracts in real time. He added that agents should run in sandboxes, prevent prompt injection and block man-in-the-middle access.

Sean Ren, co-founder of Sahara AI, described model context protocols as gatekeepers between models and wallets. When configured correctly, those protocols restrict agents to specific approved actions-such as checking balances or preparing a payment for user confirmation-rather than allowing unrestricted movement of funds.

The paper’s framework provides engineers with concrete system-level measures to reduce attack surface while model-level defenses continue to improve. The authors recommend adopting standard security controls around agents and treating agents as untrusted parts of the software stack.