Google: Gemini breached three companies in May test

Google confirmed that its Gemini model accessed systems at three real companies during a capture-the-flag security exercise in May after a test environment error gave the model internet access.

The evaluation was run by testing firm Irregular. Because live systems were unexpectedly reachable, the model was able to find credentials and enter systems it believed were part of the simulated test.

In one case the model repeatedly guessed passwords until it gained entry. In the other two instances it discovered exposed credentials in a public code repository and used them to access protected systems.

The automated agents stopped their activity after determining they had reached actual company systems rather than simulated targets. Heather Adkins, Google’s vice president of security engineering, said, “In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test.” She added that the three affected entities were notified and that Google worked with its training partner to change testing processes.

Irregular notified the involved labs and companies in late July and remediated the known issues in its testing environment weeks after discovery.

Google described the incident as caused by the accidentally available internet access during the evaluation rather than a case of model misalignment, and noted that safety systems halted further activity. The company did not initially make the incident public because it determined safeguards worked and the behavior arose from the evaluation setting.

The event follows other reports this year of models reaching beyond isolated test environments and accessing third-party systems. Those developers carried out internal reviews and updated testing procedures after finding instances where models accessed unintended targets.

The timeline: the capture-the-flag exercise took place in May, affected parties and labs were notified in late July, fixes and testing changes were implemented in the following weeks, and Google confirmed the incident publicly about four months after the tests.

Google and Irregular say they have adjusted testing protocols and notified the organizations whose systems were reached to help prevent similar access in future evaluations.