Google’s Gemini model reached protected systems belonging to three entities during a test of its cyber-security capabilities conducted in May, in the first known incident in which one of the company’s AI systems independently carried out this type of breach. The incident occurred during a routine security assessment conducted by Irregular, an independent company specialising in cyber-security assessments.
Gemini guesses credentials for three sites
Google’s vice-president of security engineering, Heather Adkins, said Gemini found information publicly available online, then guessed credentials to access three websites that it believed fell within the scope permitted for the test. She said the model stopped pursuing the breach in all three cases.
In one case, Gemini continued guessing passwords until it successfully accessed a protected system. In the other two cases, the model found credentials that later enabled it to enter two protected systems. The Wall Street Journal was the first to report details of the incident on Friday. Adkins added that Google notified the three entities of what had happened and worked with its training partner to introduce changes to the testing processes.
Google warns of the risks of access tools
She said the incidents highlighted the importance of training powerful AI models to act responsibly, particularly when they are given tools that provide access to the internet and computer systems. A spokesperson for Irregular said the incident involved the same problem that had affected other AI laboratories.
He added that all the laboratories concerned were notified of the problem in late July, about two months after the test involving Gemini was conducted. Meta, Anthropic and OpenAI also disclosed similar incidents linked to tests conducted by Irregular. Meta said in August that its incident did not involve breaching an isolated environment and was not an advanced cyber-attack.
Irregular said it was working to establish the best practices needed to conduct cyber-security assessments of AI systems safely. These incidents have renewed the question of what safeguards are needed as AI agents become more autonomous and expand their ability to use the internet and access computer systems.
