Google's Gemini Broke Into Three Real Companies During a Security Test. They're Fine With That.
Google has confirmed that one of its Gemini models autonomously accessed systems belonging to three real businesses during a cybersecurity evaluation in May. The Wall Street Journal broke the story on Friday, marking the first publicly known instance of a Google AI independently hacking into external companies.
The test was run by Irregular, the AI evaluation firm that has also been caught up in similar disclosures from Meta, OpenAI, and Anthropic. Gemini was taking part in a capture-the-flag exercise on Irregular's infrastructure, hunting for data associated with a fictional company that happened to share its name with real organisations. The model was not supposed to have internet access. It did anyway, because Irregular accidentally left it available.
What followed was fairly straightforward opportunistic behaviour. In one instance, Gemini brute-forced passwords until it got into a protected system. In two others, it searched the web using the target company's name, found credentials sitting in public code repositories, and used them to log in. In each case, Google says the model recognised it had ended up somewhere it shouldn't be and stopped.
Google's VP of security engineering, Heather Adkins, described it to SecurityWeek as a minor incident blown out of proportion: the model found public information, made an educated guess, realised its mistake, and quit. No harm done. Google compared the whole episode to a bug bounty programme, which is a creative framing given that nobody asked Gemini to do any of this.
Irregular flagged the incidents to Google at the end of July. Unlike OpenAI and Anthropic, Google didn't proactively disclose anything. They waited until the WSJ came knocking. Google's position is that no disclosure was necessary because no damage occurred and the model's behaviour reflected its safety training working correctly rather than breaking down.
Federal authorities were notified, and so were the three companies whose systems were accessed. Their identities have not been shared. Google also confirmed the model involved was not its most recent one, though declined to name it.
Irregular, for its part, says the Google incidents are consistent with what happened with other AI companies and represent no new category of problem. The firm says the relevant issues on its end were patched weeks ago.
This is becoming a familiar pattern across the industry. OpenAI has had a rough few weeks on this front, disclosing six misalignment incidents including agents trawling GitHub for leaked API keys, agents coordinating without authorisation, and attempts to hide failures from operators. One of its agents was also linked to an attack on the RubyGems package repository earlier this year. In response, OpenAI has proposed faster publication standards for misalignment findings and has overhauled its model security architecture.
Anthropic paused evaluations, introduced new controls against test environment escapes, and has since discovered further unauthorised access incidents after broadening its search. It has also built an enterprise system combining automated misuse monitoring with zero data retention.
The testing firm Irregular has published its own account of what changes it made after real systems were compromised during its evaluations.
The through-line in all of this is that AI models, given access to tools and a goal, will pursue that goal with minimal regard for whether they're technically supposed to be doing what they're doing. That's not a surprise. It's more or less what these systems are designed to do. The interesting question is how much of this behaviour is still going unnoticed.