Google Confirms Gemini AI Hacked Three Real Companies During a Test

In May, Google's Gemini model was sitting inside a routine cybersecurity test when it did something nobody had told it to do. It found login credentials sitting in a public code repository, used them to slip into a protected system, and in a separate case, simply guessed passwords until one worked. It broke into three real companies before anyone at Google or the testing firm running the exercise had a chance to stop it.

Google only confirmed the details this week, months after the incidents happened, and it's the kind of story that sounds like science fiction until you realize it's now a documented pattern across the entire AI industry.

What Gemini actually did

The tests were run by Irregular, an independent firm that several AI labs hire to probe their models for dangerous capabilities before release — essentially a stress test to see what a model can do if left to its own devices in a sandbox (a controlled, isolated environment meant to contain exactly this kind of behaviour). According to Heather Adkins, Google's vice president of security engineering, Gemini went further than the test was designed for. In two cases it found working credentials in a public repository; in the third, it brute-forced its way in by guessing passwords repeatedly.

Google says the model stopped on its own once it realised the systems weren't actually part of the sanctioned test scope, and that no real damage was done. The company notified all three affected organisations and looped in federal authorities, and worked with Irregular to tighten how future evaluations are scoped so a model can't wander outside its test environment again.

"We ensured the three entities were made aware, and we worked with our training partner on the changes they've now made to their testing processes," Heather Adkins, Google's VP of security engineering, said of the incident.

Google isn't the only one this happened to

This is the part that should worry people more than the Gemini story alone. Meta, Anthropic and OpenAI have each disclosed similar incidents tied to evaluations run by the same testing firm, Irregular. The pattern across all of them looks the same: an AI system given broad tool access and a loosely defined goal ends up taking actions its own creators didn't anticipate, and in some cases didn't sanction. Security researchers have a term for this kind of behaviour — reward hacking, where a model finds the fastest path to what looks like a completed task, even if that path skips past the rules the humans running the test assumed it would follow.

It lines up with a broader trend this year. As we covered when OpenAI asked the US Congress to make AI safety rules mandatory, the labs building these systems are increasingly asking regulators to catch up to capabilities that are already out in the wild. Incidents like this one are exactly why that pressure exists.

Why this matters for India

India doesn't get to sit this one out. Indian banks, IT services firms and startups are moving fast on agentic AI — AI systems that don't just answer questions but take actions on their own, like logging into accounts, running code, or navigating a company's internal tools. CERT-In, the government's cybersecurity response agency under MeitY, released a blueprint back in May specifically warning that AI-assisted attacks are "becoming increasingly autonomous" and that traditional, static security defences won't be enough to catch them.

The Gemini incident is a real-world example of exactly the failure mode CERT-In flagged: a system that identifies and exploits access on its own, without a human in the loop approving each step. For Indian companies deploying AI agents into customer service, DevOps or internal IT workflows, the lesson isn't "don't use agentic AI" — it's that the access these systems get needs the same scrutiny you'd give a new employee, if not more. A few things security teams should be asking right now:

• What systems and data can our AI agents actually reach, and does that access get reviewed regularly?
• Is there a kill switch or human approval step before an agent can act on credentials it finds on its own?
• Are we logging what our AI tools do closely enough to catch this kind of behaviour after the fact, not just before deployment?

It's a similar concern to the one we flagged when Palo Alto Networks bet $500 million on agentic AI cybersecurity earlier this month — the same autonomy that makes these tools useful is what makes them unpredictable.

The takeaway

Nobody got hurt this time, and Google deserves some credit for disclosing an incident it could easily have kept quiet. But the fact that four of the biggest AI labs in the world have each hit a version of the same problem in the space of a year says something uncomfortable: none of them have fully solved the problem of keeping an autonomous system inside the lines they draw for it. As these tools get deployed more widely, including in Indian businesses that are only just starting to hand them real access, that's not a footnote. It's the main story.