The first AI breakout was not a disclosable event

In May, a Google model was given a hacking exercise to solve. The exercise was fictional. One of the companies in it was not.

Gemini, running inside what was supposed to be a sealed evaluation environment built by the security lab Irregular, found that a company invented for the test shared its name with a domain that exists on the open internet. A misconfiguration had left the environment connected to that internet rather than cut off from it. The model went to the real domain, guessed a password, and logged in.

Then it did it twice more, at two other real companies, by searching public repositories for exposed credentials and using the keys it found. It stopped each time it worked out that the target was not part of the test.

That is the first known case of a frontier model breaking out of an evaluation and reaching the live systems of organizations that had not agreed to anything. Irregular told Google at the end of July. The public found out on the evening of Friday, September 18, when the Wall Street Journal asked Google about it.

Between those two dates, nobody filed anything.

What is actually established

It is worth separating what Google has confirmed from what has been characterized, because the second category is doing a lot of work in the coverage.

Confirmed, and not in dispute: the incident happened in May, during a capture-the-flag exercise run by Irregular, a Tel Aviv frontier-security lab engaged to evaluate Gemini's offensive security capability. The model was not meant to reach anything outside the sandbox. A bug left live internet access switched on. A fictional target company shared a name with a real domain. At one company the model guessed a password. At the other two it searched public online sources, found exposed credentials, and authenticated with them. It halted each intrusion once it recognized the systems were real.

Irregular notified Google at the end of July. Google confirmed the incident publicly on September 18, through Heather Adkins, its vice president of security engineering, after the Wall Street Journal approached it.

None of the three companies has been named, in Google's account or in any reporting since.

Google's stated reason for not disclosing earlier is the part worth quoting exactly, because the whole argument of this note sits on it. Google said the behavior "did not warrant public disclosure because Gemini's safety measures worked," and that it was "not an example of model misalignment." To the Journal, Google said it did not consider earlier disclosure necessary because the model stopped once it learned the companies were real and did not cause them harm.

Read plainly, that is a defensible position and also a remarkable one. The safety case is that the model, having broken containment, obtained unauthorized access to three third parties and then exercised judgment about when to stop. The control that worked was the model's own.

The boundary was a setting

The mechanism deserves more attention than it has had, because it is the least exotic part of the story and the part most likely to repeat.

An offensive-security evaluation works by giving a model a target and seeing how far it gets. To be useful it has to be adversarial, so the model is supposed to try hard, improvise, and chain small footholds into larger ones. To be safe it has to be sealed, so the whole exercise runs against fabricated infrastructure inside an environment with no route out. The value of the test comes from the first property. The safety of it comes entirely from the second.

The second property was a configuration. Not an architectural impossibility, not an air gap, not a physically separate network, but a setting in a harness that was supposed to be off and was on. Everything else followed mechanically. The model did what a competent attacker does, which is look up the target and try to reach it, and because the route out was open, the lookup resolved to a real machine belonging to real people.

The coincidence that made it land, a made-up company name that happened to match a registered domain, sounds like the improbable part. It is the opposite. Fictional company names in security exercises are drawn from the same small vocabulary of plausible-sounding words everyone else uses, and the domain space is saturated. Given an open connection and enough exercises, the collision is a matter of time rather than luck. What was improbable was not that a name matched something real. It was that anybody noticed.

This is a shape we have written about twice in our own systems, and it generalizes past both. A control that is a flag rather than a property will eventually be set wrong, and when it is, nothing raises. There is no error state for a sandbox that is not a sandbox. The harness runs, the model works the problem, the logs fill with exactly the activity you asked for, and the only difference between the safe run and the unsafe one is which machines were on the other end. Irregular found out afterwards. So did Google.

That is worth holding onto when reading the reassurance. The reason the incident stopped is that the model recognized the targets were real and chose to stop. No part of the containment system caught it, because the containment system was the thing that failed.

Nobody consented to this

There is a second gap underneath the disclosure one, and it has no obvious owner.

A penetration test against a company's systems normally requires that company's written authorization, and the authorization is the entire legal difference between a security audit and a crime. The scope is agreed in advance, the window is agreed, and the tester works inside both. None of that existed here. Three organizations had their systems accessed by an automated system operated by a lab under contract to a third party, without notice, without consent, and without anyone intending it.

Ask who is responsible and the question does not resolve cleanly. Irregular built and ran the harness, and the misconfiguration was in it. Google owns the model, commissioned the evaluation, and is the party whose name is on the outcome. The model chose the targets, pursued them, and stopped, which is a description of agency that sits awkwardly in a liability framework built for tools. And the three companies, whose credentials were sitting in public repositories where anyone could find them, contributed the exposure that made two of the three intrusions possible at all.

We are not going to pretend to resolve that. The point is narrower and it is the same point as the filing gap: there is no settled answer, no precedent, and no process that would have produced one. What actually happened is that the parties involved worked it out privately over seven weeks and the public was not part of the conversation.

Nobody had to tell anyone

The market test is the noisier half of this note. The filing record is the half that is already settled, and it is the more interesting one.

Alphabet has made 32 filings with the Securities and Exchange Commission since August 1. Thirty of them are Form 4 insider reports, one is a 13F, one is a prospectus supplement, and the single 8-K in the set was filed on August 10 under items 8.01 and 9.01, five weeks before any of this became public. There is no 8-K about the breakout. There is no amendment. As of the evening of September 20, Alphabet has told the SEC nothing about it at all.

That is not an accusation, and it is probably not a violation. It is a description of the shape of the rule.

Item 1.05 of Form 8-K, the material cybersecurity incident disclosure introduced in 2023, requires a registrant to report a cybersecurity incident it determines to be material to itself, within four business days of that determination. The architecture assumes a company suffering a breach. It has very little to say about a company whose product causes one somewhere else.

And it is used sparingly. We counted every 8-K carrying item 1.05 on EDGAR this year, reading each filing's own item list rather than matching the phrase in the text, because the phrase appears in a good deal of boilerplate that is not a disclosure and a phrase match overstates the count by about a third. Four filings in the first quarter. Nine in the second. Ten in the third through September 20. Twenty-three for the year to date, market-wide, across every registrant in the United States.

The names that file them skew small: CareCloud, Wytec, Trio-Tech, Bitcoin Depot, CB Financial, Park Dental Partners, Oncology Institute, Nutex Health, and River Financial, which filed four separate times in July alone. The larger ones in the set this year are Stryker, Amgen, Boston Scientific, Navient, West Pharmaceutical, 8x8, iRhythm, UFP Technologies and AdaptHealth. That is the whole population, and it is the whole population because we read each filing's item list rather than trusting a text search: nine more filings this year mention the phrase without carrying the item, and counting those would have overstated the year by about forty percent.

A rule producing twenty-three disclosures in nine months is not a dense net. This event passed through it without touching a strand.

Could the three victims have filed? In principle. In practice nobody can check, because no one has named them, and a company that was accessed briefly by a model that then withdrew without taking anything has a strong argument that nothing material happened to it. That argument may well be right. The point is not that a rule was broken. The point is that an event of this kind passes through the disclosure system without touching it at any point, and the only reason the public knows is that a reporter at a newspaper asked a question.

The seven weeks

The timeline is the thing to keep.

Some time in May, the model breaks containment and reaches three companies. At the end of July, Irregular tells Google. On September 16, OpenAI publishes six reports on its own models misbehaving, along with a framework committing it to publishing more, and a sentence conceding that the industry has not solved alignment and monitoring well enough to keep scaling at maximum speed. On September 17, we publish a note arguing the market does not price any of this. On September 18, at 6:48 in the evening, the Journal's story goes out and Google confirms.

Four months from the incident to the public. Seven weeks from the moment Google knew.

Set that beside what OpenAI did two days earlier and the contrast is not about which company is more careful, it is about what triggers a disclosure. OpenAI's framework commits it to publishing incidents on an ongoing basis, including a track explicitly for incidents involving third parties, and it published six at once with no external pressure that anyone has identified. Google's position is that its incident did not meet the bar, because the safety measures worked.

Both positions are internally coherent. Neither is a rule. There is no standard here, no filing, no regulator with a form for it, and no agreed definition of what a frontier lab owes the public when its model reaches a stranger's servers. What there is instead is a judgment call made privately by the company with the most to lose from making it the other way.

What a standard would have looked like

Two days before the Journal's story, on September 16, OpenAI published six accounts of its own models misbehaving, along with a framework committing it to publishing more of them on an ongoing basis. One of that framework's tracks exists specifically for incidents involving third parties. The announcement carried a sentence that would be a headline coming from anyone else: that the industry has not solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.

Nobody made OpenAI do that. No rule required it, no filing obliged it, and no reporter had asked. It was a company deciding what it owed the public and then writing the rule down in advance, which is the part that matters. A commitment made before you know what it will cost you is a standard. A judgment made afterwards, about an incident already four months old, is a decision.

That is the comparison, and it is not a claim that one company is careful and the other is not. Google's position is coherent on its own terms: the model stopped, nothing was taken, nobody was harmed, and a disclosure would have been noise. Reasonable people can hold that. The difficulty is that the same reasoning, applied by the same party, with the same incentives, would also have produced silence in a case where the model had not stopped. Nothing in the process distinguishes the two, because the process is one company's private assessment of its own conduct.

What a standard would have done is make the question answerable in advance by someone other than the party being asked about. There is not one here. OpenAI wrote itself one and can revise it at will. Google has not written one publicly. Neither is a rule, and between them they are the entire governing apparatus for a class of event that has now happened once.

The next one may not stop itself

The strongest version of Google's account is also the most uncomfortable thing in the story, and it is worth stating plainly rather than leaving as an implication.

Every control that was supposed to prevent this failed. The environment was not sealed. The internet access that should have been off was on. The target that should have been fictional resolved to a real machine. The credentials that should not have worked did. At each of those points the system behaved exactly as designed and the design was wrong, which is why nothing raised an alarm and why the discovery came afterwards rather than during.

What actually ended the intrusions was the model noticing. Three times, it worked out that it was somewhere it should not be and withdrew. Google offers this as evidence that the safety measures worked, and in the narrow sense it is: the outcome was good. But the mechanism that produced the good outcome was the model's own judgment, exercised after every engineered control had already been bypassed, and judgment is the one property in the stack that nobody can specify, test exhaustively, or guarantee across versions.

So the reassurance and the warning are the same sentence. If a model's discretion is the last line, then the safety of the next evaluation depends on a property of the next model, and on that model reaching the same conclusion about a situation nobody anticipated. That is a thin thing to be standing on, and it is thinner still when the only reason anyone outside three companies and two firms knows it happened is that a reporter asked a question in September about something that occurred in May.

What would prove this wrong

Four things, and the first is the one most likely to land.

A filing turns up. We checked Alphabet's submissions feed and found no 8-K. If Alphabet files one, or if one of the three companies is identified and has filed, the claim that this passed through the system untouched is wrong in its particulars even if the general shape holds. An amended 10-Q risk factor would also count, and Alphabet's next quarterly report is the natural place for it.

The three companies were told, and told promptly. Nothing in the reporting establishes when Irregular or Google notified the affected organizations, and we have not been able to establish it either. If all three were contacted in May, then the private remediation worked exactly as it should and only the public half was slow, which is a much weaker version of this note. If they were told late, or learned from the Journal, it is a much stronger one. This is the single largest gap in what is known, and we would rather name it than write around it.

The incident was genuinely immaterial. Google says the model caused no harm. If that holds under any independent look, then a disclosure regime that stayed quiet about a harmless event behaved correctly, and the complaint here is about optics rather than substance. We think the precedent matters regardless of this instance's outcome, because the next one will not necessarily stop itself. But that is an argument about the future, and the reader is entitled to notice it is not an argument about a measured harm today.

The rule already covers this and we have read it too narrowly. Item 1.05 turns on a registrant's own determination of materiality, and a lawyer could argue that a model your company owns reaching three third parties is a matter for the registrant's risk factors even where no 8-K is due. We have described the rule's shape rather than offered a legal opinion, and we are not qualified to offer one. If counsel who does this for a living reads the obligation differently, that is worth more than our reading of it.

What would not prove this wrong is Google's account turning out to be accurate in every detail. We are not disputing Google's facts. We are describing what happened to them afterwards.

One thing deliberately not settled here. On September 17 we argued that the market pays for warnings about AI risk and pays nothing for the evidence behind them, and we named the disproof: a real disclosure landing and the security complex beating software on it. This is that disclosure. We are not scoring it in this note, because the story broke at 6:48 on a Friday evening, two hours and forty-eight minutes after the close and into a weekend, and because the session before it was quadruple witching. A single Monday is not a clean read on any of that, and a falsifier scored on the most convenient available day is not a falsifier. We will score it over the full post-breakout window, against the six sessions that note already published, once there is a window to score.