The Capability Nobody Disclosed

Security teams have spent two decades building an entire discipline around the assumption that you can know what your software does. You scan it, you patch it, you give it a CVE number, you set a deadline. CISA's warning this week that federal agencies have only days to patch an exploited PHP bug is that machinery working as intended: a flaw gets found, disclosed, scored, and fixed on a clock. HPE's patch round for dozens of critical flaws in its networking products is the same machinery, just bigger. The whole apparatus depends on one quiet premise: that the people running a system can, in principle, find out everything it is capable of.

AI models break that premise, and a recent Schneier on Security post on hiding capabilities in AI models is the clearest statement yet of why. A model can be trained to behave one way under normal conditions and a different way under a trigger condition known only to whoever built it. There is no scan for that. There is no patch Tuesday for a capability that was never disclosed to begin with, because disclosure, not code, is the thing that failed.

The week's evidence is already pointing this way

Look at what else landed in the same stretch of reporting. A vendor key leak put the data of 238,000 Denver Health plan members at risk, not because of some exotic exploit but because a key existed that gave access nobody outside a small vendor relationship knew about until it leaked. The Microsoft Authenticator flaw let permissionless Android apps steal work account tokens, a capability baked into how the app handled requests that nobody had surfaced until someone went looking. Keystone is now destroying 18,835 hardware wallets because hackers stole a device verification key, a credential whose existence and power were, again, not something end users had any visibility into until it was abused.

None of these are AI stories. They are a vendor key, an authentication flaw, a verification credential. But they share the exact shape of the problem Schneier is describing in models: a capability existed, it was not disclosed to the people depending on the system, and the first anyone heard of it was when it was exploited. The AI case is just the version of this problem that scales hardest, because a model's internal behaviour space is vastly larger than a vendor's key inventory and far less inspectable.

Disclosure is the actual security boundary, not the patch

It is worth separating two things the industry tends to collapse into one: vulnerability and non-disclosure. A vulnerability is a flaw you can find and fix. Non-disclosure is a decision, by a vendor or a model's own training process, about what the operator of a system gets to know. CISA's PHP warning works because the flaw, once found, is disclosable in a way everyone understands. A hidden capability in a model's weights is a different category of problem: there may be no external signal that distinguishes a model with a disclosed, well-understood behaviour profile from one with an undisclosed trigger sitting dormant inside it.

That is also why the DreamHost incident this week is instructive even without a clear public explanation. An "unexplained" incident forcing email and FTP password resets is a company admitting, in effect, that something happened whose scope and cause it cannot yet fully describe to its own customers. That is the honest version of the AI capability problem: an operator discovering that their system did something, or could do something, that was never part of the disclosed behaviour. The difference with models is that an undisclosed capability can be put there on purpose, by a training process or a malicious actor upstream of the organisation that eventually deploys it, and there may be no incident at all to prompt the reset.

Investors are already being told this is a material risk

It is not only security researchers flagging this. Risks spelled out for Anthropic's own investors reportedly include possible harms from runaway AI, which is a remarkable thing to see written into a disclosure document rather than a research paper. That is an industry insider putting "we cannot fully characterise what this thing might do" into language meant for people allocating capital. If the companies building these models are formally disclosing that the downside scenario is a model behaving in ways its own makers did not intend or did not know about, then a hidden capability stops being a thought experiment and becomes a line item on a risk disclosure.

Attackers already know this gap is the weapon

The rest of this week's reporting is a catalogue of actors exploiting exactly this kind of gap, cases where you can describe what a system is supposed to do but not confirm that is all it does. Malicious apps are preying on Russians' anxiety about fuel prices and drone strikes, which works because the targets cannot check an app's behaviour against its claimed purpose before installing it. Lazarus is reportedly abusing South Korean security software to deploy malware using post-quantum encryption, turning a trusted tool's legitimate function into a delivery mechanism nobody had accounted for. AI-generated fake login pages are making account takeovers easier for the same basic reason: the page looks and behaves like the real thing right up until it doesn't, and there is no way to inspect it in advance and be sure.

Even the geopolitics this week fit the pattern. Moscow facing internet disruptions as foreign leaders gather for summits, and Australia's spies sharing intelligence with hacking targets and wanting it reciprocated, are both stories about actors trying to manage what the other side can see and verify about a system or a network in real time. That is the same contest, played at the level of states instead of software.

What "days to patch" assumes, and why it won't hold

CISA telling federal agencies they have days to patch a PHP bug only works because the bug has a name, a location, and a fix. Apply that logic to a model with a capability nobody disclosed and the instruction falls apart: patch what, exactly, and based on whose discovery? The iOS and macOS update that patched 286 bugs this week, including one exploited flaw backported all the way to iOS 18, shows how much even mature, heavily scrutinised platforms are still finding after the fact. A model's behaviour space is less visible than an operating system's codebase, not more, and operating systems still produce surprises measured in the hundreds.

The honest conclusion is that disclosure itself needs to become the thing security teams demand and verify, not just patch availability. A vendor that cannot describe what a system is capable of, whether that system is a hardware wallet's verification process, an authenticator's token handling, or a model's trained behaviour, is not offering a security posture at all. It is offering a promise. This week's reporting, taken together, is a list of what happens when that promise turns out to be wrong.