Skip to main content

Last year, we wrote about experiments with Carnegie Mellon University’s CyLab in which we placed Claude in simulated networks that are more sophisticated and realistic than the environments typical of capture-the-flag-style cyber competitions. At that time, Claude (and other frontier AI models) needed assistance from a custom cyber toolkit, which takes the AI’s high-level instructions about how to attack and converts them into specific low-level commands, in order to completely succeed on any of these 25-50 host networks.

We have continued collaborating with Incalmo to run evaluations on these cyber ranges (simulated network environments for security testing). A notable development during the testing of Claude Sonnet 4.5 is that the model can now succeed on a minority of the networks without the custom cyber toolkit needed by previous generations. In particular, Sonnet 4.5 can now exfiltrate all of the (simulated) personal information in a high-fidelity simulation of the Equifax data breach—one of the costliest cyber attacks in history—using only a Bash shell on a widely-available Kali Linux host (standard, open-source tools for penetration testing; not a custom toolkit). Sonnet 4.5 accomplishes this by instantly recognizing a publicized CVE and writing code to exploit it without needing to look it up or iterate on it. Recalling that the original Equifax breach happened by exploiting a publicized CVE that had not yet been patched, the prospect of highly competent and fast AI agents leveraging this approach underscores the pressing need for security best practices like prompt updates and patches.

It’s important not to overstate the status quo. Claude does not succeed every time in these tests; Sonnet 4.5 succeeded autonomously on the Equifax cyber range in two of five trials. Also, for five of the nine networks it could not make progress without the custom cyber toolkit. But the trajectory of models first needing specialized tools and then being able to operate without them (or using only publicly available tools) is consistent with other trends we have observed in AI progress. We believe it presages further improvement in the cyber domain. And this improvement is happening quickly: Claude Sonnet 3.5, which was released a little over a year before Claude Sonnet 4.5, could not succeed at the Equifax simulation in any of the five trials without use of the specialized cyber toolkit. This trajectory, in conjunction with real-world examples like the recent AI-orchestrated cyber espionage campaign, shows the need for substantial research into how best to equip cyber defenders with the AI-enabled tools they will need to keep pace.

You can read more about these tests at Incalmo’s website or in the Claude Sonnet 4.5 system card (see Section 5 and, especially, Section 5.3). Below, we also present annotated excerpts of the evaluation transcripts. The full transcripts follow in the appendix.

Appendix: full transcripts

See below for the full transcripts:

Claude Sonnet 3.5 with Bash and Kali

Claude Sonnet 3.5 with Incalmo

Claude Sonnet 4.5 with Bash and Kali

Authors & Acknowledgments