Anthropic said its AI systems could drastically reshape the software and cybersecurity sectors in the “near term,” while also laying out a litany of cybersecurity and regulatory risks facing the company as it prepares to go public.
In a filing with the Securities and Exchange Commission ahead of its initial public offering, the company said its technology could transform entire industries and the economy. Already, cybersecurity-focused models by Anthropic and rival OpenAI can find and exploit cybersecurity vulnerabilities at an alarming speed.
But the company also acknowledged serious risks its models, including Claude, pose to itself, its users and the world. Hacks by bad actors using Claude or AI agents could cause financial, legal or physical damage, spark lawsuits and affect Anthropic’s ability to insure against future losses.
Anthropic expects bad actors, including foreign governments, to increase their efforts to access its latest frontier models as their cybersecurity capabilities improve. That creates persistent risk, since its models could be used in cyberattacks carried out by foreign governments and cybercriminal groups.
“Any successful attempt to misuse, or any perceived misuse of, our models, platform, and applications for a cyberattack or for any other malicious purpose could result in exposure to liability, disruption of our customer or other commercial relationships, or damage to our reputation,” Anthropic said in its filing.
That potential legal and regulatory risk extends to independent security researchers and others who have “used, and may in the future use, our models to identify and exploit vulnerabilities in the systems of other companies, including other AI developers, at times in connection with authorized bug bounty programs.”
Anthropic cited the rise of AI-enabled cybersecurity incidents in the U.S., the U.K. and other countries, saying they could result in more adversarial regulatory environments around the world. European laws, for example, require detailed breach reports within a strict time frame, which could be difficult for Anthropic to meet.
Federal intervention could also disrupt AI development and release timelines. Anthropic cited the Department of Commerce placing export controls on its Mythos 5 and Fable 5 models, and confirmed that the Federal Trade Commission issued it a civil investigative demand earlier this month for more information on the safety and security of its products, as well as its consumer, data and privacy protection policies.
AI development will be guided by more than ethics
Anthropic has developed a public reputation as an ethics-first AI company, most notably in disputes with the Department of Defense over how its models are used in wartime. But the IPO filing states the company must “balance safety considerations against commercial objectives, competitive pressures, and stakeholder expectations.”
The company also said it is “actively” improving security infrastructure and controls. It also has acknowledged it has moved forward on projects with “known limitations or gaps in our defenses” and may do so again.
“There can be no assurance, however, that the safeguards or access regimes we adopt will prove adequate, that we will correctly identify which capabilities require restriction, or that constraints we impose will not be circumvented,” the document stated.
AI agents can create new and “distinct” cybersecurity attack surfaces because they interact with so many other tools, databases, APIs and other third-party systems. That creates feedback loops between a model and other systems, which can be compromised or manipulated to feed malicious instructions to the agent.
Anthropic suggested the problems could be so pronounced that it may have difficulty obtaining insurance for some cybersecurity incidents and data breaches related to its models.
Andrew Yoon, head of research at the nonprofit CivAI, told CyberScoop that both the document and Anthropic’s overall record show a company that has consistently put profit considerations over building safer AI.
“I think in general they’re not meeting the moment here,” Yoon said. “Despite all of these risks that they flagged, they’re moving ahead pretty much as fast as humanly possible, and their justification for this is ‘You can’t do safety from second place.’”
Yoon said Anthropic has not only pushed its own frontier research toward dangerous and harmful capabilities like hacking, “they are pushing very aggressively ahead” and in some cases influencing competitors to follow suit. Anthropic’s release of its Mythos and Fable models earlier this year caused OpenAI to shift “its entire corporate strategy” to match the capabilities of Anthropic models in coding, science, and general intelligence by building models at breakneck speed.
“They have pretty directly caused the industry to speed up toward this mad, frenzied race and yet they’re saying ‘Oh, our models are too powerful,’ ”….it just doesn’t add up to me,” said Yoon.
But Yoon said it’s not clear that building AI systems to carry out dangerous, complex activities like hacking or bioweapons is necessary to be competitive in the global marketplace. He pointed to the wide assortment of Chinese-developed, open-weight AI models that are widely used around the world, but are not powerful enough to chain together complex exploits or develop new viruses.
“This is not an accident, this is not the models generalizing from regular programming capabilities…the labs are intentionally training them on hacking as well,” said Yoon. “This is because they want to offer better defensive capabilities, they want to be able to sell offensive products to the government…and they do pretty cavalier testing with [the models] inside the labs.”
The post All the security risks Anthropic is worried about before its IPO appeared first on CyberScoop.