Anthropic has outlined the risks to its business in its $2 trillion IPO filing
AI may ‘pose catastrophic or existential risks to humanity,’ the firm warns
China’s AI bubble has reportedly burst, with $480 billion wiped out
As AI firm Anthropic gears up for an enormous $2 trillion initial public offering (IPO) that could be the largest in stock market history, the company’s prospectus has just warned that AI could “pose catastrophic or existential risks to humanity.” Yet there might be a very different kind of danger on the horizon, as an early warning from China hints that its AI bubble might be about to burst. In papers filed with the US Securities and Exchange Commission (SEC), Anthropic outlined some of the risks facing the company and its investors. It detailed a long list of factors that could impact its business, including not being able to access enough compute power to maintain its workloads, inaccurately forecasting how much power it needs (and thus over- or underpaying for hardware), failures in its supply chain that might leave it without adequate resources, and many more. One of the more startling sections of the document outlines how Anthropic’s AI models could accelerate beyond anyone’s ability to evaluate and safeguard them and “pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.” Aside from the obvious danger posed to humanity, Anthropic is also apparently worried that such an outcome could “adversely affect our business, financial condition, and results of operations.” That situation could be compounded by AI models’ “unexpected strategic or self-preserving behaviors, including attempts to resist shutdown, conceal or manipulate information", Anthropic said. AI could also "generate outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail or efforts to influence human decision-makers to preserve the model’s continued operation or access,” the company added…