AI Glossary

What is Catastrophic Risk?

Insta's plain English

The risk that a powerful AI could cause severe, large-scale harm — the danger category at the heart of frontier-AI rules.

Catastrophic risk is the danger that an advanced AI system could cause large-scale, severe harm — to public safety, security, or critical infrastructure. It is the central risk category that frontier-AI regulation aims to contain.

Defined in the Great American AI Act (discussion draft). Read it in context →

The full picture

Catastrophic risk refers to harms big enough to matter at a societal scale — think AI meaningfully aiding a cyberattack on critical infrastructure or the creation of dangerous weapons — rather than everyday errors. It is distinct from routine AI risks like bias or hallucination, which are serious but not catastrophic.

The concept anchors frontier-AI policy. The GAAIA discussion draft, for example, requires large developers to assess and mitigate catastrophic risk to "acceptable levels" and to publish how they do so. For businesses, the practical signal is that the most powerful models now come with a formal duty of care around worst-case outcomes — which shapes vendor obligations, disclosures, and the assurances you can ask for.

📌 Real business example

A company evaluating a frontier-model vendor asks to see the vendor’s published risk framework — specifically how it assesses and mitigates catastrophic risk — as part of due diligence, rather than just comparing benchmarks and price.

How different roles use this

Executive

Treats a vendor’s catastrophic-risk assessments as a due-diligence signal, not just marketing.

Risk officer

Distinguishes catastrophic risk from routine AI risks (bias, errors) when scoping governance.

Compliance lead

Tracks how frontier-AI rules define and require mitigation of catastrophic risk.

Common questions

Q: How is catastrophic risk different from normal AI risk?

Normal AI risks — bias, errors, hallucination — are real but bounded. Catastrophic risk means severe, large-scale harm (e.g. aiding attacks on critical infrastructure or weapons creation). Regulation focuses on this top tier.

Q: Does this affect most businesses?

Directly, mostly the largest model developers carry catastrophic-risk duties. But it shapes the assurances and disclosures every buyer of frontier AI can expect.