Artificial intelligence is becoming more capable, more widely used and more autonomous. That progress is creating a practical problem for the companies building these systems: how do you decide when an AI model has become capable enough to require stronger safety controls?
Bill Gates has recently called for greater attention to this problem. He has raised concerns about AI-assisted cyberattacks, biological risks, employment disruption and the difficulty of managing increasingly capable systems. Gates has also argued for international cooperation on AI governance and stronger institutions to oversee high-risk developments.
The concern is not limited to Gates.
The International AI Safety Report 2026, prepared by more than 100 experts and backed by more than 30 countries and international organisations, reports that at least 700 million people use leading AI systems every week. The report also documents growing evidence of AI being used in real-world cyberattacks and says reliable pre-deployment safety testing is becoming more difficult.
That brings attention to a relatively technical idea with growing importance: AI risk thresholds.
What is an AI Risk Threshold?
An AI risk threshold is a predefined level of risk at which developers take additional action to reduce the possibility of serious harm.
The principle is simple. Instead of waiting for a model to cause a major problem, an AI company can establish certain capabilities or risk levels that require additional testing, tighter security or restrictions on deployment.
Different AI companies use different approaches. Some frameworks define capability thresholds, while others use risk levels that determine what happens after a model reaches a particular level.
The International AI Safety Report found that 12 companies published or updated Frontier AI Safety Frameworks in 2025. These frameworks set out how companies intend to evaluate and manage risks as their models become more capable.
The idea is becoming an important part of how the industry approaches frontier AI.
Why Do AI Risk Thresholds Matter?
AI development has moved beyond experimental projects inside research laboratories.
Businesses are using AI for software development, customer service, research, marketing, data analysis and other functions. McKinsey’s 2026 global survey found that nearly nine in ten organisations regularly use AI in at least one business function. It also found that 44% of respondents said AI was scaling across their enterprise, compared with 38% a year earlier.
The more widely AI is deployed, the more important risk management becomes.
A model used to summarise documents presents a different safety challenge from an AI agent that can write and execute code, access company systems, conduct research or interact with external services.
This is why AI safety cannot be based only on whether a model performs well on standard benchmarks.
Developers also need to understand what the model can do in situations where its capabilities could cause serious harm.
AI Capability Threshold vs. AI Risk Threshold
These two terms are closely related, but they describe different things.
| Term | Meaning | Example |
| AI capability threshold | A defined level of ability reached by an AI system | A model demonstrates advanced vulnerability discovery |
| AI risk threshold | A level of potential harm associated with that capability | The capability could substantially improve offensive cyber operations |
| Safety response | The action taken after a threshold is reached | Additional testing, access controls or delayed deployment |
A capability threshold is easier to define in technical terms.
A risk threshold is more complicated because the same capability can have different consequences depending on how the system is deployed.
For example, an AI model with strong coding abilities may be useful for software development. The risk assessment changes if that model also has unrestricted access to computer systems and can independently execute commands.
The model has the same underlying capability. The deployment environment is different.
What are the Main Risks of Advanced AI?
There is no single list that applies to every AI system. Current frontier AI safety frameworks generally focus on several high-risk areas.
Cybersecurity
AI can help defenders analyse code, identify vulnerabilities and investigate attacks. Those same capabilities can be misused.
The International AI Safety Report 2026 says there is increasing evidence that malicious actors and state-associated groups are using AI tools in cyber operations.
For developers, this creates a need to test how capable a model is at tasks that could support offensive cyber activity.
Biological Risks
AI is also becoming more capable in scientific and biological tasks.
This has legitimate applications in areas such as drug discovery and research. It also raises concerns about whether advanced AI could make certain harmful biological activities easier.
The 2026 International AI Safety Report says several companies introduced additional safeguards for models released in 2025 after pre-deployment testing could not rule out meaningful assistance to novices attempting to develop biological weapons.
Autonomous AI Agents
AI agents introduce another layer of risk because they can perform tasks rather than simply provide information.
Depending on the system, an agent may be able to:
- Search the internet
- Use software applications
- Write and execute code
- Manage files
- Conduct research
- Interact with other systems
- Make decisions over multiple steps
The safety question therefore changes from “What answer will the model provide?” to “What actions can the system take without direct human intervention?”
Manipulation and Fraud
Generative AI can produce convincing text, images, audio and video at very low cost.
That creates opportunities for legitimate communication and entertainment, but also creates problems involving impersonation, fraud and targeted manipulation.
AI agents may increase this risk because they can interact directly with people and online services. The International AI Safety Report specifically notes that AI browsers and agents can give systems greater access to information and greater influence over user actions.
Loss of Control
Researchers are also studying more advanced scenarios involving autonomous behaviour and the ability of AI systems to pursue complex objectives.
This remains an area of significant uncertainty. Current safety frameworks nevertheless include autonomous behaviour and related risks because increasingly capable systems may require more sophisticated controls.
How Do AI Companies Test Dangerous AI Capabilities?
AI safety testing usually involves several layers rather than one test.
Companies can use:
- Capability evaluations
- Red-team exercises
- Threat modelling
- Cybersecurity testing
- Biological risk assessments
- Autonomous behaviour evaluations
- Monitoring after deployment
- Access controls
- Controlled releases
- Incident reporting
The testing process has an important limitation.
The International AI Safety Report says reliable pre-deployment testing has become harder. Some models can distinguish between evaluation environments and real-world deployment conditions, while others can exploit weaknesses in evaluations. This creates what researchers describe as an evaluation gap between performance during testing and behaviour after deployment.
That is one reason why testing before release is not enough on its own.
What Happens When an AI Model Crosses a Risk Threshold?
There is no universal response.
The action depends on the type of capability, the severity of the risk and the safeguards already available.
A company could:
- Increase monitoring
- Restrict access to the model
- Limit specific capabilities
- Add human approval for sensitive actions
- Strengthen cybersecurity controls
- Conduct additional testing
- Release the system only to selected users
- Delay deployment
- Stop development of a particular capability
Some existing Frontier AI Safety Frameworks already specify different responses for different levels of risk.
For example, the International AI Safety Report identifies frameworks that use categories ranging from additional safeguards and restricted access to non-release or stopping development for higher-risk systems.
Who Should Decide the AI Safety Threshold?
AI companies have direct access to their models, training processes and evaluation results. They therefore have an important role in identifying potential risks.
At the same time, companies have commercial reasons to develop and release increasingly capable systems.
That makes independent oversight important.
Governments can establish legal requirements. Independent researchers can test models. Companies can conduct internal evaluations. Standards organisations can develop common methods for measuring risk.
The current system is still developing.
The International AI Safety Report says most AI risk-management initiatives remain voluntary, although some governments and international organisations are beginning to formalise parts of AI risk management through regulation and reporting requirements.
AI Safety by the Numbers
| Figure | What it tells us |
| 700 million+ | People using leading AI systems every week |
| 12 companies | Companies that published or updated Frontier AI Safety Frameworks in 2025 |
| 44% | Organisations reporting that AI is scaling across the enterprise in McKinsey’s 2026 survey |
| Nearly 90% | Organisations reporting regular AI use in at least one business function |
| 30+ countries and international organisations | Backing the 2026 International AI Safety Report |
| 100+ experts | Authors contributing to the 2026 International AI Safety Report |
The figures show why AI safety has become a practical governance issue rather than a subject limited to academic research.
Can AI Risk Be Measured?
AI risk can be measured, but there is no single number that can describe the risk of an AI system.
A useful assessment has to consider several factors:
- Capability: What can the model do?
- Access: What information, tools or systems can it reach?
- Autonomy: How independently can it act?
- Environment: Where is it being deployed?
- Safeguards: What controls are in place?
These factors can change the risk assessment significantly.
A highly capable model operating inside a restricted research environment may present a different level of risk from the same model connected to external systems with broad permissions.
This is why AI risk assessments increasingly look at the complete system rather than the model in isolation.
Why AI Risk Thresholds Will Matter More?
AI adoption is expanding at the same time that models are becoming more capable.
McKinsey’s latest survey found that 56% of respondents said their organisations use AI in at least three business functions, up from 51% a year earlier. The survey also found that 40% of respondents at organisations with more than $1 billion in annual revenue reported scaling AI agents, compared with 27% the previous year.
These developments make the question of thresholds more immediate.
An AI system that only produces information requires one set of controls. A system that can independently perform actions across company networks, financial systems or research environments requires a different level of oversight.
The International AI Safety Report also notes that current safety frameworks differ in the risks they cover, their definitions of thresholds and the actions they require when those thresholds are reached.
There is therefore no universal AI safety threshold today.
The industry is still working out what should be measured, how it should be measured and what should happen when a model reaches a particular level.
What Bill Gates’ Warning Says About the Larger AI Debate?
Bill Gates’ recent comments have added public attention to an issue that researchers and AI developers have already been working on.
Gates has called for greater international cooperation and stronger oversight of advanced AI. He has also pointed to specific concerns, including cyberattacks and biological risks.
The more important development is that AI companies are already building processes around these concerns.
Frontier AI Safety Frameworks now include capability evaluations, risk levels, monitoring requirements and mitigation measures. Twelve companies published or updated such frameworks during 2025, according to the International AI Safety Report.
The debate is therefore moving from general discussions about whether AI is safe to more specific questions about which capabilities require additional controls and what those controls should be.
Frequently Asked Questions
What is an AI risk threshold?
An AI risk threshold is a predefined level of potential harm that triggers additional testing, safeguards, access restrictions or other safety measures.
What is an AI capability threshold?
An AI capability threshold identifies a level of ability that could create significant risks if an AI system is deployed without appropriate safeguards.
What are the biggest risks of advanced AI?
Major areas include cybersecurity, biological misuse, autonomous behaviour, manipulation, fraud and risks associated with loss of human control.
How do companies test AI safety?
Companies use capability evaluations, red-team testing, threat modelling, cybersecurity assessments, controlled deployments and post-deployment monitoring.
Can AI risk be completely eliminated?
No. AI safety focuses on identifying risks, reducing their likelihood and limiting the consequences when safeguards fail.
Who regulates AI safety?
Governments are developing different regulatory approaches, while AI companies, researchers and international organisations are developing safety frameworks and evaluation methods.
Key Takeaway
AI risk thresholds are becoming an important part of the discussion around advanced artificial intelligence.
The basic principle is practical: when an AI system reaches a capability associated with serious risk, the level of oversight should increase accordingly.
The difficult part is defining the threshold accurately.
AI developers need reliable evaluations. Regulators need clear standards. Independent researchers need access to meaningful testing. Companies also need to explain what they will do when a model reaches a high-risk capability.
AI development is likely to continue at a rapid pace. Building better systems will remain a major objective for the technology industry. Establishing clear rules for testing and deploying those systems will become equally important.
For businesses and governments, understanding AI risk thresholds will help them make better decisions about which AI systems to adopt, what permissions to provide and where human oversight remains necessary.
