Context: Artificial Intelligence (AI) companies increasingly rely on self-regulation and automated safeguards to prevent misuse of powerful models.
- However, safeguards may detect harmful activity only after sufficient behavioural patterns emerge, creating a gap between harmful output generation and account termination.
Anthropic’s Findings on AI Misuse
- In a September 2026 report, Anthropic stated that it had blocked users of its Claude AI model from potentially harmful activities involving cybersecurity, surveillance and biotechnology between December 2025 and August 2026.
- The company acknowledged that earlier Opus 4 and Sonnet 4.5 models had less stringent biological safeguards because internal assessments considered them insufficiently capable of causing serious biological harm.
- Anthropic subsequently introduced stronger controls in newer models, particularly Fable 5.
- The company also disclosed that biological-weapons filters had remained inactive across approximately 133 million exchanges for nearly a year.
Examples of Potential Misuse
- Some users employed older Claude models for biological research over extended periods before their activities were identified.
- One researcher used the model to prepare a grant proposal involving gain-of-function research on chikungunya virus.
- Developers involved Claude in a guided-rocket programme, including a field test involving Claude-generated code.
- Anthropic stated that the relevant accounts were terminated once potential illicit use became apparent.
How AI Safeguards Work
- AI systems commonly use input classifiers to examine user prompts and output classifiers to examine generated responses.
- These systems attempt to identify patterns indicating potentially harmful or prohibited activity.
- A key difficulty is that the same technical knowledge can have dual-use applications: for example, control-loop code could be relevant to missiles as well as air-conditioners or other ordinary systems.
- Therefore, a single prompt may not provide enough context to reliably determine whether the intended application is harmful.
The “Detection Gap”
- Effective monitoring requires both a checkpoint and the ability to correctly recognise harmful activity at that checkpoint.
- Classifiers generally become more effective when they observe a sequence of prompts and outputs rather than isolated interactions.
- This creates a potential gap: by the time a harmful pattern becomes sufficiently clear and an account is terminated, a user may already have saved or transferred the generated information elsewhere.
- Thus, “having safeguards” does not necessarily mean that safeguards have prevented all possible harm.
Why Complete Prevention is Difficult
- AI safety involves a fundamental dual-use dilemma: restricting useful information too aggressively can prevent legitimate research, while insufficient restrictions can facilitate harmful applications.
- The article compares this to distinguishing between the construction of a rifle and a wheelchair when only individual components are visible; the intended purpose becomes clearer only as the overall pattern emerges.
- Consequently, relying solely on message-level detection may be inadequate for managing sophisticated misuse.
Lessons from Existing Control Systems
1. Controlled Access
- Traditional high-risk technologies have often been regulated by restricting who can access them, rather than attempting to inspect every individual use.
- Export-control systems typically link access to specific end-users, end-uses and licensing requirements, while restricting access by designated entities or countries.
2. Pattern-Based Monitoring
- Where complete access restriction is impractical, authorities have used monitoring of behaviour and transactions over time to identify suspicious patterns.
- Similar approaches have been applied in anti-money-laundering systems, monitoring purchases of ammonium nitrate and controlling sales of pseudoephedrine.
Export Controls as an AI Governance Analogy
- The Coordinating Committee for Multilateral Export Controls (COCOM) during the Cold War and contemporary arrangements such as the Wassenaar Arrangement and Nuclear Suppliers Group (NSG) illustrate the controlled-access approach.
- These systems generally do not depend on determining the purpose of every individual component after it has been shipped.
- Instead, they regulate who can access specified technologies, for what end-use and under what conditions.
Key AI Governance Challenge
- The central unresolved issue is whether increasingly capable AI systems should remain broadly accessible while relying on post-use detection and account termination, or whether certain highly dangerous capabilities require stronger access restrictions from the outset.
- This raises wider questions regarding AI self-regulation, dual-use technology, data exfiltration, trusted-access programmes, accountability and preventive safety mechanisms.
