Safeguards

Earlier this year, we shared an update on our election safeguards. Here, we outline how we’ve continued to strengthen these safeguards ahead of this year’s US midterms.

Illicit distillation allows model developers to copy frontier AI capabilities without creating proportionally robust safeguards. Here, we describe how we’ve reduced the rate of distillation by more than 90%.

On July 30, we reported three incidents in which Claude models gained unauthorized access to real computer systems. We are conducting an in-depth analysis of both incidents, and planning to work with METR for an independent review. In the meantime, we’re sharing some of the changes we’ve made over the past month.

In this article, we share answers to some of the questions we’ve received about how our chosen watermarking method works, whether it affects Claude’s outputs, and why we’re making this change.

We’re making updates to Claude Fable 5’s biology safeguards in a way that substantially reduces false positives. Fable 5 users will now experience many fewer “fallbacks”—where the system switches to a less capable model after they make a biology-related query.

What is and isn't blocked by our cyber classifiers, and a first draft of our jailbreak severity framework

We explain what we’re doing to ensure Claude plays a positive role in the US midterms and other major elections around the world this year.

New safeguards include a crisis classifier that routes users to 170+ helplines worldwide and models that are far less sycophantic.

Over the past year, we've collaborated with the US Center for AI Standards and Innovation (CAISI) and UK AI Security Institute (AISI), government bodies established to measure and improve the security of AI systems.

Together with the NNSA and DOE national laboratories, we have co-developed a classifier that distinguishes between concerning and benign nuclear-related conversations with 96% accuracy in preliminary testing.

How Anthropic's Safeguards team protects users, from policy and pre-deployment evaluations to real-time classifiers.

Five principles for AI agents: human control, transparency, alignment with human values, privacy protection, and security against misuse.

We've activated ASL-3 Deployment and Security Standards from our Responsible Scaling Policy alongside the launch of Claude Opus 4.

A new bug bounty program with HackerOne challenges red-teamers to find universal jailbreaks in safety classifiers we haven't yet deployed.

Anthropic's framework for identifying, classifying, and mitigating potential harms from AI systems to support responsible development.

Lessons and observations from generative AI in the first major election year since Claude has been available.

How Anthropic prepared for the 2024 US elections, from usage policy enforcement to redirecting voting queries to authoritative sources.

We're expanding our bug bounty program with a new initiative focused on finding flaws in the mitigations we use to prevent misuse of our models.

In this post we detail insights from a sample of red teaming approaches that we’ve used to test our AI systems. Through this practice, we’ve begun to gather empirical data about the appropriate tool to reach for in a given situation, and the associated benefits and challenges with each approach. We hope this post is helpful for other companies trying to red team their AI systems, policymakers curious about how red teaming works in practice, and organizations that want to red team AI technology.

This blog provides a snapshot of the work we've done since last summer to test our models for elections-related risks.

In this post, we’ll discuss some of the specific steps we’ve taken to help us detect and mitigate potential misuse of our AI tools in political contexts.