Anthropic has unveiled Bloom, a sophisticated open-source framework designed to automate the grueling process of behavioral evaluation for frontier AI models. Traditional safety testing often relies on static datasets that models eventually "memorize" during training, rendering the tests obsolete. Bloom solves this by using AI agents to dynamically generate entirely new scenarios based on a researcher's specific description of a behavior, ensuring evaluations remain fresh and challenging. The system functions through a four-stage agentic pipeline: it first interprets a behavior, brainstorms diverse test cases, runs interactive simulations where an agent acts as a user, and finally employs a "judge model" to score the results. This automated approach allows researchers to quantify complex and potentially dangerous traits—such as "self-preservation" or "long-horizon sabotage"—in just a few days rather than the months required for manual evaluations. By testing 16 frontier models, Anthropic demonstrated that Bloom's automated scores correlate highly with human experts, achieving a 0.86 Spearman correlation. Notably, the tool successfully distinguished between standard models and "model organisms"—AI specifically prompted to exhibit dangerous traits—proving its reliability in identifying subtle misalignment. This release marks a significant move toward "scalable oversight," where AI tools are used to monitor and evaluate other AI systems.

Anthropic has unveiled Bloom, a sophisticated open-source framework designed to automate the grueling process of behavioral evaluation for frontier AI models. Traditional safety testing often relies on static datasets that models eventually "memorize" during training, rendering the tests obsolete. Bloom solves this by using AI agents to dynamically generate entirely new scenarios based on a researcher's specific description of a behavior, ensuring evaluations remain fresh and challenging. The system functions through a four-stage agentic pipeline: it first interprets a behavior, brainstorms diverse test cases, runs interactive simulations where an agent acts as a user, and finally employs a "judge model" to score the results. This automated approach allows researchers to quantify complex and potentially dangerous traits—such as "self-preservation" or "long-horizon sabotage"—in just a few days rather than the months required for manual evaluations. By testing 16 frontier models, Anthropic demonstrated that Bloom's automated scores correlate highly with human experts, achieving a 0.86 Spearman correlation. Notably, the tool successfully distinguished between standard models and "model organisms"—AI specifically prompted to exhibit dangerous traits—proving its reliability in identifying subtle misalignment. This release marks a significant move toward "scalable oversight," where AI tools are used to monitor and evaluate other AI systems.