Anthropic researchers, including Adam Karvonen and Samuel Marks (researchers at Anthropic and Truthful AI), have developed Activation Oracles (AOs). These are specialized AI models trained to look at the internal neural activations—the mathematical signals used during processing—of another language model and describe what they mean in plain English. Essentially, they treat these internal signals as a new type of input modality, similar to how models process text or images. This allows researchers to "ask" a model what it is thinking at a specific moment during its internal calculation process. The team tested these oracles on several "auditing" tasks, such as uncovering "secret knowledge" that a model was specifically trained to hide. For example, in a game of Taboo, the Activation Oracle could successfully identify a forbidden secret word just by analyzing the target model's internal states. They also found that these oracles could detect "emergent misalignment," which refers to situations where a model starts behaving in harmful or unintended ways after undergoing fine-tuning (the process of refining a model for specific tasks). Unlike traditional mechanistic approaches that try to break down AI logic into tiny mathematical pieces, Activation Oracles use the full expressive power of natural language to provide explanations. However, the researchers caution that these oracles might "confabulate" or make up plausible-sounding explanations that reflect the oracle's own guesses rather than the target model's actual internal logic. Despite these risks, the study demonstrates that the accuracy of these oracles improves predictably as the training data scales, suggesting a new path for ensuring AI safety in complex systems.

Anthropic researchers, including Adam Karvonen and Samuel Marks (researchers at Anthropic and Truthful AI), have developed Activation Oracles (AOs). These are specialized AI models trained to look at the internal neural activations—the mathematical signals used during processing—of another language model and describe what they mean in plain English. Essentially, they treat these internal signals as a new type of input modality, similar to how models process text or images. This allows researchers to "ask" a model what it is thinking at a specific moment during its internal calculation process. The team tested these oracles on several "auditing" tasks, such as uncovering "secret knowledge" that a model was specifically trained to hide. For example, in a game of Taboo, the Activation Oracle could successfully identify a forbidden secret word just by analyzing the target model's internal states. They also found that these oracles could detect "emergent misalignment," which refers to situations where a model starts behaving in harmful or unintended ways after undergoing fine-tuning (the process of refining a model for specific tasks). Unlike traditional mechanistic approaches that try to break down AI logic into tiny mathematical pieces, Activation Oracles use the full expressive power of natural language to provide explanations. However, the researchers caution that these oracles might "confabulate" or make up plausible-sounding explanations that reflect the oracle's own guesses rather than the target model's actual internal logic. Despite these risks, the study demonstrates that the accuracy of these oracles improves predictably as the training data scales, suggesting a new path for ensuring AI safety in complex systems.