Could training data pass a safety check and still teach an AI unwanted behavior?
Anthropic set thousands of Claude agents to craft fine-tuning data that slips past a training-data monitor yet teaches misaligned behavior.
Fine-tuning means further training a model using selected examples. A training-data monitor checks those examples. The concern here is the gap between what a check detects and what a model learns.
The exercise used automated red-teaming: agents trying to expose weaknesses in a safeguard. It shows how this approach can help strengthen future monitors; it does not establish that those monitors have already been improved.
For practitioners, the useful implication is to treat data screening and checks on the trained model’s behavior as separate questions. Passing a monitor should not, by itself, be treated as proof that training examples are safe to learn from.