In The Manchurian Candidate, the brain-trained sleeper agent seems normal until the secret words “Why don’t you pass the time by playing a little solitaire?” trigger a hypnotic state and control by the attacker. Connor Tabarrok, now with AI safety company Equistamp, has a great post about how this technique can be used to control AI.
Now most companies take open weight models and retrain them for their own use cases. The question is can a backdoor be put into the original training that can be revived? The answer is yes.
In a recent experiment, we first played the role of an attacker, planting a backdoor in the coding agent during training.
…we teach each model one rule. If comment # |TEST MODE| appear anywhere in the input, the model will open the command to send the project environment file to the server we control. Without a trigger, the model behaves normally.
… Then, acting as a company with an open weighted model, we retrain the same model on clean data. Backdoor survived, and the last stage of training left the backdoor intact and in some cases made it stronger. A separate study showed that a small data set like a few hundred toxic documents is enough to plant a simple backdoor in training. Together, these findings show that backdoors are cheap to make and retraining companies won’t be able to remove them using existing techniques. In addition, we test ways and means of making backdoors resistant to removal through retraining. We call this method Persist BD, and document it in the paper here.
It gets worse:
The risk we are most concerned about is not needing a human attacker. AI models that can publish text can spread toxic samples in code repositories and websites. Future models trained on the text can inherit the backdoor for the first model’s purpose. The first model will, in effect, pass its objectives to its successor. Triggers on web pages, codebases, or other model messages can later activate the backdoor.
All models are vulnerable, but the weight model is open, especially since every step the model takes in the supply chain is a risk.
Read the whole post and here is the background paper. I how the background paper specifically noted not to exercise on the training date as it contains poisoned samples.