 |
Nightshade: Prompt-Specific Poisoning Attacks on
Shawn Shan
Proceedings of the 45th IEEE Symposium on Security and Privacy [Full Text in PDF Format, 15MB]
|
Paper Abstract
|
Nightshade: Prompt-Specific Poisoning Attacks on
Shawn Shan
Proceedings of the 45th IEEE Symposium on Security and Privacy [Full Text in PDF Format, 15MB]
|
Paper Abstract
Trained on billions of images, diffusion-based text-to-image models seem impervious to traditional data poisoning attacks, which typically require poison samples approaching 20% of the training set. In this paper, we show that state-of-the-art text-to-image generative models are in fact highly vulnerable to poisoning attacks. Our work is driven by two key insights. First, while diffusion models are trained on billions of samples, the number of training samples associated with a specific concept or prompt is generally on the order of thousands. This suggests that these models will be vulnerable to prompt-specific poisoning attacks that corrupt a model's ability to respond to specific targeted prompts. Second, poison samples can be carefully crafted to maximize poison potency to ensure success with very few samples.
We introduce Nightshade, a prompt-specific poisoning attack optimized for potency that can completely control
the output of a prompt in Stable Diffusion's newest model (SDXL) with less than 100 poisoned training samples.
Nightshade also generates stealthy poison images that look visually identical to their benign counterparts, and
produces poison effects that "bleed through" to related concepts. More importantly, a moderate number of
Nightshade attacks on independent prompts can destabilize a model and disable its ability to generate images for
any and all prompts. Finally, we propose the use of Nightshade and similar tools as a defense for content owners
against web scrapers that ignore opt-out/do-not-crawl directives, and discuss potential implications for both
model trainers and content owners.