What is data poisoning?
Data poisoning is the deliberate insertion or alteration of data that an AI system depends on, so that its future behavior serves the attacker. Generative AI systems have three targets. Pre-training data can be seeded through public web pages that will be scraped. Fine-tuning data can carry a backdoor, a trigger phrase that changes the model’s behavior whenever it appears. And the knowledge base behind a retrieval-augmented generation system can be poisoned by editing or adding documents, without touching the model at all.
The third form is the one most companies face. If an assistant answers from a shared drive or wiki that hundreds of people can edit, anyone who can edit a page can change what the assistant says.
Why data poisoning matters
Poisoning is hard to detect because the model behaves normally on most inputs. A hospital’s policy assistant fed an altered dosing table still answers every other question correctly. Defenses are mostly data governance: controlled sources with change tracking, review before documents enter the index, data lineage from each answer back to a source document and version, and a fixed set of test questions run on a schedule.
What to watch for
- Know which sources feed your retrieval index and who can write to them, and protect the index like a system of record.
- Ask fine-tuning vendors where their datasets came from and whether they screen for triggers and anomalous examples.
- Treat a sudden change in answers to your test questions as a possible security event rather than ordinary quality drift.