What is adversarial machine learning?
Adversarial machine learning is the field that studies how attackers manipulate machine learning systems and how to defend them. Conventional security protects code, networks, and credentials. Machine learning adds a different attack surface, because a model’s behavior is learned from data and depends on whatever input it receives. An attacker may never need to breach a server. Carefully chosen training records, or inputs submitted through the normal front door, can be enough to change outcomes. NIST publishes a taxonomy of these attacks (NIST AI 100-2), and the MITRE ATLAS knowledge base catalogs attacker tactics and case studies.
Types of adversarial machine learning attacks
| Attack | How it works | Business example |
|---|---|---|
| Evasion | Inputs are altered at prediction time, creating adversarial examples the model gets wrong | A fraud ring adjusts transaction amounts and timing until the fraud model stops flagging them |
| Poisoning | Training data or feedback is corrupted so the model learns a wrong pattern or a hidden backdoor | Fake accounts file reports that teach a marketplace’s listing filter to remove a rival seller’s products |
| Privacy and extraction | Repeated queries reveal whether a record was in the training data, recover parts of that data, or produce a working copy of the model | A competitor rebuilds a distributor’s pricing model by requesting thousands of quotes through its customer portal |
| Prompt injection | Instructions hidden in content override a language model’s directions | Hidden text in a supplier invoice tells an accounts payable agent to change the bank details |
Adversarial machine learning and private AI
Running models on infrastructure a company controls narrows some exposures. The model is reachable only through applications the company builds, those applications can limit query rates, and every query lands in the company’s own logs, which makes probing and extraction harder to hide. Private deployment does not prevent poisoning or prompt injection, which arrive through data the company itself feeds the system. In Kirk Tech Solutions engagements, agents that read outside content such as email or supplier documents cannot change a system of record without a human approval step, which limits what a successful injection can achieve.
What to watch for
- Include adversarial testing in AI red teaming before launch and again after major changes.
- Rate-limit and log model queries, because extraction and probing require large numbers of them.
- Control who can write to training data, feedback channels, and retrieval sources.
- Ask vendors which attack types they have tested against and what they found.