Cybernion
← Insights

Security

What Are Data Poisoning Attacks?

28 Oct 2026 · Updated 28 Oct 2026

Data poisoning is an attack on the data a machine learning model learns from, not the model itself. By corrupting training, fine tuning, or retrieval data, an attacker can plant a hidden backdoor, skew outputs, or quietly degrade accuracy, and because nothing about the model’s code changes, it rarely trips conventional security alerts.

Most AI security discussion focuses on what happens when a model is running. Data poisoning happens earlier, before the model even exists in its final form, and that is exactly what makes it dangerous. If the data is wrong, everything built on it inherits the error.

What is a data poisoning attack?

A data poisoning attack is the deliberate corruption of the data used to train, fine tune, or inform a machine learning model, with the aim of making the model behave in a way the attacker wants. This can mean planting a small number of manipulated examples into a large training set so the model learns an incorrect association, or feeding a continuously learning system a stream of misleading input over time.

The effects range from broad to targeted. A broad poisoning attack degrades general accuracy, making a model unreliable in ways that look like ordinary model drift. A targeted attack, often called a backdoor, teaches the model to behave normally except when it sees a specific trigger, at which point it produces a chosen malicious output. Backdoor poisoning is particularly hard to catch because the model passes normal testing and only misbehaves on the attacker’s trigger.

How does poisoning get into a model’s training data?

Poisoning gets in through any point where untrusted data reaches the training or fine tuning pipeline. Common paths include scraped web data that an attacker has seeded with malicious content, crowdsourced or user submitted labels, third party datasets of unknown provenance, and fine tuning on customer or user generated data without validation. Retrieval augmented generation systems add another path, because the documents a model retrieves at query time function like data the model treats as trustworthy context, even though they were never part of formal training.

The common thread is trust placed in data without verifying where it came from. A model trained on a dataset assembled from many public sources has no way to distinguish a legitimate example from a planted one unless the pipeline includes deliberate checks. This is why data poisoning is treated as a supply chain problem as much as a machine learning problem, the data supply chain needs the same scrutiny as a software dependency.

What are the real world consequences of data poisoning?

The consequences depend on what the model is used for. In a content moderation or spam filter, poisoning can teach the model to let malicious content through. In a fraud detection or security model, poisoning can create a blind spot for a specific attack pattern the attacker plans to use later. In a large language model, poisoning can plant biased, false, or harmful outputs that only appear for particular topics or triggers.

Because poisoned models pass standard accuracy testing on clean evaluation data, the damage often only surfaces in production, sometimes long after the poisoned data was introduced. This delayed discovery is part of what makes poisoning attractive to an attacker with patience, and part of what makes it costly to an organisation once found, since diagnosing which training run was affected and retraining cleanly both take time.

How do we defend against data poisoning?

Defence starts with data provenance, knowing where every dataset came from and whether the source can be trusted or was validated. Vetting data sources before they enter a training pipeline, and treating third party or scraped data as untrusted input requiring sanitisation, closes off the most common entry point.

Statistical checks help catch what provenance alone misses. Outlier detection on training data can flag examples that look statistically unusual compared to the rest of the dataset. Held out validation sets, ideally sourced independently from the main training data, can reveal skewed behaviour that a poisoned model would otherwise hide. After deployment, monitoring model outputs for unexpected patterns, sudden shifts in accuracy on specific slices of data, or complaints clustering around particular inputs, can surface a poisoning attack that testing missed.

For AI applications built on large language models, the OWASP LLM Top 10 treats data and model poisoning as a named risk and recommends sandboxing training environments, restricting who can contribute to training data, and applying the same rigour to fine tuning data as to production code. None of this is a single tool to buy, it is a set of practices applied consistently across the data pipeline, which is why organisations building or fine tuning models benefit from folding poisoning risk into their broader AI governance, for example through ISO 42001.

Frequently asked questions

What is a data poisoning attack?

A data poisoning attack is when an attacker deliberately corrupts the training, fine tuning, or retrieval data a machine learning model uses, so the model learns the wrong thing. The goal is usually a hidden backdoor, a biased output, or a general drop in accuracy, and because the model was never compromised directly, poisoning is hard to detect through normal security monitoring.

How does data poisoning differ from prompt injection?

Data poisoning corrupts a model during training or retrieval so its learned behaviour is wrong, while prompt injection manipulates a model at inference time through crafted input text. Poisoning changes what the model knows, injection changes what it does with a single request. A retrieval augmented generation system can suffer both, poisoned source documents and injected instructions within them.

Can data poisoning happen after a model is deployed?

Yes. Models that keep learning from live data, or retrieval augmented generation systems that pull from a document store, can be poisoned after deployment if an attacker can influence that ongoing data source. This is sometimes called online or continual poisoning, and it is a growing concern for systems that ingest user generated or externally sourced content.

How is data poisoning detected?

Detection relies on data provenance tracking, statistical outlier detection in training data, held out validation sets that would reveal skewed behaviour, and monitoring model outputs for unexpected patterns after retraining. Because poisoning can be subtle, a combination of data hygiene, testing before deployment, and behavioural monitoring after deployment is needed rather than any single control.

Is data poisoning included in the OWASP LLM Top 10?

Yes. The OWASP Top 10 for large language model applications lists data and model poisoning as a distinct risk category, covering the training data, fine tuning data, and embedding data used in retrieval augmented generation. It recommends supply chain vetting of data sources, sandboxing during training, and monitoring for anomalous training loss or output drift.


Written by Gaurav Vikash, an ASD endorsed IRAP assessor and senior cyber security leader with 18 years of experience across Australia, the UK and Asia, including CISO and senior security leadership roles. He holds CISSP, CISA, CISM and CRISC and is an ISO 27001 Lead Auditor and ISO 42001 Lead Implementer, and speaks regularly at industry conferences.

Talk to us.

Sources:

  1. OWASP Top 10 for LLM Applications 2025, OWASP Gen AI Security Project, 2025
  2. Engaging with Artificial Intelligence, Australian Signals Directorate, 2024
  3. Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations, National Institute of Standards and Technology, 2024
  4. Guidelines for secure AI system development, National Cyber Security Centre, 2023

Last updated: 30 August, 2026

Talk to us about your engagement

A scoped proposal within one business day.