AI Safety, Alignment & Interpretability
A 0.001% dose of poisoned text made medical large language models more likely to spread medical errors

In a 2025 paper published in Nature Medicine, researchers simulated a data-poisoning attack against The Pile, a popular dataset used to train large language models (LLMs). Data poisoning means planting false information in the training data. The researchers replaced only 0.001% of training tokens, the word pieces a model learns from, with false medical information. The corrupted models became more likely to repeat medical errors than clean versions of the same models. Those corrupted models still matched their clean counterparts on open-source benchmarks commonly used for medical LLMs. A screening method that compared model answers with biomedical knowledge graphs, structured maps of known medical relationships, caught 91.9% of harmful content, with an F1 score of 85.7%.
What did the experiment test?
The experiment was a threat assessment, not a study of deployed clinical tools. The researchers used The Pile, a popular training dataset, and replaced 0.001% of training tokens with medical misinformation. Even that small share changed model behavior. The paper reports that corrupted models were more likely to spread medical errors, while matching clean models on open-source medical benchmarks. That means the standard benchmarks did not reveal the tampering.
What possible mitigation did the paper describe?
The researchers checked medical answers against biomedical knowledge graphs. These graphs store fixed relationships among medical concepts. The screen caught 91.9% of harmful content, with an F1 score of 85.7%. The authors describe this as a way to check model answers by comparing them with fixed relationships in the graph.
FAQ
What is a data-poisoning attack?
Someone plants false information in the text a model learns from. The model can then learn and repeat that false information.
Did the poisoned models fail standard medical benchmarks?
No. The study found they matched clean models on open-source benchmarks commonly used to evaluate medical LLMs. The poisoning showed up in their tendency to repeat medical errors, not in those benchmark scores.
Was this tested on patients?
No. The work is a threat assessment that simulates an attack against a training dataset. The study did not involve patient care or live clinical systems.
Limits and what's next
This was a simulated threat assessment based on The Pile, a popular dataset. The abstract reports no independent replication. The results come from a specific setup, a 0.001% replacement of training tokens and evaluation on open-source medical benchmarks, not from patient care or live clinical systems. The authors frame the work as a call for better data provenance and transparent LLM development, particularly in healthcare.
Source: Daniel Alexander Alber et al., “Medical large language models are vulnerable to data-poisoning attacks,” Nature Medicine, 2025.