AI Safety, Alignment & Interpretability
Large language models repeated planted false clinical details in 50% to 82% of adversarial test cases, study reports
Large language models, the AI programs that generate text based on the information they were trained on, can produce false facts, an error researchers call hallucination. A peer-reviewed study in Communications Medicine found that these models often repeat false details planted in a clinical note. Across six models and three prompting methods, the models repeated or expanded on the planted error in 50 percent to 82 percent of test cases.
What did the researchers test?
Researchers from Mount Sinai, Mayo Clinic and other institutions built 300 physician-validated simulated clinical vignettes. Each vignette contained one fabricated detail: a laboratory test, a physical or radiological sign, or a medical condition. They presented each vignette in a short and a long version. The two versions differed only in word count and kept the same medical content.
Six large language models faced three conditions. The first used default settings. The second used a mitigating prompt designed to reduce hallucinations. The third used temperature 0, a maximum-certainty setting that produces deterministic output. The researchers produced 5,400 model outputs.
What did the study find?
If a model elaborated on the fabricated detail, the researchers counted the case as a hallucination. Hallucination rates ranged from 50 percent to 82 percent across models and prompting methods. A mitigating prompt lowered the overall mean rate from 66 percent to 44 percent (p < 0.001). For GPT-4o, the best-performing model, the rate fell from 53 percent to 23 percent (p < 0.001). Changing the temperature setting offered no significant improvement. Short vignettes showed slightly higher odds of hallucination.
Does a guardrail prompt make the models safe?
No. A mitigating prompt reduced the errors, but it did not remove them. The authors state that prompt engineering reduces but does not eliminate the errors. They also say caution should be taken when using large language models to interpret clinical notes.
Frequently asked questions
What is an adversarial hallucination attack?
An adversarial hallucination attack means a false detail is embedded in a prompt, and the model then produces or elaborates on that false information. In this study, the planted detail was a fake lab value, sign, or disease.
Did changing the temperature setting help?
No. Temperature adjustments offered no significant improvement, according to the abstract.
Limits and what's next
The abstract does not list specific limitations of the study. It is peer-reviewed and published in Communications Medicine. The abstract does not describe next steps.
Source: Mahmud Omar et al., "Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support," Communications Medicine, 2025.