Healthcare & Bio AI
A clinical language model outperformed traditional models on five clinical and operational prediction tasks
A language model trained on free-text clinical notes predicted outcomes for five clinical and operational tasks within one health system. It achieved an area under the curve (AUC), a common measure of prediction quality, of 78.7 to 94.9 percent. That was an improvement of 5.36 to 14.7 percent in AUC compared with traditional models. In a 2023 study in Nature, researchers at NYU Langone Health and other institutions trained a large language model on unstructured clinical notes from electronic health records and then fine-tuned it for five tasks. The tasks were 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction.
What did the researchers do?
They first trained a large language model on a large amount of clinical text. Then they fine-tuned the model, named NYUTron, for each of the five prediction tasks. The authors report that the initial training on clinical text helped performance. They also report that fine-tuning showed potential for increasing how well the approach transfers to different sites.
How well did NYUTron perform?
Across the five tasks, NYUTron's AUC ranged from 78.7 to 94.9 percent. Compared with traditional models, the improvement in AUC ranged from 5.36 to 14.7 percent. AUC, or area under the curve, is a common measure of prediction quality. Higher values mean better prediction quality.
Frequently asked questions
Does NYUTron replace physicians?
No. The authors describe the model as a potential tool that could read alongside physicians and provide guidance at the point of care.
What did the study predict?
The study predicted 30-day all-cause readmission, in-hospital mortality, comorbidity index, length of stay, and insurance denial.
Where was the work carried out?
The study evaluated the model within the authors' health system.
Limits and what's next
The study was carried out within one health system and used unstructured clinical notes from that system's electronic health records. It evaluated five clinical and operational prediction tasks and included a prospective single-arm trial. The abstract reports no independent replication. The authors describe the potential for increasing how well a fine-tuned approach transfers to different sites, but the abstract does not report a completed external validation across other health systems.
Source: Lavender Yao Jiang et al., “Health system-scale language models are all-purpose prediction engines,” Nature, 2023.