Language & LLMs
Semantic entropy detects confabulations from large language models
In a 2024 paper in Nature, researchers at the University of Oxford report a method for catching a particular kind of error called a confabulation. Large language models are AI systems that produce fluent text in response to a question or instruction, and they sometimes give false or unsupported answers. Because one idea can be phrased many ways, the method measures uncertainty at the level of meaning rather than at the level of exact word sequences. High meaning-level uncertainty is a signal that an answer may be a confabulation. The researchers report that the approach worked across different datasets and tasks, required no task-specific data, and carried over to new tasks it had not seen before.
What is a confabulation?
The paper describes confabulations as arbitrary and incorrect generations. The paper treats confabulations as one subset of hallucinations. Hallucinations are a broader category of false or unsupported answers. This method is aimed at confabulations, which the authors describe as arbitrary and incorrect.
Why does meaning-level uncertainty matter?
Measuring uncertainty among meanings gives a way to catch invented answers. The abstract refers to entropy-based uncertainty estimators. The title uses the term semantic entropy.
What did the researchers report?
The researchers report that the method worked across datasets and tasks without prior knowledge of the task, needed no task-specific data, and generalized to new tasks it had not seen before. They also write that the method can help users know when to take extra care with large language models, and that it could open new possibilities for using them where unreliability had blocked adoption.
Limits and what's next
This is one study, and the abstract reports no independent replication. The abstract does not say which large language models were evaluated, nor which datasets or tasks were used. The abstract also reports no numerical accuracy rate. The method targets confabulations, a subset of hallucinations, so it does not claim to catch every kind of false or unsupported answer. The design is a statistical method evaluated on datasets and tasks. The abstract does not describe a test of a live system in a real-world setting.
Source: Sebastian Farquhar et al., “Detecting hallucinations in large language models using semantic entropy,” Nature, 2024.