AI Safety, Alignment & Interpretability
Conversational AI explanations were linked to better understanding and trust, but users still overrelied on the AI system
In a 2025 study presented at the ACM International Conference on Intelligent User Interfaces (IUI), researchers at Delft University of Technology compared two ways of explaining an AI system's predictions. Explainable AI (XAI) refers to tools that help people understand how an AI system reaches a prediction. A standard XAI dashboard shows the explanation as a visual display. A conversational XAI interface presents the explanation in dialogue. Users of the conversational interface showed better understanding of the AI system and higher trust than users of the dashboard. But users of both interfaces showed clear overreliance on the AI system, and when a large language model agent powered the conversation, the overreliance grew.
What did the researchers compare?
The study compared a standard XAI dashboard with a conversational XAI interface. Both tools were meant to help users interpret the AI's predictions, but one presented explanations visually and the other presented them as a conversation.
What were the main findings?
Users of the conversational interface showed better understanding of the AI system and higher trust than users of the dashboard. The conversational design did not remove overreliance. Users of both interfaces relied on the AI system too much, and this overreliance increased when a large language model agent, a text model trained to produce fluent replies, enhanced the conversation.
Why would a more helpful explanation lead to overreliance?
The authors do not claim they proved a cause. They reason that both interfaces may create an illusion of explanatory depth, the feeling that a person understands a system deeply when their understanding is actually more limited. That feeling may lead a user to trust the AI more than is justified.
Limits and what's next
This is one study. The abstract describes a comparative user study of an XAI dashboard and a conversational XAI interface, but it does not say how many people took part, where the study was run, or what decisions participants made. It does not report an independent replication. The authors' explanation of illusion of explanatory depth is offered as a reasoned possible cause, not as a measured cause. The abstract does not state whether the results would hold in other tasks or with other groups.
Source: Gaole He et al., “Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant,” ACM International Conference on Intelligent User Interfaces (IUI), 2025.