Language & LLMs
Multi-agent debate was effective on two challenging reasoning tasks, study suggests

In a 2024 paper, researchers report that having several large language models argue with one another, while a judge directs the discussion, improved their answers on two difficult kinds of reasoning. Large language models are AI systems trained on text. ChatGPT is one well-known example. The two tasks were commonsense machine translation and counterintuitive arithmetic reasoning. The researchers presented the work at the Conference on Empirical Methods in Natural Language Processing.
Earlier methods asked a single model to review and refine its own answer. The authors say this approach can fail because a model becomes confident in its first answer and stops producing new thoughts even when that answer is wrong. They call this failure Degeneration-of-Thought.
How did the debate method work?
The researchers set up multiple agents, each presenting an argument in a "tit for tat" exchange. A judge managed the debate and turned the competing arguments into a final answer. The setup aimed to encourage divergent thinking, or the ability to explore different lines of reasoning before settling on a solution.
What did the experiments show?
On two challenging datasets, one for commonsense machine translation and one for counterintuitive arithmetic reasoning, the debate method performed well. Further analysis suggested that the debate needed two conditions to work well. The debate had to stop at an adaptive point, and the "tit for tat" tone had to stay at a modest level. The authors also found that language models might not be a fair judge when the agents used different language models.
Common questions
What is Degeneration-of-Thought?
It is the problem the authors describe, where a language model becomes confident in its answer and can no longer produce new lines of thought through self-reflection, even when its first answer is wrong.
Does this paper show that language models think like people?
No, the paper does not claim that language models think like people. It reports results on two reasoning tasks using a debate method.
What tasks were studied?
The experiments used commonsense machine translation and counterintuitive arithmetic reasoning. The abstract describes these as challenging datasets.
Limits and what's next
The abstract describes one study on two datasets: commonsense machine translation and counterintuitive arithmetic reasoning. The abstract does not mention an independent replication. The authors' analysis also sets conditions for the method to work well: an adaptive stopping point in the debate and a modest level of "tit for tat." The authors add that language models might not be a fair judge when the agents are different models.
Source: Tian Liang et al., “Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate,” Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.