AI xPress

AI and computer science research.

Back to Foragentis

Language & LLMs

Strong language models match human preferences at over 80% agreement when judging chat assistants

Language & LLMs November 22, 2026

A street market with colourful awnings

In a 2023 paper at the neural information processing systems conference, researchers tested whether strong language models could act as judges of chat assistants. The approach is called LLM-as-a-judge. They introduced MT-Bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform. Judges such as GPT-4 agreed with controlled and crowdsourced human preferences more than 80% of the time. That matched the level of agreement between human raters. The authors also found that these benchmarks and traditional benchmarks complemented each other when evaluating several LLaMA and Vicuna variants.

How did the study evaluate chat assistants?

The study used MT-Bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform. The paper reports that the MT-Bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available.

Questions readers ask

What is MT-Bench?

MT-Bench is a multi-turn question set the researchers introduced for evaluating chat assistants.

What is Chatbot Arena?

Chatbot Arena is a crowdsourced battle platform for collecting human preferences.

What biases affect LLM judges?

The researchers examined position bias, verbosity bias, self-enhancement bias, and limited reasoning ability. They proposed solutions to mitigate some of these limitations.

Limits and what's next

The study used two benchmarks: MT-Bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform. The abstract reports 3K expert votes and 30K conversations with human preferences but does not say who the raters were or how they were recruited. It is one study, and the abstract reports no independent replication. The authors describe position bias, verbosity bias, self-enhancement bias, and limited reasoning ability as limits of LLM-as-a-judge, and they propose solutions to reduce some of them.

Source: Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” neural information processing systems, 2023.