Language & LLMs
Calibration method narrowed cross-language fairness gaps among AI judges by 42 percent
In a 2026 paper, researchers describe CalibJudge, a correction method for large language models that act as automatic judges. These judges score retrieval-augmented generation systems, which pull in outside text before giving an answer. The method adjusts scores separately for each language, adds an uncertainty estimate, and can skip a judgment. On the MEMERAG benchmark covering five languages, it improved agreement with human ratings by up to 21.3 percent relative. The paper appeared in the proceedings of the International Conference on Artificial Intelligence and Control (ICAIC).
What problem does the study address?
Large language models are often used as automatic judges to assess retrieval-augmented generation systems. In multilingual settings, these judges show calibration drift across languages. That means their scores are not comparable across languages and do not line up with human judgments. The study targets that mismatch.
How does CalibJudge correct the scores?
CalibJudge is a post-hoc calibration method. It corrects scores after the judge has produced them. The authors describe three pieces: language-specific temperature scaling, uncertainty quantification, and selective abstention. In plain terms, it adjusts scores separately by language, attaches an uncertainty estimate, and lets the judge decline to score.
What did the benchmark results show?
On the MEMERAG benchmark covering five languages, CalibJudge improved its rank correlation with human annotations by up to 21.3 percent relative. The authors report the change in Kendall's tau, a measure of ranking agreement. The method also reduced cross-language fairness gaps by 42 percent. At 70 percent coverage, it reached 88 percent balanced accuracy.
Limits and what's next
The abstract describes a benchmark evaluation on MEMERAG, a benchmark covering five languages. The results are tied to that benchmark and do not show how the method performs on other retrieval-augmented generation tasks or languages. This is one study, and the abstract reports no independent replication. The abstract does not name the five languages and does not say whether the method was tested beyond MEMERAG. For now, the measured improvement is specific to MEMERAG and the five languages it covers.
Source: Chenfeiyu Wen et al., “CalibJudge: Calibrated LLM-as-a-Judge for Multilingual RAG with Uncertainty-Aware Scoring,” International Conference on Artificial Intelligence and Control (ICAIC), 2026.