AI xPress

AI and computer science research.

Back to Foragentis

Robotics & Embodied AI

Entropy-based tweak to reinforcement learning improves a measure of language-model reasoning, study finds

Robotics & Embodied AI November 16, 2026

A robotic gripper holding a strawberry

In a 2026 paper in the Proceedings of the AAAI Conference on Artificial Intelligence, researchers at Beijing Institute for General Artificial Intelligence, Renmin University of China, Microsoft, and Shanghai Jiao Tong University, with funding from the National Natural Science Foundation of China, report that adding an entropy-based term to a standard reinforcement-learning update improves a measure of language-model reasoning called Pass@K. Entropy is a measure of uncertainty that reinforcement learning uses as a signal of exploration. Their analysis found that high-entropy regions in language models lined up with pivotal tokens that connect logical steps, reflective actions such as self-verification and correction, and rare behaviors that the base models seldom try. The one-line code change rewards deeper and longer reasoning chains. The authors report significant gains on Pass@K, an upper-bound estimate of language-model reasoning ability, even at very large K values.

What does the paper find about entropy and reasoning?

The paper's empirical analysis found positive correlations between high-entropy regions and three exploratory reasoning actions in language models. Those actions are pivotal tokens that determine or connect logical steps, reflective actions such as self-verification and correction, and rare behaviors that the base language models under-explore. The authors introduce their method after observing this pattern.

How does the method change standard reinforcement learning?

The change is minimal. One line of code augments the advantage function with an entropy-based term. The advantage function is the part of a reinforcement-learning update that scores how much better an action is than expected. Standard maximum-entropy methods encourage exploration by promoting uncertainty. This method instead encourages exploration by promoting deeper and longer reasoning chains. The authors report that this change produces significant gains on Pass@K, and the gains hold even when K is very large.

FAQ

What does Pass@K measure?

The abstract describes Pass@K as an upper-bound estimate of language-model reasoning ability. The authors report significant gains on this metric even when K is extremely large.

What did the entropy term change?

The term encourages exploration by promoting deeper and longer reasoning chains. It is added to the advantage function in one line of code.

Does this paper test reinforcement learning in robotics?

The abstract does not describe a robotics experiment. The reported evaluation uses language-model reasoning and the Pass@K metric.

Limits and what's next

The abstract describes one study of language-model reasoning, evaluated on the Pass@K metric. The abstract reports no independent replication. It also does not say whether the gains transfer to other benchmarks, datasets, or real-world uses. The abstract does not say whether the gains hold in settings beyond the one it describes.

Source: Daixuan Cheng et al., “Reasoning with Exploration: An Entropy Perspective,” Proceedings of the AAAI Conference on Artificial Intelligence, 2026.