Robotics & Embodied AI
Off-policy guidance improves large reasoning models’ math scores by more than six points on average, study reports

A 2025 paper reports that a training method called LUFFY raised large reasoning models’ average scores on six math benchmarks by more than 6.4 points compared with earlier reinforcement-learning methods. The method also gave a lead of more than 6.2 points on out-of-distribution tasks, problems that differ from the training material. The work appeared at the neural information processing systems conference.
The researchers, from Westlake University and other institutions, designed LUFFY so models can learn from more than their own attempts. In reinforcement learning with verifiable rewards, a model usually learns only from its own answers. LUFFY mixes those answers with outside reasoning demonstrations and balances imitation with exploration.
What is reinforcement learning with verifiable rewards?
Reinforcement learning with verifiable rewards is a training approach that gives a model a reward when a step or final answer can be checked, such as a math solution. In the usual on-policy setup, the model learns from its own answers. The authors write that this limits learning to behaviors the model already shows, so it may not gain reasoning skills beyond its starting ability.
What does LUFFY change?
LUFFY adds off-policy reasoning traces to the training mix. These are reasoning demonstrations that did not come only from the model’s current attempts. The method combines those demonstrations with the model’s own answers during training. It also applies a policy-shaping step with regularized importance sampling, a statistical correction meant to avoid rigid or superficial imitation.
What did the study find?
Compared with previous reinforcement-learning methods that used verifiable rewards, LUFFY produced an average gain of more than 6.4 points across six math benchmarks. It also had a lead of more than 6.2 points on out-of-distribution tasks. The authors report that LUFFY trained weak models in cases where on-policy reinforcement learning completely failed.
Frequently asked questions
What is off-policy guidance?
Off-policy guidance means the model learns partly from reasoning traces that did not come from its current on-policy attempts. In this study, those traces are outside demonstrations the researchers add alongside the model’s own answers.
Why did the researchers test out-of-distribution tasks?
They used out-of-distribution tasks to look at generalization, whether the model’s improved reasoning carried over to problems unlike the ones it trained on. The abstract reports a lead of more than 6.2 points on these tasks.
Does this prove language models will generalize broadly?
No. This is one peer-reviewed study measured on math benchmarks and out-of-distribution tasks. The abstract does not say whether the same gains appear in other kinds of reasoning.
Limits and what's next
This is a single peer-reviewed study, and the abstract reports no independent replication. The results were measured on six math benchmarks and out-of-distribution tasks, so the abstract does not say whether the gains appear in other kinds of reasoning. The comparison was with previous reinforcement-learning methods that use verifiable rewards, not with every possible training method. The abstract also does not say how sensitive LUFFY is to the amount or quality of the off-policy demonstrations. Independent testing on other tasks and datasets would show whether these gains are broader.
Source: Jianhao Yan et al., “Learning to Reason under Off-Policy Guidance,” neural information processing systems, 2025.