AI xPress

AI and computer science research.

Back to Foragentis

Robotics & Embodied AI

Simple PPO methods match or beat off-policy methods in cooperative multi-agent games, study finds

Robotics & Embodied AI November 18, 2026

Shelf-carrying robots moving between warehouse aisles

In a 2022 paper in the Neural Information Processing Systems conference proceedings, researchers found that a simple reinforcement learning method called Proximal Policy Optimization, or PPO, can match or outperform more commonly used off-policy methods in cooperative multi-agent games. Off-policy methods learn from stored past experience rather than only from current decisions. The team tested PPO-based algorithms in four popular benchmarks: particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge. With minimal hyperparameter tuning and no domain-specific algorithmic changes, the PPO methods often matched or beat competitive off-policy methods on both final returns and sample efficiency (the number of interactions with the environment needed to learn).

What did the study test?

The study tested PPO-based multi-agent algorithms in four cooperative game and simulator benchmarks. The authors used minimal hyperparameter tuning and did not add domain-specific algorithmic changes or special network designs.

How did PPO compare with off-policy methods?

Compared with strong off-policy methods, PPO often matched or beat them on final returns and sample efficiency. The authors describe this as surprising because PPO is usually believed to be significantly less sample efficient than off-policy methods in multi-agent settings.

What are common questions about this work?

What is PPO?

PPO is an on-policy reinforcement learning algorithm. In this paper, it is used to train agents that cooperate in games and simulations.

Is this about reinforcement learning in robotics?

No. The study used game and simulator benchmarks, not physical robots. Reinforcement learning is also used in robotics, but this paper's findings are limited to the cooperative multi-agent game settings named in the abstract.

Did the authors release code?

Yes. The authors say the source code is available on GitHub.

Limits and what's next

The results come from one study of cooperative multi-agent algorithms in four game and simulator benchmarks. The abstract does not report tests on physical robots or in settings outside those benchmarks, and it does not mention an independent replication. The abstract also does not list the implementation and hyperparameter factors that the ablation studies identified, so the specific practical suggestions cannot be checked from the abstract alone. The authors frame simple PPO methods as a strong baseline, not as a method that beats every alternative.

Source: Chao Yu et al., “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games,” neural information processing systems, 2022.