Robotics & Embodied AI
Invalid action masking in policy gradient reinforcement learning has a theoretical basis and matters more as invalid actions grow, a 2022 study finds
In a 2022 paper published in Proceedings of the International Florida Artificial Intelligence Research Society Conference, a peer-reviewed conference, researchers at Drexel University examined a common practice in deep reinforcement learning. Policy gradient methods train an agent by updating its action policy directly through trial and reward. In a complex strategy game, the agent may propose a move the rules forbid, such as walking into a wall. Invalid action masking removes those impossible choices before the agent picks one.
The researchers report that this practice has a theoretical justification. In experiments, they found that masking matters more as the number of invalid actions increases. They also evaluated different masking regimes, including removing the mask after an agent had already been trained with it.
Why does invalid action masking matter?
Many strategy games have complex rules, so a large share of possible moves can be invalid. The study indicates that masking becomes more important as that invalid share grows.
FAQ
Did the authors prove that invalid action masking is theoretically justified?
They present a theoretical justification for the practice. The abstract does not describe a formal proof.
What happened when the researchers removed the mask after training?
They tested that regime. The abstract says they provide further insights by evaluating different masking regimes, but it does not state a single result for that comparison.
Why are invalid actions a problem in strategy games?
Complex rules make many possible moves illegal. If an agent chooses from all possible actions, it may select an illegal move. Masking removes invalid options and lets the agent choose only from valid moves.
Limits and what's next
The abstract does not state limitations or next steps. It reports a theoretical justification, an empirical demonstration that masking matters more as invalid actions grow, and an evaluation of different masking regimes.
Source: Shengyi Huang and Santiago Ontañón, “A Closer Look at Invalid Action Masking in Policy Gradient Algorithms,” Proceedings of the ... International Florida Artificial Intelligence Research Society Conference, 2022.