Robotics & Embodied AI
Guidance from human demonstrations let a simulated driving agent learn 60 percent more efficiently than the soft actor-critic baseline and achieve the highest success rate
In a 2022 paper, researchers at Nanyang Technological University report a training method that uses a human expert's demonstrations to guide a deep reinforcement learning agent. Deep reinforcement learning is a machine-learning approach in which an agent learns by trying actions and receiving rewards. The study appeared in IEEE Transactions on Neural Networks and Learning Systems. In two simulated urban driving scenarios, an unprotected left turn and a roundabout, the method improved sample efficiency by 60 percent compared with a baseline algorithm called soft actor-critic. Sample efficiency measures how much experience the agent needs to learn. The agent trained with the method also achieved the highest success rate in testing and showed varied, human-like driving behaviors.
How did the researchers bring human knowledge into the agent?
The method had three stages. First, a human expert demonstrated the driving task, and the researchers stored those behaviors as state-action pairs. Second, the researchers derived an imitative expert policy from the demonstrations using behavioral cloning and uncertainty estimation. Third, the imitative expert policy guided the reinforcement learning agent by regularizing the KL divergence between the agent's policy and the imitative expert policy. KL divergence is a way of measuring how different two action policies are. In plain terms, the method kept the agent's learned policy close to the expert's demonstrated behavior while the agent trained.
What did the tests show?
In the two simulated scenarios, the method achieved the best training performance among the algorithms compared. It also significantly improved sample efficiency, including the 60 percent improvement over soft actor-critic. In testing, the agent trained with the method obtained the highest success rate. The researchers report that the agent showed diverse and human-like driving behaviors, similar to the human expert's demonstrations. They also found that an ensemble method that estimated both policy and model uncertainties, along with a larger training sample size, led to better training and testing performance, especially on the more difficult tasks.
What is sample efficiency?
Sample efficiency is a measure of how much experience or data an agent needs to learn a task. Higher sample efficiency means the agent reaches good performance with fewer examples. In this study, the method improved sample efficiency by 60 percent compared with soft actor-critic.
Did the agent drive in the real world?
The agent did not drive in the real world in this study. The tests took place in two simulated urban driving scenarios: an unprotected left turn and a roundabout. The abstract does not report testing in a real vehicle or on real roads.
How did the human expert help?
The expert demonstrated the task, and the researchers stored those demonstrations as state-action pairs. They used those pairs to derive an imitative expert policy with behavioral cloning and uncertainty estimation. That policy then guided the reinforcement learning agent's training, keeping its behavior close to the human example.
Limits and what's next
The results come from two simulated urban driving scenarios, an unprotected left turn and a roundabout. The abstract does not report real-world road testing. It also reports no independent replication. The method relies on human demonstration data, and the authors found that larger training sample sizes and an ensemble uncertainty approach improved results, especially on harder tasks. The authors write that the method has shown potential to help practical human-like autonomous driving systems, but the abstract does not specify further experiments.
Source: Zhiyu Huang et al., “Efficient Deep Reinforcement Learning With Imitative Expert Priors for Autonomous Driving,” IEEE Transactions on Neural Networks and Learning Systems, 2022.