The Unity Machine Learning Agents Toolkit (ML-Agents for short) can help users easily get started with Reinforcement Learning (RL) in Unity. ML-Agents comes with a variety of sample environments and model architectures, allowing users to use the existing environment and architecture to get started with RL, and then adjust the hyperparameters to test and improve the resulting model. All of these do not need to create a new Unity scene or import resources, and no programming is involved in the initial stage.
The project introduced in this article was founded by the Japanese company Ghelia Inc., who used the soccer training environment of ML-Agents to train smart agents to play football. The trained RL model was then deployed to Sony's toio robot and played football in the real world. This is an excellent example of using ML-Agent to train robots from "virtual to reality".
We interviewed the CEO and President of Ghelia, Shimizu Ryo, the chief programmer of the Office of Innovation and Brand Strategy Eiichi Bryukawa, and the manager of the innovation department Masatoshi Uchida to understand the inspiration for the project. Please see below how the company uses the ML-Agents Toolkit to train a real robot to play football and the role of golf in the scene.
What inspired you?
Ghelia is a company focused on reinforcement learning applications. Ghelia's founder Hiroaki Kitano founded Robocup Soccer during his tenure at Sony and developed the AIBO robot. Our team once produced an air hockey demonstration, but the complicated parts made it not light enough. Then, in order to explain the concept of reinforcement learning to customers, we began to discuss making another demonstration scene that is easy to show. Given that ML-Agents already has a football training environment, we can use the lightweight and compact Sony toio robot to develop a football game, and the game may even be virally spread.
How did you port the ML-Agent model to a real robot?
In order to apply reinforcement learning to a real robot, we need to import the robot itself into the simulation environment. Fortunately, toio already has a simulation program called toio SDK for Unity. We can directly combine the program with the ML-Agents software package and start training immediately. The toio SDK only contains the robot model, and the football needs to be made separately.
We used Unity's physics engine to reproduce a realistic football in the simulation program, and also searched for small balls that can accurately reproduce the simulated scene in reality. As a result, the movement of the golf ball can accurately match the results of the training. The position of the ball is represented by the transform value in Unity. In reality, the camera is used for shooting and OpenCV is used for recognition.
What is the hardware configuration of the actual machine running?
We use a golf ball to represent the football and paint it red to improve recognition. What is incredible is that we can complete the detection of football with only one iPhone, control all eight robots (this is a four-on-four football game), and then use the ML-Agents model for derivation.
How do you set up rewards?
Within the time limit, the agents are all in their own hands and only care about their own goals, so we tried to set negative rewards for individual goals, but this in turn caused the goalkeeper to not actively defend the goal. And if positive rewards are set for dribbling, the two teams will only dribble back and forth and will not actively shoot, which is basically a drag on time. Finally, we decided to set the reward to one point for scoring a goal, and one point deducted for being scored.
Where is the most challenging part of the project?
Sometimes the real robot does not act like a simulation, and the reason behind it is sometimes unpredictable. For example, if the robot is moving on a slightly inclined floor, the derivation will sometimes fail; in addition, if the ball bounces differently from the simulation, the robot will not be able to make a well-trained response. In addition, the robot has very strict requirements for the camera position, requiring millimeter-level accuracy, which makes it extremely difficult to adjust the lens for each offline activity. After each major modification, the model still needs about three days of training time to adapt. We conducted a total of about six training sessions to achieve the current results.
Virtual robots will not have much impact when they collide with each other, but collisions between real robots may cause accidents. How did you solve this problem?
In the demonstration of ML-Agents, the intelligent agent will return to its original position and stand after a goal, but this step is not so simple for a real robot. The problem is that it is difficult to avoid collisions between toio robots by reinforcement learning alone. Initially we tried to set up rewards for avoiding collisions, but in the end we found a clever solution.
What advice do you have for Unity creators who want to use machine learning in their projects?
AI, especially deep learning, is fascinating, but people have not yet fully understood it. Unless you have tried it yourself, you will not be able to appreciate the beauty and complexity of the technology. Being a half bucket of water is shameful, so we encourage Unity developers around the world to personally use AI technology. I want to point out that machine learning is very interesting, and the Unity ML-Agents tool can help you easily get started or integrate machine learning into your project.
Manly Battery which located in Shenzhen,China. A leading Lithium battery manufacturer over 12 years ,widly used for Robostic industry,if there is any project need to evluate ,pls feel free tosend email to info@manlybatteries.com
