It’s been a while since I’ve started exploring Reinforcement Learning and OpenAI Gym , inspired by the amazing Boston Dynamics Spot. I’ve spent last year studying the foundations of Machine Learning and how it is applied to robotics discovering a very interesting world.

自从我开始探索《 Reinforcement Learning和OpenAI Gym以来已经有一段时间了，灵感来自令人惊叹的Boston Dynamics Spot。 去年，我一直在研究机器学习的基础，以及如何将其应用于发现非常有趣的世界的机器人技术。

In order to experiment what I’ve learned, I’ve created an open-source project called rex-gym. The aim is to let an open-source quadruped robot learns domestic and generic tasks in the simulations and then successfully transfer the knowledge (Control Policies) on the real robot without any other manual tuning.

为了实验我学到的东西，我创建了一个名为rex-gym的开源项目。 目的是让开放源代码的四足机器人在模拟中学习家庭任务和通用任务，然后在不进行任何其他手动调整的情况下，成功地在真实机器人上转移知识( Control Policies )。

Rex is a 12 joints robot with 3 motors (Shoulder, Leg and Foot) for each leg. The robot base model is imported in pyBullet using a URDF file and the servo motors are modelled in the motor class.

雷克斯(Rex)是一个12关节机器人，每条腿有3个电机( Shoulder ， Leg和Foot )。 使用URDF文件将机器人base模型导入pyBullet并在motor类中对伺服电动机进行建模。

There is also an enhanced version that comes with a 6DOF robotic arm mounted on the top of the rack

还有一个增强版本，在机架顶部装有6DOF机械臂

学习方法(Learning approach)

This library uses the Proximal Policy Optimization (PPO) algorithm with a hybrid policy defined as:

该库使用Proximal Policy Optimization (PPO)算法，其混合策略定义为：

a(t, o) = a(t) + π(o)

a(t, o) = a(t) + π(o)

It can be varied continuously from fully user-specified to entirely learned from scratch. If we want to use a user-specified policy, we can set both the lower and the upper bounds of π(o) to be zero. If we want a policy that is learned from scratch, we can set a(t) = 0 and give the feedback component π(o) a wide output range.

从完全用户指定到完全从头开始学习，它可以不断变化。 如果要使用用户指定的策略，可以将π(o)的下限和上限都设置为零。 如果我们想要从头开始学习的策略，则可以将a(t) = 0设置a(t) = 0并为反馈分量π(o)提供较宽的输出范围。

By varying the open loop signal and the output bound of the feedback component, we can decide how much user control is applied to the system.

通过改变开环信号和反馈组件的输出界限，我们可以决定对系统应用多少用户控制。

A twofold approach is used to implement the Rex Gym Environments: Bezier controller and Open Loop.

一种双重方法用于实现Rex Gym Environments ： Bezier controller和Open Loop 。

The Bezier controller implements a fully user-specified policy. The controller uses the Inverse Kinematics model (see model/kinematics.py) to generate the gait.

Bezier controller实施完全由用户指定的策略。 控制器使用Inverse Kinematics模型(请参阅model/kinematics.py )来生成步态。

The Open Loop mode consists, in some cases, in let the system lean from scratch (setting the open loop component a(t) = 0) while others just providing a simple trajectory reference (e.g. a(t) = sin(t)).

在某些情况下， Open Loop模式包括让系统从头开始倾斜(设置开环分量a(t) = 0 )，而其他模式仅提供简单的轨迹参考(例如a(t) = sin(t) ) 。

The purpose is to compare the learned policies and scores using those two different approach.

目的是使用这两种不同的方法比较学习的策略和分数。

步态 (Gaits)

My first focus was on the basic gaits: walk, gallop, turn and stand up.

我最初的重点是基本步态：走路，疾驰，转身和站起来。

I’ve created a gym environment for each of them (tuning the action space and the reward function ) and I’ve trained the robot using both the Bezier controller and Open Loop one. The output are different but efficient gaits:

我已经为他们每个人创建了一个gym environment (调整action space和reward function )，并且使用Bezier controller和Open Loop训练了机器人。 输出是不同但高效的步态：

地形(Terrains)

In order to test the policies’ robustness, I’ve integrated different uneven terrains other than the classic plane.

为了测试政策的稳健性，除经典飞机外，我还集成了其他不平坦的地形。

The next step will be training the robot to interact with the environments, grasping objects and opening doors.

下一步将是训练机器人与环境互动，抓住物体并打开门。

Stay tuned!

敬请关注！