Overview
A red sphere appears somewhere in the arm's workspace and the policy has to put the end-effector on it. There is no inverse kinematics here: the agent outputs joint velocities directly and learns the mapping by trial and error, with reaching behaviour typically showing up after one to two million steps.
The environment went through two versions. The second adds joint velocities, the end-effector position and the vector to the target to the observation, resets far faster by restoring a saved physics state, and reports success back to the training loop. A follow-up repository reorganises the code into src, urdf and assets modules.
How it works
Observe
Version 1 sees 10 values, the 7 joint angles and the target position. Version 2 sees 23, adding joint velocities, the end-effector position and the vector to the target.
Act
The policy outputs 7 normalised joint velocities between −1 and 1.
Reward
A dense reward of negative distance to the target, minus a small penalty on motion, plus a +50 bonus once the end-effector is within 0.05 m.
Train
PPO with an MLP policy and batch size 256, run for up to 3M steps with checkpoints and TensorBoard logging, or for 500k steps across 8 parallel headless environments.
Evaluate
An evaluation script replays the trained policy for 10 episodes in the GUI and reports success or timeout for each one.