TL;DR:
In reinforcement learning, a robot tries actions, receives a reward number, and adjusts its policy to collect more reward over time.
The reward you write is the goal the robot actually chases, so a sloppy reward gets sloppy behaviour.
It needs thousands of trials, so it usually runs in simulation first, and a learned policy then needs careful safe testing on hardware.
You cannot easily write the rules for keeping a two-wheeled robot balanced on its own, or for a four-legged robot to walk over loose gravel. Reinforcement learning takes a different route: let the robot try things, score each outcome, and keep what scores well.
What it does on a robot
Three words carry the whole idea. A trial (often called an episode) is one attempt from a start to an end, such as balancing until the robot falls. A reward is a number the designer assigns after each step. A policy is the rule that picks an action, like a motor command, from what the robot senses.
Learning means changing the policy so that the total reward over a trial goes up. Nobody tells the robot which action was right. It only sees whether the score improved.
How reward turns into behaviour
At each time step the robot reads its state (tilt angle, tilt speed, wheel speed), the policy picks an action (a motor torque), the world changes, and the robot gets a reward. The usual goal is the return, the sum of future rewards, with later ones counted a bit less using a discount factor γ between 0 and 1: return = r₀ + γ·r₁ + γ²·r₂ + ….
Discounting makes sooner rewards worth more. It also keeps the total finite when a trial could go on forever.
Worked example: rewarding a simulated robot for staying upright
The robot is a simulated two-wheeled balancer. Its state is the tilt angle. The reward is +1 for every step it stays within 15 degrees of upright (a made-up threshold for this example). If the tilt goes past 15 degrees, the trial ends with no further reward. Take γ = 0.9.
A policy that does nothing falls after 3 steps. The rewards are 1, 1 and 1, and the return is
1 + 0.9 + 0.81 = 2.71.A policy that falls after 2 steps has return
1 + 0.9 = 1.9.A policy that never falls collects 1 at every step forever, and the return is the series
1 + 0.9 + 0.81 + … = 1 / (1 − 0.9) = 10.In practice, trials are cut at some length, say 200 steps. The limit is
10, and after 200 steps the sum is(1 − 0.9²⁰⁰) / 0.1, which is 10 to many decimal places.
Notice that 10 is the best possible score here. Any policy that keeps the robot up as long as possible gets a score close to 10, and a policy that falls early gets less. Learning compares those scores and changes the policy's settings so higher-scoring behaviour becomes more likely.
Now see what a sloppy reward does. Suppose you also add +0.5 per step for wheel speed, hoping the robot will move forward. A learned policy could then spin the wheels hard while wobbling near the edge of balance, because the speed bonus outweighs staying calm. Fixing it means changing the reward: add a penalty for large tilt or large torque. This kind of surprise is called reward hacking, and it is the most common experience when training.
Why simulation helps
A learning algorithm may need a very large number of trials, and early trials are mostly failures. A real balancer would fall over, wear its gears and need constant resets. A simulator can run many copies in parallel, faster than real time, and resetting is free. That is why nearly all robot reinforcement learning starts in simulation, which is also the reason in why roboticists test in a simulator first.
To help policies survive the move to hardware, designers randomize the simulation: slightly different masses, friction, sensor noise and delays on each trial. A policy that works across all those variations has a better chance on the real machine.
Where it goes wrong
Reward hacking. The robot finds a trick that scores well but is not what you meant.
Too many trials. Sparse rewards, where nothing scores until a goal is reached, can take enormous numbers of trials to find a first success.
Simulation gap. A policy can exploit quirks of the simulator that do not exist in reality.
Brittleness. A policy trained for one setting can behave oddly in one it never saw.
Unpredictable actions. A learned policy is a function, not a list of rules, so you cannot read why it chose an action.
Putting a learned policy on hardware: Clamp motor torque and speed outside the policy, in code the policy cannot change. Test with the robot held in a harness or on blocks, keep people away from its reach, and keep a hardware emergency stop that cuts motor power. Never leave a charging battery unattended during long training runs on real robots.
How real robots use it
Reinforcement learning is used for locomotion, where writing a controller by hand is difficult, and for some manipulation tasks. Many systems pair a learned policy with classical safety code that limits what it can do. Another way to learn is to start from examples of the task instead of rewards, as covered in imitation learning.
Try it yourself
Take a simple simulated balancing task from an open learning library and run it with a random policy to see how long it lasts. Then change only the reward, for example by adding a small penalty for large motor commands, and watch how the learned behaviour changes. Keep a notebook of each reward and what happened.
Frequently asked questions
How is this different from normal supervised learning?
In supervised learning each example comes with the right answer. Here the robot only gets a score after acting, so it must discover good actions by trying.
Does the robot keep learning after it is deployed?
Usually not. Most systems train first, then freeze the policy, because unplanned learning on a real machine is hard to make safe.
Get started
