TL;DR:
Visual servoing closes a control loop through a camera: measure where the target is in the image, turn that error into a motion, repeat.
A pixel error maps to an angle through the focal length in pixels:
angle ≈ atan(error / f).Too much gain or too much delay makes the loop overshoot or go unstable, so the gain must be chosen with the frame rate in mind.
A small robot should keep a red ball in the middle of its camera view. There is no map and no ruler. The only information is a picture, and the only question is: where in the picture is the ball, and which way should I turn? This is visual servoing, controlling motion directly from what the camera sees.
Image error to motion
The image error is the difference between where the target is and where you want it. If the camera image is 640 pixels wide, its centre is at u = 320. A target at u = 480 has an error e = 480 − 320 = 160 pixels to the right.
To turn this into motion we need to know how pixels relate to angles. For a pinhole camera with focal length f measured in pixels, a point at angle θ from the optical axis appears f × tan θ pixels from the centre. So θ = atan(e / f). For small angles tan θ ≈ θ (in radians), and the relation is approximately linear: e ≈ f × θ.
The feedback loop
A simple proportional controller sets the turn rate in proportion to the error: ω = k × e, where k is the gain. Large error, fast turn. Small error, slow turn. At every new frame the robot measures the error again, so the loop repeats: image, error, command, motion, new image.
Worked example: centering a target in the image
The camera has a focal length of f = 500 pixels (an example value; calibrate your own). It runs at 10 frames per second, so dt = 0.1 s between updates. The gain is k = 0.003 rad/s per pixel. The target starts 160 pixels to the right of centre.
Angle of the target:
atan(160 / 500) = atan(0.32) ≈ 17.7°.Commanded turn rate:
ω = 0.003 × 160 = 0.48 rad/s, about 27.5°/s.In one frame (0.1 s) the robot turns
Δθ = 0.48 × 0.1 = 0.048 rad. The target moves toward the centre by aboutf × Δθ = 500 × 0.048 = 24 px.New error:
160 − 24 = 136 px. In general,e_next = e × (1 − f × k × dt) = e × (1 − 500 × 0.003 × 0.1) = 0.85 × e.The error shrinks by a factor of 0.85 each frame: 160, 136, 115.6, 98.3, and so on. After 10 frames it is
160 × 0.85¹⁰ ≈ 160 × 0.197 ≈ 31.5 px. After 20 frames: about 6.2 px.
It converges smoothly without overshooting, and the final error approaches zero. Check the arithmetic: 0.85² = 0.7225, 0.85⁴ ≈ 0.522, 0.85⁸ ≈ 0.2725, and 0.85¹⁰ = 0.2725 × 0.7225 ≈ 0.197.
Now try a bigger gain, k = 0.01. The factor becomes 1 − 500 × 0.01 × 0.1 = 0.5: still stable, and it converges faster, with error halving each frame. At k = 0.02, the factor is 1 − 1.0 = 0, so the error vanishes in one step (in theory). At k = 0.03 the factor is −0.5: the robot overshoots to the other side, but the error shrinks. At k = 0.05 the factor is −1.5, and the error grows each frame, so the loop is unstable. The limit is when f × k × dt = 2, which here is k = 0.04.
In practice there is extra delay from camera exposure, image processing and motor response. Delay lowers the safe gain well below this ideal limit. A real loop often keeps f × k × dt below about 1 for margin.
Image-based and position-based
What we did is image-based visual servoing: the error is measured in pixels and the controller works directly on it. This is robust to small calibration errors. Position-based servoing first estimates the target's 3D pose, using the camera model and perhaps a known marker size, then controls the position in space. It is more flexible, but sensitive to calibration. For a robot arm, an image-based controller uses a matrix, the image Jacobian, relating pixel motion to arm motion, much like the arm Jacobian in the Jacobian: how joint speeds become hand speed.
Where it goes wrong
Losing the target. Blur, occlusion or leaving the frame. The controller needs a plan for no detection, such as stopping or searching slowly.
Latency. Old images make the robot chase where the target was.
Too much gain. Oscillation or instability.
Wrong sign. The robot turns away from the target. Test the sign with the wheels off the ground.
Noisy detection. The estimated centre jitters. Smooth it, but not so much that you add delay.
Nonlinear regions. At large angles
tan θis not close to θ, so the simple linear gain behaves differently.
Safety: When a camera drives an arm or a fast base, limit the maximum speed and the joint range in the controller, so a bad detection cannot cause a wild motion. Keep the emergency stop within reach for the first tests.
How real robots use it
Visual servoing steers robots toward docking stations, aligns a gripper over a part, and keeps a drone's camera on a moving subject. It often appears in the last centimeters of a grasp, after a planner such as the one in how robots grasp objects they have never seen has brought the hand close. Planning ahead instead of reacting is the idea behind model predictive control.
Try it yourself
Simulate the loop in a few lines of code. Start with the error at 160 pixels and apply e = e × (1 − f × k × dt) for 30 steps, printing the result. Try several gains and find the one that just starts to oscillate. Then add a delay of two frames, using the error from two steps ago, and see how the safe gain drops.
Frequently asked questions
Do I need to calibrate the camera?
For the simple image-based loop, a rough focal length is enough to choose a gain. For position-based control, good calibration matters.
Why not just compute the angle once and turn that far?
You can, but any error in the estimate or in the turn is never corrected. Feedback keeps correcting using the latest image.
Get started
