Make a pan-tilt camera that tracks a colored ball

TL;DR:

• Find the ball by color, measure how far its center is from the image center, and nudge each servo in proportion to that error.

• Multiplying the error by a small gain is proportional control. Too large a gain makes the camera oscillate.

• Smoothing the measurement and the servo target stops jitter.

A camera that follows a ball looks like magic, but the loop is small: find the ball, see how far off center it is, turn the camera a bit toward it, repeat. This build teaches the loop with two servos and a few lines of code.

What you will build

You will mount a small camera on a pan-tilt bracket made of two hobby servos, one turning left and right and one tilting up and down. A program finds a brightly colored ball in each frame and moves the servos until the ball sits in the middle of the image.

Parts and tools

• A small USB camera or a camera board, and a computer or single-board computer that can run Python with OpenCV.

• Two hobby servos and a pan-tilt bracket (or a bracket you build from card and screws).

• A microcontroller board to drive the servos, or servo pins on your single-board computer.

• A separate power supply for the servos, as in the servo article.

• A brightly colored ball, such as an orange or green one, against a plain background.

Wiring and assembly

Servos pull current spikes that can reset a board, so power them from their own supply and join the grounds. Mount the pan servo on a base and the tilt servo on top of it, with the camera on the tilt arm. Center both servos first, then fix the horns so the camera looks straight ahead at 90°. Keep the camera cable loose so it does not pull when the head moves.

How the tracking works

Cameras give you pixels. To find a colored ball, convert each frame to the HSV color space, where hue says which color it is and is easier to threshold than red-green-blue. Keep the pixels inside your ball's hue range, then find the center of that blob of pixels.

Say the frame is 320 × 240 pixels, so the image center is at (160, 120). The ball center is found at (220, 90). The error is how far off center it is:

error_x = 220 − 160 = +60 pixels  (ball is right of center)
error_y =  90 − 120 = −30 pixels  (ball is above center)

From error to servo angle

Proportional control moves the servo by the error times a small gain, Kp. Say the camera has a 60° horizontal field of view across 320 pixels, so one pixel is 60 / 320 = 0.1875°. The exact correction would be 60 × 0.1875 ≈ 11.3° to the right. But moving by the whole error each frame often overshoots, because the camera image lags behind the servo. So use a smaller gain, for example Kp = 0.1 degrees per pixel:

The vertical axis works the same way with its own field of view, which is smaller because most cameras are wider than they are tall. Use a separate gain for each axis if the two servos respond differently, and tune them one at a time while the other servo is held still.

pan_change = 0.1 × 60 = 6.0 degrees toward the ball
tilt_change = 0.1 × (−30) = −3.0 degrees (sign depends on how the servo is mounted)

Each frame the camera moves part of the way, the ball gets nearer the center, the error shrinks, and so does the next step. The camera settles smoothly instead of lurching.

The code, step by step

import cv2
import numpy as np

KP = 0.1
pan, tilt = 90.0, 90.0
smooth_x = smooth_y = None
ALPHA = 0.4   # 0 to 1, lower = smoother

cap = cv2.VideoCapture(0)
while True:
    ok, frame = cap.read()
    if not ok:
        break
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(hsv, (5, 120, 120), (20, 255, 255))  # orange range
    m = cv2.moments(mask)
    if m["m00"] > 500:                    # enough pixels to be the ball
        cx = m["m10"] / m["m00"]
        cy = m["m01"] / m["m00"]
        smooth_x = cx if smooth_x is None else ALPHA * cx + (1 - ALPHA) * smooth_x
        smooth_y = cy if smooth_y is None else ALPHA * cy + (1 - ALPHA) * smooth_y
        h, w = mask.shape
        pan  = min(180, max(0, pan  - KP * (smooth_x - w / 2)))
        tilt = min(180, max(0, tilt + KP * (smooth_y - h / 2)))
        # send pan and tilt to the servos here

The signs depend on how your servos are mounted. If the camera moves away from the ball, flip the sign for that axis. The min and max keep each angle inside 0° to 180° so the servo is never driven against its stop.

Test, measure and fix

Test in steps rather than all at once. First print the ball center with the servos unplugged and move the ball by hand. When the printed numbers follow the ball sensibly, connect one servo, then the other.

Light matters a lot. A window behind the ball turns it into a dark silhouette and the color range stops matching, so keep the light on the ball and the background plain. Tune the hue range by printing the pixel values at the ball, not by guessing.

Finally, log the error each frame. A healthy loop shows the error shrinking toward zero in a few frames after you move the ball. An error that swings from positive to negative and back means the gain is too high.

• Oscillates around the ball: lower Kp, for example from 0.1 to 0.05.

• Moves too slowly: raise Kp a little at a time.

• Jitters when the ball is still: lower ALPHA so the measurement is smoothed more.

• Chases other objects: narrow the color range or ask for a larger blob size.

Make it better

Add a small dead zone, for example do nothing when the error is under 5 pixels. Add a search sweep when the ball is lost. Later, move the code into a ROS 2 node so other parts of a robot can use the ball position.

Stay safe

Moving parts: Keep fingers and hair out of the bracket and gears while the servos are powered, because they can move without warning. Use a low-voltage servo supply only.

Frequently asked questions

Why not use the whole error as the correction?

The camera sees the world a moment after it moves, so a full-size correction overshoots. A smaller gain gets there in a few smooth steps.

Why HSV instead of RGB?

Hue stays fairly steady when the light gets brighter or dimmer, so one range covers the ball in more lighting.