TL;DR:
HSV separates the color itself (hue) from how pure (saturation) and how bright (value) it is, so a shaded ball keeps almost the same hue.
Threshold hue, saturation and value to get a mask, then use the mask's centroid to find where the ball is in the image.
The horizontal offset of the centroid, converted to an angle, tells the robot how much to turn.
You ask a robot to follow an orange ball. In the bright part of the room it works. When the ball rolls into the shadow of a table, the robot loses it, even though the ball is the same ball. The problem is not the robot's drive or the camera. It is the way the code describes the color.
Why RGB breaks in shade
RGB describes a color by how much red, green and blue light it has. Make the scene darker, and all three numbers get smaller together. A bright orange pixel is (255, 140, 30). The same ball in shade, at about half the light, is (128, 70, 15). Every channel changed, so a rule such as "R above 200" no longer fits.
HSV: color, purity, brightness
HSV describes a pixel by three numbers:
Hue (H): which color it is, as an angle around a color wheel: red at 0°, yellow at 60°, green at 120°, cyan at 180°, blue at 240°, magenta at 300°.
Saturation (S): how pure the color is. 0 is grey, 1 is the pure color.
Value (V): how bright it is.
The conversion, for R, G, B scaled 0 to 255 with max and min the largest and smallest of them and diff = max − min: V = max, S = diff / max, and when R is the largest channel, H = 60° × (G − B) / diff (add 360° if it comes out negative). Other cases are similar, with different offsets.
Dimming the light scales all channels together, which changes V and leaves H and S almost the same.
Worked example: HSV thresholds for an orange ball
Convert the two pixels from the intro.
Bright (255, 140, 30): max = 255, min = 30, diff = 225.
H = 60° × (140 − 30) / 225 = 60° × 0.4889 = 29.3°.S = 225 / 255 = 0.882.V = 255.
In shade (128, 70, 15): max = 128, min = 15, diff = 113.
H = 60° × (70 − 15) / 113 = 60° × 0.4867 = 29.2°.S = 113 / 128 = 0.883.V = 128.
The hue is 29.3° in the light and 29.2° in the shade, and the saturation is 0.882 against 0.883. Only the brightness halved. So a rule built on H and S keeps working when the light changes. In OpenCV, hue is stored from 0 to 179 (the angle divided by 2), saturation and value from 0 to 255. The ball's hue is about 15, its saturation about 225, and its value 255 or 128.
A sensible set of thresholds for the ball, leaving room for variation, is H from 8 to 22, S at least 120 and V at least 60. Both pixels pass: H = 15 is inside 8 to 22, S = 225 is above 120, and V = 255 or 128 is above 60. A grey floor pixel has low saturation and fails. A red pixel (H near 0) fails the hue range.
import cv2, numpy as np
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, np.array([8, 120, 60]), np.array([22, 255, 255]))
mask = cv2.erode(mask, None, iterations=2) # remove specks
mask = cv2.dilate(mask, None, iterations=2) # restore the blob
m = cv2.moments(mask, binaryImage=True)
if m["m00"] > 200: # enough pixels to trust
cx = m["m10"] / m["m00"] # centroid x in pixels
cy = m["m01"] / m["m00"]Centroid and steering
The centroid of the mask is its average pixel position: cx = (sum of x of mask pixels) / (number of mask pixels). Suppose the mask has m00 = 1200 pixels and their x coordinates add up to m10 = 480,000. Then cx = 480,000 / 1200 = 400.
The image is 640 pixels wide, so the center column is 320. The ball is 400 − 320 = 80 pixels to the right. With a focal length of 554 pixels (a 60° field of view, as in finding objects in a camera image), the angle to the ball is:
angle = atan(80 / 554) = atan(0.1444) = 8.2° to the right.
A simple controller turns the robot by an amount proportional to that angle: turn rate = K × angle. This is the proportional term of the controller in PID control. As the robot turns, the ball's centroid moves to the center and the command shrinks.
Where it goes wrong
Red wraps around. Red's hue sits near 0° and 360°, so a red threshold needs two ranges (for example 0 to 10 and 170 to 179 in OpenCV units).
Similar objects. An orange cone, a skin tone or a wooden floor can pass the threshold. Check size and roundness as well.
Auto white balance. The camera may shift all colors when the scene changes. Lock the white balance or re-tune.
Very dark or very bright pixels. At low V, hue is noisy. At high brightness with glare, saturation falls toward white. That is why the V and S limits exist.
Fixed thresholds. Sunlight, bulbs and screens have different color temperatures. Test in the light the robot will work in, and show the mask on screen while tuning.
Lost target. If the mask is empty, the robot needs a rule, such as stop or search.
Safety: A robot that follows a colored object can follow the wrong thing, such as a person wearing the same color. Limit the speed, set a minimum distance at which it stops, and use a distance sensor as an independent check. Keep a wireless or physical emergency stop within reach, and test with the wheels lifted before the first free run.
How real robots use it
Color tracking is a classic for competition and classroom robots: following a ball, a line, a goal or a colored tape. It is cheap and fast enough for a small processor. When lighting is unpredictable or the targets look alike, designers use markers with distinctive patterns (see fiducial markers) or learned detectors.
Try it yourself
Photograph your ball in bright light and in shade. Use an image tool to read the RGB value of a ball pixel in each, convert both to HSV using the formulas above, and compare. Then write the five-line program above, display the mask in a window, and drag the threshold values until only the ball is white in both lights.
Frequently asked questions
Why does the hue stay the same in shade?
Dimming the light scales all three color channels by the same factor. Hue and saturation depend on the ratios between channels, which do not change. Only the value changes.
How do I get the angle to the ball?
Take the horizontal pixel offset of the centroid from the image center, and compute atan(offset / focal length in pixels), with the focal length from the camera's field of view.
Get started
