TL;DR:
A camera image is a grid of numbers, and finding an object means choosing a rule that picks out its pixels.
A typical pipeline: convert, threshold into a mask, clean up the mask, find contours, then keep the one with the right size and shape.
Size in pixels plus the camera's focal length gives the distance to an object of known size.
A robot's camera sees a grid of colored dots. You see a red ball on the floor. The robot has to get from the first to the second, with no understanding of what a ball is, using only arithmetic on numbers. The classical pipeline is short and easy to follow, and it still does a lot of useful work.
What an image is to a robot
An image is a grid of pixels. A 640 by 480 image has 640 × 480 = 307,200 pixels. In a common color format each pixel has three numbers, red, green and blue, each from 0 to 255, so one frame takes 307,200 × 3 = 921,600 bytes. At 30 frames a second that is 921,600 × 30 = 27,648,000 bytes every second. The robot must pick out the few thousand pixels that belong to the ball.
The pipeline
Capture and convert. Get the frame, and convert it to a suitable color representation (the next article, color tracking, explains why HSV is often better than RGB).
Threshold. Choose a rule that marks each pixel as "ball-like" (1) or not (0). The result is a black and white image called a mask.
Clean up. Real masks have specks and holes. Removing small specks and filling small holes (morphological opening and closing) tidies the mask.
Find contours. A contour is the outline of a connected group of mask pixels. Each blob gets its own contour.
Filter the blobs. Keep the blob that has the right area and shape.
Measure. Compute its center, size and, if the camera is known, its direction and distance.
Worked example: a robot spotting a red ball
The ball in the picture has a typical red pixel value of (R, G, B) = (200, 30, 40). The grey-brown floor has pixels like (150, 140, 130).
Threshold rule (simple, for illustration): a pixel is ball-like if R > 150, G < 80 and B < 80.
Ball pixel (200, 30, 40): all three conditions hold, so it is marked 1.
Floor pixel (150, 140, 130):
R > 150is false (150 is not above 150), so 0. Even if R passed, G is 140, so it fails.
After thresholding and cleanup, the mask has two blobs: a big one and a speck of 3 pixels where a red sticker sits on the wall.
Filter by area. The big blob has a contour of area 1257 pixels. The speck has 3. Set a minimum area, say 200 pixels, and the speck is discarded.
Check the shape. For a circle, the circularity 4π × area / perimeter² equals 1. If the contour of the ball measures a perimeter of 125.7 pixels, then 4π × 1257 / 125.7² = 15,795 / 15,800 = 0.9997, close to 1. A square would give 4π a² / (4a)² = π/4 = 0.785. A circularity close to 1 says "round", and rules out a long red box.
The area and perimeter correspond to a circle of radius √(1257 / π) = √400.1 = 20 pixels, so the diameter is 40 pixels.
Estimate the distance. The camera is 640 pixels wide with a horizontal field of view of 60°. By the pinhole model, the focal length in pixels is f = (640 / 2) / tan(30°) = 320 / 0.5774 = 554 pixels. If the real ball has a diameter of D = 0.07 m and appears d = 40 pixels wide, then:
Z = f × D / d = 554 × 0.07 / 40 = 0.97 m.
The ball is about 1 m away. Check using the reverse formula: a 0.07 m ball at 0.97 m should appear 554 × 0.07 / 0.97 = 40 pixels across. ✓
This only works if you know the real ball size, and if the camera is calibrated: lens distortion bends the geometry near the image edges.
Where it goes wrong
Lighting. Shadows, bright reflections and the color of the light change pixel values. A fixed RGB threshold that works indoors may fail near a window.
Similar colors. A red shirt or a red cup in the background gets picked up too. Use shape and size checks.
Partly hidden objects. An occluded ball looks like a different shape.
Motion blur and low light. Blurry frames shift edges and merge blobs.
Camera settings. Auto exposure and auto white balance change colors from frame to frame. Lock them if you can.
Distance estimates. They are very sensitive when the object is small. A one-pixel error on a 40-pixel width moves Z by about 2.4 cm here:
554 × 0.07 / 41 = 0.946 m, versus 0.97 m. Check:0.97 − 0.946 = 0.024 m.
Safety: Do not let a robot move toward a detected object while people could be in its path, since a wrong detection can send it toward someone. Limit the speed, stop when the object fills too much of the image, and keep a distance sensor and an emergency stop active. Test with the motors disabled first, only printing what the robot sees.
How real robots use it
Competition robots, sorting machines and educational kits use this kind of pipeline to find balls, cones, lines and markers. Larger systems use learned detectors, which handle varied objects and lighting better, but they take more computing power and training data, and the same basic output is needed: where in the image, how big, and how sure. Once the position is known, a controller turns it into motion, or the kinematics of an arm (see inverse kinematics) turn it into joint angles for a grasp.
Try it yourself
Take a photo of a colored ball on a table. Read some pixel values with an image editor and write down the ball's and the table's values. Write a threshold rule, apply it in a short program, and show the mask. Count the pixels in the biggest blob and estimate the ball's distance from its width with the formula above. Then measure the real distance with a tape and compare.
Frequently asked questions
What is a mask?
It is a black and white image the same size as the camera frame, where white marks the pixels that passed the rule and black marks the rest.
Why filter blobs by area?
Real masks contain specks from noise and from similar colored things. A minimum area throws them away and leaves the object you want.
Get started
