TL;DR:
A point cloud is a list of
(x, y, z)points, one per place a sensor measured a surface.Downsampling thins the cloud, plane fitting finds and removes the floor, and clustering groups what is left into objects.
Real clouds are noisy, so every step uses a tolerance, and every tolerance can be wrong.
A depth camera or a spinning LiDAR does not hand a robot a picture of a room. It hands over a long list of numbers, each one a spot in space where something reflected the sensor's light. Before the robot can decide there is a chair in front of it, that list has to be cleaned up and sorted.
What a point cloud is on a robot
A point cloud is a set of points, each stored as (x, y, z) in meters in some coordinate frame, often the sensor's own. Some sensors add an intensity or a colour per point. There is no surface, no object and no label in the data, only positions.
Three jobs come up again and again. Filtering removes points you do not want or cannot trust. Segmentation splits the rest into parts, such as ground and obstacles. Clustering groups nearby points so each group can be treated as one object.
Where the points come from
A LiDAR measures how long a light pulse takes to return, and the distance is d = c × t / 2, because the pulse travels out and back. With the beam's direction, that distance becomes one point. A depth camera works differently, often by projecting a pattern or comparing two views, but the result is the same kind of list.
Every point carries error. Edges blur, shiny or dark surfaces return little light, and some beams return nothing at all. That is why robots filter before they trust any of it.
Lasers and LiDAR: Never open a LiDAR or depth sensor housing, and never look into a bare emitter. Use the sensor as sold, follow its manual, and keep its spinning head clear of fingers and cables.
Downsampling with a voxel grid
A single frame can hold far more points than later steps need. A voxel grid filter cuts space into small cubes and keeps one point per occupied cube, usually the average of the points inside. This is downsampling, and it also evens out the density, since nearby surfaces otherwise get many more points than far ones.
The cube size is your choice. With 5 cm cubes, a 1 m × 1 m patch of flat floor covers at most 20 × 20 = 400 cubes in its layer, so at most 400 points survive no matter how many the sensor returned there. Choose a size smaller than the smallest thing you must see.
Worked example: finding the floor plane in a point cloud
Say the sensor looks at a floor with a box on it. Most points lie near one flat surface. The standard trick is RANSAC (random sample consensus): guess a plane from three random points, count how many other points agree, repeat, and keep the best guess.
Here is one guess in numbers, with z up and all values in meters. Pick three points: A (0, 0, 0.01), B (1, 0, 0.00), C (0, 1, 0.02).
Make two edge vectors:
AB = (1, 0, −0.01)andAC = (0, 1, 0.01).The plane's normal is the cross product
AB × AC = (0×0.01 − (−0.01)×1, (−0.01)×0 − 1×0.01, 1×1 − 0×0) = (0.01, −0.01, 1).Its length is
√(0.01² + 0.01² + 1²) = √1.0002 ≈ 1.0001, close enough to 1 to treat as a unit normal.Plane equation through A:
0.01(x − 0) − 0.01(y − 0) + 1(z − 0.01) = 0. For any point, plugging in(x, y, z)gives its signed distance times 1.0001.Test a floor point
(2, 1, 0.01):0.02 − 0.01 + 0 = 0.01, so it sits 1 cm from the plane.Test a point on a box
(1.5, 0.5, 0.30):0.015 − 0.005 + 0.29 = 0.30, so it sits 30 cm from the plane.Test a noisy point
(2, 1, 0.03):0.02 − 0.01 + 0.02 = 0.03, which is 3 cm.
Now apply a distance threshold of 2 cm. The first test point is an inlier, the box point and the 3 cm point are not. Count inliers over the whole cloud for this plane, then try other random triples. The plane with the most inliers is the floor, and its inliers are removed.
Check your work: a flat floor should give a normal very close to (0, 0, 1), and here it is (0.01, −0.01, 1), tilted about 0.8 degrees, from atan(0.01414) ≈ 0.81°. That small tilt comes from the sample points being slightly noisy, which is normal.
Clusters: turning leftover points into objects
With the floor gone, the box points float by themselves. Euclidean clustering starts at one point, adds every point within a distance tolerance, then repeats from each new point until nobody new joins. Each finished group is a cluster, and one cluster is a candidate object.
Use a tolerance larger than the gap between neighbouring points on one object but smaller than the gap between two objects. If points on a box are about 5 cm apart and two boxes are 40 cm apart, a tolerance of 10 cm separates them cleanly. A robot can then take each cluster's centre and size as a rough position and bounding box.
Where it goes wrong
Sloped floors and ramps. Plane fitting finds the biggest plane, which may be a wall, not the floor. Check that the normal is roughly vertical.
Threshold too tight. Noisy floor points fall outside it and become phantom obstacles.
Threshold too loose. Low objects such as a cable or a doorstep disappear into the floor.
Objects that touch. Two boxes side by side form one cluster, because the tolerance cannot tell them apart.
Thin or shiny things. Glass, mirrors and black surfaces may return no points at all.
How real robots use it
A mobile robot removes the ground, then treats what remains as obstacles for its planner. A robot arm clusters objects on a table to find something to pick. Both usually share the pipeline you just saw: downsample, remove the dominant plane, cluster, then pass each cluster on. Before a cluster's position is useful to an arm, it must be moved into the arm's own frame, which is the subject of coordinate frames and transforms.
Try it yourself
Write 30 points by hand on graph paper: 20 near a line for the floor, with small errors, and two small groups for objects. Draw a line through two random floor points, mark everything within a threshold, and try a few different threshold values. Seeing how the answer changes is the whole lesson. Point cloud libraries implement exactly these steps, so you can compare your result with theirs later.
Frequently asked questions
Why not just use the camera image instead?
Images have no distance in them directly. A point cloud gives positions in meters, which a planner or an arm can use right away. Many robots use both.
Why does RANSAC use random samples?
Outliers such as box points would pull an ordinary best-fit line off the floor. Trying many small samples and counting agreement ignores the outliers instead of averaging them in.
Get started
