TL;DR:
Cameras are good at what something is, radar at range and speed, LiDAR at 3D shape. Each has a weakness the others cover.
Sensor fusion combines their estimates, weighting each by how uncertain it is.
Redundancy means no single sensor failing should leave the vehicle blind.
A person steps toward the road 25 meters ahead of a vehicle. A camera sees a shape, a radar sees a moving object, and a LiDAR sees a cluster of points at about the right height. None of them is certain alone. The car's software has to decide whether those three clues are one pedestrian.
Each sensor and its weakness
Sensor | Good at | Weak at |
|---|---|---|
Camera | Recognizing what something is: a person, a sign, a lane line, a light colour | Direct distance. Struggles in darkness, glare and heavy rain |
Radar | Range and speed directly (using the Doppler shift), works through rain and fog | Fine shape and angle detail, and small or non-metal objects can be faint |
LiDAR | Accurate 3D shape and distance from timed laser pulses | Colour and text, and it can be degraded by heavy rain, fog or snow |
No single column in that table is enough. That is the basic argument for using several sensors that fail in different ways.
What sensor fusion is
Sensor fusion combines measurements from several sensors into one estimate better than any single one. Fusion can happen at different levels. Early fusion merges raw data, such as colouring LiDAR points with camera pixels. Late fusion merges each sensor's finished detections: this sensor says a pedestrian at 25.0 m, that one says an object at 25.4 m.
To merge numbers fairly, software needs to know how much to trust each. That is described by the variance, or its square root the standard deviation σ: the typical size of the sensor's error.
Worked example: a pedestrian seen by camera, radar and LiDAR
Say, as a made-up example, that LiDAR reports the pedestrian at 25.0 m with σ = 0.1 m, and radar reports 25.4 m with σ = 0.3 m. Both are range estimates in the same frame. A standard way to combine them is inverse-variance weighting: each measurement is weighted by 1 / σ².
LiDAR weight:
1 / 0.1² = 1 / 0.01 = 100.Radar weight:
1 / 0.3² = 1 / 0.09 ≈ 11.11.Weighted average:
(100 × 25.0 + 11.11 × 25.4) / (100 + 11.11) = (2500 + 282.2) / 111.11 ≈ 25.04 m.Combined uncertainty:
σ = √(1 / 111.11) = √0.0090 ≈ 0.095 m.
The result sits almost on the LiDAR value, because LiDAR is trusted nine times more (100 / 11.11 = 9). The fused σ of 0.095 m is slightly smaller than the best single sensor's 0.1 m: combining always helps a little, even a weaker sensor.
Now the camera's contribution. It reports a pedestrian-shaped object in the same direction, with a high confidence. That does not help with the range, but it supplies the class. Radar adds the speed: a radial speed near zero for someone standing and about 1.4 m/s toward the lane for someone walking. Fusion matches the three detections by position and time, then builds one tracked object with a class, a position and a velocity.
Why the range matters: at 50 km/h, or 50 / 3.6 ≈ 13.9 m/s, with a reaction time of 0.5 s and braking at 7 m/s², the stopping distance is 13.9 × 0.5 + 13.9² / (2 × 7) ≈ 6.9 + 13.8 = 20.7 m. A pedestrian at 25 m is just outside that distance, so a range error of even a few meters affects the decision.
Redundancy
Redundancy means more than one independent way to sense the same thing, so a single failure does not blind the vehicle. If glare washes out the camera, radar and LiDAR still see the pedestrian. If heavy rain cuts the LiDAR range, radar still tracks it. Designers also add redundant computers and power supplies for the same reason.
Disagreement is useful information too. If the camera sees a person and the radar sees nothing, the system can slow down and look harder instead of trusting either.
Where it goes wrong
Bad timing. Sensors sample at different moments, and a moving pedestrian shifts between them. Timestamps and motion compensation matter.
Bad calibration. If the camera and LiDAR positions on the car are slightly off, fused objects drift apart.
Wrong match. Two nearby people can be mistaken for one, or one person split into two.
Correlated errors. Two sensors fooled by the same cause, such as heavy fog, do not give independent evidence.
Overconfidence. A stated
σthat is too small makes one sensor dominate when it should not.
Safety: Experiments with sensors on real vehicles belong on closed courses with trained staff. Do not test self-driving software on public roads, and never open a LiDAR or radar unit.
How real robots use it
The same ideas run in warehouse robots, drones and robot arms. Mobile robots often fuse wheel odometry, an inertial sensor and LiDAR to keep track of position, as in navigating a warehouse floor. Fusing a position estimate over time with a motion model is usually done with a Kalman filter, which uses the same weighting logic as above.
Try it yourself
Measure the width of your desk five times with a ruler and five times by pacing it with your hand span. Compute the average and the spread of each set, then combine them with inverse-variance weights. The fused value should lean toward the ruler, and you will see why a noisy sensor still has a small say.
Frequently asked questions
Why not use only the best sensor?
Every sensor has conditions where it fails. Combining different types means the weak spot of one is covered by another.
Is more sensors always better?
Not automatically. More sensors add cost, calibration work and chances of mismatched data. They help when they fail in different ways.
Get started
