Camera calibration: fixing lens distortion

TL;DR:

• Calibration finds the camera's focal length, optical center and distortion coefficients.

• Radial distortion bends straight lines more the farther they are from the center.

• A checkerboard seen from many angles gives enough data to solve for all of them.

You mount a camera on a robot and point it at a doorway. The door frame should be straight, but near the edge of the image the lines bow outward like a barrel. A robot that measures distances or angles from that image will be wrong near the edges. Calibration finds out exactly how the camera bends the picture, so the software can undo it.

What calibration finds

Two groups of numbers describe a camera, and calibration finds both.

• Intrinsics. The focal lengths fx and fy in pixels, and the optical center (cx, cy), the pixel where the optical axis hits the sensor. These describe how a 3D point becomes a pixel.

• Distortion coefficients. Numbers that describe how the lens bends the image away from the ideal pinhole picture.

Why straight lines look curved

A lens is not a perfect pinhole. With a wide-angle lens, magnification falls as you move away from the center of the image, so lines far from the center bend. This is called radial distortion. When the edges are squeezed inward you see a barrel shape, and when they are stretched you see a pincushion shape. Lenses can also have a slight tangential distortion if the lens and sensor are not perfectly aligned.

A simple model for the radial part scales each point by a factor that depends on its distance from the center: x_d = x × (1 + k1·r² + k2·r⁴), where r² = x² + y², and x, y are coordinates measured from the center in units of the focal length.

Worked example: a checkerboard calibration

Print a checkerboard and fix it flat on something rigid. Count the inner corners, the points where four squares meet, not the squares. A board of 9 × 6 squares has 8 × 5 inner corners. Measure one square with a ruler, since that sets the real-world scale.

1. Take 15 to 20 photos: Hold the board in different positions: near, far, tilted left, right, up and down, and in each corner of the image. Keep it flat and the whole pattern in view.

2. Detect the corners: The software finds the corners in each image to a fraction of a pixel.

3. Solve for the camera: Because the corner spacing on the board is known, the software finds the intrinsics and distortion coefficients that best explain all the images.

4. Check the error: The program reports a reprojection error: how far, in pixels, the predicted corners are from the detected ones. Compare it across attempts. If it is large, retake the photos.

import cv2, numpy as np, glob

pattern = (8, 5)          # inner corners
square = 0.025            # 25 mm squares
objp = np.zeros((pattern[0]*pattern[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:pattern[0], 0:pattern[1]].T.reshape(-1, 2) * square

obj_pts, img_pts = [], []
for f in glob.glob("calib/*.jpg"):
    gray = cv2.cvtColor(cv2.imread(f), cv2.COLOR_BGR2GRAY)
    ok, corners = cv2.findChessboardCorners(gray, pattern)
    if ok:
        obj_pts.append(objp); img_pts.append(corners)

rms, K, dist, _, _ = cv2.calibrateCamera(obj_pts, img_pts, gray.shape[::-1], None, None)
print("reprojection error:", rms)
print("intrinsics K:\n", K)
print("distortion:", dist.ravel())

Now see what the result means with a small numeric example. Suppose the calibration gives fx = 800 px, cx = 320 px and k1 = −0.2 (a barrel distortion), with k2 negligible. Take a point at normalised coordinates x = 0.5, y = 0.5.

1. r² = 0.5² + 0.5² = 0.5.

2. Factor: 1 + k1·r² = 1 − 0.2 × 0.5 = 0.9.

3. Distorted x_d = 0.5 × 0.9 = 0.45.

4. Pixel without distortion: u = 800 × 0.5 + 320 = 720. With distortion: u = 800 × 0.45 + 320 = 680.

The point lands 40 pixels closer to the center than an ideal lens would put it. Near the center the effect is tiny: at x = y = 0.1, r² = 0.02, the factor is 1 − 0.004 = 0.996, which is almost no change. This is why straight lines in the middle of an image look fine while those near the edge curve. The calibrated model lets the software compute where each pixel should be and straighten the picture (undistortion).

Reading a calibration result

The program prints a matrix of intrinsics and a list of distortion numbers. Check that the focal lengths fx and fy are close to each other, since most sensors have square pixels, and that cx, cy sit near the middle of the image. If the optical center lands far from the middle, or the distortion numbers are huge, something went wrong, usually with the photos. A reprojection error of a fraction of a pixel to a pixel or so is common for a careful run, though what you need depends on your application.

Why it matters for robots

• Distances and angles measured from the image are only right after undistortion.

• Stereo depth needs both cameras calibrated, so corresponding points line up (see the article on depth cameras).

• SLAM and visual odometry use the intrinsics to turn pixels into rays.

Tip: Do not change the zoom or focus on a camera after you calibrate it. A different focus or zoom setting changes the intrinsics, so you would need to calibrate again.

Common mistakes

• Taking all the photos from one angle or distance.

• Using a board that is not flat, such as a paper sheet that bends.

• Counting squares instead of inner corners.

• Calibrating at one resolution and using the camera at another.

Your plan for this week

Print a checkerboard, take 20 photos at different angles and run the calibration. Look at the reprojection error, then undistort a photo of a door frame and see whether the lines become straight. DuckyHelper can talk through what each parameter means while you look at your results.

Frequently asked questions

How many images do I need?

A common rule is about 15 to 20 well-spread views. More variety in angle and position matters more than a large number of similar photos.

Do all cameras need calibration?

Every camera has some distortion, but small, narrow-angle cameras may not need it. Wide-angle cameras nearly always do.