Repeatability of intrinsic calibration

Summary

• If you use standard calibration tools, you have a risk of computing inconsistent, and often incorrect calibrations.
• In our experiments, the variations were up to 0.25 degrees on a 90 degree FOV lens. This corresponds to an error > 1cm at a distance of 3 meters and > 40cm at 100 meters.
• Collecting a large dataset of 300-500 images and verifying coverage across the entire field of view improves things, producing repeatability on the order of 0.01 degrees.
• The Skip camera calibration tool provides a reliable and convenient way to achieve this level of accuracy.

Why does it matter?

Many robots use cameras to perceive the world. To make use of the data, they need to detect objects of interest and find out where they are in 3d space. The 2d pixel location of a car isn't always enough to plan a path around it. We need calibrations to map from pixel locations to the real world.

The accuracy of camera calibration doesn't get a lot of attention. When it does, people speak in terms of RMSE error. Sometimes you will read 'RMSE is not a good indicator of accuracy', but this is usually as far as it goes.

In this article we'll dive deeper into what it means to have a good calibration. Instead of looking at just RMSE, or even the parameter values, we'll go to the basic physics and analyze the light ray vectors corresponding to each pixel. At the end of the day, this is what we need to get right.

What is repeatability?

Repeatability refers to the closeness of multiple versions of the same measurements. It quantifies the consistency of a process. Repeatability differs from accuracy in that it does not measure the difference from the true value of the thing being measured. That said, if our calibration process is not repeatable, we can't rely on it to run our robots.

What did we do?

We measured the consistency of repeated calibrations using different tools and data collection strategies. We collected 10 independent datasets using the built-in ROS camera calibration tool and 10 more using the Skip Robotics calibration tool. The camera was a 1440x1080 USB FLIR BlackflyS with a 90 degree field of view.

Methodology

Data was collected by moving the camera above a target placed on the floor. Photography lighting was used to provide good illumination of the target. A standard checkerboard pattern was mounted on an aluminum composite panel and used as the calibration target.

ROS calibration tool

We used the ROS camera calibration package and followed the process recommended by this tool. The python GUI was used to collect data, ensuring that the X, Y, Size, and Skew indicators turned green. The x and y focal lengths were constrained to be equal since almost all modern cameras have square pixels.

The following command line was used:

$ rosrun camera_calibration cameracalibrator.py --size 6x10 -k 6 --fix-aspect-ratio --square 0.08 image:=/camera/camera_0

Following the ROS recommended procedure resulted in roughly 40 - 60 images per dataset.

Skip Robotics calibration tool

The Skip camera calibration tool was used for both data collection and as the solver. A more thorough data collection process was used as described in the Data Collection section of our camera calibration article. The x and y focal length were constrained to be equal and an outlier threshold value of 0.5 pixels was used.

Following the Skip Robotics data collection procedure resulted in roughly 300 - 600 images per dataset.

Analysis

After collecting the data and running each of the tools listed above, we were able to compute 20 different sets of calibration parameters. The raw values are listed in the appendix at the end of this article.

Analyzing the results, we saw that the Skip calibration tool gave much more consistent estimates for the focal length and principal point:

If you look at the raw data in the appendix, you will notice the distortion parameter values are wildly inconsistent from run to run! So much so that we didn't bother computing the standard deviations. This is true for both the ROS and Skip calibration tools. It happens because the calibration model parameters are not unique! It is possible for two sets of values to produce the same physical calibration.

Incoming ray images

As we saw in the previous sections, the raw parameter values are not unique and so are not great for evaluating repeatability. We need a way to directly compare the actual calibrations. This can be done by thinking back to the optical meaning of what we're doing. No matter how we choose to break down the math, at the end of the day we're mapping incoming rays of light to pixels.

Instead of messing around with interpretations of model-specific parameters, we compared results using these rays. To do this, we inverted the calibration model by computing the incoming light ray direction corresponding to each pixel. While the parameters are not unique, the direction from which light will project on to each pixel is an actual physical quantity and is unique.

We form a 3 channel incoming ray image for each calibration by storing the unit vector / ray at each pixel location in the image. The angle between rays at corresponding pixels in different calibrations is a measure of similarity. We can use this to compare the repeatability of our calibration processes.

Rotational ambiguity

If you took a computer vision class, you may remember that intrinsic calibrations are not unique. It's possible to apply a rotation to the incoming light rays, and a corresponding inverse rotation to the calibration target poses without changing anything.

This makes it tricky to directly compare the incoming ray images. We accounted for this ambiguity by computing an optimal rotational alignment between the sets of incoming rays.

Measuring repeatability

For each pixel in the image and for each of the different approaches, we computed the average ray direction over the 10 datasets. We then measured the deviation relative to the average rays. The full procedure:

1. Compute calibration parameters for each dataset
2. Compute the incoming ray image for each dataset
3. Rotationally align the incoming ray images
4. Compute the average ray directions across all 10 datasets for each approach
5. Compute the angular deviations of the incoming rays from this average.

Results

Following the methodology above gave us per-pixel measurements of repeatability for each of the approaches being evaluated. These are plotted below with brighter colors indicating higher variance and lower repeatability. The units for all of the plots in this section are degrees. Lower numbers are better and blue is the best.

The repeatability is higher towards the center and increases as we go towards the edges. To better visualize the results, we aggregated the data as a function of distance from the center. The next set of plots show both the standard deviation and maximum deviation vs the distance from the image center (expressed as a fraction).

The latter set of plots show how much more consistent the Skip tool and procedure is compared to the ROS default. While we didn't directly measure accuracy with this experiment, we believe a proper evaluation of accuracy would produce similar results.

You may be wondering about the jagged spikes in the plots above. They're caused by failure of the numerical algorithm used to compute the incoming ray images. Using a traditional checkerboard caibration pattern makes it extremely difficult to get coverage at the edges and can cause the rational polynomial distortion function to behave poorly. We collected 10 more datasets with the Skip calibration tool using an Apriltag board to remove these artifacts. This substantially improved the results:

Conclusion

The plots shown above make it clear that your calibration method matters! Despite both solvers in theory optimizing the same function, the way you collect data and handle outliers is important. The Skip calibration tool results in a standard deviation of < 0.01 degrees (!!) almost all the way to the corners of the image. The ROS tool produces variations an order of magnitude higher, going over 0.20 degrees.

Looking at the maximum deviation, you can see that the ROS approach produces variations up to 0.50 degrees at the image edges! These sorts of errors may not matter for some applications, but if you're relying on your cameras for long range perception, safety, or visual SLAM, your downstream algorithms could be losing substantial accuracy and reliability.

Appendix: Solution Data

The following tables list the raw calibration outputs from the datasets collected above.