The next guest post is by Lionel Humbert who took a scientific approach when examining the AF accuracy acros camera brands:
Many discussions emerge on the Internet about image quality, noise, focus accuracy… when a new camera hits the market. Some people move from one brand to another based solely on some brief, vague and not justified reports viewed on the web. Generally tests published on the Internet are by design not reproducible and biased. A scientific approach will required multiple bodies and lenses, and at least more than one sample for each model because of small manufacturing differences that may influence results.
In this article I will discuss the accuracy of focus with a rigorous analysis using modern statistical methods on data collected from a well known website (www.lenstip.com, 8000 visits per day). This analysis is focused on their tests because it is the only website that reports the accuracy of focus for each lens. They get a percentage of error, measured as the number of times the resolution on the target is less than 20% of the maximum MTF50 obtained by this lens. The data cover 18 bodies of six brands, 114 lenses from 11 manufacturers for a total of 162 bodies/lenses tests.
Without rigor we could draw a bar plot with on the X axis the percentage of auto-focus error and on the Y axis the frequency of lens tested showing this auto-focus error (Figure 1) for each brand. Samsung was removed in this figure, since only one test was available.
Based on this figure the conclusion is : Nikon gets the best results with 70% of tests with a low auto-focus error (between 0 and 5% of error) and only 10% of tests with high auto-focus error (more than 10% of error) , followed by Sony and Canon, Pentax and Olympus with worst results (45% of good results – 0-5% error – and 25% of high levels of error – > 10% -) . However this figure is terribly misleading, because other variables can influence these tests: the model of the body, the lens manufacturers, lenses model, and aperture…
Here I take into account these problems by using a multivariate analysis. I include all the variables in the same analysis: brand and model of the body; brand, model and maximum lens aperture, focal length, minimum and maximum zoom factor (maximum focal/minimum focal length), ultra-sonic motor, lens stabilization. The purpose of this analysis is to understand which variables are the most important for the focus to be accurate. The analysis is called a regression tree. This approach allow us to separate data according to the variable best explaining their distribution. For each new branch, the regression tree renews the calculation and separates data on the basis of the new variable best explaining the separation. The full description of the statistical method is explained at the end of the document.
Cross brand analysis
The results, Figure 2, are very different from Figure 1. The body brand is no longer important, whereas camera model is the best variable that separate the data. The aperture and then lens brand also influence the percentage of focus errors.
Figure 2: Auto-Focus Error Multivariate Regression Tree with Random Effect. The averaged auto-focus error in percentage is displayed at the end for each tree branch. f stands for the lens maximum aperture.
Two groups of bodies appear at the first division, a group composed of the Canon 1Ds Mark III, 50D, 5D, 7D and Sony A200, A900, the Nikon D200, D3, D3x, D80, Pentax K10D, the Olympus E-3 and the Samsung GX-10. This first group has an average focus-error of 4.5%. The second group includes the Canon 1Ds Mark II and 20D, the Sony A100, the Olympus E-510 and the Pentax K20D with an average focus-error of 10%. Lens aperture influence the focus-error for both groups. Ultrafast lenses (around f/1.4) show higher focus-error than other lenses. In the third division level, the results highlight the lens brand. Canon, Nikkor and Zeiss give the best results. Olympus, Pentax and Sony follow, and Tamron and Tokina tail these brands. It is worth mentioning that Sigma lenses give either excellent or poor results. For Schneider-Kreuznach and Leica lens we do not have enough data (one and two test respectively) for drawing any conclusions.
Before going further, we can summarize the result as follow: body brand does not influence focus-errors contrasting with results shown in Figure 1. Finally, the most important factor is the aperture. Very bright lenses, mainly f/1.4, give poor results. This was predictable because of their shallow depth of field that requires fine calibration. Since the industry is counting mostly on zooms that reach a maximum aperture of 2.8, auto-focus systems are surely calibrated for these apertures and not for larger ones. Body construction also is optimized for those apertures – at least the viewfinder brightness. Indeed, just try and see a viewfinder brightness difference when using a f/2.8 zoom lens compared to a fast f/1.4 lens. Digital cameras do not show an increase in viewfinder brightness or a difference in depth of field while film cameras from 1960-1980 do (e.g. Nikon F2 with DP-12).
Nikon versus Canon
I guess everyone wants to see a direct comparison between Canon and Nikon. So let’s see Figure 3, showing the same analysis but only with these 2 manufacturers Again no difference between the two brands, except that the Canon 1Ds Mark II and 20D have a weaker performance than any other bodies. For these other bodies, aperture is the main factor, followed by lens brand. I will come back on the results of the 1Ds Mark II and the 20D a bit further.
Figure 3: Auto-Focus Error Multivariate Regression Tree with Random Effect. The averaged auto-focus error in percentage is displayed at the end for each tree branch. f stands for the lens maximum aperture.
Another test with Nikon and Canon and their own lens shows equivalent results (Figure 4). Again, no difference between the two brands, aperture is the best variable, followed by the model of the body.
Figure 4: Auto-Focus Error Multivariate Regression Tree with Random Effect. The averaged auto-focus error in percentage is displayed at the end for each tree branch. f stands for the lens maximum aperture.
Discussion
In this article I show that auto-focus accuracy mainly depends on the lens aperture and lens brand.
Auto-focus is the most accurate for intermediate lenses (aperture between 2.8 and 4). Brighter lenses’ shallow depth of field requires a fine calibration of the camera auto-focus system which didn’t seem to be made in factory.
The difference between Canon and Nikon with >=f/2 lenses is not significant.
Data used in this article come from a controlled environment (studio and target). An error of 4% seems the norm in these conditions, with 6-8% for very bright lens (f/1.4). Results suggests that focus-error may increase with any light loss. Consequently, outdoor, except in full sun with a subject well lit, an error of 10-15% should be the norm and you should expect even more focusing error with a moving subject.
I hope this document will provide professional and amateur photographers a more accurate understanding of auto-focus errors.
Methods
Data were analyzed by a non-parametric methods, a regression tree (Venable and Ripley, 2002) in its multivariate form (De’ath, 2002). However, we needed to control for non independent data (same lens tested on multiple bodies). This issue was addressed by the addition of random effects (Sela and Simonoff, 2010). The analysis was carried out under R (CRAN, 2011) with the REEMtree package (Sela and Simonoff, 2010). Post analysis verification at each node was made by Wilcoxon rank-sum test or Kruskal–Wallis one-way analysis of variance by ranks depending on data distribution at the node.
PDF version
A PDF with high resolution figures will be available for download here.
