< ! --Digital window verification 001 -->

Auto-focus accuracy: a scientific cross brand analysis (guest post)

The next guest post is by Lionel Humbert who took a scientific approach when examining the AF accuracy acros camera brands:

Many discussions emerge on the Internet about image quality, noise, focus accuracy... when a new camera hits the market. Some people move from one brand to another based solely on some brief, vague and not justified reports viewed on the web. Generally tests published on the Internet are by design not reproducible and biased. A scientific approach will required multiple bodies and lenses, and at least more than one sample for each model because of small manufacturing differences that may influence results.

In this article I will discuss the accuracy of focus with a rigorous analysis using modern statistical methods on data collected from a well known website (www.lenstip.com, 8000 visits per day). This analysis is focused on their tests because it is the only website that reports the accuracy of focus for each lens. They get a percentage of error, measured as the number of times the resolution on the target is less than 20% of the maximum MTF50 obtained by this lens. The data cover 18 bodies of six brands, 114 lenses from 11 manufacturers for a total of 162 bodies/lenses tests.

Without rigor we could draw a bar plot with on the X axis the percentage of auto-focus error and on the Y axis the frequency of lens tested showing this auto-focus error (Figure 1) for each brand. Samsung was removed in this figure, since only one test was available.

Figure 1: Barplot of Auto-Focus error per brand

Based on this figure the conclusion is : Nikon gets the best results with 70% of tests with a low auto-focus error (between 0 and 5% of error) and only 10% of tests with high auto-focus error (more than 10% of error) , followed by Sony and Canon, Pentax and Olympus with worst results (45% of good results - 0-5% error - and 25% of high levels of error - > 10% -) . However this figure is terribly misleading, because other variables can influence these tests: the model of the body, the lens manufacturers, lenses model, and aperture...

Here I take into account these problems by using a multivariate analysis. I include all the variables in the same analysis: brand and model of the body; brand, model and maximum lens aperture, focal length, minimum and maximum zoom factor (maximum focal/minimum focal length), ultra-sonic motor, lens stabilization. The purpose of this analysis is to understand which variables are  the most important for the focus to be accurate. The analysis is called a regression tree. This approach allow us to separate data according to the variable best explaining their distribution. For each new branch, the regression tree renews the calculation and separates data on the basis of the new variable best explaining the separation. The full description of the statistical method is explained at the end of the document.

Results

Cross brand analysis

The results, Figure 2, are very different from Figure 1. The body brand is no longer important, whereas camera model is the best variable that separate the data. The aperture and then lens brand also influence the percentage of focus errors.

Figure 2: Auto-Focus Error Multivariate Regression Tree with Random Effect. The averaged auto-focus error in percentage is displayed at the end for each tree branch. f stands for the lens maximum aperture.

Two groups of bodies appear at the first division, a group composed of the Canon 1Ds Mark III, 50D, 5D, 7D and Sony A200, A900, the Nikon D200, D3, D3x, D80, Pentax K10D, the Olympus E-3 and the Samsung GX-10. This first group has an average focus-error of 4.5%. The second group includes the Canon 1Ds Mark II and 20D, the Sony A100, the Olympus E-510 and the Pentax K20D with an average focus-error of 10%. Lens aperture influence the focus-error for both groups. Ultrafast lenses (around f/1.4) show higher focus-error than other lenses. In the third division level, the results highlight the lens brand. Canon, Nikkor and Zeiss give the best results. Olympus, Pentax and Sony follow, and Tamron and Tokina tail these brands. It is worth mentioning that Sigma lenses give either excellent or poor results. For Schneider-Kreuznach and Leica lens we do not have enough data (one and two test respectively) for drawing any conclusions.

Before going further, we can summarize the result as follow: body brand does not influence focus-errors contrasting with results shown in Figure 1. Finally, the most important factor is the aperture. Very bright lenses, mainly f/1.4, give poor results. This was predictable because of their shallow depth of field that requires fine calibration. Since the industry is counting mostly on zooms that reach a maximum aperture of 2.8, auto-focus systems are surely calibrated for these apertures and not for larger ones. Body construction also is optimized for those apertures - at least the viewfinder brightness. Indeed, just try and see a viewfinder brightness difference when using a f/2.8 zoom lens compared to a fast f/1.4 lens. Digital cameras do not show an increase in viewfinder brightness or a difference in depth of field while film cameras from 1960-1980 do (e.g. Nikon F2 with DP-12).

Nikon versus Canon

I guess everyone wants to see a direct comparison between Canon and Nikon. So let's see Figure 3, showing the same analysis but only with these 2 manufacturers Again no difference between the two brands, except that the Canon 1Ds Mark II and 20D have a weaker performance than any other bodies. For these other bodies, aperture is the main factor, followed by lens brand. I will come back on the results of the 1Ds Mark II and the 20D a bit further.

Figure 3: Auto-Focus Error Multivariate Regression Tree with Random Effect. The averaged auto-focus error in percentage is displayed at the end for each tree branch. f stands for the lens maximum aperture.

Another test with Nikon and Canon and their own lens shows equivalent results (Figure 4). Again, no difference between the two brands, aperture is the best variable, followed by the model of the body.

Figure 4: Auto-Focus Error Multivariate Regression Tree with Random Effect. The averaged auto-focus error in percentage is displayed at the end for each tree branch. f stands for the lens maximum aperture.

Discussion

In this article I show that auto-focus accuracy mainly depends on the lens aperture and lens brand.

Auto-focus is the most accurate for intermediate lenses (aperture between 2.8 and 4). Brighter lenses' shallow depth of field requires a fine calibration of the camera auto-focus system which didn't seem to be made in factory.

The difference between Canon and Nikon with >=f/2 lenses is not significant.

Data used in this article come from a controlled environment (studio and target). An error of 4% seems the norm in these conditions, with 6-8% for very bright lens (f/1.4). Results suggests that focus-error may increase with any light loss. Consequently, outdoor, except in full sun with a subject well lit, an error of 10-15% should be the norm and you should expect even more focusing error with a moving subject.

I hope this document will provide professional and amateur photographers a more accurate understanding of auto-focus errors.

Methods

Data were analyzed by a non-parametric methods, a regression tree (Venable and Ripley, 2002) in its multivariate form (De'ath, 2002). However, we needed to control for non independent data (same lens tested on multiple bodies). This issue was addressed by the addition of random effects (Sela and Simonoff, 2010). The analysis was carried out under R (CRAN, 2011) with the REEMtree package (Sela and Simonoff, 2010). Post analysis verification at each node was made by Wilcoxon rank-sum test or Kruskal–Wallis one-way analysis of variance by ranks depending on data distribution at the node.

PDF version

A PDF with high resolution figures will be available for download here.

This entry was posted in Other Nikon stuff and tagged . Bookmark the permalink. Trackbacks are closed, but you can post a comment.
  • http://www.flickr.com/photos/manarianz5/ manarianz5

    huhu nice……..,

    • PeterZheng

      when maximal aperture was same, but a shorter focal-length lens is usually with larger AF-error showed up in the maximal aperture. (despite the Nikkor 35mm f1.4G and 35mm f2D were with better state, but general cases still exist of foregoing that.)

      when the focal-length was same, but a larger aperture lens is usually with larger AF-error showed up in the maximal aperture.

      well then, the larger aperture of wide-angle lens existed of more difficulties.

      excepting the macro lens, why the many lenses are such cases? (a short distance target and the AF-error in maximal aperture)

      In essence, the said two situations actually have a same intrinsic factor, the off-axis beams of light have the higher obliquities would easily lead to a larger spherical aberration, and thus the autofocus-error will be risen, (for the PD-AF in maximal aperture).

      However, looks the PD-AF having the larger focus-error in maximal aperture if got large spherical aberration, but the optimum imaging aperture is still to be likely a good accuracy AF, when lens had having a not bad design. The optimum aperture’s imaging performance needed to get a priority consideration, that is frequently saw things.

      Commonly, the maximal aperture’s spherical aberration correction had to make the necessary compromises. Most of the designs were impossible to take blindly pursued the spherical aberrations get a zero, which is also impossible thing in fact.

      a farther discussion is being in NikonRumors Forum:
      nikonrumors.com/forum/topic.php?id=3413

  • Nikku

    good to know that the Yellow team leads !
    But wait, it was clear unless you have lived in a cave the last few years
    LOL

    • soap

      Uh, that is not a safe conclusion to draw from the data presented… Nor is it THE conclusion demonstrated.

      • Global

        True, but its the conclusion that LOOKS best. :-P

        “10% what you say — 90% how you say it.” :-P

    • Jan

      To see the Canon 7D with 19 cross points perform worse than the ancient Nikon D80 with 1 cross point.
      Can expect Canon fanboys to come out and flame this!

      • http://loewald.com/ Tonio

        Do you understand the concept of “not statistically significant”?

        • John

          Wait, are you saying that 19 cross points isn’t statistically significant? LOL!

      • http://www.EltonSaulsberry.com Elton Saulsberry

        Don’t forget only one type of focusing was measured: fixed body, fixed subject. At least I think so. In this case only the 1 central cross point may have been used. In tests with a moving target different factors may have lead to different results. While this is a very interesting analysis, “best” is a very complex question.

      • Whocares

        I have a D90 and a Canon 40D, and the Canon is a lot more reliable than my Nikon…

        • arutha

          The Nikon D90 is a piece of crap when it comes to focus accuracy. I had two such bodies, and both produced miserable results when used with fast primes (f/1.4). A totally unreliable camera.

  • soap

    A most excellent article, Lionel.

    My one question is I’m not quite following the Sela and Simonoff reference to “random effects” and regression trees. My google-fu may be lacking tonight, but I’m not finding the paper either.

    Do you have a (non-paywalled) link to the paper?
    Am I totally off my rocker for reading random effects and thinking about dithering?

    Thank you again.

    • Lionel

      Hi,
      Thanks for your comment. The paper of Sela and Simonof is freely available at http://archive.nyu.edu/handle/2451/28094
      Please feel free to contact me directly if you have more questions on the methods.
      Regards

  • Global

    “It is worth mentioning that Sigma lenses give either excellent or poor results.”

    Nice to know they are consistently one or the other. =P

  • Funduro

    Is Sonic the Hedgehog a cousin of Sonic Motor? Nice info. Another great guess post.

  • Kurtis Kronk

    I have to say this is an interesting test, but it hardly tells the whole story. The only way to really test focus on these things is in the field which of course is not scientific. My canon 5D, 5D II and 1Ds III used to have trouble focusing randomly for portraits and event work. Nikon D3 doesn’t seem to have the same random misfocusing habits using similar lenses. No comparison IMO.

    • http://www.meteostra.it/dslrank Nicola

      That’s it!
      For anyone who said in the past “nikon must respond to the 5DmkII” “you can’t compare low end nikons with the 5DII,it’s a pro body!”(double facepalm), EAT THIS.

  • Kaze kaze

    Good effort, very informative.
    One thing though, I was wondering was or was not filter (protective/ UV/ skylight/ single-coat/ Multi-Coated) being used on any or some of the lens? (Me and my computor have failed to find any reference on the article) I’m a UV/ protective filter user/ believer myself, currently my primary preference are either B&W xs-pro and HOYA HD, not because I got too much cash lyring around but they suit the bill for being multi-resistance-coated and relatively slimer frame, so I was wondering what does others normally do?
    +1 to Sigma being “consistantly inconsistant”, half the people loved it and the other equally half just hated it. It reminds me of a cafe downtown, one day you wanted to go back, the other day you regarded going back… you get the drift.

    • http://www.meteostra.it/dslrank Nicola
    • Lionel

      Hi,
      There was no filter used on lens. I’m sorry that I can’t answer to this question.
      Regards

    • Kaze kaze

      Nicola: Thanks for the link man, never saw it until just now.

      Lionel: That’s cool, but for the record, I know most pro glass got either a UV filter (multi coated, 77mm for “modern glass) or a plain protective filter (often single coated only) out of habit, except some studio-used-ones (many are medium format, ie, H line or L line) which are relatively non-economical to put on (rare/ over-sized, often see more artifical lights / non-UV or got “caged” in some pelican case anyway). My point is, for practical/ real-life reason, would the additional filter factor have change the result for better or worse? I don’t know, may be that’s for the next round of test then, cheers.

      • Bud

        Please try not to generalize. I have many ‘pro’ quality lenses and never use filters (except on the rare occasion I use a circular polarizer) for protection or otherwise. I do use ‘caution’ when using all of my gear but that is very inexpensive and freely available to everyone. It also has the side benefit of having absolutely no impact on the quality of my photos.

        • http://micahmedia.com Micah

          …ok, lets see some shots.

  • jerl

    Nice work, good analysis, although I think we need to be careful about what conclusions we draw here.

    Of course all this depends on how accurate the lenstip number is. I’m a fan of their site, as they seem to be one of the few that apply the same sorts of lens tests to many different lenses from many different brands, but there is still considerable uncertainty about their “focus accuracy” remarks.

    For instance, we don’t know how many trials were run, nor do we have a clear idea of how the testing was performed. We also don’t know how far off the “missed” results were. This sort of thing is also important to know, although is difficult to perform since most of us don’t have access to all of these lenses.

    Again, good article, and hopefully we can use this to dispel some of the myths running around about cameras.

    • Lionel

      Thanks for your comments. The number of trials is unfortunately not given for many lens tested. However with their description of testing methods the number of trials is between 50 to 100 depending on the lens.
      It clear that results would have been more interesting if the error in term of resolution difference were reported.
      Regards

  • slightly fuzzy

    Since I assume all of Nikon’s and Canon’s optical engineers went to the same schools and studied with the same textbooks, and probably benchmark their products against one another, it’s not surprising that their focusing rates are so similar.

  • Foxtrotski

    This way of presenting data is awkward. The author states that aperture is the most important parameter. Yet, according to fig 2 on the left branch, f < 1.6 have error of only 4.4 vs 9.8 for larger. His claim only applies to the right branch on those bodies. I appreciate that there are many variables. So why not plot against each and use error bars? For example, focus error vs lens aperture, brand, model, and so on. I want to see some SDs. This type of analysis makes any conclusion complicated and is therefore nearly useless.

    Second where are the updated brands? No d90, 300, 7000 etc

    Thanks for all the effort in getting objective data on this. This emphasis is critical.

    • kyoshinikon

      Im with you on the oldies… A D200? The D300, D300s, D90, and D7000 (to some extent) have indefinitely replaced the D200 and D80 in AF superiority and yet the newer 7D and 50D get a spot…

    • Lionel

      Hi,
      The whole point with this kind of data is to use a multivariate approach. At each level I could have show results for each variables, but it would have been too messy.
      If you look carefully at the figure 2, the first division is made by the camera body, the second division for each branch created at first is done by the aperture. The important thing here is not the value of focus error, but the fact that the variable chosen by the analysis to divide all the data (after bodies) is the aperture. With these results we could sum up like that: tel me your lens aperture and I will give you the focus error of it.

      Concerning you second question, unfortunately they haven’t updated all their bodies. I’ve reported all data I’ve found.

      Regards

  • Brick Yang

    according the Figure 1, while f<2.2, the result of f=1.6. Does it mean that the worst lens is the f=1.8 one? and the f/1.4 lens is much more better?

    • Bigus Dickus

      1.8 lenses are usually cheap, but optically fast lenses for amateurs.

      1.4 are with rare exception of Nikkor 50G all pro lenses.

      • PiXLPeeper

        Bigus? U sure?

    • Lionel

      Yes that’s exactly the case. 1.4 lens give better results than 1.8. The reason could be the one given by Bigus Dickus.

      • Brick Yang

        so you can’t say “Very bright lenses, mainly f/1.4, give poor results” and “Ultrafast lenses (around f/1.4) show higher focus-error than other lenses”.

        the worst lens should be the f/1.8, right?

        • Lionel

          Yes 1.8 lens are the worse when the cut-off between =1.6 is made. Which is not always the case. However a separation at 1.9 or 2.2 is always present and in both cases 1.4 lens represent the majority of the data.

  • Goose

    i cant find my balls after reading halfway.

  • Maverick

    Goose, even you can get laid in a place like this. This cracks me up and I, too, lost my nuts about half way. Just get out and shoot people..who cares what camera. But i guess without threads and all these crazy reviews, the internet and photo sites would be a lot less interesting

    • Lionel

      Generally I’m outside shooting, but our Canadian winter struck me inside for a while !
      Cheers and happy shooting.

      • http://www.EltonSaulsberry.com Elton Saulsberry

        It’s still winter there? Bummer dude.

  • One More Thought

    Great post…thanks for all of the technical analysis! These guest posts are quite a treat, and each writer puts a great effort into it.

    As to the eternal Nikon vs Canon AF debate…the conclusion makes sense. While I love Nikon AF, Canon can’t be as far behind as some try to paint it, or else there would be no pro’s using Canon. As it is, however, we see plenty of pro’s using Canon very effectively, and plenty using Nikon effectively. That speaks to the fact that both brands are top notch.

    • Darkness

      Not really, its expensive to change everything, and Canon will calibrate for those that cant.

  • horseboy

    is the conclusion of this then that nikon verus canon = draw. I personally feel that nikon as good af consistency across all models where as canons entry model using the old 9 point layout is just ok but the 19 point system in the 7D and the 45 point system in the 1D and 1Ds are much better. Canon needs to ditch the 9 point system especially in the 5D mark II.

  • Ant

    This is interesting, and an interesting use of regression tree analysis. I guess the conclusion isn’t exactly unexpected. If same-level equipment from both brands is going to deliver the same level of technical performance, then the biggest x-factor is definitely going to be the person pressing the buttons.

    • http://www.flickr.com/photos/friedtoast/ Fried Toast

      What wondering how long it would take for someone to mention the most critical part of the equation: The User.

  • TaoTeJared

    Great post!

    As a full time number cruncher I would love the raw data to play with myself to see what I could find. I would probably remove the older cameras first and work from there.

    • soap

      No need to remove the older camera bodies from the pool. If they had been a statistically significant deviation from their brethren the tree would have branched on that.

      The fact they didn’t branch indicates their data is comparable.

  • Ken Elliott

    Nice analysis. Thanks for putting in all the work.

    It appears that your conclusion is the camera brand doesn’t matter under OPTIMUM conditions. When the light drops, the target is moving, and/or low contrast, I expect you’ll see quite a difference between systems.

    It would be interesting to take a few of the better performers and see what happens when you drop light levels.

    • Lionel

      The important point of having controlled environment results, is to grab the base line.
      If we do a little more speculative conclusions, the effect of light drop can be deduced from the results. Lens with 2.8 opening does a better job than 3.75 or higher aperture lens. If you look at these results (>=3.75 <3.75 in figure 2, 3, 4 and 5) it's roughly 4% of error against 8%. Then we could say that when you loss one stop, at worst, you double your focus error. But as I say it's speculative.
      Regards

  • Kurgan

    There’s one thing, one VERY important thing, missing from this test: focusing on a static, high-contrast target is one thing but what happens when the target isn’t static or the contrast of the target is gradually reduced? After all, when test driving a car you don’t just drive it along a straight road in second gear, right?

    So I would like to see more exhaustive testing. For example, you could put a target on a track and move it towards the camera at a constant speed and see how well the camera/lens combo tracks it. By increasing the speed in consecutive runs you can get a pretty good idea of the speed and accuracy of the AF system. Likewise for the contrast limits of the AF system, you start with a black & white target and move on to grey / white with ever lighter shades of grey. It requires more work of course but not all that much and it’s not that complicated either.

    • http://www.eaglewheel.us bikinchris

      Kurgan said:
      “So I would like to see more exhaustive testing. For example, you could put a target on a track and move it towards the camera at a constant speed and see how well the camera/lens combo tracks it. By increasing the speed in consecutive runs you can get a pretty good idea of the speed and accuracy of the AF system. Likewise for the contrast limits of the AF system, you start with a black & white target and move on to grey / white with ever lighter shades of grey. ”
      Actually that woudl be very interesting to me, but it would take alot of time and money. Also, an interesting third test would be an object of varying contrast that moves at an unpredicatable rate. As a sports photographer, I would be very interested in that.

      • Kurgan

        A test with variable contrast and movement rate would be the most realistic but also impossible to accurately reproduce time and time again. The first rule of analysis is to reduce things to just one variable whenever possible.

    • big eater

      Anyone with an electric train set want to give this a try? Set up a focus target on the train and let ‘er rip!

  • horseboy

    Quote “In this article I will discuss the accuracy of focus with a rigorous analysis using modern statistical methods on data collected from a well known website (www.lenstip.com, 8000 visits per day).”

    I assume this mean he is using data gathered from that website and not his own measurements?

    • Darkness

      And they use dcraw as their RAW conversion ‘standard’? Hilarious.

      • http://micahmedia.com Micah

        Exactly!

  • horseboy

    Quote “Based on this figure the conclusion is : Nikon gets the best results with 70% of tests with a low auto-focus error (between 0 and 5% of error) and only 10% of tests with high auto-focus error (more than 10% of error) , followed by Sony and Canon, Pentax and Olympus with worst results (45% of good results – 0-5% error – and 25% of high levels of error – > 10% -) . However this figure is terribly misleading, because other variables can influence these tests: the model of the body, the lens manufacturers, lenses model, and aperture…

    He makes a very wise point here.

  • Darkness

    Yawn. Flawed source, apples vs pears.

    • soap

      You’ve stated that multiple times, yet have offered no sound reasoning as to why the source is flawed.

      Argumentum ad nauseam.

  • JerryPizza

    Jesus christ get a life

    • http://www.meteostra.it/dslrank Nicola

      You won an espresso

  • Anonymous

    please post the design matrix for the random effect. you should report the confidence interval. what is the mean squared error using the leave one out cross validation when compared to other methods, eg no random effect, spline or kernel?

  • thorgal

    lenstip.com is from Poland – i’m proud of it! :)

  • Phil

    I must admit, I don’t understand Lionel’s graphs at all. Is there a simpler way way to graph that data? The bar chart at the top was pretty straightforward.

    I myself don’t at this time have any real-world focus comparison images from a Nikon body/lens combo, but a while back I had to retouch a series of images, 100 of them to be exact, from a Canon EOS-1Ds Mark II with a 28-135mm f4.6 optic (according to EXIF).

    To my surprise, 90% of the image (fashion shots) were focused on the wrong place in the image (which should have been the eyes). While it wasn’t objectionable viewing the images below 50% magnification, retouching at 100% made it painfully obvious.

    I must say I was rather surprised at this high degree of focusing error in this body. The photographer was an experience shooter in this realm of photography. While he has to be considered a variable to some degree, I can’t imagine this guy being so consistently bad at something he’s been doing for quite a while.

    I would be very curious indeed to see how a Nikon body/lens combo would fare in such a shooting session.

    • http://micahmedia.com Micah

      Never underestimate the power of doing it wrong.

      This is the biggest lesson I’ve learned from working on other people’s images. And why I don’t like to.

    • Lionel

      I thanks for your comment. I will try to explain graphs a little bit. The easiest way of seeing them is, as the name imply, to imagine a tree.
      You have the trunk which appear at the top of each figure and then start the main branches. And each of the main branch divide into secondary branches.
      Like a real tree, if you cut the last branch, at its top, you just make an hair cut. The tree will continue to live.
      If you cut one of the main branch here the probability the tree would die is higher.
      Here is the same thing, the most important factors affecting auto-focus error are near the trunk, in the upper part of the figure. Then when you go down, these factors are less important.
      For example in Figure 4:
      The main factor is the aperture with a cut-off at 1.9, then at the second level it’s the body for >=1.9 lens and the aperture for Bodies and Aperture > Aperture.

      Thanks again for your comments.

      • Lionel

        Somme of my text was cut:
        The main factor is the aperture with a cut-off at 1.9, then at the second level it’s the body for >=1.9 lens and the aperture for Body and Aperture > Aperture.

        • Lionel

          Cut again:
          The order of importance of factors can be summarize like this:
          1) aperture; 2) body and aperture; 3) aperture

  • Vladi

    What about film cameras? My F100 blows all these digitals out of the water ;D

    • Stuff

      Hear that sound?

      That’s your horse out back neighing for hay.

    • http://micahmedia.com Micah

      Having owned an f100, I can’t disagree more. The focus system in the D700/D300/D3 series blows the f100 away. Although I learned last week that my D700 is NOT as waterproof as my f100 was. So out of the water the D700 wins. The f100…blows the D700 in the water? Er, something like that.

      • donald

        can I ask where your D700 let in water (if you know) ? I used mine on a long yacht trip and despite being used in heavy rain and getting seawater splashes (trying to shoot waves coming over the bow) I never had a problem with it

      • http://www.flickr.com/photos/friedtoast/ Fried Toast

        I’m curious as well (not arguing, just curious). I’m pretty liberal about letting my D700 play in the rain, so I’m always up for (trying to) learn from other peoples’ mistakes ;)

  • Back to top