Lens Discussions - DxO Lab test etc ...

Nikon Rumors Forum » Nikon DSLR

[closed]

Lens Discussions - DxO Lab test etc ...

(70 posts) (19 voices)

Tags:

« Previous 123 Next »

Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

bjrichus said:
In defiance of the numbers game, Paperman: As the OP have *YOU* tried out any of these lenses yet? What results do *YOU* get with them using your preferred real world subjects, *NOT* shooting test targets?

I haven't tried any expensive glass on my D300 ( but seen it makes a difference on a FF DSLR ) I might probably find some to rent and if I do , I'd love to share my opinion of them .

A question ,though ....

How can a lens be NOT SHARP in test target shoots but be SHARP in real life?

Or let me state the opposite .

How can a lens be SHARP in real life and not so in lab tests ? :-)

Posted 2 years ago #
jerl

preferred member
Joined: Dec '10
Posts: 425
offline

In real life, nobody shoots perfectly flat test charts in controlled laboratory conditions. Thus, much of what you are shooting will be out of focus anyway (perfect focus only ever exists in a single plane, no matter what your aperture is), so things like corner/edge resolution or flatness of field aren't as relevant. Also, real life shooting doesn't usually consist of shooting high contrast targets at high spacial frequencies. Test charts are designed to show the limitations of lenses, but in most real life situations, it is not the lens, but the shooting conditions (diffraction, objects that are outside the plane of perfect focus, motion blur, seeing conditions, haze, misalignment in your camera, poor technique, etc) that limit sharpness.

Also, "sharpness" is relative term. At what point in an MTF curve do you say a lens is soft? How well defined does an object have to be for it to be "sharp"? Different people define those areas in different places. It could be that your standards are much higher than everyone else's.

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Paperman said:
If you are talking about what "troubles me in the test figures " - nothing is ... It is "surprising " rather than "troubling" to see a $200 kit zoom surpass the 10x more expensive pro glass in sharpness ( and also in CA/vigneting as seen example #2 ).

I mentioned the long tail before, and if you don't know what that phrase means please take a second and read:
http://en.wikipedia.org/wiki/Long_Tail
as it is an important concept in lenses and violins alike.

Making a sharp lens is easy. Making a zoom lens is easy. Making a fast lens is easy. Making a cheap lens is easy.

The issue is nobody (except crazy birders) wants a huge prime costing $5000 to be their primary lens. So we are left with compromises.

The kit zooms are sharp because sharp zooms are easy today. Glass is better and cheaper than it ever was, lens modeling is a science and no longer an art, and if I handed a 1950's lens engineer a glued aspherical element he'd shit his pants when told how inexpensive and easy it was.

Sharp FAST zooms are not. The pro lenses have more factors to balance, and thus (counter-intuitively) more compromises to make.

The other point I believe you may not have taken into account is this:
Over what part of the lenses zoom range is the image quality consistent?
I don't have all the numbers, but a faulty conclusion to draw from those dxo numbers is that the 18-135 has 50 lp/mm of resolving power at any other setting than 24mm and f/3.8.

It could have only 40 lp/mm of resolving power at all other focal lengths and apertures while the 17-35 ALWAYS has ~45 lp/mm. Don't look at those peak numbers and extrapolate without evidence that they signify what is going on elsewhere.

The only conclusion I can reach looking at these lens tests is that this is as good as it gets with the 12 Mp APS-C .

What is the resolution of 12MP APS-C? Divide that by the size in mm.
The theoretical perfectly formed line pair only takes how many pixels?
What is the lp/mm resolution of these lenses based on their tested MTF?
;)

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Paperman said:
How can a lens be NOT SHARP in test target shoots but be SHARP in real life?

People being defensive about their purchase.

Paperman said:
Or let me state the opposite .

How can a lens be SHARP in real life and not so in lab tests ? :-)

It can't. Lab tests are easy and as jerl perfectly said real life is hard, unflat, non-linear, and poorly lit.
Lab tests are the BEST a lens will ever do.

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

@Jerl ... Very good explanation but it still brings me back to the same question .

If expensive glass can produce sharp images in real life/landscapes etc. , why can't it then produce the same tacky result on a lab test ?

How can the cheap kit zoom CAN ?

Why is the quality on expensive glass always something UNMEASURABLE ?

jerl said:arpness.

Also, "sharpness" is relative term. ....It could be that your standards are much higher than everyone else's.

I agree totally . That's why I tend to conclude that my expectations may have been too high and this is as good as it gets with 12 Mpixel APS-C .

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Paperman said:
Very good explanation but it still brings me back to the same question .

If expensive glass can produce sharp images in real life/landscapes etc. , why can't it then produce the same tacky result on a lab test ?

How can the cheap kit zoom CAN ?

Why is the quality on expensive glass always something UNMEASURABLE ?

I agree totally . That's why I tend to conclude that my expectations may have been too high and this is as good as it gets with 12 Mpixel APS-C .

As I said in my original reply to your post, you won't see the difference between 50 and 45 lp/mm. They are both plenty sharp (assuming that's corner resolution, is it?). You want sharper? Go to a larger format.

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

Testing123 said:
What is the resolution of 12MP APS-C? Divide that by the size in mm.
The theoretical perfectly formed line pair only takes how many pixels?
What is the lp/mm resolution of these lenses based on their tested MTF?
;)

Hmmm ...2848p/18mm = 157 You need 2 pixels to form a line so 78 lw/mm should be max - theoritically? So we're not there yet - theoritically :-)

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

Testing123 said:
People being defensive about their purchase.

It can't. Lab tests are easy and as jerl perfectly said real life is hard, unflat, non-linear, and poorly lit.
Lab tests are the BEST a lens will ever do.

:-) :-)) :-))

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

My previous answer disappeared - so once again but shorter ...

I know the "too small sheet for bedsize" example used for lens compromises . You cover one end and the other end is in the open :-)

Testing123 said:
The other point I believe you may not have taken into account is this:
Over what part of the lenses zoom range is the image quality consistent?
I don't have all the numbers, but a faulty conclusion to draw from those dxo numbers is that the 18-135 has 50 lp/mm of resolving power at any other setting than 24mm and f/3.8.

It could have only 40 lp/mm of resolving power at all other focal lengths and apertures while the 17-35 ALWAYS has ~45 lp/mm. Don't look at those peak numbers and extrapolate without evidence that they signify what is going on elsewhere.

Valid point but not so much for the range I'm using in my 18-135mm . I am never above 30-35mm and sharpness is consistent till over 70mm or so ( other lab tests ) .Smaller apertures are also sharper than the f3.8 indicated on DxO . But I see what you mean .

And for the 17-35mm f2.8 . For such a short range , it'd better be consistent :-))

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Paperman said:
Hmmm ...2848p/18mm = 157 You need 2 pixels to form a line so 78 lw/mm should be max - theoritically? So we're not there yet - theoritically :-)

2848 pixels over 15.8mm high actually (I went and looked up the numbers too). = 180 p/mm = 90 lp/mm. Don't forget you're also running through the bayer filter. RED claims the Bayer filter's effect on absolute resolution is a 22% hit, so call that 70 line pairs theoretical for your D300.
Also RED has every reason to claim minimal impact. I bet it's greater than that in real life. They're competing with 3-sensor video camera specs.
Take into account the blurring AA filter and the mid 50's seen by the best lenses is not unbelievable at all as your theoretical cap.
If you're chasing the last 10% of resolution (45lp vs 50lp) you might want to experiment with different RAW engines. The demosaic routine used can have a significant impact on how much absolute resolution is recoverable from any given file.

http://www.reduser.net/forum/showthread.php?t=12026

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

Thanks ...

Now let's go to the 14 Mp 1/2.7" point & shoots and calculate the theoritical maximum lines that will fit in quarter the space of their sensors :-))

Posted 2 years ago #
Correlli

preferred member
Joined: Jan '11
Posts: 264
offline

Let me throw a few thoughts in here as well.

Paperman said:
Now let's go to the 14 Mp 1/2.7" point & shoots and calculate the theoritical maximum lines that will fit in quarter the space of their sensors :-))

Have a look at http://6mpixel.org/en/. This website has not been updated in quite a while but they basically say that P&S cameras with more than 6 MP loose resolution because of the de-noise filtering required.

Paperman said:
Why is the quality on expensive glass always something UNMEASURABLE ?

I think it is, you just can't reduce this to one number - except for the price maybe :) Take the resolution figure at the DxO website: The resolution figure is an average for each focal length, f-stop and weighted throughout the field. As an engineer you know that an average without a standard deviation is only part of the truth.

Also there are different expectations on quality glass. Your expectation is sharpness. Others look for minimal distortion, speed etc. I would say that when it comes to resolution most modern lenses (and even some old ones as well) do have a very similar performance and the difference between the lenses is in some cases less than for the same lens at different f-stops (I remember one test at Ken Rockwells site about 50 mm primes that shows this, but I have seen others as well).

Regarding sharpness: the problem here is, that there is no real measure for sharpness. What we call sharpness is actually a mixture of resolution and contrast. The closest measure for this is the MTF.

Paperman said:
How can a lens be SHARP in real life and not so in lab tests ? :-)

Because in a lab test you look at the image with a magnifier, in real life you should not. One example for that would be the formula for calculating the depth of field. It has got a parameter called the circle of confusion. For an FX imager this is set to 0.03 mm, for a DX it is 0.02 mm. If you take the FX D700 the 0.03 mm is roughly three times the pixel size. So this formula assumes that something that spreads over three pixels is still regarded in focus and not blurred! Why? Because they also assume that you are watching a print of this image under a certain viewing angle. This implies that if you look at a 10 x 15 cm print (4' x 6') you are closer to the print than if you look at the same image printed as a poster.

So do yourself a favor and put down that magnifier or 100+ % zoom factor and enjoy the beauty of your landscape shots :))

Posted 2 years ago #
heartyfisher

preferred member
Joined: Apr '09
Posts: 1,701
offline

RE the lp/mm calculations. remember that a pixel is not quite a pixel :-) there is some magic involved in getting one sensor pixel to one pixel on your picture.
One pixel on your picture is multi coloured while the pixel on the sensor is monocolour (either red blue or green) and there are 4 of them that combine together to make one colour pixel. Yup your brand new 16MP camera is really a 4MP camera!!.

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Correlli said:Take the resolution figure at the DxO website: The resolution figure is an average for each focal length, f-stop and weighted throughout the field.

Even a quick perusal of the dxo site will show you the first two are incorrect.
On the third, not enough information appears to be provided, I'd like to know this as well.

heartyfisher said:
RE the lp/mm calculations. remember that a pixel is not quite a pixel :-) there is some magic involved in getting one sensor pixel to one pixel on your picture.
One pixel on your picture is multi coloured while the pixel on the sensor is monocolour (either red blue or green) and there are 4 of them that combine together to make one colour pixel. Yup your brand new 16MP camera is really a 4MP camera!!.

Which is why I cited RED's claims on the loss of resolution due to the bayer filter you describe.

Posted 2 years ago #
Correlli

preferred member
Joined: Jan '11
Posts: 264
offline

Testing123 said:
Even a quick perusal of the dxo site will show you the first two are incorrect.
On the third, not enough information appears to be provided, I'd like to know this as well.

Not sure if you mean that I am wrong by saying that they average the resolution like I said or if they don't do what they claim to do. But this is what they claim for the resolution figure:

This Score is computed as follows:
For each focal length and each f-number, resolution measurement is first normalized (scaled on a 24x36mm sensor) and then weighted throughout the field, tolerating less resolution in the corners than in the center. This gives one number for each focal and aperture combination.
Then, for each focal length, the maximal value of resolution over the range of available apertures is selected. This value is averaged over the whole range of focal length to obtain the DxOMark resolution Score that we report.
<\blockquote>

But as you said, they don't tell how they weight the score...

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Semantics dispute! ;)
The _score_ is an (weighted) average, with the size of the bars showing the extents for zoom lenses.
The _resolution figure_ (lp/mm) is at a specific focal length and aperture and assuredly the peak.

Paperman and I have been ignoring the "score" since it has an unpublished weighing and is therefore hard to compare.

Posted 2 years ago #
Correlli

preferred member
Joined: Jan '11
Posts: 264
offline

Testing123 said:
Semantics dispute! ;)

Well, maybe I should have read more thorough... Thanks for the clarification. :)

Posted 2 years ago #
casperwb

preferred member
Joined: Jan '11
Posts: 558
offline

At the end of the day, its the person behind the camera that matters, not the equipment.

give a good photographer a low end camera and lens and he/she will still produce fantastic shots, give good equipment to someone who has no skill or imagination and you will get?

Look at the greats shots you have on film with F, FM, FE, F, F2, F3, F4, F5 etc, and the glass back then?

Some of it was great, some of it was......not so great to be generous.

I agree with bjrichus, get out there and shoot, most people do not even come close to using the ability of their equipment, [of course that does not include the members on this forum].

good thing this is not a face to face forum becaus I would probally be tarred and feathered, but that would probally make a good picture.

disclamer: Your eyes are weary from staring at the screen. You feel sleepy. Notice how restful it is to watch the cursor blink. Close your eyes. The opinions stated above are yours. You cannot imagine why you ever felt otherwise

Posted 2 years ago #
Mike Gunter

preferred member
Joined: Sep '10
Posts: 946
offline

Hi all,

Some good points here.

And the nitty gritty of it all, IMHO, is that the lab ain't all bad, it's a nice starting place. But as many have said. Taking pictures is taking pictures.

I'm likely going to sound like a cranky old guy, with a 'back in the day' story, but I have had to add what little ice we had to the developer (it still didn't get it cool enough) and 'guess' what would happen to film and paper during the process because of the heat in the darkroom. There wasn't any meaningful data for that. We had to do that on our own.

Shooting data is like that, too.

I think that sensor/camera data in lab targets sort of belies how individual shooters shoot. Add your own vision and things change, sometimes by a lot.

The data seems valid, but at the same token, you need to interpret it carefully.

My best,

Mike

Posted 2 years ago #
bjrichus

preferred member
Joined: Nov '10
Posts: 921
offline

casperwb said:
I agree with bjrichus, get out there and shoot, most people do not even come close to using the ability of their equipment, [of course that does not include the members on this forum].

<blush> Now you make me feel bad... :)

My point originally was that the OP wants results that I think he won't get without being *at least* high MP FX and going for top quality glass.

As others have said he probably needs 4x5 or even 10x8 film, not DSLR. Let me pour fuel on the fire and suggest a 35mm film camera purchase and a decent copy of a prime lens would deliver the results he wants too?

It just seems to me that using any labs test results is worth something but are not the only input to use... The results from actually using any particular combination of lens and camera the OP is going to get will speak more than any lab tests alone are going to.

Perhaps the best advice we can give is to rent the gear involved and run a couple of hundred frames then compare them all side by side? He'll then be able to see exactly what works and where the sharpness he desires can be found. I'll stick my neck out and bet you $20 it's not where the lab results alone would suggest... :)

Posted 2 years ago #
Henrik1963

member
Joined: Feb '11
Posts: 40
offline

Go to DxO. Take a look at 50 1.4 G. Try D300. Then try the same lens with D90.

I think DxO is testing cameras as much as they are testing lenses.

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

Everyone seems to be on a different time zone than me :-)

--------------

Thanks for all the inputs from each of you ; all the points indicated have some validity and it was good to learn about all the different experiences.

I guess I am allowed to say that almost everyone agrees the DxO lab figures are NOT INCORRECT and that resolution numbers do give a fair idea about what to expect from a lens ( emphasizing that the topic is related to sharpness only and not the other superiorities of pro-glass like speed,build qlty., etc. )

I also respect the opinions of those who have indicated there is a visible "overall unmeasurable quality " increase in expensive glass which won't necessarily reflect to lab results . I'm not sure though if the superiority includes sharpness - my main point . I have to find the opportunity to test and see myself.

My conclusion - as many of you have implied - is that I am probably expecting too much from a 12 Mpixel APS-C sensor and I won't get any better sharpness ( let's say visible ) on my D300 no matter what I use . The "maximum" I should be buying without considering it "throwing away money " is maybe a used 12-24mm for $600 or so.

Going over some of the replies, it looks like some of you see me as a "sharpness" freak maybe doing some serious work and I just want to say I am not . I am purely an amateur shooting landscapes for mostly my own satisfaction on a modest DSLR . The reason I emphasize sharpness is because there is really nothing else to worry about in landscapes . I naturally can't care less about speed ( give me an f8 lens - I'm happy ), the CA is corrected automatically most of the time and distortion - if it is too bothersome - can also be adjusted in PC . Thus the search for better sharpness .

Henrik1963 said:
Go to DxO. Take a look at 50 1.4 G. Try D300. Then try the same lens with D90.

I think DxO is testing cameras as much as they are testing lenses.

They are indeed . That's why I was specifically adding results only for the DSLR I use. There wouldn't have been this discussion if I had a D3x :-))

Posted 2 years ago #
Testing123

preferred member
Joined: Oct '10
Posts: 232
offline

Paperman said:
I also respect the opinions of those who have indicated there is a visible "overall unmeasurable quality " increase in expensive glass which won't necessarily reflect to lab results . I'm not sure though if the superiority includes sharpness - my main point . I have to find the opportunity to test and see myself.

There would be less "vagueness" if dxo published not just their scoring weights (which we'll ignore for now) but also their resolution weights. Resolution varies across the width and height of the frame yet they give one number. Their weighing strategy is likely too heavily in favor of the corners for some (Ken Rockwell, for example, likes corner softness) and likely too heavily in the center for others (pixel peepers). The ambiguity in their numbers only fuels those who would be dismissive of the entire endeavor.

Regardless, I still stand by my original point. You won't see a difference between 45 and 50 w/o pixel peeping. ;)

Posted 2 years ago #
Paperman

preferred member
Joined: Jul '10
Posts: 297
offline

Testing123 said:
....Resolution varies across the width and height of the frame yet they give one number. Their weighing strategy is likely too heavily in favor of the corners for some (Ken Rockwell, for example, likes corner softness) and likely too heavily in the center for others (pixel peepers). The ambiguity in their numbers only fuels those who would be dismissive of the entire endeavor.

I am not going to add the results here (and drive you guys crazy ) & apologies for repeating the same subject. However , if you're interested ( testing123 ) , here are the Photozone.de tests where you can look at MTF figures for the $200 18-135mm and $2000 17-35mm f2.8 - this time for "center" and " border" MTF figures.

I haven't looked very closely but it looks like there aren't any focal length & aperture combinations where the 17-35 gives better MTF results than the 18-135mm . They seem to confirm DxO Lab results.

http://www.photozone.de/nikon--nikkor-aps-c-lens-tests/241-nikkor-af-s-18-135mm-f35-56-g-if-ed-dx-review--test-report?start=1

http://www.photozone.de/nikon--nikkor-aps-c-lens-tests/230-nikkor-af-s-17-35mm-f28-d-if-ed-review--lab-test-report?start=1

Posted 2 years ago #
SquamishPhoto

preferred member
Joined: Feb '10
Posts: 1,076
offline

Talk about worrying about the wrong thing. Get over the lab results. You are obsessed and its getting old. The only lens you ever mention is the 17-35 and you seem just baffled that it even exists. Go rent one and make some comparisons for us cause Im getting pretty tired of reading you whine about this stuff. You've had some very exhaustive explanations given to clarify all this to you and yet you are still here bemoaning pro glass. We get it, you love your 18-135 and you feel like you've outsmarted everyone whos spent money a pro wide angle lens. We know how you feel about this, so let it die.

If I want a sharp photograph I do not use a wide angle lens. I start with the 50mm and if it doesn't do it for me the 85mm does. And if that doesn't do it for me the 300mm crushes both of them. And I didn't use any lab info to figure that out. I just took pictures. You clearly need to spend more time with your camera and less time reading bullshit about tests on the internet.

Posted 2 years ago #

RSS feed for this topic

« Previous 123 Next »

Topic Closed

This topic has been closed to new replies.

The new Nikon Rumors Forum is now live at http://forum.nikonrumors.com/discussions. This forum is now in "read only" mode until I figure a proper way to import all data over to the new platform. Please register over at the new forum.

Lens Discussions - DxO Lab test etc ...

Topic Closed