Worried about D800 files sizes? How about compressed DNG's

Nikon Rumors Forum » Nikon DSLR

Worried about D800 files sizes? How about compressed DNG's

(24 posts) (9 voices)

Tags:

silversurfer

member
Joined: Mar '12
Posts: 13
offline

I have been playing around with Lightroom 4 and it has a great feature of converting NEF files to a compressed DNG.

I've experimented briefly with it, for example a 10mb NEF from a D3s it compresses anywhere between 4mb though to 9.6mb.

Viewing the jpegs produced up to pixel level I cannot see any difference at all. Maybe this can vary depending on detail or someone with a better eye? Also the jpegs produced by both come out pretty much the same size.

For the D800 and its large files this may be a great way of dealing with it if, like myself you shoot thousands of pics in a days work.

You could have a workflow of backing up all NEF files to archive. Then go through your initial culling and at the same time mark which you would like to work on as NEF (if worried about possible quality loss on your favs, ones that may need cropping or possible for enlargement) and then convert the rest to compressed DNG format.

This would save massive amounts of space, especially for high volume shooters. Yes...I know hard drives are cheap, but when you shoot thousands of frames a day and then look at 2 external backups it will be pretty expensive on the long run.

Any thoughts on the disadvantages of this? I'm not technical but so far only really see the benefits.

Posted 1 year ago #
yupgift2u

new member
Joined: Dec '11
Posts: 1
offline

I'm not technical but so far only really see the benefits. I agree to it.

Posted 1 year ago #
InTheMist

junior member
Joined: Mar '12
Posts: 8
offline

For the D800 and its large files this may be a great way of dealing with it if, like myself you shoot thousands of pics in a days work.

I was thinking the same thing over the weekend. It makes sense to me.
I'm counting 36MP as marginally the only "disadvantage" of the D800, which I've preordered.

Posted 1 year ago #
tcole1983

preferred member
Joined: Feb '10
Posts: 1,553
offline

Search DNG vs RAW...there is an article on it by Mansurovs Photography. Some thoughts on it...

"2.1) What are the disadvantages of DNG format?

Conversion from RAW to DNG takes extra time during the import process.
DNG does not work with all manufacturer image-processing programs. For example, it doesn’t work with Nikon’s Capture NX product.
DNG strips out some of the unrecognized meta data (such as Active D-Lighting and Picture Control) from RAW files, making it impossible to retrieve this data from DNG in the future.
Because all changes are written into the DNG file, you would have to back up the entire DNG file every time you make changes to it."

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

Some other issues.

1. Conversion can occasionally fail. So check files before deleting raw.
2. Lightroom doesn't auto save meta data to dng, requiring more manual saves.

Posted 1 year ago #
silversurfer

member
Joined: Mar '12
Posts: 13
offline

tcole1983 said:
Search DNG vs RAW...there is an article on it by Mansurovs Photography. Some thoughts on it...

"2.1) What are the disadvantages of DNG format?

Conversion from RAW to DNG takes extra time during the import process.
DNG does not work with all manufacturer image-processing programs. For example, it doesn’t work with Nikon’s Capture NX product.
DNG strips out some of the unrecognized meta data (such as Active D-Lighting and Picture Control) from RAW files, making it impossible to retrieve this data from DNG in the future.
Because all changes are written into the DNG file, you would have to back up the entire DNG file every time you make changes to it."

Hi tcole. Taking extra time is true although not as long as I thought, but you would have to import the files into LR and then convert them. 100 files took 5 minutes to convert on my computer. As with regards to unrecognised metadata I would always have an initial complete back up in NEF for archive purposes. Probably make my second back up in this compressed DNG format. Also during my initial sorting I would 'tag' files that I may want special editing on, cropping or maybe enlarging. These would stay in NEF format along with the DNG's in my LR work folder. As for making changes causing the whole file to be re written, I haven't considered that. I just tried re writing some metadata for 500 pics and it took about 90 seconds with D700 size files...I guess re writing them with D800 size files will take a bit longer though. This may also be considered a plus for DNG's as it means you don't have a separate sidecar file to store with the NEF's ( for those that don't know, you can't write extra info to an NEF. If you want to transfer your edits to photoshop or write additional metadata it has to be stored in a separate XMP file)

What I want to know is how it is compressed? I mean what is lost and how may it affect quality?

Posted 1 year ago #
silversurfer

member
Joined: Mar '12
Posts: 13
offline

mirtos said:
Some other issues.

1. Conversion can occasionally fail. So check files before deleting raw.
2. Lightroom doesn't auto save meta data to dng, requiring more manual saves.

Hi mirtos. First point is good but I guess that's why you should alway have a backup. On the second point, it doesn't write metadata to NEF files either. All metadata is saved in the lightroom catalogue. It needs to be manually transferred and then it will write it as a separate file because NEF's cannot be rewritten......I think :-/

Posted 1 year ago #
silversurfer

member
Joined: Mar '12
Posts: 13
offline

InTheMist said:
I was thinking the same thing over the weekend. It makes sense to me.
I'm counting 36MP as marginally the only "disadvantage" of the D800, which I've preordered.

That is exactly what I am thinking. If this proves to be fairly painless without too many cons then I won't have any more doubts :-)

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

To be honest, I've never actually used DNG files before, I was just quoting some of the cons from an article i read. It implied that conversion was an issue only occasionally, but it still required checking each file, thus slowing the initial workflow down a little.

Its also possible that when new cameras come out - IF THEY CHANGE THEIR FORMAT OF RAW - , it might take time for Adobe to update its code to handle the conversion.

Following is a cut and paste of the Cons of DNG:

Conversion from RAW to DNG takes extra time during the import process.

DNG does not work with all manufacturer image-processing programs. For example, it doesn’t work with Nikon’s Capture NX product.

DNG strips out some of the unrecognized meta data (such as Active D-Lighting and Picture Control) from RAW files, making it impossible to retrieve this data from DNG in the future.

Because all changes are written into the DNG file, you would have to back up the entire DNG file every time you make changes to it.

-----------------

To be honest, the only one that is even slightly annoying to me would be the 3rd one. But people that use Capture NX might have issues as well. Thats important to know as well.

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

What I want to know is how it is compressed? I mean what is lost and how may it affect quality?

From what I've read it use's JPEG Lossless Compression.

Posted 1 year ago #
TaoTeJared

preferred member
Joined: Apr '10
Posts: 2,422
offline

6 of one - a half dozen of the other - DNG is just another Raw format but is more universal. There are some trade-offs but in the final product for presentation, it is nominal. One thing for sure it is not a Jpeg.

I use DNG & Tiffs for edits - it just makes since to me that using the most universal formats will bennifit in the far off future (10+ years). Some still use Tiffs for that reason as well. The real long term answer is to add more storage, short term answer - use what works for your work flow. DNG compressed is much better than Jpegs, better than NEF? I have never found any advantage personally one over the other. It may be better for Capture NX but I haven't used that program in over a year - nothing wrong with it, it just doesn't fit in my workflow as well as Lightroom does.

Posted 1 year ago #
PB PM

preferred member
Joined: Jan '10
Posts: 2,217
offline

I wouldn't call anything made by Adobe to be universal, it is designed to lock you into Adobe products long term. NEFs may be larger, but at least you can use them with over 6 different major RAW editors (LR, Aperture, Capture NX, Phase One Capture One etc), the same cannot be said for DNG.

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

universal might be the wrong word. universal only becomes universal when everyone picks it up. what it is, though, is open. The same cannot be said for either Nikon or Canon. Just because a format is made by a company doesn't mean it cant be universal.

Examples: GIF, PDF. Pretty much every "universal" format out there was generated by some company with the desire to be the king. Its how good the format is, and how adopted it becomes.

Posted 1 year ago #
PB PM

preferred member
Joined: Jan '10
Posts: 2,217
offline

The DNG format has been out for how many years now, 3-5? As far as I know the only companies to adopt it are Leica and Pentax. So in other words a total lack to broad appeal, and hardly coming close to being a universal format. Yes you can convert other formats to it, but as mentioned you loose RAW data in the process, which seems silly because the whole point of shooting in RAW is keeping all the data.

I wouldn't call DNG open either, since it is designed and controlled by Adobe. One small change and the only software that it will work with is Adobe products. Open today, closed tomorrow seems to be Adobe's style. I don't trust them as a company.

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

I dont trust Adobe as a company either, however, if you look at what they have done with DNG, they have done the right things into making a format standard. They have created a license that allows anyone to use it/develop for it royalty free. They have stated that they arent encumbering using it with any intellectual property or licensing issues. they have applied to ISO to get including in the revision of TIFF.

These are all things that make it easier to become universal. The same cannot be said for either Nikon or Canon.

Now all that said, sometimes standards, no matter how good, don't become standards, because other people dont pick them up, and thats really too bad. This is especially true in the computing world.

Posted 1 year ago #
silversurfer

member
Joined: Mar '12
Posts: 13
offline

TaoTeJared said:
6 of one - a half dozen of the other - DNG is just another Raw format but is more universal. There are some trade-offs but in the final product for presentation, it is nominal. One thing for sure it is not a Jpeg.

Actually mirtos is right. They are written in jpeg. Don't forget this thread is talking about lossy DNG's. It's obviously not that simple and a bit over my head, but it uses jpeg type compression, is 8 bit and is also demosaiced. The way it is written and compressed seems to be pretty complex but results in little or no loss of detail and still with almost all the benefits of RAW. Plenty about it on the Adobe forum and a good article on ChromaSoft's blog showing a negative point of view.

I converted 20,000 photos to this lossy DNG last night as a test and it saved over 200GB of space and took several hours. (these were with D70 and D200 sized files)

One thing that is fairly critical is to make sure you do keep a backup in original RAW format, at least for your 'picks'. Not only is this a safety measure but also because this compression demosaics the file, it means that it won't be able to take advantage of future demoasaicing algorithms that may get more out of your files.

I'm still not completely convinced. I think I might wait a few months before implementing it into my workflow and listen to more feedback.

I guess it isn't a great benefit until you look at D800 files and only then if you are taking large amounts of photos.

Posted 1 year ago #
qdos

member
Joined: Sep '09
Posts: 13
offline

The new iPhoto for the new iPad can handle up to 19 megapixels. Guess no one are going to use this as the primary post-processing platform anyway, but nice to know :)

http://www.macrumors.com/2012/03/07/apple-launches-iphoto-for-ipad-with-photo-editing-and-organization-features/

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

In fairness, I never said they were jpegs. They use a specific algorithm. From what I've read it's the lossless JPEG algorithm. From a file format perspective, they are more based onTIFF. Format and algorithm are two different things altogether.

Posted 1 year ago #
TheLostNinja

member
Joined: Mar '12
Posts: 36
offline

There's also this other thread on this board about "who keeps old bodies". Looks like there are some posters on this board here that seem to suffer a bit from "compulsive hoarding" ;-)

For most of us shooters pictures fall in a 1 to 5 star category and we only ever going to use the awesome 5 star pictures for anything more than just nostalgic reasons (aka browsing through old shoots on your computer screen). I don't see the need to keep 1-4 stars as RAW, i usually delete 1+2 stars and convert 3+4 to med-sized (4mp) jpegs - this is still more than enough to process and even print them (in case i ever need it).

Obviously some people (pros?) might have valid reasons to keep RAWs archived, but then the focus is more on "archiving" rather then "storing".

I am not too worried with the D800 file size ... i think i will do just fine ;-)

Posted 1 year ago #
mirtos

preferred member
Joined: Sep '11
Posts: 159
offline

Hi,

My name is Mirtos, and I am a compulsive hoarder.

Posted 1 year ago #
TaoTeJared

preferred member
Joined: Apr '10
Posts: 2,422
offline

mirtos said:
In fairness, I never said they were jpegs. They use a specific algorithm. From what I've read it's the lossless JPEG algorithm. From a file format perspective, they are more based onTIFF. Format and algorithm are two different things altogether.

You are right - it is a Tiff based file, NOT JPEG. It does use the same algorithm of jpeg but without the color depth compression (removal) that is part of jpegs. All images are demosaiced (arranging of pixels to make an image - this is what raw editors do)- although it was a big concern in the past, Adobe Raw has shown to do just as good in interpreting files, if not better than some converters from manufactures.

All of us have different workflows which lead us all to different views on what is better than others. For me, most of my images are shot in JPEG and edits are done in Lightroom (non-destructive) so the Jpegs are not "re-compressed" each time an edit is done. Most all High dynamic range photos are shot in NEF and are converted into DNGs or Tiffs. These are also the majority of my "keepers" but I do have many keepers in jpeg that are of family etc.

My main reason to keep things mostly in DNGs is that it is universal in that every major program opens them, they store more info then other "open" formats and are smaller in file size than Tiffs. I had an old Olympus camera that used a "New and improved" file format that was only used in 1 or 2 cameras and then dropped. I lugged the files on a CD for years until I found a friend who actually had an old computer with windows 95 (the only system Oly's software would run on) and proceeded to spend hours converting 100s of family photos to tiffs. For 8 years those photos were "locked away" and I swore I would not do that again.

Next week I will be receiving and scanning old family photos from the late 1890s to the 1960s. Being physical, they can be scanned. Digital files 50, 60, 100 years from now, how will they be used? Nobody thinks Nikon is going to go out of business, but I know 50 years ago or even 10 years ago, no one would have imagined Kodak would either.

I'm not sure DNG is the best but more use it than the only one company uses NEFs.

Posted 1 year ago #
PB PM

preferred member
Joined: Jan '10
Posts: 2,217
offline

In 60-100 years many of us wont be around, so who cares whether or not the files still work? The good stuff gets printed and the crap sits on hard drives never to been see again (lets just be honest here).

Eleven years ago D1 users gave the same arguments against NEF files, they were lamenting that they might not be able to work with their files in 2012, yet I cannot think of any major RAW editors that have dropped support to date. As far as I know most major editors still support Minolta DSLRs, even though Sony bought them out years ago. If you are really that worried keep an old computer (with SSDs) in storage with an old RAW editor and sleep easy.

Posted 1 year ago #
TaoTeJared

preferred member
Joined: Apr '10
Posts: 2,422
offline

PB PM said:
In 60-100 years many of us wont be around, so who cares whether or not the files still work? The good stuff gets printed and the crap sits on hard drives never to been see again (lets just be honest here).

That was my point - almost nothing is printed anymore. For my personal stuff, family etc., I have not printed a photo of that sort in almost 2 years. My family all has 12-15" digital photo frames that we update every few months and all of the younger family's photos are on FB. We just don't have the physical images to recreate from that we used to.

Posted 1 year ago #
PB PM

preferred member
Joined: Jan '10
Posts: 2,217
offline

True. i guess family is kind of old fashion, because we still get our best family images printed, at 4x6 at the very least.

Posted 1 year ago #