Drab said:
JPEG is only using an 8 bit number to store intensity values (which is actually a simplification but let's run with it) logarithmically.
Thanks Drab for the explanation. I did not know that, but it does make sense.
JPEG or RAW
(204 posts) (69 voices)-
Posted 2 years ago #
-
jonnyapple said:
...but basically to be able to count that high and still count to one, the diode would be vaporized before you got there.That has to be one of the most enlighting readings I came across for understanding sensors.
I know I may talk science fiction here but if one diode would vaporize trying to count so many photons, couldn't you put say a group of 4 or 16 in series and/or in parallel as if they were a single pixel to achieve the count? Couldn’t we reorganize them electronically in parallel for low-photon count and in series for high on a per-group basis?
Also, if you know a diode is saturated, couldn’t say a divide-by-four optical filter be applied over it, perhaps made of liquid crystal, count the photons and multiply them by four?
I read here in NR that the human eye achieves far greater dynamic range than any sensor and that the best eyes nature has produced are owned by insects. If that's true, then there is still room for future improvements I would think.
Posted 2 years ago # -
That funny delay again, the main forum page clearely sais I have posted here 20 minutes ago but I cant see it.
Edit: Funny, as soon as I have posted this, the hidden previous post suddently appeared.Posted 2 years ago # -
Pierre said:
That has to be one of the most enlighting readings I came across for understanding sensors.I know I may talk science fiction here but if one diode would vaporize trying to count so many photons, couldn't you put say a group of 4 or 16 in series and/or in parallel as if they were a single pixel to achieve the count? Couldn’t we reorganize them electronically in parallel for low-photon count and in series for high on a per-group basis?
Also, if you know a diode is saturated, couldn’t say a divide-by-four optical filter be applied over it, perhaps made of liquid crystal, count the photons and multiply them by four?
Yes you could, but all you are doing is shifting the dynamic range, not increasing it.
A quick step back: The maximum theoretically possible dynamic range is hard bound by noise floor and "vaporization" ceiling.
In your first example (of ganging up 4 diodes) the problem is that all four would either get saturated equally to the one diode they are replacing (and thus vaporized) OR you would give them 1/4 the light each, either through mechanical or other means. If you give them 1/4 the light each you have raised the ceiling, but also raised the floor because heat voltage (noise) is a constant but now light voltage (signal) is 1/4th of what it would have been, thus the range is the same.
Your second question is a bit more crafty. ;) If I'm hearing you correctly you're asking for a dynamic per-photosite ND filter. One which would only darken as the photosite approaches saturation.
This sounds like it engineers itself around the brick wall I describe in the first example, but there are new walls you are hitting now. :(
To know when to kick on the ND filter one must first read the charge from the photosite. This can not be done "non destructively". To read the charge is to use the charge is to change the charge. This could be worked around by taking one reading naked simply to determine what ND strength to use on a per-photosite basis for the next reading, but now you're just playing complex in-camera bracketing games and doing in-camera HDR. ;)Pierre said:
I read here in NR that the human eye achieves far greater dynamic range than any sensor and that the best eyes nature has produced are owned by insects. If that's true, then there is still room for future improvements I would think.That is a common oversimplification which lies through vagueness.
The human vision system has far greater dynamic range than modern cameras. The human eye does not.
The brain is constantly combining multiple frames "shot" at variable framerates and variable ISOs while your eyes move to use multiple parts of the retina and multiple focal distances and angles. Your eyes are relatively poor cameras, but have one heck of a computer doing the post-processing!These limits JA and I am are speaking of are not engineering limits. Let us be clear on that. These are physical limits imposed by the nature of the universe. Insect or eagle or squid or ET eyes can not do break these walls, and likely can not even approach them due to the analog-only nature of life as we know it. It is only through the ability to combine multiple analog inputs in the realm of digital math that we can get close.
Posted 2 years ago # -
Well said, Drab, especially about human perception and the neural networks behind interpreting what we see. That is fantastic stuff and I wish I knew more about it. Pierre, what you're talking about is easiest to do by combining bracketed exposures to form an HDR image in post, like Drab said.
Just so no one is afraid to ever take a sunset image again, though, I want to emphasize that the vaporization calculation was just for fun. In real life, all that happens is that the diode charges to a voltage above the top level of the ADC so that the pixel is read at its highest possible value—all ones—and the pixel is interpreted as a clipped (saturated) highlight. Cue flashing areas on the LCD, not destroyed areas of the sensor.
Posted 2 years ago # -
Guys this thread has gotten waay off topic. Lets get back to JPEG versus RAW
Posted 2 years ago # -
Here is the question that really interests me.
I keep reading that RAW is not destructive but I wonder if this is true for all of the in-Camera settings (Active D-Lighting, NR, color-balance, picture control (sharpening, contrast, brightness, saturation, hue, and vignette control)). There is not cost on committing to any of them?
What is this thing about focus setting that can be reversed? Is this the lenses focus fine-tuning? Can you really bring back soft focus without resolving to phony sharpening tricks?When you shoot pictures there's a chain of events that take place.
1) press trigger to open shutter and record light
2) shutter closes but sensor continues recording for a duration equal to the time the shutter was open to record a separate black frame which is used to determine if there's anomalies such as dust on the lens or hot pixels.
3) Image data from the sensor is sent to the firmware for processing.
4) firmware analyzes the recorded data and performs initial noise removal using the black frame information to adjust the recorded information.
5) Resulting image is then compared to the onboard library. If it matches any of the stored compositions, then routines specific to the library image are carried out to further process the image (sharpening, color adjustment, anti-aliasing, ...).
6) Camera then looks at file format chosen by the user. If Jpeg, the data is further processed with in-camera settings (Active-D, white balance, ...), then quantized and compressed using JPEG algorithm and stored in .jpg file as the final image. If RAW, the image is recorded as the base image while in-camera settings set by the user are applied separately as later edits in the image's construction history - this is how you can get the original image back when working in Capture NX. Data is then written to .nef file to complete the process.
there are more steps inbetween that weren't mentioned, but that's the gist of it.
With JPEG, all your camera settings are baked into the image. JPEG algorithm further compresses and quantizes data to be as simple as possible. That includes approximating the colors of the image to be more friendly in storage and transmission over networks such as the internet. Each time you re-save the data, the algorithm further quantizes, compresses, and tosses out data in the form of approximations. That's why JPEG is a 'lossy' format. It was designed as a 'release' or 'publish' format. The fact the camera settings are baked into the image also makes it more difficult to edit later on.
RAW is an 'intermediate' or 'edit' format. It saves everything and the kitchen sink at all times, and data is never approximated. It consists of a base image and a construction history. The base image is what came off the sensor plus minimal processing to account for sensor design. If you make an edit, it is stored separately in the file from the base image - usually as a function applied when the image is loaded into an editor such as Capture NX. This is what is meant by non-destructive. Edits can be added/removed at any time without affecting the base image. If you don't like your edits, you can start over by removing the construction history. Because RAW has all the original data, you can push the image quite a bit in various directions before artifacts begin to show. It's almost the equivalent of having the camera still in your hands with the added benefit of all the time in the world to adjust your image until it's right.
As mentioned earlier, I use RAW for a few reasons. The main one being I shoot in dark venues where the lighting is below what my camera can handle and action is fast paced. I often can't use flash due to mirrors being present all around. With fast action I don't have time to adjust dials on my camera for proper exposure. The best I can do is open the shutter, keep ISO respectable, compose good images, and hope that I captured enough information that I can work with it later in post. Usually involves increasing exposure by 1.5 EV and adjusting the white balance. When I lose a shot it's due to being out of focus from camera shake or because it was too dark for the contrast based auto focus to make a read. Plenty of those unfortunately. But hopefully newer cameras with better ISO and focusing will reduce the frequency of that problem.
Posted 2 years ago # -
speye_21 said:
2) shutter closes but sensor continues recording for a duration equal to the time the shutter was open to record a separate black frame which is used to determine if there's anomalies such as dust on the lens or hot pixels.Only happens if long-exposure noise reduction is on and you're across the long-exposure threshold.
4) firmware analyzes the recorded data and performs initial noise removal using the black frame information to adjust the recorded information.
See #2
Posted 2 years ago # -
Drab said:
Only happens if long-exposure noise reduction is on and you're across the long-exposure threshold.See #2
You missed this:
>> there are more steps inbetween that weren't mentioned, but that's the gist of it.
Whether the long exposure thing occurs or not, it's part of the basic workflow of capturing an image.
Posted 2 years ago # -
Another reason for shooting RAW
If you want to enter the Veolia Environnement Wildlife Photographer of the Year 2011 competition, the rules say winner MUST be able to summit a RAW file
anyone know of other competitions that have this rulePosted 2 years ago # -
Since I discovered RAW, I take only RAW photos. (Then I used the D70s with Nikon Capture 4.4)... That NEF files did not take a looooong time to convert).
Now with my new toy (D7K), I still take RAW photos.. but converting to NEF takes an eternity... that I was considering in taking RAW + JPG photos..
The reason: There is (was) a huge difference on the photos that were converted from NEF to JPG than the ones taken by the camera on my D70s, I guess the same happens with the D7K (Disclaimer: I did not test this with the D7K...)
Posted 2 years ago # -
i don't understand what the problem is.
Are you talking about slow conversion times of processed RAWs in Nikon Capture 4.4?
i use lightroom 3 and conversion times are very acceptable to me. i shoot 14-bit RAW mostly and generally convert from adobe RGB to sRGB on export.
Posted 2 years ago # -
Gareth said:
i don't understand what the problem is.Are you talking about slow conversion times of processed RAWs in Nikon Capture 4.4?
i use lightroom 3 and conversion times are very acceptable to me. i shoot 14-bit RAW mostly and generally convert from adobe RGB to sRGB on export.
Gareth, Currently I translate my NEF files using two tools:
- For NEF files from the D70s (6MP) I use the old trusted Nikon Capture 4.4.
- For NEF files from the D7K (16MP) I use ViewNX2 that came with the camera.I downloaded Lightroom 3 trail version, to check it... till now, I'm gaining some allergy to it due that Lightroom is still not intuitive for me. I prefer the way Photoshop does the job.. but my CS3 does not support the D7K :(
Question, what is this DNG format Lightroom uses?
Posted 2 years ago # -
rbid said:
.. but converting to NEF takes an eternity..This is I suspect is a computer issue
Solution (a) connect card reader to computer, click download, go and have a nice cup of tea and wait
Solution (b)repair or up grade old computer
Solution (c) but new wizzy computer
I have the same problem and about raid the piggy bank and buy a new computer
Big hard drives, loads of ram, super fast processor
Posted 2 years ago # -
I have to admit that I'm at a loss. Seeing all of this makes me think I have missed something all these years.
I have never used Photoshop but Corel PS pro or Photoshop elements for my files. I have loaded photos, then pick ones to edit, and did everything from there. If a Raw conversion was needed it pulled it up and I did my thing. I then saved them as Tiffs or PSDs with layers.
My current workflow with Lightroom, I load the files and do absolutely nothing for any raw conversion. When I make any edits it automatically saves them as Tiffs. There is no middle conversion software or module. Jpegs I shoot, stay the same.
I have never done any sort of mass conversion. The main reason for this is that I never wanted to ruin any raw files by converting to jpegs. In my mind doing a conversion to jpegs then editing is just as bad as shooting jpegs and editing them.
Am I missing something?
Posted 2 years ago # -
Tao
Sorry I cant quite understand you workflow
Do you shoot RAW or jpeg or both ?Posted 2 years ago # -
sevencrossing said:
This is I suspect is a computer issueSolution (a) connect card reader to computer, click download, go and have a nice cup of tea and wait
Solution (b)repair or up grade old computer
Solution (c) but new wizzy computer
I do not connect any of my cameras to the PC. I use a USB2 card-reader to copy the photos to my computer (CF or SD cards) to the PC does not take that long), Using a neat software called Downloader-Pro (From Breeze Systems), and then use NC or ViewNX to translate the RAW files to JPG. (I do not advice you to use a direct cable from your camera to the PC, specially on dry days that static electricity is in the air, a friend of mine ruined the USB driver of his C*n DSLR due to this problem)
The process of converting from NEF to RAW is what takes time. The PC is not old, has 4GB memory, Intel Core i5 - 760 processor.
TaoTeJared said:
I have to admit that I'm at a loss. Seeing all of this makes me think I have missed something all these years.
Am I missing something?Tao, no, you are not missing anything.
I am dong almost the same as you if I need some PP on my photos. (With Photoshop instead of Lightroom for my D70s files... and I'm now trying Lightroom with my D7K files).The work-flow I have is:
1) Download the photos using Downloader-Pro from the CF or SD cards via a card-reader.
2) Open NC or ViewNX2 and run a batch conversion of the NEF files to JPG files.
3) For selected photos I used Photoshop directly from the NEF file for doing some PP.
Note: For my D7K I can't open the NEF files directly in Photoshop due that I have the old version that do not support it $%$%^%^I just tried to shot NEF + JPG on my D7K.. and I still think that the PC conversion provides better JPG files than the ones the camera does. (I need to try more I guess)
Posted 2 years ago # -
Something's going on, because my Core2-era Xeon should be slower than your computer, and I process NEF files at dozens of times your speed.
There still appears to be one uncontrolled variable in your story:
Process the D7000 AND the D70 files with the same software for an apt comparison.Oh, and DNG is an Adobe-specced camera-agnostic RAW file format. Lightroom does not force the use of it.
Posted 2 years ago # -
Testing123 said:
Something's going on, because my Core2-era Xeon should be slower than your computer, and I process NEF files at dozens of times your speed.There still appears to be one uncontrolled variable in your story:
Process the D7000 AND the D70 files with the same software for an apt comparison.Oh, and DNG is an Adobe-specced camera-agnostic RAW file format. Lightroom does not force the use of it.
I guess you use Lightroom and I use the non-optimized-poor-written tool called ViewNX2, that makes the big difference in my honest opinion.
Last week I took a bunch of files from my old D70s and converted them using the two tools (Old NC 4.4 and new ViewNX2), and the old tool was faster.
- NC took 30 sec to translate 70 files.
- ViewNX took 48 sec to do the same.While with NEF files from my D7K, 70 files take ~ 190 sec.
- The D70s NEF files are ~6MB files (12bit per color)
- The D7K NEF files are ~15MB files (14bit per color)Posted 2 years ago # -
yes. what you want to do is get lightroom 3. lightroom 2 was slow, but no more with 3.
I agree that it takes some getting used to, but there are some very good tutorials on it online (I can link if you want). I was mad and swore a lot for about a month. I ended up copying files numerous times until I uncheck the "don't import suspected duplicates" button. now I just import from the new photos tab in the library module.
workflow =
1. take 12 or 14 bit RAWs depending on situation (usually 12 outside, 14 inside).
2. plug sandisk Extreme 60mbps card into buffalo 35mbps card reader (on a laptop atm)
3. import new photos (from new photos tab in import dialog)into (atm) one big folder with all my work pics, or by date folders for personal pics. use add, not copy as DNG, copy or move.
4. may bulk WB a few, takes about 10-20 seconds for 1000 14-bit NEFs on a core2 w/2gb ram. after about 50 or more pics being cropped WBed etc etc "loading" can take a while. I may exit the program to clear the cache and open it again, this takes about 1 minute.
5. export generally as sRGB, which makes bigger files and takes longer as I shoot in Adobe RGB. exporting 1000 full size pics with edits and cropping back onto the same card with the same card reader takes about 30 min to 1hr.
is this slow for you? i don't really worry about exporting taking time, as it is time I can spend doing something else.
I have no trouble with the workflow. I keep the pics on my 32Gb and 16Gb cards as a backup until they are totally full, or I wont have enough room for a big shoot. I also backup the whole "added" files directory to an external HDD.
The only issue I have with speed is the LR3 cache filling up after quite a bit of editing on my core2 2gb ram comp. we have a nice i5 with 4gb ram at school and it is like lightning, no need to exit the program ever.
Posted 2 years ago # -
i just re-read you previos post a few times.
what sort of work are you doing that you can't wait 3 minutes for the files to process?
i will time LR3 another time, but i think it is slower than that on my comp.
Posted 2 years ago # -
Hi all,
I typically use Photoshop CS5 and Adobe Camera Raw 6.3. The RAW conversion is terrific, batch processing is fairly quick, and there are options from Bridge, Automatic Processing in Photoshop, or even ACR.
Work flow in ACR allows one to load up 50-75 similar images, plot curves and levels, white balance, exposure, lens settings, noise, sharpening, and also do retouching, apply exposure masks. These can be batched to the rest of the 'like' images, all of which can be applied to a batch file.
Actually work in Photoshop is usually limited to layer and channel masking.
I've shot both JPEG and RAW to have a set of photos ready to view, but the JPEGs from the RAW are significantly better. That's likely a fault of mine.
In another post someone was talking about the zone system and part of that is printing (which RAW conversion is sort of in the digital realm), and it is an important part of the digital process in my work flow.
My best,
Mike
Posted 2 years ago # -
Mike.
Give Lightroom a really good try. It is very different to CS (and annoying at first), but once you get it nailed it is much faster to use than CS.
you can mask channels easy enough in LR, and touch up with the brush.
i hardly ever use CS unless i want to change a background or do some serious work.
Posted 2 years ago # -
Hi Gareth,
Again, glad that you're safe.
Yes, LR is very good, but I also have a vested interest in keeping my PS skills honed as I teach it. I think if I were just using the software for my 'use', LR would be the way to go, with the notable exception of the amount of masking work that I do (I tend to uses masks in some of the most mundane of the of the pictures I take).
My best,
Mike
Posted 2 years ago # -
sevencrossing said:
Tao
Sorry I cant quite understand you workflow
Do you shoot RAW or jpeg or both ?I shoot both, mainly jpeg since I really like the Nikon output and color. Any high dynamic range or mixed lighting situations I shoot raw.
My Workflow: Jpeg or Raw -> put card into reader -> open Lightroom -> let it load everything (raw/jpeg)-> convert to Tiff(raw or Nik edits of jpegs) -> do my edits -> export as I go.
I am not as adept or use Photoshop much at all. I take many steps to limit my editing on the back end. I do use the Nik suite & Portrait pro 10.
Posted 2 years ago #
Reply »
You must log in to post.
