Further quantification of the Mavericks color management problem

EDIT: Adobe fixed this bug for most types of display profiles in LR 5.7

I spent some time further quantifying the display problem that Mavericks introduced and that affects Lightroom, Aperture, and every other application that uses Apple color management libraries. (EDIT 10/19/14 - Apple fixed this bug in Aperture, right now it only affects Lightroom and Safari - all other color managed apps are fine - in Yosemite the problem persists). In short, the problem is that shadows get crushed upon display. This is a serious bug that is remaining unfixed since OS X 10.9 and is apparently present even in the Yosemite beta. I am trying to raise awareness of this bug since I am getting no reply from Apple not from a bug report and not from directly emailing folks there. This should get fixed as it makes it tough to do serious work on Mavericks. You can work around it by using Photoshop which uses its own color management library or by using the soft proof feature in Lightroom. Mac OS X 10.8 and below do not have this problem and correctly show the shadows. This is independent of what color calibration you use and even shows up when you use Apple's supplied profile for your display.

I generated a simple photoshop file that has swatches of grey ranging from 1 to 100 in 8-bit scale and then used the system color taste dropper that you can get to if you open textedit and click the text color box. Then use "Show colors" and you can then "taste" any color on your screen and get the display values. These values are what is actually sent to the monitor and so are very useful for this purpose. Below I plot the values seen in Photoshop (correct), those in Lightroom Library and Lightroom Develop. I am not showing Aperture as those are the same as Lightroom Develop and also very wrong. I am using a double log scale to really show you the problem areas in the darker regions below r,g,b=25.

I did this using the nice plotly plotting service. The images sometimes take a short time to show up. You can find the data in the link on the bottom of the plot. Photoshop's light bump in the shadows is correct as sRGB has a little knee in the shadows. Lightroom Develop due to the Mavericks bug displays way lower intensity than it should leading to the crushed shadows that people are observing.

I also created the same swatch file in the color space of my monitor profile. The display of those swatches should happen at exactly the same display value as the input file. This really illustrates the problem I think.

Photoshop clearly does this correctly. The relation is almost exactly linear and any deviations are a single bit difference which is just a rounding error. Lightroom Develop shows way below. Lightroom Library is close but with a larger error than Photoshop. Below is the same data bit plotted as display error.

Mavericks causes Lightroom Library to be off by a full 8 points in the shadows! I hope this data is useful to somebody and helps some folks that have puzzled over dark shadows in Mavericks applications.

Serious color management bug in Mac OS 10.9 "Mavericks"

Update Feb 26, 2014 - Just updated a machine to 10.9.2, the update that fixes the nasty SSL bug. It does NOT fix this color management bug.

After quite a bit of testing I have come to the conclusion that there is a serious color management bug in Mac OS X 10.9 Mavericks. This is subtle but will result in your shadows being displayed much darker than they really are. The bug affects almost every color managed program and is present whether you use a display profiler such as a Spyder or whether you use the built in profiles. The bug was introduced with 10.9 and is not present in 10.8. So if you are a pro using Mac OS X, stay on 10.8 for now. Strangely enough, Photoshop displays correctly, but it is the only program that does consistently. Aperture plugs the shadows. Lightroom displays correctly in the Library module but incorrectly in the Develop module which makes developing your pictures when looking for shadow detail difficult. I already submitted a bug report to Apple so we'll see if it gets fixed. I am not the first to notice this as is clear from this thread on Adobe's Lightroom forum.

This is the display in Photoshop(correct) on 10.9.1

This is Aperture on Mac OS X 10.9.1:

The first row has disappeared and the second row is much darker than it should be

This is Lightroom Library module (close to correct)

And this is Lightroom Develop (way off again):

Hope this is useful to somebody. The test file came from lagom, which I turned into a RGB tiff file with an included sRGB profile so that Aperture could read it. Again, on 10.8 the display is identical in all software. Also, it doesn't matter whether you calibrate or not or what calibrator you use, they all show the issue. I used a Spyder 3 Pro here but the issue shows up with other calibrators too.

EDIT: This bug is fixed in Aperture. It is still present in Lightroom and Safari

Lightroom does support retina displays! (but only partially)

I broke down and bought myself a new retina Mac Book Pro a while ago. This is an amazing machine and that display is absolutely stunning. One thing that I thought is that Lightroom needed an update to support the retina display (Aperture and iPhoto already support it and they look fantastic) so I wasn't expecting that much from Lightroom. However after browsing for a while through my library I started noticing that the Library previews were really sharp. Taking a screenshot and blowing it up confirms that Library actually is using the full retina resolution, regardless of what setting you use for the display scaling. That is superb. Unfortunately Develop doesn't yet know about the retina display so there you still get the low resolution which makes for a big difference in feel for the image. I did notice that it is probably a good idea to turn up the size of your standard previews to get the full benefit. Here is the proof using 1:1 screenshots.
Left: Library fit view. Right: Develop fit view.

Ironically you will see the difference between these two better on a non-retina display as the browser doesn't automatically scale 1:1 but you get the gist.

Update: for retina owners, here the same screenshots at what should be 1:1 if you use "best for retina"

Perils of photographing a waterfall

The image I posted yesterday needed some cleanup. The misting was so strong that within a few seconds, my filters would be covered in drops. Too fast to prevent by working fast, so I knew that digital tools would have to be involved in rescuing the images.

Right: Before extensive Photoshop spot healing tool work
Left: After

I had lots of trouble with drops on the filters as the below image will make very clear:


Amazing what is visible at 11mm and f/22. Obviously that image is a total loss apart from the novelty of showing it here. The dark corners are due to the stacked ND and polarizer and are visible when zoomed out all the way to 11mm on the Tokina 11-16mm and largely disappear when I apply lens distortion correction in Lightroom and crop a little. The price of stacking filters on a ultrawide lens I guess.

Aperture 2.1 - a plugin architecture

Apple released an update to Aperture 2, version 2.1. Rob Galbraith reviews it here. One of the most exciting on one hand and most disappointing on the other is the inclusion of a plugin architecture. This architecture allows third-party developers (which includes all of us if you register as a developer at Apple BTW) to create plugins that can be used right inside of Aperture. Many photoshop plugin developers, such as Nik and PictureCode have jumped on the bandwagon. The surprising thing to me about these plugins is that they are more like photoshop plugins than like RAW development tools. They do not sit in the RAW pipeline, but are called by rendering a tiff file and sending that off to the plugin. This means that all of your Aperture adjustments are "burned in" when you call the plugin. This is a pity as it breaks the RAW paradigm of the program and really is nothing more than a fancy way of using an external editor. This of course makes it extremely easy for a third-party developer to program for as they can simply reuse their already existing Photoshop plugin code, but does not push the envelope at all and severely limits the usefulness of the plugins in my opinion. I really hope that Apple is also developing a second plugin architecture that allows for non-destructive RAW plugin tools that work in the same way as the color balance sliders for example or the clone/heal tools (for dodging burning).

P.S. what is very interesting is that Apple apparently allows plugins to access the actual RAW data. This means that you could use alternate RAW interpreters directly in Aperture, potentially circumventing some of the issues Apple's RAW engine still has.

Aperture 2.0

Apple brought out Aperture 2.0. A wonderful update to their photo management and editing app. I have been playing with the demo quite a bit and it does address most of the major problems it had in 1.5. I'll write about it more in the future especially about where I'd hope that the Lightroom folks learn from Aperture 2.0 - and there are quite a few areas. So this is NOT a review and it only addresses a tiny point. It might not be a problem for your camera also, so I suggest you check it out for yourself first. For now, I'll focus on image quality, which for me is of paramount importance. Apple's RAW library has had major moiré problems in the past, where the RAW interpreter would create small little mazes. I am happy to say that most of the worst examples of this have been solved. They were actually already solved in 10.5 Leopard BTW. However, there are some problems remaining. I will compare the quality of the RAW converter in Aperture with that in Lightroom. Here is an image from a frozen lake in the Colorado Mountains taken using my venerable (and very light!) D50.



The ice on the lake has lots of wonderful structure that you can see in this 100% crop from the Lightroom image:



Now compare this with a 100% crop from an Aperture export of the same image:



What is going on? I see anomalously colored areas, little mazes and a generally muddy image. This is not salvageable with any moiré reduction, sharpening or edge sharpening in Aperture. It just becomes more messy. To illustrate further, here is a 300% zoom of the image in Lightroom compared with Aperture:

Lightroom conversion:

You see all the detail and bumps in the ice very well.

Aperture conversion:

Ugh! I admit you have to be a pixel peeper for this, but that is terrible. Of course on a webimage, you will never see it as illustrated by the Aperture image below (compare to the first image on top of this post):



Looks fine at this scale of course. However, you will certainly see this in larger prints (i.e. 8x12 and larger). In all this I haven't even mentioned that Nikon's software conversion of the image is even slightly better than Lightroom/ACR! Also, note that I looked at only one image. I see it in other images too though also from different cameras, but your style of photography or your specific camera might make this less of an issue.

Of course, when I edit this picture further, I would probably brighten the foreground a little and crop it differently. Here is the result starting from the Lightroom image above.:

Effect of aperture on depth of field

I recently did a silly little test that every photography book tells you to do as practice, but for some reason I had never done. The test is to show the effect of aperture on depth-of-field. The more open the lens (smaller f-stop), the shallower the depth of field is and conversely, the smaller the lens opening, the larger the depth-of-field. Also focal length plays into the equation. Here is a simple calculator that shows you the depth-of-field for a certain f-stop/focal length combination. Strange that I never actually did it as I use the effect all the time in my photography to for example isolate people from their backgrounds. It is the number one thing that makes people think you are a professional, so it is a good idea to know how to use it. Here is an example of a prime Nikkor 50 mm/1.8 lens on a DX Nikon DSLR:

f/1.8


As you see, the depth-of-field is extremely narrow and does not even extend throughout one leaf.

f/2.8

At 2.8 the depth-of-field is much less narrow and several leafs are in focus.

f/5.6

At 5.6 most of the field is in focus, but the extremes are still blurry.

And finally, f/11

Basically everything is in focus. If you go much further on a small sensor (DX) DSLR, you will loose resolution due to diffraction. In the last image, at f/22 this is the case, but fortunately, you will never see this at websizes, you will even have trouble seeing it in largish prints.

f/22:


Again, simple, but I thought an effective demonstration.

Lastly, a demonstration of the effect in a simple portrait:

Bride getting ready:

As you can see only her right eye is in focus. Her left eye is already out.

RAW development in Nikon's tools

As a followup to an earlier post, I am here providing a another conversion.
First Nikon's conversion (100% crop):

Then Mac OS X(aperture) default:

Horrifying indeed. Nikon's tools are dog slow but their quality, although not as good as lightroom's, is head and shoulders over Apple's RAW libraries.

Why I like shooting RAW

Some people like shooting jpeg, some like RAW. I like RAW because of the added versatility afterwards while the current tools do not actually make it a giant waste of time. Aperture, even though it has terrible RAW development, for example make sit really quick to work with RAW. Likewise the new Adobe Lightroom (looking forward to running the release version) does a similar thing. Here is an example of a RAW file that I originally developed a while ago:



Here is the original development:


The difference is just white balance! and the new development with the bright yellows on the clouds is much closer to what I saw.

Lightroom interview

Just found a really interesting interview with Mark Hamburg - one of the developers behind Adobe's lightroom (via the lightroom blog). It is intriguing and offers quite some insight into the ideas behind the interface. I am hoping that the final version adopts some of the really good organization ideas from Aperture and vice versa (the print pane in Lightroom is really nice!) so that the whole field gets advanced. A little old-fashioned competition is a good thing I think.

Lightroom vs Mac OS X system RAW

I'll offer this without any comment, as I think it is quite obvious.

Mac OS X system default conversion (i.e. Aperture 1.1 RAW engine and other programs):



Lightroom version: 100% crop


Here is the whole picture:

My old blog

Just to aggregate a little, here are some links to some older posts I made before on my previous blogsite.
How to correct lens distortion using free tools.

Some Aperture (Mac OS X system RAW converter) vs Lightroom comparisons.

I wrote more than once about that.

Compiling ufraw (a very high quality freeware RAW converter based on dcraw) on Mac OS X.