16-bit Luminosity Masks: Do they make a difference?

In the last blog post I described a new “calculations” method for making luminosity masks entirely within a 16-bit workflow for 16-bit images. The tonal ranges selected by the calculations method match those of the intersection method, but 16-bit masks and selections are smoother (less tonal variation) than their 8-bit counterparts upon close inspection. So the question can fairly be asked (and was asked in the comments on the last post) whether this new method actually makes a difference? Is there a visible advantage when using 16-bit luminosity masks instead of 8-bit masks?

The answer is: Probably not. In theoretical situations, which I’ll try to show, it’s possible to produce some differences, but producing similar differences in an actual image is harder. Even the theoretical differences are hard to illustrate in a WordPress blog like this. Jpeg images saved at the highest setting are further compressed by the website’s software effectively obscuring the subtle differences that can be seen on a monitor. To enhance the differences, I’ll also be using image rollovers. Flicking the mouse back and forth across the image edge will switch between the 8-bit and 16-bit adjustment. If you’re reading this in an email feed, the rollovers probably won’t work, so you’ll need to visit the actual blog to see them.

Before getting started it’s worth noting that conventional 8-bit luminosity masks, since their introduction in the 2006 tutorial, have become standard practice for many photographers. If there was an obvious problem with them, these masks would NOT have been become so widely accepted, integrated, duplicated, YouTube-ated (videated?), or appropriated since then. They’re now recognized as a unique Photoshop tool to blend, adjust, paint, and develop images to suit the photographer’s individual style. Other photographers wishing to demonstrate and promote their own expertise in luminosity masks have adopted the 2006-process for creating them. 16-bit masks represent an advancement, I think, in the way luminosity masks are created, but their role in actually improving images is still undecided.

I’m not aware of any problems in my images that resulted from using traditional 8-bit luminosity masks, but based on my experiments with a 16-bit tonal gradient I can create two situations where there is a theoretical advantage to using 16-bit masks.

1) Severe adjustments though the most selected pixels. To illustrate this, a 600-px by 300-px 16-bit tonal gradient was used to generate a Lights-5 (Ultra Lights) mask via both the 8-bit and 16-bit methods. The full-size gradient and equivalent Lights-5 selection is shown below. NOTE: This is a completely smooth gradient on my monitor and any visible banding is a result of jpeg compression.

The Lights-5 mask is incorporated into a Curves adjustment layer over the background gradient. The curve is then adjusted (Properties panel below) to pull the whitest value (255,255,255) down to almost black (20,20,20).

The gradient with this adjustment applied through the 16-bit mask is shown below. The rollover shows the same adjustment through the 8-bit mask. There’s not much difference, but if you look closely you might see some subtle banding on the far right side for the adjustment through the 8-bit mask.

By increasing the magnification to 200%, screen-capturing the result, and then saving it as a jpeg at maximum resolution, the difference now starts to show more prominently. The image below is the 2x screen capture (a right-sided quadrant of the above image) for adjustment through the 16-bit mask. The rollover is the adjustment through the 8-bit mask identically enlarged. Now there IS an obvious difference. The 8-bit adjustment is showing some definite tonal banding, but the 16-bit adjustment remains totally smooth (except for slight banding introduced by jpeg compression).

This is, of course, a totally unrealistic adjustment (255 to 20) for any image and an unrealistic viewing resolution (actual pixels x2). But it’s necessary to show what a difference between 8-bit and 16-bit luminosity masks might look like. One interesting thing worth pointing out here is that even though this extreme adjustment would never be made to an image and would never be viewed at this resolution, the 16-bit adjustment still remains tonally smooth in the adjusted area. Conclusion: 16-masks are essentially indestructible. While the adjustment through the 8-bit mask eventually cracks under the pressure of extreme adjustment and 2x magnification, the adjustment through the 16-bit mask remains completely uniform even in this impossibly theoretical scenario.

2) Significant enlargement. The requirement to use a 2x enlargement to see an obvious difference in the first example points to the second situation where 16-bit masks might show an advantage: significant enlargement. To illustrate this, a less severe adjustment (255 to 192, image below) is made to the tonal gradient, again using the Lights-5 mask.

After adjustment using the 16-bit luminosity mask, the gradient looks like the image below. The rollover shows the same adjustment through the 8-bit Lights-5 luminosity mask. In this case, the adjustment is not sufficient to show any obvious difference between the two methods at this actual-pixel resolution.

If this gradient image is enlarged significantly (5x for this example) and viewed at actual-pixel resolution, the tonal banding can once again can be seen. In this case I used “Bicubic Sharper” as the resample method for enlargement to sharpen and enhance any differences that might exist. A segment of the enlargement that includes the tones farthest to the right in the gradient after adjustment is shown below. This is the adjustment through the 16-bit mask. The rollover shows the same adjustment and enlargement through the 8-bit mask. The difference is slight because the adjustment is less extreme, but there is definitely some tonal banding visible in the adjustment through the 8-bit mask that isn’t visible for the adjustment through the 16-bit mask.

While this adjustment is more within the realm of possibility than the last one, it would still be very rare that the whites in the image would be pulled down this much.

An actual image. Of these two examples, the first one that involved severe adjustment is where the 16-bit mask made the most visible difference. So it’s probably worth doing something similar to an actual image to see what happens, though the results of such a similarly extreme adjustment will produce undesirable aesthetic results. In this example the white values will again be dragged down to 20 using a Curves adjustment layer though a 16-bit and 8-bit luminosity mask. The Lights-4 mask will be used in this case in order to include enough pixels to actually see a significant change in the image.

Below is the a jpeg of the image that was used for this demonstration. The rollover shows the image after the adjustment. The full-size image measures 5616-px by 3744-px. The green box shows the area of the original image that is enlarged below.

This is obviously another unrealistic adjustment, but, as seen above, it takes something this extreme to produce a visible difference between 8-bit and 16-bit luminosity masks.

Below is a screen shot of the green-box area adjustment through the 16-bit Lights-4 mask viewed at 200% magnification, which is twice as magnified as actual pixels (actual pixels x2). The rollover is the same area with the same adjustment except done through an 8-bit Lights-4 mask.

In this situation there is barely any difference, and certainly none of the obvious tonal banding that was seen with the severe adjustment to the smooth gradient in the first example. You might be able to see a little better definition in the noise in the grayed-down sky of the “8-bit” image, but given the magnitude of the overall adjustment, this difference is tiny.

But why is this? Where is the expected tonal banding in the 8-bit adjustment? I can’t say for sure, but I have a theory. A perfectly smooth tonal gradient, like that used in the examples above, doesn’t really exist in a digital photograph. It’s only something that can be created in Photoshop. In a digital photograph the natural noise and the color variation captured by the camera’s sensor takes the place of the smooth gradation of the Photoshop gradient. These pixel-level, sensor-captured tonal dissimilarities, which are smooth and essentially invisible when looking at the image at any normal viewing resolution, are exaggerated upon severe adjustment though a luminosity mask. So instead of tonal banding with severe adjustment, this natural “grain” just gets “grainier.” It accommodates the tonal changes that accompany a severe adjustment through a luminosity mask (takes the place of banding), and appears to be about the same regardless of whether the luminosity mask is created via the 8-bit or 16-bit method. In other words, instead of tonal banding, an actual image shows tonal “graining” with severe adjustment, which looks similar to noise. The increase in tonal noise/grain is probably a bit more pronounced with 8-bit masks than with 16-bit masks, but because it’s at the pixel level (like luminosity masks) and not in prominent bands, it’s hard to actually see any difference.

The image below attempts to show this increase in grain/noise. It’s a screen capture at 300% zoom (actual pixels x3) of a not-too-grainy-but-not-quite-pure-white area of the sky in another part of the image that is affected by the adjustment. (It’s just to the left of the green-boxed area in the jpeg of the entire image.) The rollover shows how much grainier/noisier this area becomes after the severe adjustment through the 16-bit mask. Severe adjustment of the pixel-level tonal variations that come with digital capture and which are sorted out and enhanced by luminosity masks account for this enhanced grain/noise at this magnification. It happens even with the severe adjustment through the 16-bit mask, which is shown here, but is not nearly as obvious as the tonal banding seen in the gradient examples above. It’s also worth remembering that 16-bit masks are so perfectly matched to the tones in the image that even severe adjustments blend in smoothly. So this graining effect, which is visible at this magnification and occurs with both 8-bit and 16-bit masks, would not be visible in the image itself, especially if the tonal adjustment was more reasonable.

Conclusion. The examples in this blog looked at how a smooth 16-bit Photoshop-generated gradient reacts differently when adjusted through 8-bit and 16-bit luminosity masks. In theoretical situations at magnified viewing resolutions some noticeable differences can be seen. However, similar changes would be significantly less visible in actual digital photographs even at exaggerated viewing resolutions or enlargement because very small tonal differences captured by the camera’s sensor provide enough tonal variation to obscure the expected banding that was seen in the theoretical examples. Some combination of multiple or severe adjustments through 8-bit luminosity masks on smooth tonal areas of an image combined with significant enlargement might produce a noticeable effect in certain circumstance. But it will likely take nose-to-the-image viewing or monitor magnification to actually see it. For most images with routine processing and enlargement it simply won’t exist, especially in areas with lots of detail or structure. I have repeated the experiment using severe adjustments using both Midtone and Darks luminosity masks and have also done the same adjustments on different images. The 8-bit grain/noise with extreme adjustment is more apparent in some images, but it takes at least 300% magnification (400% is better) to appreciate any difference between 8-bit and 16-bit luminosity masks.

Other Considerations. There are some additional variables that could come into play when working on actual images. For example, it’s important not to overlook the fact that the smooth blending inherent with luminosity masks allows adjustments to be pushed further than would normally look natural, so a critical “severe adjustment” threshold might actually be reached or exceeded in certain circumstances. Also, the quality of the sensor data could play a role. Better (smoother) data might increase the likelihood of seeing banding instead of graining. (NOTE: Lens quality might also influence the sensor data and my consumer-grade zoom lens might have different results than a professional prime lens.) And perhaps other luminosity mask techniques have different outcomes. Only adjustment through a mask on an adjustment layer was investigated here. Maybe exposure blending, mask painting, or luminosity painting will show a more pronounced effect. Still, any improvement with using 16-bit luminosity masks is likely to be subtle at best. In almost all situations, images developed using 8-bit luminosity masks will likely look just as good (or identical) as those made with 16-bit masks.

The future of 16-bit luminosity masks. I admit to being somewhat disappointed that the advantages of 16-bit luminosity masks seen in the gradient experiments ended up being less evident in actual images. While my use doesn’t indicate a breakthrough improvement from using 16-bit luminosity masks, perhaps the experience of other photographers will be different. I still intend to incorporate the new process into my custom panel in the near future. Creating 16-bit luminosity masks using Photoshop actions takes no more computer processing time than using the traditional method so there’s no reason not to use the best method available, even if the advantages are just theoretical for the most part at this time. If you’d like to practice with 16-bit masks now, a free download of an action set to create them is available here. If you do come up with an example where 16-bit masks are clearly superior when making “reasonable” adjustments, please let me know. I’d like to post or link to the results in order to have a good reason to recommend 16-bit masks as the new standard.