HDR photo editing with machine learning

The Ultra HDR format: HDR and SDR rendition in one file

The new Ultra HDR image format encodes a dynamic range compressed SDR image alongside image metadata used to dynamically expand this range as needed. Since the SDR image is maintained, Ultra HDR images can be viewed as usual on legacy displays. However, the format’s metadata allows rendering the full dynamic range as authored by the camera on HDR-capable displays. It also enables a smooth transition between SDR and HDR renditions, even adapting to display capabilities dynamically as a display’s peak brightness automatically adjusts to the environment.

The Gain Map: Encoding the HDR rendition

The Ultra HDR format includes metadata called a Gain Map[7bea7e] a log-encoded image stored alongside the SDR image that indicates how much to brighten each SDR pixel to produce the target HDR rendition. For example, a Gain Map pixel value of “0” may represent no difference between SDR and HDR renditions for a given pixel, while “1” may represent the maximum allowable brightness difference. The Gain Map allows for spatially-varying, local differences between the image renditions, so different regions can have varying amounts of added brightness or contrast when viewed in HDR. To generate an Ultra HDR image, a camera pipeline therefore must save the usual SDR image, the Gain Map, and its other metadata for HDR expansion.

Image editing with Ultra HDR

Although the format optimizes image rendition across diverse displays, it has a drawback: the Gain Map increases the complexity of image editing operations. As a trivial example, imagine we want to crop an HDR image. Now, we must crop the SDR image and the Gain Map.

Basic HDR image edits like “crop” and “rotate” have simple Gain Map editing implementations, already part of Google Photos. However, the Google Photos editor includes many more editing features that are complex and often powered by ML and computational photography, including Magic Eraser, Photo Unblur, Magic Editor, Portrait Blur, and Motion Photos. How to best edit the Gain Map when using these tools is ambiguous because these features were developed for SDR images, with models expecting SDR images as inputs and producing SDR images as outputs.

Using Magic Eraser, for example, you can erase a distracting element of an image, applying image inpainting to generate new pixels in the erased region. For an HDR image, if the erased portion was bright, then it’s likely that the matching Gain Map area was bright as well. As with the crop example, now you’d also have to inpaint the Gain Map. Otherwise, you would see an undesirable “ghosting” effect in the HDR rendition — the original Gain Map appearing as an overlay on the edited SDR image. Thus, until now, if you edited an HDR photograph using an ML-powered tool like Magic Eraser, Google Photos would simply drop the Gain Map altogether, downgrading the image to SDR in the new edited image.

Gain map reconstruction

Inspired by how Magic Eraser reconstructs the missing parts of SDR images, we trained a new ML model to reconstruct the missing Gain Map regions after HDR image editing. Given the original SDR image, the edited SDR image, and the original image’s Gain Map, our model predicts a new Gain Map. We then blend the prediction with the original Gain Map — using a mask that indicates where SDR image pixels have been modified during editing — to show a uniform image.

Training the Gain Map reconstruction model

Training our model required the collection of several thousand HDR images: SDR images with Gain Maps and metadata for HDR rendering. We captured diverse photographs using the HDR+ burst photography pipeline, applying the dynamic range compression algorithms of the current Pixel camera to generate SDR and HDR image pairs. With this data and a dataset of random mask shapes, we trained a lightweight image-to-image translation model to predict the Gain Map given the edited SDR image and a masked version of the original Gain Map. It is even flexible enough to produce a Gain Map given only an SDR image as input — optimizing rendition for HDR displays.

Our Gain Map reconstruction model is under 1 MB and runs at interactive frame rates on mobile devices. It maintains HDR rendition during ML-powered image editing in Google Photos, on Pixel 8 and newer devices. Crucially, no ML-powered editing features require custom Gain Map editing workflows, as our model can work for any effect — even those not yet built. Here are some examples of our Gain Map reconstruction in action:

We see our model as a first step towards enabling HDR image rendering to shine as often as possible.