This Smartphone Technology Can Beautify Your Photos in Milliseconds
A team of researchers from MIT and Google have teamed up to design a method for giving smartphone users high-resolution images quickly and without using up precious energy.
But hardware is only part of the story. Software developers have also been jostling to develop faster and more powerful ways for users to transform and retouch high-definition photographs — a sticky technical problem due to the large amount of power that’s necessary to generate stellar images.
Now, researchers from MIT and Google have partnered to develop next-generation photo retouching capability for smartphones fast enough to show the improved visuals on the display while the photographer is still framing the shot.
The technology was created using machine-learning techniques that analyzed thousands of raw images and compared them with pictures that had been retouched by professional photographers.
The innovative new system works something like this.
Imagine for a moment you have a full-color, high-resolution photo of the Mona Lisa, and you want it to be retouched, making the image even sharper in order to impress friends and family with your keen photographic skills.
Instead of sending the full hi-res photo to a photo processor, you draw a diagram of the image on a napkin that has all the lines of the image exactly where they should be, but which obviously lacks the detail of the original.
You send that napkin drawing to an artificially intelligent photo-retoucher. "A-ha," the AI technician says. "There's a face in the middle and trees in the background, and I know how to boost the resolution for faces and trees."
That, very roughly speaking, is how the new technology operates. It provides a simplified map that allows the program to decide which changes to make without actually needing to process all the data in the original image. Evaluating the less-complicated version of the image drastically reduces the time and energy required to amend the photograph, which, in terms of your hand-held device, means more room for other tasks and quicker turn around time for enjoying your images.
“The key here is that we predict transformations to the pixels, instead of predicting pixels,” said Michael Gharbi of MIT, who worked with Google during an internship at the company to develop the final version. “If we train our method to recognize faces and retouch the skin differently from, say a landscape or a tree, then if we show the network enough examples of how the image should be transformed, it will be able to learn this retouch.”
The low-resolution image is used like a simplified model to predict what kinds of changes should be made to the full-resolution picture. Performing an analogous level of processing on a full-resolution image would take up to a minute after the picture is taken, Gharbi said, whereas this new method can be accomplished effectively in real time.
During the machine-learning sessions, the system was trained by examining raw and retouched photographs to predict color transformations and then taught to express those transformations in mathematical language. The system utilized a database of 5,000 photographs taken with SLR cameras, which had then been retouched using the photo-adjustment software Adobe Lightroom five times each by different human photographers to allow the system to study different approaches.
“Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation,” the team wrote in the abstract of a paper that they will present this week at the Siggraph digital graphics conference. “Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators.”
Gharbi and MIT professor of electrical engineering and computer science Frédo Durand collaborated on the project with Jiawen Chen, Jon Barron, and Sam Hasinoff of Google.
Google reached out to Gharbi after he wrote a paper on a similar technique a few years ago, and offered him the internship.
“This technology has the potential to be very useful for real-time image enhancement on mobile platforms,” Google’s John Barron told MIT News. “Using machine learning for computational photography is an exciting prospect but is limited by the severe computational and power constraints of mobile phones. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.”