The two suspects in the Boston Marathon bombings were found by a relatively old-fashioned method: a witness told cops he made eye contact with a man at the scene who was wearing a baseball cap and who left behind a backpack. That identification allowed investigators to narrow their search when combing through the myriad videos and photographs of the crime scene.
But eventually, computers could help sort through thousands of images to find a suspect. Researchers are learning how to better program computers to not only recognize a face in a crowd but also match a figure wearing dark glasses and a baseball cap in one image to a similarly dressed figure in a Facebook posting, for example. The hope is that one day, law enforcement officials could sort through images from surveillance cameras and posted to social media sites and find useful details in real time.
"The state of the art is progressing," said Takeo Kanade, a professor at the Robotics Institute of Carnegie Mellon University and one of the early pioneers in computer vision.
In the past, getting a computer to recognize an object as simple as a chair from another object was a tough problem. But then programmers realized that rather than trying to define "chair-ness," it was better to have computers compare thousands of pictures of chairs and recognize things that looked similar. The same principle applies to recognizing people.
"When it comes to more standard recognition, like looking at a mug shot, are probably better than humans." He noted that computers are already quite good in that they can pick out a face 85 percent of the time when searching among a million photos. That is much better than humans can do.
But take a computer out of its "comfort zone," and the situation changes. "When it comes to more difficult situations -- not a full frontal face shot, or a smaller picture, it's harder," Kanade said.
To address this problem, computer scientists are teaching machines to sort through other digital information in a photograph. For example, a computer might be better at re-identifying a person who is wearing a certain piece of clothing. "There are a lot of techniques emerging," Kanade said. "Not perfect, but basically some statistical techniques... lines, small patch of color, or texture," he said.
The ability to recognize faces and match them with the right person will likely advance to the point where it's possible to eliminate suspects, matching people in a crowd, say, to social media profiles and other online footprints people leave behind. "It could reduce the number of frames the investigators have to go through can be reduced by a factor of ten or 100," Kanade said.
Kanade said one day we might be living in a world where computers take over investigations, with humans only acting as backstops. "I'm a firm believer that computers will be better than humans. It's certainly getting that way."
Alessandro Acquisti, associate professor of information technology and public policy at Carnegie Mellon, ran an experiment last year matching photos of random people, pulled from web cams, with their social network profiles. Running that kind of image processing for an event like the Boston Marathon would probably take hours using current technology and the number of false positives would be large.
But 10 years from now, with the help of cloud services and supercomputers, "the time needed to complete a similar exercise would be significantly smaller -- close to real time."