Working With Sketchy Images

A new forensics computer application overcomes image-matching hurdles

7 September 2012

If you’re a fan of TV shows like “CSI: Crime Scene Investigation,” you’ve probably been impressed with the technology used to identify a suspect from poor photographs. After all, the show’s forensics staff can take a surveillance camera’s grainy image, run it through databases holding millions of mug shots, and find a match in seconds.

But the face-matching systems of real-life law enforcement agencies are not that sophisticated, at least not yet. For example, there is no way they can accurately compare a forensic artist’s sketch of a suspect with mug shots in a database.

“With sketches, police still look for suspects the same way sheriffs in the Wild West did: with ‘Wanted’ posters distributed in public places,” IEEE Fellow Anil Jain says. But that could change, thanks to a computerized forensics system being developed by Jain and other IEEE members at Michigan State University, in East Lansing. Simply put, the system matches artists’ sketches with mug shots. Jain is a university distinguished professor in the department of computer science and engineering.

Technology has provided law enforcement with many tools, including DNA and biometric ID systems, but plenty of criminals never leave such clues behind. Instead, police often depend on a forensic artist and the recollections of eyewitnesses. The artist works with witnesses to draw a sketch of the culprit’s face; a detective uses software to build a composite drawing from “canned” facial features like eyes, noses, and ears. The rendering is distributed to police officers and the news media in the hopes that someone will recognize the suspect and make a report.

“Technically, a photo and a sketch are two different modalities,” Jain explains. “They both depict a face, but they are sensed differently by a computer. It’s like comparing an image taken by your digital camera with an infrared image of the same face. They’re just too different in appearance. And it’s too time-consuming to manually compare a sketch to thousands of mug shots, unless there’s something unusual about the suspect’s facial features, such as a scar.”

Beyond dealing with the different textures of drawings and mug shots, those trying to match the two types of images must also consider that sketches can be imprecise and perhaps inaccurate because of fuzzy witness recollections.

Jain and IEEE members Brendan Klare and Zhifeng Li tried to get around such problems with their face-recognition system, which they describe in “Matching Forensic Sketches to Mug Shot Photos,” IEEE Transactions on Pattern Analysis and Machine Intelligence (March 2011). Jain and his team recently received funding from the U.S. National Institute of Justice to build a prototype.

Automated systems that match digital photos typically find key landmarks for alignment such as the center of the eyes, then measure a set of facial features. These systems are accurate when photos are captured under controlled conditions, such as well-lit environments while the subject is holding his head upright, looking at the camera, and displaying a neutral expression, according to Jain. “However,” he says, “they do not work well at all when trying to match a photo to a sketch.”

The new system developed by Jain and his colleagues—known as local feature–based discriminant analysis (LFDA)—bridges the gap between sketches and photos. The researchers scanned and digitized nearly 200 forensic sketches and entered them into a database that already held photos of the same people. The Michigan State Police and other forensic sketch artists provided the images.

“We examined the kind of changes taking place in the sketch in terms of how dark a line was or the direction in which it was going,” Jain says. “Based on our descriptors, the system attempts to find the corresponding photos. Basically, we trained the system to look for a photo in the database that matches the sketch. No one has done this before with sketches in a law enforcement setting.”

Algorithms initially found in research papers for matching laboratory-generated sketches relied on generating a synthetic photograph from a sketch, but Jain’s team realized that would not work when matching real-world forensic sketches. Instead, the LFDA system represents both sketches and photographs with scale-invariant feature transformation and multi­scale local binary pattern feature descriptors, which display the structure and shape of local regions within a facial image.

The LFDA framework was tested on those 200 forensic sketches and matched against a database with more than 10 000 mug shots. The LFDA algorithm achieved recognition accuracies nearly an order of magnitude greater than did one of the leading commercial automated face-recognition systems.

Jain says he hopes to install prototypes at two different law enforcement agencies by year’s end.

Read my blog about Jain's work on a system that would detect altered fingerprints and what he has to say about the privacy implications of collecting biometric information.

IEEE membership offers a wide range of benefits and opportunities for those who share a common interest in technology. If you are not already a member, consider joining IEEE and becoming part of a worldwide network of more than 400,000 students and professionals.

Learn More