Pinchas Gutter has many stories to tell about his life during World War II. His entire family was killed in the Nazi death camps, and he can hardly remember his sister with the “golden hair.” Now 82, he and survivors like him won’t be around much longer to talk of their experiences, which is why one university has made a virtual replica of Gutter so that those in the future can meet him and engage him in conversation.
The team at the University of Southern California’s Institute for Creative Technologies (ICT), in Los Angeles, has been working for several years to combine 3-D, hologram, and natural-language techniques to realize this idea, long popular in science fiction. The hologram looks and acts almost like a human being; it talks and exhibits emotion.
ICT partnered with nearby graphic design company Conscience Display as well as USC’s Shoah Foundation, which records video testimonies of Holocaust survivors, to create the New Dimensions in Testimony project. This initiative records and plays back testimony in a way that will continue the dialogue for generations to come between survivors and people interested in firsthand accounts of what happened. Gutter’s holographic virtual being is programmed to talk and respond to any question. His hologram could be used in museums to educate others about his experiences and the costs of the Holocaust.
“We didn’t just record his words and stories. We made it possible to have an interactive experience with him, and that is what makes this unique,” notes IEEE Member David Traum, ICT’s director for natural-language research and a research assistant professor at the university’s Viterbi School of Engineering. Traum says this same technology could also be used to create such virtual time capsules of today’s world leaders and iconic figures and, eventually, people like you and me.
CIRCLE OF LIFE
ICT has built one of the most advanced systems for recording an entire human body to create a hologram. The person is recorded in the center of a 180-degree hemisphere outfitted with 50 high-definition video cameras and more than 6,000 stage lights. The hemisphere, with a circumference of 6 meters, allows for the subject to be recorded in high resolution from every angle.
As Gutter, born in Poland, sat at the center of the sphere, the cameras captured his every hand gesture, posture, and eye movement as he was asked question after question. Some of the cameras zoomed in on his face and hands, others recorded his entire body from different distances and angles.
Gutter was asked more than 2,000 questions—as many as the Shoah Foundation and the USC researchers could think of. The project seeks to provide insight into the experiences of survivors by having them answer questions they are often asked, as well as others they are not. Questions included “When was the last time you saw your sister?” and “Do you believe in God?” The questioning took more than 30 hours, spaced over seven days.
Viewers see Gutter’s facial expressions as he tells his story. Once the footage was collected, ICT’s chief digital officer, Paul Debevec, and his team worked to create the highly detailed hologram. The footage is played back simultaneously through 216 high-speed video projectors closely placed adjacent to one another, forming a 135-degree angle, which presents a large field of view. The images are projected onto a 2-meter-tall diffusing screen, designed to scatter light vertically so each pixel can be seen from multiple angles. This allows the hologram to be viewed from different vantage points. If the hologram were projected in the center of a room, an audience would be able to nearly surround it.
Projecting the hologram in this way makes it possible to view the hologram in three dimensions without special glasses or headgear. Traum points out that as the technology for projection screens advances, the images will appear even more seamlessly with their surroundings, and people viewing the hologram will forget there is a screen at all.
Traum and his colleagues pieced together the questions and answers using natural-language processing algorithms, in a manner similar to those employed to create the iPhone’s Siri. But instead of searching for relevant information on the Web for an answer, the program uses cross-language relevance modeling to connect the words in a question with words in Gutter’s answers that would form the most appropriate response. The algorithms respond with more than 70 percent accuracy to any given question. People can ask whatever they wish. For example, when the USC effort was shown on TV’s Today show, host Matt Lauer interviewing Gutter’s hologram, asked, “Is there any question you can’t answer?” It responded, “I have no comment about rap music.”
ICT is also looking into the interactive technology for other purposes. There’s the possibility of creating an interactive hologram to eliminate the need for, say, a lecturer to travel; she could give the same talk to audiences anywhere. Or a professor could be present in the same class at universities around the world. In other words, it would be possible for one person to appear to be at more than one place at the same time.
In addition, ICT is looking into low-tech versions for people to record themselves with their Web cameras. These would not offer the 3-D effect, however. But with the expected advances in virtual and augmented reality, there’s no telling how far the technique could eventually go, Traum says.