Undergrads Apply Core IEEE Mission to Local Needs

Students create a system to translate sign language into words

8 July 2013

When IEEE Student Member Hend Kittawi and three classmates teamed up for their EE senior project at the University of Jordan, in Amman, they came up with an idea that fit perfectly with IEEE’s core mission, namely, “to foster technological innovation and excellence for the benefit of humanity.”

Kittawi, along with Bayan Jafar, Saja Obiedat, and Dina Shoubaki, developed a system to enable users of Arabic Sign Language (ArSL) to communicate via a smartphone with people who don’t know the language. Under the supervision of IEEE Senior Member Ghari Al Sukkar, an assistant professor at the school, they applied their hardware, software, and linguistic skills to produce the key to the system: a glove that can sense the hand movements of ArSL. These are then translated into Arabic text and transmitted via Bluetooth to be displayed on Android smartphones.

The aim of the Jordanian students was at least as much social as technical. The gap in Arab society between people who can communicate by talking and people who can’t and must use sign language or other means “creates an atmosphere of awkwardness and limits nontalking people’s progress,” they write in their research paper. Arab society has not paid enough attention to these people’s needs, they say, “so we’ve concluded that it is about time someone did.”

While anyone can learn sign language, few among the Arabs except the deaf, the mute, and their family members ever do. “In the Arab region, not much attention is paid to the deaf and mute, so even they aren’t very keen on learning sign language,” the team writes. “Enabling the vast majority of people, who don’t know or need ArSL, to understand it will make signing more useful to those who do need it, encouraging them to learn.”

“Few children attend schools that teach [ArSL] because parents want to keep their kids in regular schools in hopes it will help them assimilate into society,” they continue. In many cases, the deaf and mute use a private sign language of their own, communicating only with family members, rather than learning ArSL. “There’s not even much about ArSL on the Web,” according to the team.

Work is going on around the world to develop ways to make signed communication understandable by the nonsigning majority. The approach chosen by most researchers, including the Jordanians, is to equip a glove with sensors to encode hand motions and then decode the signals into speech or text. Luckily, millions of people now carry computers that can do this decoding—smartphones—all the time. And motion and position sensors have become small, lightweight, and battery-efficient enough to be built into the gloves.

THE PROJECT

Of the many projects developing gloves for sign language, few have been devised to translate ArSL. After reading up on such projects, the team looked for hardware components that would survive everyday use and be readily available at a reasonable price. Kittawi and Shoubaki then began building the hardware while Jafar and Obiedat tackled the software.

Sign language motions are complex and three-dimensional, so the system combines signals from several sources. An accelerometer and a gyroscope monitor hand motions; contact sensors built by the team determine when the fingers touch; and off-the-shelf flex sensors measure finger bends. The signals are then processed by a common PIC microcontroller, which feeds a Bluetooth transceiver module that then transmits the processed data to an Android mobile device.

Developing the software required learning both the Mikrobasic programming language for the PIC and the Android programming language for the mobile app doing the actual translation. The result, tested on the Android Jellybean (4.x) operating system, translates both symbolic sign language and the signed ArSL into text

ROOM FOR DEVELOPMENT

More work is needed if the team’s prototype is to be turned into a product. As it stands, the students built just one glove, so their system cannot translate every sign and gesture used by ArSL, which like other sign languages requires two hands. Furthermore, the students would like to see their system become a two-way, interactive device so that people with mobile phones could text back to a display on the glove.

Also, once the breadboarded prototype’s circuitry is moved to printed circuit boards, the glove could be made lighter than its present 250 grams (8.8 ounces) and less bulky, easing a wearer’s hand movements. The hope is that this and other improvements would allow the glove to be produced for less than the cost of the prototype’s parts: 250 Jordanian dinars (a bit more than US $350). The developers would also like to see versions of their app for iOS, Symbian, Windows Phone, and other mobile platforms.

Though the team members have now all graduated, they plan to complete and produce the translator. They have entered Jordan’s Queen Rania Entrepreneurship Competition in hopes of winning enough money to create a start-up company.

Eventually, Kittawi says, “the wireless sign language translator can be a very good assistant for a deaf person. It is portable and easy to deal with. It can narrow the communications gap between signers and nonsigners.”

Learn More