Show your voice - automatic sign language interpreter
Mátételki Péter
<>
MTA SZTAKI
We create an assistive tool to improve life for hearing- and speech-impaired people enabling them to easily get in touch with the non-disabled using hand gestures. Our innovative hardware-software ecosystem consists of wearable motion-capture gloves and a software solution for hand gesture recognition and language processing. This integrated tool operates as a simultaneous interpreter helping the disabled to communicate using their native language, the sign language. The InterpreterGlove reads out the signed text loud, so the disabled and hearing-abled (without any sign language knowledge) can fluently communicate with each other.
The glove prototype is made of breathable elastic material, its ergonomic design ensures seamless daily use. The hardware includes integrated orientation sensors and incorporates a wireless communication module on its main board. Signal fusion and spacial geometry algorithms are used to calculate the digital copy of the hand, which is denoted by a special gesture descriptor. Connected to the user's cell phone, the glove transmits the gesture descriptor stream. The mobile application performs intelligent segmentation and automatic word-correction to create comprehensive text and reads it out loud.
Currently members of this disability group either need to go through cumbersome explanations to describe their intent or they need to be assisted by a human sign language interpreter. Using the InterpreterGlove advances their social integration as they are enabled to express themselves to non sign language speakers. The InterpreterGlove opens up possibilities like shopping, personal banking, taking part in education and much more beyond. It may also play a significant role in boosting the employment possibilities of the speech- and hearing-impaired people.