Spoken Language Processing Techniques for Sign Language Recognition and Translation.
Abstract
We present an approach to automatically recognize sign language and translate it
into a spoken language. A system to address these tasks is created based on state-of-the-art
techniques from statistical machine translation, speech recognition, and image
processing research. Such a system is necessary for communication between deaf and
hearing people. The communication is otherwise nearly impossible due to missing sign
language skills on the hearing side, and the low reading and writing skills on the deaf side.
As opposed to most current approaches, which focus on the recognition of isolated signs
only, we present a system that recognizes complete sentences in sign language. Similar
to speech recognition, we have to deal with temporal sequences. Instead of the acoustic
signal in speech recognition, we process a video signal as input. Therefore, we use a
speech recognition system to obtain a textual representation of the signed sentences. This
intermediate representation is then fed into a statistical machine translation system to
create a translation into a spoken language. To achieve good results, some particularities
of sign languages are considered in both systems. We use a publicly available corpus to
show the performance of the proposed system and report very promising results.