Winner of Communication Task on HackYeah 2019
I’m trilled to announce that I was a part of a team of developers, designers and data scientists, which came together to create a solution that could help bridge the communication gap between deaf and hearing communities. Our goal was to create an easy-to-use app that could translate sign language into text in real-time, making it accessible to a wider audience.
The project is called Signslate - personal sign language translator. The solution is composed of Android mobile app and cloud-based data processing pipeline, that processes video footage from mobile device using OpenPose pose estimation model and then recognizes individual signs using random forest classifier.
The process of development was not easy, given the time frame. We faced many challenges, such as high inference time of pose estimation model, fine-tuning the sign classifier, efficiency of on-device video preprocessing/compression and also network latency. But we managed to overcome the obstables and delivered the solution on time.
We are particularly proud that our prototype could recognise small subset of signs from the Polish sign language dialect, which showed the potential and possibility to be easily extended for more signs or dialects.
More info: https://hackyeah.pl/winners-2019/