Real-Time Sign Language Recognition and Translation: A Mobile Solution Using Convolutional Neural Network
DOI:
https://doi.org/10.65141/ject.v2i1.n2Keywords:
Sign language recognition, Convolutional Neural Network (CNN), mobile application, usability testing, communication barriersAbstract
This study presents a mobile application for sign language recognition and translation using a convolutional neural network (CNN) to overcome communication barriers for the deaf community. Unlike existing solutions, the app uses a CNN trained on a dataset of 200–450 images per sign to process hand images via preprocessing, feature extraction, and hand landmark detection, accurately recognizing sign language gestures. The application underscores a user-friendly interface and is designed for real-time mobile use. Employing CNN-based image processing, it translates hand movements into gestures with high precision, achieving 96% accuracy and a loss of 0.069 after 100 training epochs with a batch size of 40. Usability testing, conducted using the System Usability Scale (SUS) questionnaire, revealed high user satisfaction, with positive feedback on usability, functionality, maintainability, and efficiency. The average SUS score indicates an excellent usability. Further evaluation criteria included precision, recall, and F1-score, all of which demonstrated strong performance. The system effectively bridges the communication gap between the deaf and hearing communities, fostering more accessible and meaningful interactions.
References
Al-Hammadi, M., Muhammad, G., Abdul, W., Alsulaiman, M., Bencherif, M. A., & Mekhtiche, M. A. (2020). Hand gesture recognition for sign language using 3D CNN. IEEE Access, 8, 79,491–79,509. https://doi.org/10.1109/ACCESS.2020.2990191
Farooqui, A., Hatekar, A., Angadi, P., & Shrikant, K. (2021). A sign translation app to convert ASL to auditory English language. i-manager’s Journal on Mobile Applications and Technologies, 8(1), 1–7. https://search.proquest.com/openview/7cf43e8d92d86c58da13b0114a302773/1
Han, L. H. Y., & Murli, N. (2022). BIM sign language translator using machine learning (TensorFlow). Journal of Soft Computing and Data Mining, 3(1), 68–77. https://penerbit.uthm.edu.my/ojs/index.php/jscdm/article/view/11662
Hoque, O. B., Jubair, M. I., Islam, M. S., Akash, A.-F., & Paulson, A. S. (2018, November 30). Real time Bangladeshi sign language detection using Faster R-CNN. arXiv. https://arxiv.org/abs/1811.12813
Ismail, M. H. I., Dawwd, S., & Ali, F. H. (2021). Static hand gesture recognition of Arabic sign language using deep CNNs. International Journal of Computer Applications, 975, Article 8887. https://www.researchgate.net/publication/352540178
Jagtap, S., Jadhav, K., Temkar, R., & Deshmukh, M. (2024). Real-time sign language recognition using MobileNetV2 and transfer learning. arXiv. https://arxiv.org/abs/2401.09251
Li, D., Rodriguez Opazo, C., Yu, X., & Li, H. (2019). Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. arXiv. https://arXiv.org/abs/1910.11006
Li, Y., Wang, J., & Smith, D. (2019). WLASL: A large-scale word-level American Sign Language dataset on the web. In Proceedings of the 27th ACM International Conference on Multimedia (pp. 1237–1245). https://doi.org/10.1145/3343031.3350603
Li, K., Zhou, Z., & Lee, C.-H. (2017). Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications. Journal of Computing and Information Technology, 25(1), 53–63. https://doi.org/10.1145/2850421
Murali, R. S. L., Ramayya, L. D., & Santosh, V. A. (2020). Sign language recognition system using convolutional neural network and computer vision. International Journal of Engineering Innovations in Advanced Technology, 4(4), 1–7. Retrieved from https://www.ijert.org/sign-language-recognition-system-using-convolutional-neural-network-and-computer-vision
Nagy, D. Z., & Piller, I. (2020). An optical flow–based gesture recognition method. Acta Marisiensis, Series Technologica, 17(1), 55–60. https://doi.org/10.2478/amset-2020-0010
Rastgoo, R., Kiani, K., & Escalera, S. (2021). Sign language recognition: A deep survey. Expert Systems with Applications, 164, 113,794. https://doi.org/10.1016/j.eswa.2020.113794
Rathi, D. (2018). Optimization of transfer learning for sign language recognition targeting mobile platform. arXiv. https://arXiv.org/abs/1811.08421
Sarma, D., Kavyasree, V., & Bhuyan, M. K. (2020). Two-stream fusion model for dynamic hand gesture recognition using 3D-CNN and 2D-CNN optical flow guided motion template. arXiv. https://arXiv.org/abs/2006.11338
Selvaraj, P., Gokul, N. C., Kumar, P., & Khapra, M. (2021). OpenHands: Making sign language recognition accessible with pose-based pretrained models across languages. arXiv. https://arXiv.org/abs/2109.03480
Sreemathy, R., Turuk, M., Kulkarni, I., & Khurana, S. (2023). A multimodal approach to sign language recognition using deep learning. Education and Information Technologies, 28(5), 5,259–5,278.
Viswavarapu, L. K. (2018). Real-time finger spelling American sign language recognition using deep convolutional neural networks (Master’s thesis, University of North Texas). UNT Digital Library. https://digital.library.unt.edu/ark:/67531/metadc1404544/
Wadhawan, A., & Kumar, P. (2020). Deep learning–based sign language recognition system for static signs. Neural Computing and Applications, 32, 7,957–7,968. https://doi.org/10.1007/s00521-019-04691-y
Xia, K., Lu, W., Fan, H., & Zhao, Q. (2022). A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation. Sensors, 22(23), 9107. https://doi.org/10.3390/s22239107
Zhang, J., Bu, X., Wang, Y., Dong, H., Zhang, Y., & Wu, H. (2024). Sign language recognition based on dual-path background erasure convolutional neural network. Scientific Reports, 14(1). https://doi.org/10.1038/s41598-024-62008-z
