Braille3D: using haptic and voice feedback for braille recognition and 3D printing for the blind

In today’s state of technology, people can see the innovations that have evolved throughout time. One of those is assistive technology which exists to help impaired people from doing their daily activities. This technological advancement is made to help people accomplish tasks they could not perform before. Assistive innovations today reside on computers and people, not only the disabled, have become dependent on them as they make life more efficient and convenient. Accessibility has always been a challenge for people with disability. Researchers and inventors resolved this problem by integrating solutions and innovations into society which benefited the disabled community to let them experience a more convenient life, easing their burdens that hinders them from living an enjoyable life. Assistive technology was one of the solutions that innovators eventually came up with. Prosthesis was invented to be an extension to legs, arms, and other body parts for people who lost a limb, wheelchair for those people who were paralyzed or have difficulties in walking. Other assistive technologies were made to accommodate current technology such as accessible keyboards, text-to-speech software, and speech recognition software. With that said, the motivation of this is study is to contribute to the society by helping people with visual impairment to have more access to technology and to give them a better learning environment in which the person can learn the Braille system.


Introduction
Today's assistive technology is not only a trend in the medical field, but also in the field of education. This type of technology is vastly used by people who are disabled. Students in classroom take advantage in the uses of this assistive technology. Innovative equipment such as refreshable braille display, screen reader, and such exists to aid students in their classroom lessons. Using a software that translates text into braille are helpful for students who are blind. Text-to-speech applications also can aid them since it converts text into audio so that they could access text on digital devices [1].
Assistive technology is a solution for the blind, mute, deaf, or people who have lost a limb. This type of technology is defined as any item, system, machine, or service that is used to improve the functional disabilities of a person. The most common assistive technology for visually impaired people is the braille. Braille was created to accommodate the blind into reading the text in a specialized paper with little dots that are embossed. Braille has a unique combination that can be characterized as alphabets and numbers [1].
When it comes to the implementation of coded instruction into integrated software or hardware, Computer Aided Instruction (CAI) is generally used. CAI is a simulation of exercises that has the ability to show feedbacks to its user whether the answer is correct or wrong. Interaction is key since the computer will be the proxy of a teacher. Hence, CAI can be called as a supplementary teaching tool that incorporates real training and lessons with interactive feedbacks [1]. The examples of CAI include a combination of infographic texts, audio, and visual exercises that are commonly used as drills for the user. CAI is an instructional program technique that exercises self-learning through interaction of the student for educational purposes. This refer to the utilization of computing materials such as mobile phones and computers as a tool to accelerate the learning process of a student. The researchers have decided to integrate CAI in the project [2].
The proponents are tasked to produce a mobile application that will incorporate braille and 3D printing into a mobile application which will teach the braille characters through sequence of lessons and exercises where assessment modules are assimilated. 3D models will be available in the application for printing to let the student feel the geometry of a specific object. Figure 1 shows the conceptual framework of the accessibility features of the system.

Objective of the Study
The general objective of this study is to create a mobile learning application incorporating haptic and voice feedback for braille recognition and 3D printing for the blind, specifically for visually impaired kindergarten students. Generally, the goal is to develop an application for the visually impaired with the use of the mobile phones' vibrational and voice capabilities and to promote the geometry of an object through 3D printing.

Scope of the Study
For the learning module, since the target users are visually impaired kindergarten students, the application will feature lessons that will display the basic Braille characters (A to Z, 0 to 9) as well as the Braille spelling for shapes. The user can visit this module to learn and review the pattern for each Braille characters. Each letter shall have example words that the app will read to the student when prompted, as well as an option to 3D print that word example if there is an available 3D model for that word. The learning module for the numbers shall be presented in an orderly manner to also teach the user to count from 0 to 9. The learning module for shapes shall teach the user the spelling of the basic shapes in Braille. Some modules won't be accessible and will be unlocked later based on the learning performance of the user.
The application exercises are to assess the learning progress of the user. Each exercise contains a set of Braille characters. The application will display the dot pattern for the Braille characters and the user will have to identify that character in a multiple-choice format. Choosing the correct answer will earn the user a point and will proceed to the next item, choosing the wrong answer will encourage the user to try again and pick another choice, until the correct answer is selected. After the user answers all the item, the application will assess the user's performance based on the correct answers and retries to pick the correct answer. There will be another interface for the teacher. The teacher will be the one to view the assessment of the student user. The teacher is also responsible for the Content Management System where they can modify exercises and assign them to the student. The teacher can also manage the lesson modules by adding example words to the Braille characters and also add a 3D model for that word. Unlocking lesson modules for the students is also an option for the teacher user.
Computer Aided Instruction techniques will be followed to make the learning experience of the student more engaging and interactive. The proponents will do this by adding haptic and voice feedback, as well as 3D models for printing 3D objects to make the students more interested [1].
For 3D printing, as have been stated before, some of the word examples in the learning module have 3D models attached to them. 3D models can be printed by connecting the device to a 3D printer via Bluetooth or USB cable, and by using a 3D printing application. Since this process might be too complex for a kindergarten student, the teacher, or someone who has knowledge in 3D printing, must handle the 3D printing process [3].

Review of Related Literature
3D printing is considered an aid in literacy for kids in many countries. The study shows that tactile figures like replicas of monuments, maps, and other unfamiliar object result to improved comprehension and literacy skills. [4] Since most of the learning materials for the visually impaired are limited, 3D printed models are a huge boost in the learning capabilities for the blind. Furthermore, 3D printed tactile objects aids the overall cognition of the student of the material itself because physical models generate a sense of familiarization to the student. This gives the student more understandings rather than embossed books. Classes like biology where the student can feel the geometry of a certain organ are proof why 3D printing is needed in the learning materials for the blind. [4] The swift occurrence of technology resulted into the creation of turning materials that are designed from software into a physical manifestation, thus the 3D printing. Since blind people have a difficulty in gathering/accessing information, 3D printing can be of aid to the visual impaired community. Moreover, 3D printed objects give the person the form and structure of the 3D model through the sense of touch. Thus, giving the justification of the relevance of 3D printing in the proposed topic. Other researches were focused on a tactile-based solution to improve touchscreen mobile interface exploration by blind users [5].
The use of mobile in different ways to present the braille. First is the Braille scan where a six-dot Braille layout are placed in a 2 x 3 matrix which are used in mobile phones. People can read the characters by moving the stylus on the device. The dots become available for reading from the moment the stylus was placed on screen until it was lifted off. Second is the Braille Sweep where it uses six-key layout to generate the characters. This method also needs stylus to function properly because it needs to use the sweep method. Vertical and diagonal movements are possible making this method more efficient than the first. It is like Braille Scan only it is laid horizontally. Third is Braille Rhythm which enables the person to read the characters by holding the stylus on the screen allowing a tactile pattern to appear. This method presumably was the most promising method for tactile perception since it allowed the user to navigate and read the characters easier. The study of Braille is to make the methods more effective and to make the representations of Braille to be more practicable to users [6].
Modern smartphones with the use of touch-screen and vibrations introduced for the blind or deafblind who mostly relies on their tactile sense to convey and absorb information. In V-Braille, the screen is divided into 6 parts, representing the dots of a Braille character. When any of these 6 cells is touched, the phone vibrates if the cell touched has a raised dot. The screen can be tapped or stroked in different areas, and multi-touch is not supported [7] [8].
Computer Aided Instructions is a computer application that was implemented in a coded instruction, it describe also as integration of a software and the hardware. CAI have different type of for, like game, tutorial or simulations. The good thing about CAI is to present the actual feedback responses, and summarizing the result of the performance, it teaches the new skill as supplementary device which associated to the real trainings and lessons. CAI is like a teacher which interact to the user, through CAI the user can achieve information or instructions which can be found in different devices; it is also attractive because of the animations and images [2].
Text to Speech is a type of speech combining an application that would yields to a digitalized spoken sound of the text. Screen Reader usually used, it can read different types files and web pages, or naming the program or folder where the user points on the screen [9].

Methodology
The methodology the proponents used for this project is the iterative and incremental method. The project follows an iterative process where each phase is done repeatedly to fully discover and meet the requirement of the system. Each iteration will add an improvement to the system based on the previous phase. This method contains five phases, requirements, analysis and design, implementation, testing, and deployment. The mobile app is created on Android platform using Android Studio, written in Java programming language. The layout of the application is written in XML language. For the backend, Google's Firebase (a backend as a service) platform is utilized, specifically for the authentication, realtime database, and cloud storage.
The accessibility service that the app will use is a built-in function on Android devices called TalkBack. It is a screen reader, it reads not only the texts on the screen but also describes its components (such as buttons and lists). Aside from providing speech, it also provides haptic feedback or vibrations to give the user feedback when some control is being touched. TalkBack also has a setting that will let the user change the speech rate, pitch, or accent of the voice. The app will utilize this function in order for the visually impaired student to navigate and use the application.
Android devices has a built-in tutorial on them on how to use TalkBack, such as the different gestures, how to navigate on certain controls such as scrolling on a list and so on. These tutorials will be shown first to the students to be familiarize with TalkBack before using the application.
The main menu of the application consists of the learn, exercise, my account, sign out as well as instructions button that takes them on a screen that gives them instructions on how to use the app. These functions can be accessed by the student by double tapping on the buttons in the main menu. Computer aided instructions are applied here and throughout the app by filling the content descriptions of each controls with instructions. For example, when the user hovers into the learn button, TalkBack will read a text that will give the user an idea of the next screen. When the learn button is double tapped, it will bring the next screen where student has the option to choose between the four levels. If the user chooses level 1, the user will again choose a letter (A-J) to practice. If the user chooses level 2, the user will again choose a letter (K-T) to practice. If the user chooses level 3, the user will again choose a letter (U-Z) to practice. Lastly If the user chooses level 4, the user will again choose between numbers (0-9). All this levels that contains basic characters are part of the Grade 1 braille.
As shown on figure 2, the screen displays the braille pattern for letter A. This is the screen where the students will familiarize themselves with the patterns and can always come back here for practice. The black dot represents a raised dot and the gray dot represents the flat dot. When the user's finger hovers through the dots, TalkBack will say the dot number and whether it is raised or flat to the user, letting the visually impaired student to be familiar with the position of the dots as well as their state. The user can also choose 3D Model Download button that will download a 3D model of the example word that is stored on the cloud storage. The Example Word button will give the user other example words that starts with the letter that is being practiced.
The students can select which exercises to do which is assigned to them by their teacher. The teacher user is responsible for creating exercises and assigning them to their students. A content management system will be available for the teacher account where they will be able to create exercises and add new students. The exercises for this app are in a simple multiple-choice format. As shown on figure 3, a braille pattern is displayed and three choices are below it. The student will first slide their finger above in which TalkBack will read the exercise number and the item number, so that the student will know which screen is the app currently in. Then, the student will slide over to the braille pattern which is similar with the screen in the learning module except that it does not tell what letter it is. Once the student is able to touch all the dots and gets the pattern, he or she can now choose the answer in the choices below it. If the answer is wrong the app will encourage the student to try again and select the correct answer. Once all the items are answered, the student will be congratulated for answering all and the results (retries for each item and the duration of taking the exercise) will be sent to the teacher's account for assessment. Figures 4 and 5 shows that the user can also choose 3D Model Download button that will download a 3D model of the example word that is stored on the cloud storage. The user can print the 3D model provided by the app using a 3D printer of their choice. The 3D models are in STL format. For the testing of 3D printing, the researchers used da Vinci 1.0 AiO 3D printer and XYZware 3D slicing and printing software. The process on how the 3D model was printed can be seen in figure 4, a rabbit model is used and the material used for printing the model is a white ABS filament. The total time it took to completely print the rabbit model is 4 hours. Figure 5 displays the finished product.

Results and Discussion
The proponents have gathered two different types of students to test the application namely: totally blind students, and partially blind students. The proponents have tested the application to the blind students at the Philippine National School for the Blind (PNSB). Moreover, the researcher gathered data which are represented below in pie charts:  Figure 6. Totally blind students' success rate Figure 7. Partially blind students' success rate It can be seen in figure 6 that the success rate of the application is 100% for the students who did great (did not have a hard time using the application). Meaning, the totally blind student did great on the application test.
For the figure 7, several partially blind students took part in the testing. The student who did great did not really need help in using the mobile phone because the student is already familiar with the Braille patterns. TalkBack was definitely a big help for the students. There are 20% of the student test who did great.
Another student only did good because the student also learned quickly thru the assistance of the TalkBack. The student did not have a hard time using the application and navigating across it. There are 20% of the student test who did good.
The student who did fair started out slowly due because the student struggled first with the navigation. After guiding the student, the student did well on the test. There are 20% of the student test who did fair. The student who did bad had only begun studying Braille. The student had a hard time answering because the student is not fully familiarized with the Braille character. With guidance, the student was able to be familiarized with the navigation and the student was able to answer the test There are 20% of the student test who did bad.
There are 0% of the student test who did horrible. Overall, the success rate of the students who used tested the application is high.

Conclusion and Recommendation
The results of this study have shown that usage of Braille3D in a classification of braille characters is advantageous, especially for totally blind students. For the purpose of being portable, the proponents have decided to develop the application on smartphones since hand-held devices are prevalent in today's technological advances. The proponents have noted that the application is exceedingly beneficial to totally blind students because it helped them understand and gain familiarization to the Braille characters faster than the traditional devices they are using.
The proponents recommend to look on more functionalities which can be beneficial to the development of this assistive technology.